Skip to content

Conversation

@ljgray
Copy link
Contributor

@ljgray ljgray commented Oct 4, 2025

The overall goal of this PR is to implement a task to come up with a static RFI mask from visibility data directly, without having to rely on a hard-coded set of bad channels.

This produces a flagging metric by reducing over baselines using the existing ReduceChisq code as follows:

  • The chi2 is computed over cross-pol baselines only, and excludes baselines < 5 meters.
  • Weighting is done by dividing by baseline redundancy, which highlights faint frequency bands which are only otherwise visible in the high-delay chisq metric. I'll show some plots at some point.

Flagging is then done using a simple 1D MAD metric over the 1D median in time of the chisq metric. Only nighttime data is included in the median (the hook is implemented in a ch_pipeline PR). It uses an updated version of the arPLS baseline-fitting code, which is better able to handle the lack of hard-coded static mask.

This PR does the following:

  • Updates containers.group_tasks to be able to handle multiple setup methods. This is still a bit fragile, but it works.
  • Updated the scale_invariant_rank operation to be a bit faster (although I didn't use it in the end)
  • Adds options to the MaskBaselines task to mask polarisation and to optionally combine the mask using an and operation
  • Extends the ReduceBase task with a hook for custom weights
  • Updates the penalized_least_squares implementation to support different weighting schemes. This implementation is a bit of a hack, abusing the __new__ class method to make it appear like a function, while still being able to subclass with custom hooks. I'm open to thoughts about how else to implement this.
  • Replaces the old narrowband vis flagging task with this new static RFI mask task

I think the commits should be fairly easy to follow.

This code all works and is tested - I just haven't been able to thoroughly test how well the method actually performs on a wide selection of days

@ljgray ljgray force-pushed the ljg/rfi-vis-mask-update branch 4 times, most recently from 3f35410 to 9902473 Compare December 12, 2025 23:18
@ljgray ljgray marked this pull request as ready for review December 13, 2025 01:12
@ljgray ljgray force-pushed the ljg/rfi-vis-mask-update branch 2 times, most recently from 4d1b921 to 158488e Compare December 13, 2025 01:52
@ljgray ljgray force-pushed the ljg/rfi-vis-mask-update branch from 158488e to 62834c6 Compare December 15, 2025 22:41
@ljgray ljgray force-pushed the ljg/rfi-vis-mask-update branch from 62834c6 to c8d3091 Compare December 15, 2025 23:45
@ljgray ljgray requested a review from ssiegelx December 17, 2025 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants