Project execution plan

Welcome to the sky_weaver wiki!

The following explains the data format, code development plan and instructions to use sky_weaver. All the calculations below assumes that we record 4096-channel mode baseband data using the FBFCN nodes. It also assumes 856 MHz of band, which changes if it is UHF (544 MHz) and S band (875 MHz).

Raw input

The raw data is in the form of SPEAD packets, written out as many 30GB sized binary files, independently for each bridge. For a usual 4K mode observation, we will have 64 bridges. For a usual 2-hour COMPACT observation, it will be 4096 * 7200/(4096/856e6) * 2 * 2 * 64 / 1e15 = 1.577 PB. This will hence be written as 822 files per bridge, for a total of 52608 files.

Organised input

The above raw SPEAD packets are read, and written in (F)TAFTP order DADA-format files. The first 'F' dimension denotes the 64 bridges (or subbands) that are processed independently. The rest of the dimensions are [nsamples][64 antennas][64 channels][256 samples][2 polarisations]. These are split into different files each with [TBD] samples of data

The pipeline

The pipeline consists of the following high level steps:

• Excising RFI on CPU

• Performing coherent dedispersion on GPU

• Performing beamforming on GPU

• Performing incoherent dedispersion on CPU/GPU

• Writing out TD format to file

• Reading 64 TD format files from disk, sum across frequency and write out N timeseries files