Clarifications on data properties

[AngusWright]

I've just started having a play with the tomo_challenge data, and I'm curious about some of the quirks of the dataset. To this end, I've included here a bunch of observations & questions. I understand that the challenge is designed to be "idealised", however it occurs to me that many of these quirks might work to favour machine learning codes, for example, and disadvantage other methods (I give examples of what I mean).

My main observations are below. Apologies in advance for the long list (and for any silly observations...). I've also included a handful of figures to show what I mean. In each figure I have included data from both the (noisy) tomo_challenge dataset and from (noiseless!) MICE2 dataset, for comparison. Full disclosure that I am using MICE2 as a benchmark for 'what I roughly expect to see'.

1. The colour-redshift relation shown by individual SED tracks appears chaotic. The redshift axis is clearly discretised (due to snapshots in the simulation?), which is not an issue per-se, but between adjacent snapshots the colours appear uncorrelated. Colour-redshift tracks abruptly end and restart. This behaviour is mirrored exactly in both the training and validation datasets, so may not be overly problematic for machine learning classifiers, but such behaviour could unfairly hamper template-based or hybrid approaches that (i would say fairly) assume coherent colour-redshift evolution?
   

2. The template discrimination (i.e. the ease of discerning underlying templates in colour-redshift space) appears to become better as a function of redshift, rather than worse. At a given magnitude, the colour-redshift space becomes increasingly discretised as a function of redshift.
   

3. In the above figure, you can also see that the range of galaxy colours in the tomo_challenge data is considerably smaller than in MICE2, especially in the region 0.3≤z≤0.7 where lots of blue sources are missing. Is this because the tomo_challenge templates are missing low-mass (blue) sources? This could be important prior information for bayesian codes?

4. There appear to be small gaps in redshift at low-z, in between (what I'm guessing are) snapshots of the simulation. While these aren't themselves a problem, they may be symptomatic of an underlying bug.
   

5. In the above figures, you can also see that the tomo_challenge data appear to be lacking any large-scale structure? This would bias against hybrid approaches that wish to invoke cross-correlation.

6. The noisy tomo_challenge data extend to much higher redshift, at a given magnitude, than the sources in MICE2. Not sure if this is a problem (?), but mostly just an observation. If could hamper non-machine-learning codes invoke a data-driven redshift-magnitude prior.
   

[EiffL]

• Hummmm very good points! So I know our DC2 photometry is not perfect, because we had to do a complicated matching between the base DC2 simulation and a separate galacticus run. This is somewhat explained in the simulation paper here: https://arxiv.org/pdf/1907.06530.pdf section ~5.3 But it's fair to say we expect some amount of weirdness ^^' For instance, I thiink the discontinuities you are seeing in your first plot are coming from switching from one discrete galacticus snapshot to the next.
  This being said, I hadn't compared them to Mice, nor did I expect the tracks to be that clear at higher redshift.
  I'm gonna tag @katrinheitmann and @aphearin on this issue who can probably comment more on this.

• Concerning the range of galaxy colous/lack of blue, I don't know, but are you applying similar cuts in the MICE data as what we used to build the sample here? See for instance:
  

  tomo_challenge/bin/generate-challenge-data.py

  Line 19 in 25af0db

  sel = mcal_group['mcal_s2n'][:] > 10

  
  We have cuts on SNR and size, and for instance I'm not 100% sure off the top of my head how the metacal SNR is computed here, but @joezuntz would know.

• For cross-correlation, positions are not meant to be included by default in the available data, but LSS should be there in the input DC2 simulation.

• As for how this impacts the challenge :-| this is a good question. I think we more or less assumed no external/prior knowledge, and yeah this:

such behaviour could unfairly hamper template-based or hybrid approaches that (i would say fairly) assume coherent colour-redshift evolution?

might be a limitation...

[joezuntz]

I've checked the original data that this i drawn from, and can confirm that the tracks you see are present in the original CosmoDC2 as well, and I didn't somehow screw up and add them somehow here.

• We have cuts on SNR and size, and for instance I'm not 100% sure off the top of my head how the metacal SNR is computed here, but @joezuntz would know.

The SNR is found by:

• adding noise to the objects following Ivezic, Jones, & Lupton
• getting the snr as (n_photon / background_mean) per band
• adding them in quadrature for the r, i, z bands

[aphearin]

Thanks for sharing the plots @AngusWright. So I see two distinct sources of discreteness in these plots. The first has to do with distinct redshift snapshots of the halo catalog. Dan’s matchup method was designed to alleviate this, but it doesn't resolve it entirely.

The second kind of discreteness is purely to do with color PDFs, and is not related to the finite collection of snapshots. The discreteness in color-color-z space is ultimately caused by discreteness in the SEDs of the library of Galacticus galaxies from which we sampled to make the mock, so it is sensible/expected that you are finding it in cosmoDC2 as well as DC2.

I'm not sure what is meant by this comment though:

In the above figures, you can also see that the tomo_challenge data appear to be lacking any large-scale structure?

Could you elaborate?

[AngusWright]

Hi @aphearin. Thanks for the comments. What I mean w.r.t. the large scale structure: If you look at this figure then you can see the large scale structure (over- & under-densities) in MICE as the lighter and darker vertical bands. However you don't see such structures in DC2 (except maybe at z=0.2, but this feature looks like it might be caused by an overlap of two discrete snaps in redshift? The inverse happens at z=0.1, where there are gaps between the two steps). Is this because, e.g., redshifts were randomly assigned to galaxies within the redshift steps? This would wash out structures and give you this behaviour, I would imagine.

Regarding the discreteness of the colour-redshift distribution: Ok, I understand how having a finite number of SEDs can create such overdense trails. But I'm still confused about the lack of coherence between individual redshift steps. Take the upper panel in this figure for example. At z=0.56 and g-i = 1.3, there is a very highly represented SED (the thick black line). However this SED is apparently absent at all other redshift steps (well, at least in the adjacent snaps). It's as if this galaxy SED template popped into existence at z=0.54, was assigned to lots of galaxies between 0.54<z<0.56, and then disappeared at z=0.56 again. And this seems to happen a lot; it's hard to see many continuous SED models as a function of redshift at all.

Essentially, even if there were a small number of SED templates, I would expect to see a small number of contiguous lines as a function of redshift, rather than a large number of disjoint ones? But perhaps my intuition is just wrong!

[aphearin]

@AngusWright - thanks for clarifying. Re: lack of LSS. I think this is difficult to gauge from a scatter plot of color vs. redshift. If you're concerned about a lack of LSS, the two-point function might be a more convincing way to see the LSS in DC2 galaxies.

Re: discreteness. You have a good eye! In fact, exactly the sort of selection you describe does indeed happen in the matchup method, in which over a particular redshift range only, the same exact SED (or the same small handful) is preferentially selected over and over.

[joezuntz]

Yes, there's definitely large-scale structure in there! The sample is over a wide region, so it might be that it's being washed out in the geographic average? This is a random sub-sample over 400 sq deg.

Here's a density slice in 0.5 < z < 0.6 in a small region of the underlying catalog in ra, dec:

[aphearin]

When using the "jumpiness" of a scatter plot vs. redshift as a gauge for the presence/absence of LSS, one thing to remember is that the sky area and intrinsic number density of the sample has a direct impact on the visual perception.

[AngusWright]

Hi @joezuntz @aphearin, sorry, my wording wasn't clear. Yes the on-sky distributions show some large scale structure, but the actual structures themselves don't appear very strongly as a function of reshift. That's why they don't pop out in my colour vs redshift plot. See the below cone plots (i.e. RA vs z) as a demonstration, where I've plotted equal volumes (10x10 sqdeg on-sky, z<0.5, matched number density) for DC2 and MICE2. There is clearly some structure in the DC2 catalogue, but it's considerably weaker than the level I would expect. Perhaps this isn't relevant when looking at 2-point statistics in tomographic bins with \Delta z = 0.2, but looking at 2-point stats in in tomographic bins with \Delta z = 0.05 would give quite different results, I would expect?

[joezuntz]

Hi @AngusWright - could you compare the magnitude distributions for the two samples? I'm wondering if we're seeing the fainter LSST-like objects there.

[aphearin]

Good point @joezuntz. @AngusWright - when you say you are comparing "matched number densities", could you say a bit more about how this was done?

[AngusWright]

@joezuntz @aphearin The effect doesn't seem to be caused by magnitude. See the below. I've subset the sources in both catalogues to:

• 0.1 < z < 0.3
• 22.0 < r < 24.0
• 10x10sqdeg

The sources numbers in the two catalogues are quite different too. Originally I down-sampled the MICE2 catalogue to have the same number of sources as DC2 in the volume, just to make the plotting look fairer. I've not looked at the number counts in DC2 vs MICE2, but I'm not sure that the difference in this regime is expected; especially because the MICE2 sim (with KiDS level noise) has considerably more sources in the volume than DC2 does (with LSST level noise); 32K vs 25K. But in any case, I think that this is enough to make clear that the difference isn't magnitude driven.

[joezuntz]

One note - if you're using the files I released for this challenge, it's not the full DC2 - I randomly downsampled to split into different samples.

I also apply these cuts:

• S/N (riz) > 10
• size / PSF size > 0.5

could one of these be driving the difference?

[AngusWright]

Ah cool, yes that could definitely explain the difference in absolute number between MICE2 and DC2, which is good 👍

[aphearin]

@AngusWright - would it be laborious to use a (much) larger sky area? I'm a bit worried about over-interpreting the LSS with a scatter plot made from such a small patch.

[AngusWright]

100 square degrees is half the dataset used here, but sure. Below is using (nearly) the whole tomo_challenge dataset: 20x20 sqdegrees. Compared to the same area from MICE2. But I think that this just muddies the waters? If we collapsed the hemisphere onto the RA axis then it would look ~uniform in both cases, but the underlying issue of differing LSS would still be there. For this test a thinner slice in DEC is useful. So the second plot is the full RA extent of the tomo_challenge dataset, but only a 4sqdeg slice in DEC.