expand fields to allow multiple values

[pdowler]

this comes from an archive partners slack discussion started by David Rodrigues

proposal.id
telescope.name
instrument.name
plane.energy.bandpassName

... maybe more

[dr-rodriguez]

Some examples from MAST.
Looks like we haven't been consistent in whether we should use _, ;, | to split strings:

jw01282-c1010_t005_miri_f1550c-mask1550

<bandpassName>
	F1550C;4QPM_1550
</bandpassName>

hlsp_tasoc_tess_ffi_tic00166699853-s0004-cam2-ccd2-c1800_tess_v05

<proposal>
	<id>
	       G011160_G011155_G011188
        </id>
</proposal>

hlsp_ullyses_hst-fuse_fuse-stis_sk-71d8_uv

<telescope>
	<name>
		FUSE;HST
	</name>
</telescope>
<instrument>
	<name>
		FUV | STIS/FUV-MAMA
	</name>
</instrument>

[pdowler]

What does it mean if a plane has multiple bandpassName(s)? I recognize the form of F1550C as a filter around 1550nm...

Does it mean multiple filters in the light path, so the Plane.energy.bounds would be the intersection of the two?

In a DerivedObservation, one could stack inputs in different filters to make a wider Plane.energy.bounds, possibly with meaningful sub-samples. I know there are also some "white light" images like that...

These two possibilities kind of reduce to "light in filter A and/or B" respectively (logical and/or)... would the model only need one interpretation or also a way to specify and vs or?

[pdowler]

For multiple proposals, I'm not really sure what the usage means. I can imagine a queue-scheduled observatory optimising by taking one observation that two different proposals requested and wanting to "assign" it to both of them... (see below)

If a proposal has an observing plan that includes making stacks, then DerivedObservation(s) with proposal information makes sense. That's why Proposal is common to all Observation, not just SimpleObservation.

However, if a 3rd party were to decide to stack SimpleObservation(s) from different proposals, I would not want to assign any proposal information to the resulting DerivedObservation: the proposal indicates "this thing exists because of proposal xyz" and that's just not true of all DerivedObservation(s). They are more likely to fit "this thing exists because of project abc" (which would be in Plane.provenance.project).

[pdowler]

below:

There was a radio use case where they collect data (SimpleObservation) of a large region of the sky and different parts of the data are destined for different proposals (and the implied different permissions). The solution there was that we renamed CompositeObservation to DerivedObservation and the plan would be to have the SimpleObservation (owned by the observatory) and create a DerivedObservation for each proposal. The extra catch there was that in radio one would actually want to extract different subsets of the data for each of those so it did not overlap with the "two observations include the same artifact issue".

So, that was one way that one data acquisition was "assigned" to two proposals, but it is a special case where there really are new data created by some processing.

[pdowler]

on multiple telescopes: I assume the MAST use case where is "DerivedObservation made by combining data from multiple telescopes". In radio (interferometry) there is also VLBI observations that involve mutliple telescopes and there is the "multiple dishes in a single facility" and there I don't know where the line between telescope -- collector -- detector -- correlator lies. I'm toying with the Telescope and Instrument classes and if that looks promising I might pull this out as a separate issue.

As it stands, we could not change cardinality of Telescope.name by itself because multiple telescopes are not at the same location. If we go ahead with removing WCS from CAOM for 2.5, the need for Telescope.geoLocation* would kind of go away (I think it's only useful for some spectral reference frame transforms).

[dr-rodriguez]

What does it mean if a plane has multiple bandpassName(s)? I recognize the form of F1550C as a filter around 1550nm...

Does it mean multiple filters in the light path, so the Plane.energy.bounds would be the intersection of the two?

In a DerivedObservation, one could stack inputs in different filters to make a wider Plane.energy.bounds, possibly with meaningful sub-samples. I know there are also some "white light" images like that...

These two possibilities kind of reduce to "light in filter A and/or B" respectively (logical and/or)... would the model only need one interpretation or also a way to specify and vs or?

I've actually seen both situations (and vs or).

I think the most common are DerivedObservations that are stitched spectra across multiple bandpasses (eg, HASP, HLSP). So it's the wavelength coverage of each stacked together. Our implementation in Plane.energy.bounds is the UNION of all wavelength ranges covered in those cases.

There are also a few examples where there is more than one optical element in place, for example a filter plus a grating. I believe in that situation we've ignored or glossed over any impact of the grating and just captured the wavelength range of the filter. In those cases, I think we've used something like / to separate the grating from the filter.
In the example above, F1550C;4QPM_1550 refers to a JWST MIRI observation in the F1550C filter (about 15.5 microns) that utilizes a coronagraph mask optimized for that wavelength: https://jwst-docs.stsci.edu/jwst-mid-infrared-instrument/miri-observing-modes/miri-coronagraphic-imaging#gsc.tab=0
One could argue the grating/mask/coronagraph isn't really part of the bandpass, which is fair, but we're using this field to capture that information so users can search specifically for it. Perhaps this argues for a more generalized optical element section in the model?

I can imagine situations were observers may want to have two filters so have a much narrower bandpass (eg, suppressing red light from an inefficient blue filter). The bounds I would want to record for those are the INTERSECTION rather than the UNION of the filters. That isn't always possible though, and I don't have a good feel for how frequent it happens among the MAST datasets.

I think what I would like to have is the multiple storing of bandpasses to support the first use case: each element in the bandpass list is something that will be UNION-ed together to form the final bounds range.

[dr-rodriguez]

For multiple proposals, I'm not really sure what the usage means. I can imagine a queue-scheduled observatory optimising by taking one observation that two different proposals requested and wanting to "assign" it to both of them... (see below)

If a proposal has an observing plan that includes making stacks, then DerivedObservation(s) with proposal information makes sense. That's why Proposal is common to all Observation, not just SimpleObservation.

However, if a 3rd party were to decide to stack SimpleObservation(s) from different proposals, I would not want to assign any proposal information to the resulting DerivedObservation: the proposal indicates "this thing exists because of proposal xyz" and that's just not true of all DerivedObservation(s). They are more likely to fit "this thing exists because of project abc" (which would be in Plane.provenance.project).

For the TESS mission, it's as you initially describe: it's a survey-like situation where users can propose to extract select stars at a higher cadence than the full-frame images. The mission office selects these targets and while they all have the same PI (George Ricker, the PI of the mission itself), the targets get associated with every proposal that submitted them.

Another example is the Hubble Advanced Products (HAP), specifically the multi-visit mosaics. These are DerivedObservations that are mosaics made from separate programs. The artifacts of these observations are new images of some area of the sky that are drizzled, rotated, and stacked together. The mission specifically asked us to capture every program that has been used in them so that users can search for multi-visit mosaics produced from specific programs. I believe for these we have Plane.proposal.project be HAP-MVM. That is, these observations exist because of the HAP-MVM project, but the original programs are indicated in Observation.prpID.

This same situation will happen with the spectroscopic analogue, HASP, once they start stacking spectra across visits.

[dr-rodriguez]

on multiple telescopes: I assume the MAST use case where is "DerivedObservation made by combining data from multiple telescopes". In radio (interferometry) there is also VLBI observations that involve mutliple telescopes and there is the "multiple dishes in a single facility" and there I don't know where the line between telescope -- collector -- detector -- correlator lies. I'm toying with the Telescope and Instrument classes and if that looks promising I might pull this out as a separate issue.

As it stands, we could not change cardinality of Telescope.name by itself because multiple telescopes are not at the same location. If we go ahead with removing WCS from CAOM for 2.5, the need for Telescope.geoLocation* would kind of go away (I think it's only useful for some spectral reference frame transforms).

This has been used for High-Level Science Products (HLSPs), where things like spectra from more than one telescope/instrument have been stitched together to have broader spectral coverage. I think HASP (Hubble Advanced Spectroscopic Products) currently only uses HST data, but they do combine multiple instruments and create new products from that.

[pdowler]

bandpassName

I think I agree that multiple bandpassName(s) should be the wider union usage you describe. User queries could still use bandpassName = 'F555w' to find exactly that filter or they could use bandpassName LIKE '%F555w%' to find all data using that filter. That's consistent query style to polarization (I/Q/U/V cube, for example). There would be a restricted character in bandpassName values that would be used to serialise them into a single column (for TAP queries); that separator in keywords is |, which is notably not consistent with the / usage in polarization states (from ObsCore).

The narrower multiple filter intersection thing belongs in a possible enhancement of the instrument model (eg to describe the path of the signal through components before reaching the detector. In that case, you could still construct a single bandpassName with two filter names in it using a different separator (eg F555w+F490). We could possibly reserve a character for this usage... TBD.

In either case, Plane.energy.bounds would give the correct/representative numeric band limits.

[pdowler]

telescopes/instruments

I have increased the scope of #11 to cover this topic.

[pdowler]

proposalID

I feel like proposalID itself should be a reference into an (external) proposal system where one would also find metadata that was consistent with the limited subset that is part of the Proposal class itself. Simple allowing multiple proposalID values all with the same PI, title, keywords seems like it would be confusing.

This really is part of what provenance should cover and in principle one could write a query to extract the proposalID(s) of all the member observations (admittedly, it would be complicated because the join would bloat the query result and sincve 2.4 we allow members to be Observation so there's an arbitrary sequence of joins to collect all the initial proposalID values). Does that give essentially the same result?

I'm thinking more along the lines an optional Proposal.memberProposals [0..*] : String but it would be more or less an optimization... thoughts?