-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update uses of get_crds_parameters
to allow for more efficient file opening
#228
Comments
My initial reaction is that we could add a class method @classmethod
def _get_crds_parameters(dataset):
...
return crds_parameters, observatory The need for the class method is for - with cls._datamodels_open(dataset, asn_n_members=1) as model:
- if isinstance(model, Sequence):
- crds_parameters = model._models[0].get_crds_parameters()
- crds_observatory = model.crds_observatory
- else:
- crds_parameters = model.get_crds_parameters()
- crds_observatory = model.crds_observatory
+ crds_parameters, crds_observatory = cls._get_crds_parameters(dataset) Things are a little trickier for for parameters, observatory in cls._get_crds_parameters(dataset):
# handle overrides ...
crds_refs = crds_client.get_multiple_reference_paths(...) Another complication of |
get_crds_parameters
to allow fo more efficient file openingget_crds_parameters
to allow for more efficient file opening
get_crds_parameters
returns a subset of metadata that's passed to crds for reference file selection. Since only a subset is needed ifStep
andPipeline
are processing an input file (and not an inputDataModel
) it may be possible to perform a partial read of the file to more efficiently access the same information. This issue is to layout API changes needed to accommodate this mode.The current
get_crds_parameters
uses are:In
Step.get_reference_file
:stpipe/src/stpipe/step.py
Lines 809 to 814 in 9692bfa
The use of
open_model
here seems superfluous as the input should be an openDataModel
(andopen_model
will sometimes copy the input). Within the scope of this ticket I think this usage can be ignored.In
Step.get_config_from_reference
:stpipe/src/stpipe/step.py
Lines 862 to 867 in 9692bfa
This is likely to receive as input a filename instead of a model due to usage in cmdline.
In
Pipeline.get_config_from_reference
:stpipe/src/stpipe/pipeline.py
Lines 187 to 193 in 9692bfa
This looks functionally the same as the usage in
Step.get_config_from_reference
.In
Pipeline._precache_references_impl
:stpipe/src/stpipe/pipeline.py
Lines 319 to 321 in 9692bfa
This is indirectly called from
Pipeline._precache_references
:stpipe/src/stpipe/pipeline.py
Lines 263 to 266 in 9692bfa
which calls
Pipeline._precache_references_opened
which iterates through any container.The text was updated successfully, but these errors were encountered: