Implement batching in Observations.get_product_list and add batch_size parameter
#3454
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3454 +/- ##
==========================================
+ Coverage 70.72% 70.73% +0.01%
==========================================
Files 232 232
Lines 20041 20040 -1
==========================================
+ Hits 14174 14176 +2
+ Misses 5867 5864 -3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Please don't prompt copilot to add noise here. |
|
Oops, sorry! |
bsipocz
left a comment
bsipocz
left a comment
There was a problem hiding this comment.
Thank you!
Overall this looks good, but it needs a rebase for the changelog. And while you're at it, please make the new kwarg kwarg-only.
Also, maybe consider adding a sentence about it in the narrative docs and modify one if the examples if we have suitable ones?
Thanks!
|
|
||
| @class_or_instance | ||
| def get_product_list_async(self, datasets): | ||
| def get_product_list_async(self, datasets, batch_size=1000): |
There was a problem hiding this comment.
| def get_product_list_async(self, datasets, batch_size=1000): | |
| def get_product_list_async(self, datasets, *, batch_size=1000): |
| error_msg = (f'Dataset keyword not found for mission "{self.mission}". ' | ||
| 'Please input dataset IDs as a string, list of strings, or `~astropy.table.Column`.') | ||
| raise InvalidQueryError(error_msg) |
There was a problem hiding this comment.
Why do you need to put down the error message in a variable?
|
|
||
| @class_or_instance | ||
| def get_product_list_async(self, observations): | ||
| def get_product_list_async(self, observations, batch_size=500): |
There was a problem hiding this comment.
| def get_product_list_async(self, observations, batch_size=500): | |
| def get_product_list_async(self, observations, *, batch_size=500): |
2 participants
Add
batch_sizeparameter toMastMissions.get_product_list,Observations.get_product_list, andutils.resolve_objectto allow controlling the number of items sent in each batch request to the server. This can help avoid timeouts or connection errors for large requests.I also did some refactoring in the
MastMissionsmodule to make it more consistent withObservationsas far as what is returned byget_product_list_async.