Dag | Run Id | Task ID | Map Index | Value
-- | -- | -- | -- | --
dissemination_descriptor | Accel_to_CAFE_Dissemination | manual__2025-12-16T19:40:05.370110+00:00 | extract | -1 | {'by_filter': False, 'ingest_type': 'accelerator', 'submit_date': None, 'use_tempfiles': False, 'schema_version': '1.0.2', 'submitter_name': 'Mike Conway', 'submitter_email': 'mike.conway@nih.gov', 'temp_collection': False, 'dissemination_type': 'cafe', 'dissemination_filter': {}, 'dissemination_item_id': '6941b565fbca2a8eff0835ce', 'dissemination_version': None, 'dissemination_identifier': 'Accel_to_CAFE_Dissemination-manual__2025-12-16T19:40:05.370110+00:00'}
payload_inline | Accel_to_CAFE_Dissemination | manual__2025-12-16T19:40:05.370110+00:00 | extract | -1 | True
<br class="Apple-interchange-newline">Dag Run Id Task ID Map Index Value
dissemination_descriptor [Accel_to_CAFE_Dissemination](http://localhost:8080/dags/Accel_to_CAFE_Dissemination)
[manual__2025-12-16T19:40:05.370110+00:00](http://localhost:8080/dags/Accel_to_CAFE_Dissemination/runs/manual__2025-12-16T19:40:05.370110+00:00)
[extract](http://localhost:8080/dags/Accel_to_CAFE_Dissemination/runs/manual__2025-12-16T19:40:05.370110+00:00/tasks/extract)
-1
{'by_filter': False, 'ingest_type': 'accelerator', 'submit_date': None, 'use_tempfiles': False, 'schema_version': '1.0.2', 'submitter_name': 'Mike Conway', 'submitter_email': 'mike.conway@nih.gov', 'temp_collection': False, 'dissemination_type': 'cafe', 'dissemination_filter': {}, 'dissemination_item_id': '6941b565fbca2a8eff0835ce', 'dissemination_version': None, 'dissemination_identifier': 'Accel_to_CAFE_Dissemination-manual__2025-12-16T19:40:05.370110+00:00'}
payload_inline [Accel_to_CAFE_Dissemination](http://localhost:8080/dags/Accel_to_CAFE_Dissemination)
[manual__2025-12-16T19:40:05.370110+00:00](http://localhost:8080/dags/Accel_to_CAFE_Dissemination/runs/manual__2025-12-16T19:40:05.370110+00:00)
[extract](http://localhost:8080/dags/Accel_to_CAFE_Dissemination/runs/manual__2025-12-16T19:40:05.370110+00:00/tasks/extract)
has an error with the document id embedded in the document (I think I need to add the code to make this json?
2025-12-16, 14:43:41] INFO - Use 'pickle' as serializer.: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator"
[2025-12-16, 14:43:41] INFO - Executing cmd: /tmp/venvozeb4bb2/bin/python /tmp/venv-calleay6xiwc/script.py /tmp/venv-calleay6xiwc/script.in /tmp/venv-calleay6xiwc/script.out /tmp/venv-calleay6xiwc/string_args.txt /tmp/venv-calleay6xiwc/termination.log /tmp/venv-calleay6xiwc/airflow_context.json: source="airflow.utils.process_utils"
[2025-12-16, 14:43:41] INFO - Output:: source="airflow.utils.process_utils"
[2025-12-16, 14:43:45] INFO - OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k: source="airflow.utils.process_utils"
[2025-12-16, 14:43:49] INFO - [2025-12-16T19:43:49.706+0000] {script.py:35} INFO - Extract from accel database: source="airflow.utils.process_utils"
[2025-12-16, 14:43:49] INFO - [2025-12-16T19:43:49.708+0000] {type_matrix.py:54} INFO - Parsing type matrix from /tmp/venvozeb4bb2/lib/python3.12/site-packages/accelerator_core/schema/type_matrix.yaml: source="airflow.utils.process_utils"
[2025-12-16, 14:43:49] INFO - [2025-12-16T19:43:49.710+0000] {mongo_tools.py:72} INFO - connection string: mongodb://root:***@mongo-service-0.mongo-service-headless.accelerator-dev.svc.cluster.local:27017/: source="airflow.utils.process_utils"
[2025-12-16, 14:43:49] INFO - [2025-12-16T19:43:49.723+0000] {mongo_dissemination.py:129} INFO - Disseminating document based on original source: cedar and document identifier: 5f2d2b90-3a7b-46e9-8aa8-6e9d9de88132 of type accelerator to target: cafe: source="airflow.utils.process_utils"
[2025-12-16, 14:43:49] INFO - [2025-12-16T19:43:49.769+0000] {accel_database_utils.py:175} INFO - update_operation is {'$push': {'technical_metadata.history': {'timestamp': '2025-12-16T19:43:49.769435', 'msg': 'Disseminating document original source: cedar and document identifier: 5f2d2b90-3a7b-46e9-8aa8-6e9d9de88132 as doc_id: 6941b565fbca2a8eff0835ce of type accelerator to target: cafe'}}}: source="airflow.utils.process_utils"
[2025-12-16, 14:43:49] INFO - [2025-12-16T19:43:49.771+0000] {accel_database_utils.py:184} INFO - Document 6941b565fbca2a8eff0835ce found before update.: source="airflow.utils.process_utils"
[2025-12-16, 14:43:49] INFO - [2025-12-16T19:43:49.777+0000] {accel_database_utils.py:198} INFO - Successfully updated document 6941b565fbca2a8eff0835ce with event.: source="airflow.utils.process_utils"
[2025-12-16, 14:43:51] INFO - Done. Returned value was: {'dissemination_descriptor': {'submitter_name': 'Mike Conway', 'submitter_email': 'mike.conway@nih.gov', 'submit_date': None, 'ingest_type': 'accelerator', 'schema_version': '1.0.2', 'temp_collection': False, 'dissemination_type': 'cafe', 'dissemination_version': None, 'dissemination_identifier': 'Accel_to_CAFE_Dissemination-manual__2025-12-16T19:40:05.370110+00:00', 'dissemination_item_id': '6941b565fbca2a8eff0835ce', 'use_tempfiles': False, 'by_filter': False, 'dissemination_filter': {}}, 'payload_inline': True, 'payload_path': [], 'payload': [{'_id': ObjectId('6941b565fbca2a8eff0835ce'), 'submission': {'submitter_name': 'Mike Conway', 'submitter_email': 'mike.conway@nih.gov', 'submitter_comment': ''}, 'data': {'program': {'program_name': 'CHORDS', 'preferred_label': ''}, 'project': {'project_code': 'PM2.5-ACAG-07102024-NA', 'project_name': 'Satellite-Derived PM2.5', 'project_short_name': 'PM2.5-ACAG-NA', 'project_url': '', 'project_sponsor': [{'type': '', 'name': 'Other', 'other_type': False}, {'type': '', 'name': 'Washington University in St. Louis', 'other_type': True}, {'type': '', 'name': 'Atmospheric Composition Analysis Group (ACAG)', 'other_type': True}, {'type': 'Academic Institution', 'name': '', 'other_type': False}]}, 'resource': {'resource_name': 'Atmospheric Composition Analysis Group (ACAG) North American Regional Estimates with Composition', 'resource_version': 'V4.NA.03', 'resource_short_name': 'ACAG North American Regional Estimates with Composition V4.NA.03', 'resource_type': 'Data Resource', 'resource_url': 'https://sites.wustl.edu/acag/datasets/surface-pm2-5-archive/#V4.NA.03', 'resource:doi': '', 'resource_description': 'The Atmospheric Composition Analysis Group (ACAG) North American Regional Estimates with Composition (V4.NA.03) dataset presents estimated annual average fine particulate matter (PM2.5) air pollution total and compositional mass concentrations. This dataset covers North America with 0.01 degree (~1 km) spatial resolution during 2000 to 2018. Concentrations are estimated using satellite-based air pollution observations combined with a chemical transport model and subsequent calibration to ground-based air pollution observations using geographically weighted regression. This dataset covers total PM2.5 and the following PM2.5 components: black carbon (BC), ammonium (NH4), nitrate (NO3), sulfate (SO4), organic matter, mineral dust, and sea salt. This dataset is intended for use in large-scale, long-term epidemiological studies and health impact assessments.', 'resource_domain': [{'name': 'Air Quality', 'other_type': False}], 'resource_keywords': ['Modeled Data', 'Air Pollution', 'PM2.5', 'Criteria Air Pollutants', 'Air Pollution Composition', 'Air Pollution Mixtures', 'Chemical Transport Model', 'Remote Sensing', 'Concentration Prediction', 'Annual Average', 'Geographically Weighted Regression', 'North America'], 'resource_access_type': ['Open'], 'resource_license': {'license_name': '', 'license_uri': ''}, 'resource_reference': [], 'resource_use_agreement': [{'resource_use_agreement_text': 'Users are asked to familiarize themselves with corresponding publications and contact principle researchers (as provided within subsequent sections) to ensure the most up-to-date and appropriate use of information.', 'resource_use_agreement_link': 'https://sites.wustl.edu/acag/datasets/'}], 'publication': [{'citation': 'Van Donkelaar, A., Martin, R. V., Li, C., & Burnett, R. T. (2019). Regional estimates of chemical composition of fine particulate matter using a combined geoscience-statistical method with information from satellites, models, and monitors. Environmental Science & Technology, 53(5), 2595-2611 dot: 10.1021/acs.est.8b06392.', 'citation_link': 'https://doi.org/10.1021/acs.est.8b06392'}, {'citation': 'Hammer, M. S., van Donkelaar, A., Li, C., Lyapustin, A., Sayer, A. M., Hsu, N. C., Levy, R. C., Garay, M. J., Kalashnikova, O. V., Kahn, R. A., Brauer, M., Apte, J. S., Henze, D. K., Zhang, L., Zhang, Q., Ford, B., Pierce, J. R., & Martin, R. V. (2020). Global estimates and long-term trends of fine particulate matter concentrations (1998-2018). Environmental Science & Technology, doi: 10.1021/acs.est.0c01764.', 'citation_link': 'https://doi.org/10.1021/acs.est.0c01764'}], 'is_static': True, 'payment_required': False}, 'personnel': [], 'data_resource': {'exposure_media': ['Air'], 'measures': [{'value': 'PM2.5', 'other_type': False}, {'value': 'Black Carbon', 'other_type': False}, {'value': 'Dust', 'other_type': False}], 'measurement_method': [], 'time_extent_start': '2000', 'time_extent_end': '2018', 'time_available_comment': 'None', 'update_frequency': ['Static'], 'key_variables': ['PM2.5 Concentration', 'PM2.5 Concentration Percentage Black Carbon (BC)', 'PM2.5 Concentration Percentage Ammonium (NH4)', 'PM2.5 Concentration Percentage Nitrate (NO3)', 'PM2.5 Concentration Percentage Sulfate (SO4)', 'PM2.5 Concentration Percentage Organic Matter', 'PM2.5 Concentration Percentage Mineral Dust', 'PM2.5 Concentration Percentage Sea Salt'], 'data_formats': ['NetCDF', 'ASCII'], 'example_metrics': ['Annual average air pollution concentration for grid cell containing geocoded address'], 'data_location': [{'data_location_text': '', 'data_location_link': 'https://sites.wustl.edu/acag/datasets/surface-pm2-5-archive/#V4.NA.03'}], 'includes_citizen_collected': False, 'has_api': False, 'has_visualization_tool': False}, 'data_usage': {'intended_use': [{'value': 'Geospatial exposure assessment for studies on longer-term health outcomes (e.g., chronic conditions)', 'other_type': False}, {'value': 'Geospatial exposure assessment at the individual-level or population-level', 'other_type': False}, {'value': 'Long-term studies', 'other_type': False}, {'value': 'Large-scale studies', 'other_type': False}, {'value': 'Longitudinal studies', 'other_type': False}, {'value': 'Cross-sectional studies', 'other_type': False}, {'value': 'Studies of multi-pollutant mixtures', 'other_type': False}], 'strengths': ['Dataset supports analysis of differences in composition of PM2.5 air pollution concentration across North America over multiple decades using consistent methods'], 'limitations': ['Dataset does not include estimates of model uncertainty for each prediction', 'Dataset cannot be used to compare exposures within the same grid cell or at different times within the same year', 'Data intended for use in long-term, large-scale studies (increased uncertainties expected for use at finer spatial and temporal resolutions)', 'Finest available gridded estimates do not fully resolve spatial gradients in air pollution due to the influence of input data sources with coarser spatial resolution', 'Dataset does not include estimates of model uncertainty for each prediction', 'Dataset cannot be used to compare exposures within the same grid cell or at different times within the same year', 'Data intended for use in long-term, large-scale studies (increased uncertainties expected for use at finer spatial and temporal resolutions)', 'Finest available gridded estimates do not fully resolve spatial gradients in air pollution due to the influence of input data sources with coarser spatial resolution'], 'suggested_audience': ['Researcher']}, 'temporal_data': {'temporal_resolution': [{'value': 'Annually', 'other_type': False}], 'temporal_resolution_comment': ''}, 'population_data': {'individual_level': False, 'linkable_encounters': False, 'population_studies': [], 'biospecimens_from_humans': False, 'biospecimens_type': []}, 'geospatial_data': {'spatial_resolution': [{'value': 'Grid', 'other_type': False}, {'value': '0.01 Degrees', 'other_type': True}], 'spatial_resolution_comment': 'None', 'spatial_coverage': [{'value': 'North America', 'other_type': False}], 'spatial_bounding_box': [], 'geometry_type': ['Grid'], 'geometry_source': [], 'model_methods': [{'value': 'Other', 'other_type': False}, {'value': 'Geographically Weighted Regression', 'other_type': True}, {'value': 'Chemical Transport Model', 'other_type': True}], 'geographic_feature': [{'value': 'General', 'other_type': False}]}, 'computational_workflow': {'tool_type': [], 'is_open': False, 'languages': [], 'use_tools': [], 'example_applications': []}}, 'technical_metadata': {'created': '2025-12-16T19:39:17.256220', 'modified': '', 'verified': '', 'target_schema_version': '', 'target_schema_type': '', 'original_source_type': 'accelerator', 'original_source_identifier': '5f2d2b90-3a7b-46e9-8aa8-6e9d9de88132', 'original_source_link': 'cedar', 'history': [{'timestamp': '2025-12-16T19:39:17.256229', 'msg': 'accession from accelerator with identifier 5f2d2b90-3a7b-46e9-8aa8-6e9d9de88132 in operation CEDAR_Ingest-manual__2025-12-16T19:38:39.834637+00:00'}], 'dissemination_endpoints': []}}], 'dissemination_successful': True}: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator"
[2025-12-16, 14:43:51] INFO - Pushing xcom: ti="RuntimeTaskInstance(id=UUID('019b28b0-943e-779a-9bd9-962f1bb2daba'), task_id='extract', dag_id='Accel_to_CAFE_Dissemination', run_id='manual__2025-12-16T19:40:05.370110+00:00', try_number=2, map_index=-1, hostname='accelerator-worker-0.accelerator-worker.accelerator-dev.sv
with setup of
has an error with the document id embedded in the document (I think I need to add the code to make this json?