Skip to content

Add support for NETCDF3_64BIT_DATA in write_netcdf()#633

Merged
xylar merged 7 commits into
MPAS-Dev:masterfrom
xylar:add-cdf5-handling-to-write-netcdf
Jun 6, 2025
Merged

Add support for NETCDF3_64BIT_DATA in write_netcdf()#633
xylar merged 7 commits into
MPAS-Dev:masterfrom
xylar:add-cdf5-handling-to-write-netcdf

Conversation

@xylar
Copy link
Copy Markdown
Collaborator

@xylar xylar commented May 27, 2025

For performance, this involves first writing a NETCDF4 file and then using ncks to convert it to NETCDF3_64BIT_DATA (CDF5) format.

This merge also:

  • adds a conversion of int64 format to int32, since the MPAS framework doesn't support int64
  • adds documentation for write_netcdf()
  • adds unit tests for write_netcdf(), including both CDF5 and int64 --> int32 capabilities
  • adds nco as a dependency
  • adds the logger to write_netcdf() calls in the mask creator (to keep output from conversion to CDF5 when it happens)

@xylar
Copy link
Copy Markdown
Collaborator Author

xylar commented May 27, 2025

Testing

I tested this out by creating a new Icos30 base mesh in Polaris using this method. Both the mesh_triangles.nc and base_mesh.nc are identical before and after this change, but the file format it CDF5 as expected. See #632 (comment) for the script I used to test that all variables are identical.

@xylar xylar force-pushed the add-cdf5-handling-to-write-netcdf branch from 4418341 to e1fa7e5 Compare June 3, 2025 13:43
@xylar
Copy link
Copy Markdown
Collaborator Author

xylar commented Jun 3, 2025

Testing in Polaris

I was able to run the pr test suite on Chrysalis with these changes and by branch https://github.com/xylar/polaris/tree/write-cdf5, showing that the results were bit-for-bit identical to those with the Polaris main branch without these changes, and also that files were written in CDF5 format, as expected.

@xylar
Copy link
Copy Markdown
Collaborator Author

xylar commented Jun 5, 2025

Further Testing

I have further tested these changes in a full workflow in Polaris that involved culling topography. This exposed issues with string variables in the python mask creator tools that I have now fixed in #638

Comment thread conda_package/mpas_tools/io.py
@xylar xylar force-pushed the add-cdf5-handling-to-write-netcdf branch from b81ebae to af137de Compare June 6, 2025 08:49
@xylar xylar force-pushed the add-cdf5-handling-to-write-netcdf branch from af137de to 2fab2c8 Compare June 6, 2025 09:22
@xylar xylar merged commit adbc54e into MPAS-Dev:master Jun 6, 2025
6 checks passed
@xylar xylar deleted the add-cdf5-handling-to-write-netcdf branch June 6, 2025 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant