v0.6.4

Overview

This is a release that includes important performance enhancements that allow user-provided calibration parameters for broadband and narrowband calibration, new functionalities to compute and attach split-beam angles to calibrated Sv dataset, perform frequency-differencing and masking, as well as a number of bug fixes and other improvements.

New features

• Allow passing in cal_params as a dictionary for narrowband and broadband calibration (#955)
• Add default chunk encoding for zarr output (#939)
• Add add_splitbeam_angle function to the consolidate subpackage (#916, #971)
• Add a new mask subpackage
  • Add apply_mask function to the mask subpackage (#912)
  • Add frequency-differencing function to the mask subpackage (#901)
• Allow selection of a subset of channels when combining multiple echodata objects (#892)
  • This is done via the added channel_selection input argument to combine_echodata
• Add default consolidated flag for echodata.to_zarr (#855)
• Generalize and improve efficiency of compute_MVBS (#878)
  • Allow echo_range that vary with ping_time
  • Allow Sv data that are dask arrays
• Add .nbytes to obtain that data size of an echodata object (#874)
• Add new data variables from the raw data files to facilitate EK80 calibration (#944)
  • transceiver type, transmit impedance, receive impedance, receiver sampling frequency

Enhancements and other changes

• Unify the order of dimensions of echo_range across all sonar models (#968)
• Create and use the default echopype home directory (#896, #954)
  • This default directory is at ~/.echopype
• Refactor the calibrate subpackage (#904)
  • routines for env_params and cal_params intake and range computation are now in different modules rather than methods in each classes
  • Tidy up cal_params related routines (#953)
  • Revise env_params related routines (#952)
• Improving pulse compressed broadband Sv computation and echo range computation (#944)
  • The resulting values are tested against pyEcholab and Echoview outputs
  • Note there is unresolved discrepancy between pyEcholab/Echopype outputs with Echoview outputs for the first section of Sv values
• Change input argument names for open_raw (#962)
  • Change offload_to_zarr to use_swap
  • Change max_zarr_mb to max_mb

Bug fixes

• Handling of provenance attributes in apply-mask and add-depth, especially for testing (#930)
• Fix bugs in combining multiple echodata objects (the combine_echodata function)
  • Fix meta_source_filenames bug and enable (meta)source_filenames appending of path and list (#913)
• Fix bugs in env_params intake for calibration (#952)

Infrastructure

• Update docker build for arm64 mac silicon chips (#964)
• Fix windows github actions workflow (#947)
• Remove mamba dependency (#946)
• Fix pre-commit ci and update RTD config for jupyter-book (#934)

v0.6.3

Overview

This is a minor release that includes an important performance enhancement for combining large volumes of data residing in individual files into a single entity, a number of bug fixes, and other smaller improvements.

New features

• Overhaul combine_echodata function
  • Allow combine a large number of EchoData objects exceeding memory limits (#808, #824, #830)
  • Remove reversed time check from combine_echodata (#835)
  • Add minimal ZarrCombine test (#826)
  • Order the channel coordinate in file conversion to ensure consistent combination across files (#818)
  • Revise outdated data combination behavior (#797, #799)
  • Clean up all coordinate and attribute details under combine_echodata function (#848, #849)
• Track provanance for filenames of raw data files and auxiliary files
  • Propagate xml_path as meta_source_filenames to combined echodata (#814, #852)
  • Standardize handling of source_filenames in individual and combined echodata as well as downstream datasets (#804, #806)

Under the hood enhancements

• Clean up functions for setting encoding in the converted files (#851)
• Add requests and aiohttp to dependency (#844)
• Pin netcdf4 to be <1.6 for pypi package due to ongoing netcdf-c problem (#843)
• Write Parsed2Zarr generated files to temp_echopype_output/parsed2zarr_temp_files (#832)
• Change isel to sel to fix compute_Sv to allow working with dask array (#828)
• Add open_raw(offload_to_zarr=True) integration tests (#794)

Bug fixes

• Fix regression bug with interpolating environmental variables to ping_time grid (#837, #856)
• Fix WindowsPath error with compute_Sv when run on Windows (#829)
• Fix logic problem in open_raw(offload_to_zarr=True) that sometimes cause problems (#794, #853)

Infrastructure

• Update CI to use micromamba (#805)
• Fix version string on CI (#804, #820)
• Rename ci.yaml to build.yaml for clarity (#807)

v0.6.2

Overview

This is a minor release that includes a few new features and memory efficiency-related changes that make echopype better.

New features

• Add a new subpackage consolidate that contains functions to consolidate data across the calibrated Sv dataset and the corresponding raw-converted file
• Add function consolidate.add_location to interpolate location to calibrated dataset (#749)
• Add function consolidate.add_depth to convert range_meter to depth with information on transducer tilt and depth (#738)
• Move function that swaps the channel with frequency dimension (now as consolidate.swap_dims_channel_frequency) (#738)
• Add new functionality to allow control of logging outputs (#772)

Under the hood enhancements

• Improve memory usage while converting files that require significant NaN-padding and previously would incur very large memory expansion (#774)
  • This is achieved by directly writing variables that may incur a large memory expansion into a temporary zarr store
  • Beta function that will benefit from user feedback
• Overhaul access pattern for EchoData (#762)
  • Remove previous access pattern for different groups in the raw-converted file
  • Starting from this release all groups are accessed with echodata["GROUP_PATH"], e.g., echodata["Platform"], echodata["Sonar/Beam_group1"], etc.
• Make long_name in ds_power for EK80 consistent with other sonar model (#771)
• Modify set_beam() so it returns a list (#780)
• Change the order in _save_groups_to_file so the order of groups is preserved when opening a converted netcdf file (#779)
• Remove the user option to select NMEA sentences in open_raw to ensure raw data is preserved (#778)

Infrastructure

• Update CI set up to avoid exceeding GitHub actions memory limitation (#761)
  • decrease the number of workers from 4 to 2

v0.6.1

Overview

This is a minor release that includes important bug fixes, a number of new features, and some leftover data format changes specific to parsed AD2CP data format.

Bug fixes

• We use datatree under the hood for the EchoData object, but datatree v0.0.4 had a bug in accessing subgroups in netCDF data model in Windows OS. This was fixed in datatree v0.0.6, and hence we updated our dependency and made other associated changes to support cross platform users (#732, #748)
• Fix empty Sv problem related to renaming time coordinate associated with environmental parameters used for calibration; this is a regression bug from v0.6.0 changes (#755)
• Fix a bug in compute_MVBS in selecting echo_range for specific frequency. This is a regression bug from v0.6.0 changes (#736)
• Allow parsing data from EK60 with split-beam transducers but without phase/angle data (#491, #718)
• Check filter coeffs existence in SetGroupsEK80.set_vendor before saving (#720, #724)
• Fix the check in compute_MVBS for handling different variations of NaN entries in the Sv dataset (#753)
• Fix invalid timestamp issue in AD2CP data conversion (#733)

New features

• Enhance update_platform to support a new use case (location data from fixed location) and add more consistency (#741)
• Ability to parse and store RAW4 datagram for EK80 data (#714)
• Add utility function for swapping channel coordinate with frequency_nominal (#710)
• Add ES70, ES80, EA640 to allowed data type for calibration (#759)

Changes of netCDF data model

• Reorganize AD2CP data variables into different Sonar/Beam_groupXs and different first-level groups in a form consistent with v0.6.0 changes for all other sonar models (#731); some variables remain to be discussed and may change in future releases (#719)

Enhancements

• Refactor AD2CP conversion to improve speed and memory usage through removal of xr.merge (#505)
• Update Python requirements in docs to >=3.8 (#744)

Infrastructure

• Update PR action to use PR title [all tests ci] to run the entire suite of tests and [skip ci] to skip all tests (#721)

v0.6.0

Overview

This is a major release that contains changes that enhances the compliance of echopype data model (and hence generated file structure) to the SONAR-netCDF4 convention Version 1.0 . In addition, some variables were renamed to improve intuitive understanding of sonar data, provenance and standardized attributes are added to the processed dataset (e.g. Sv), the deprecated old API (<0.5.0) was removed, and some bugs were fixed.

Changes of netCDF data model

• Move and rename the original Beam and Beam_power group to be subgroups under the Sonar group, in order to comply with the structure defined in the convention (#567, #574, #605, #606, #611)
  • Beam --> Sonar/Beam_group1: contains either raw power or power/angle data for all sonar models other than EK80. For EK80, if only complex or power/angle data exist, all data are in this group; if both complex and power/angle data exist, the complex data are in this group.
  • Beam_power --> Sonar/Beam_group2: contains power/angle when complex data occupies Sonar/Beam_group1; only exists for EK80 data when both power/angle data and complex data bothexist in the file
• Rename the coordinate range_bin to range_sample to make it obvious that this coordinate indicates the digitization sample number for the sound waveform or intensity time series, and hence it takes a form of sequential integers 0, 1, 2, 3, ... (#595)
• Rename the data variable range in the calibrated Sv or TS dataset to echo_range, so that it is not confused with the python built-in function (#590)
• Rename the coordinate quadrant for EK80 data to beam (#619)
• Add coordinate beam with length 1 for all sonar models, except for AD2CP (#638, #646)
• Rename the data variable Sp to TS since "point backscattering strength" is a Simrad terminology and target strength (TS) is clearly defined and widely used. (#615)
• Rename time dimensions in the Platform group (location_time: time1, mru_time: time2) (#518, #631, #647)
• Rename the coordinate frequency to channel for all groups, to be more flexible (can accommodate channels with identical frequencies #490) and reasonable (since for broadband data the channel frequency is only nominal #566) (#657)
• Rename the data variable heave to vertical_offset in the Platform group (#592, #623)
• Change src_filenames string attribute to source_filenames list-of-strings variable (#620, #621)
• Bring consistency to the names of the time coordinates for the Platform and Environment groups (#656, #672)

Changes of EchoData group access pattern

• The groups can now be accessed via a path in the form echodata["Sonar/Platform"], echodata["Sonar/Platform"], echodata["Sonar/Beam_groupX"], etc. using DataTree v0.0.4 functionalities. (#611)
• The previous access pattern echodata.platform, echodata.sonar, echodata.beam, etc. is deprecated and will be removed in v0.6.1.

Addition of attributes and variables in raw-converted and processed data

• Add indexing info for Beam_groupX as data variable under the Sonar group (#658)
• Add missing coordinate and variable attributes in the processed datasets Sv, MVBS, TS(#594)
• Add water_level to processed datasets (Sv, MVBS, TS) for convenient downstream conversion to depth (#259, #583, #615)
• Add additional environment variables for EK80 data (#616)
• Add missing platform data variables (#592, #631, #649, #654)

New features and other enhancements

• Add parser for Echoview ECS file (#510)
• Add provenance to raw-converted and processed datasets (#621)
• Consolidate convention specs into a single yml file for pre-loading when creating EchoData objects (#565)
• Extra visualization module can now handle both frequency and channel filtering, as well as files with duplicated frequencies (#660)
• Improve selection of Vendor-specific calibration parameters for narrowband EK data (#697)

CI improvements

• Upgrade python minimum to 3.8 and tidy up packaging (#604, #608, #609)
• Upgrade echopype development status to Beta (#610)
• Update setup-services.py to include images & volumes subtleties (#651)

Other changes

• Remove the deprecated old (<0.5.0) API (#506, #601)
• Update README in the echopype/test_data folder (#584)
• Add documentation for visualization (#655)
• Add development roadmap to documentation (#636, #688)
• Restructure and expand data format section (#635)

v0.5.6

This is a minor release that contains an experimental new feature and a number of enhancements, clean-up and bug fixes, which pave the way for the next major release.

New feature

• (beta) Allow interpolating CTD data in calibration (#464)
  • Interpolation currently allowed along the ping_time dimension (the "stationary" case) and across latitude and longitude (the "mobile" case).
  • This mechanism is enabled via a new EnvParams class at input of calibration functions.

Enhancements

• Make visualize module fully optional with matplotlib, cmocean being optional dependency (#526, #559)
• Set range entries with no backscatter data to NaN in output of echodata.compute_range() (#547) and still allows quick visualization (#555)
• Add codespell GitHub action to ensure correct spellings of words (#557)
• Allow sonar_model="EA640" for open_raw (before it had to be "EK80") (#539)

Bug fixes

• Allow using sonar_model="EA640" (#538, #539)
• Allow flexible and empty environment variables in EA640/EK80 files (#537)
• Docstring overhaul and fix bugs in utils.uwa (#525)

Documentation

• Upgrade echopype docs to use jupyter book (#543)
• Change the RTD latest to point to the dev branch (#467)

Testing

• Update convert tests to enable parallel testing (#556)
• Overhaul tests (#523, #498)
  • use pytest.fixture for testing
  • add ES70/ES80/EA640 test files
  • add new EK80 small test files with parameter combinations
  • reduce size for a subset of large EK80 test data files
• Add packaging testing for the dev branch (#554)

v0.5.5

This is a minor release that includes new features, enhancements, bug fixes, and linking to an echopype preprint.

New features

• Allow converting ES60/70/80 files and handle various datagram anomaly (#409)
• Add simple echogram plotting functionality (beta) (#436)

Enhancements

• update_platform method for EchoData now include proper variable attributes and correctly selects time range of platform data variables corresponding to those of the acoustic data (#476, #492, #493, #488)
• Improve testing for preprocess.compute_MVBS by running through real data for all supported sonar models (#454)
• Generalize handling of Beam group coordinate attributes and a subset of variable attributes (#480, #493)
• Allow optional kwargs when loading EchoData groups to enable delaying operations (#456)

Bug fixes

• The gain factor for band-integrated Sv is now computed from broadband calibration data stored in the Vendor group (when available) or use nominal narrowband values (#446, #477)
• Fix time variable encoding for combine_echodata (#486)
• Fix missing ping_time dimension in AZFP Sv dataset to enable MVBS computation (#453)
• Fix bugs re path when writing to cloud (#462)

Documentation

• Improvements to the "Contributing to echopype" page: Elaborate on the git branch workflow. Add description of PR squash and merge vs merge commit. Add instructions for running only a subset of tests locally (#482)
• Add documentation about output_storage_options for writing to cloud storage (#482)
• Add documentation and docstring for sonar_model in open_raw (#475)
• Improve documentation of EchoData object by adding a sample of the xarray Dataset HTML browser (#503)

Others

• Zenodo badge update (#469)
• Add github citation file (#496), linking to echopype preprint on arXiv

v0.5.4

Overview

This is a minor release that contains a few bug fixes and new functionalities.
The repo has migrated to use main instead of master after this release.

New features

• Adding external platform-related data (e.g., latitude, longitude) to the EchoData object via the update_platform method (#434)
• Allow converting and storing data with duplicated ping times (#433)
• Add simple functions to compute summary statistics under the metrics subpackage (#444)

Bug fixes

• Allow string info in AD2CP data packet header (#438)
• Re-attach sonar_model attribute to outputs of combine_echodata (#437)
• Handle exception in open_converted due to potentially empty beam_power group in Zarr files (#447)

Others

• Warn users of removal of old API in the next release (#443)

v0.5.3

Overview

This is a minor release that adds a few new functionalities, in particular a method to combine multiple EchoData objects, addresses a few bugs, improves packaging by removing pinning for dependencies, and improving the testing framework.

New features

• Add a new method to combine multiple EchoData objects (#383, #414, #422, #425 )
  • Potential time reversal problems in time coordinates (e.g., ping_time, location_time) are checked and corrected as part of the combine function
  • The original timestamps are stored in the Provenance group
• Add a new method compute_range for EchoData object (#400)
• Allow flexible extensions for AZFP files in the form ".XXY" where XX is a number and Y is a letter (#428)

Bug fixes

• Fix the bug/logic problems that prevented calibrating data in EK80 files that contains coexisting BB and CW data (#400)
• Fix the bug that prevented using the latest version of fsspec (#401)
• Fix the bug that placed echosounder_raw_transmit_samples_i/q as the first ping in echosounder_raw_samples_i/q as they should be separate variables (#427)

Improvements

• Consolidate functions that handle local/remote paths and checking file existence (#401)
• Unpin all dependencies (#401)
• Improve test coverage accuracy (#411)
• Improve testing structure to match with subpackage structure (#401, #416, #429 )

Documentaion

• Expand Contributing to echopype page, including development workflow and testing strategy (#417, #420, #423)

v0.5.2

Overview

This is a minor release that addresses issues related to time encoding for data variables related to platform locations and data conversion/encoding for AD2CP data files.

Bug fixes and improvements

• Fixed the location_time encoding in the Platform group for latitude and longitude data variables (#393)
• Fixed the location_time encoding in the Platform/NMEA group (#395)
• Updated EchoData repr to show Platform/NMEA (#395, #396)
• Improved AD2CP data parsing and conversion (#388)
  • Cleaned up organization of data from different sampling modes and their corresponding time coordinates
  • Fixed parsing issues that generated spikes in parsed echosounder mode amplitude data
  • Removed the Beam_complex group and put raw IQ samples in the Vendor group per convention requirements
  • Populated the Sonar group with AD2CP information