From c4dcdbda322533cadd43aa5746b65cc782320a53 Mon Sep 17 00:00:00 2001 From: Emilio Mayorga Date: Sat, 22 May 2021 19:44:26 -0700 Subject: [PATCH 1/6] docs: Data processing page, added text about environmental parameters in EK60; fixed small mistake --- docs/source/process.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/source/process.rst b/docs/source/process.rst index dd8792d85..63d7b5a6d 100644 --- a/docs/source/process.rst +++ b/docs/source/process.rst @@ -81,8 +81,8 @@ The steps for performing these analyses are summarized below: # Remove noise ds_Sv_clean = ep.preprocess.remove_noise( # obtain a denoised Sv dataset ds_Sv, # calibrated Sv dataset - range_bin_num=30 # number of samples along the range_bin dimension for estimating noise - ping_num=5, # number of pings for estimating noise + range_bin_num=30, # number of samples along the range_bin dimension for estimating noise + ping_num=5 # number of pings for estimating noise ) The functions in the ``calibrate`` subpackage take in an ``EchoData`` object, @@ -125,7 +125,9 @@ critical in biological interpretation of ocean sonar data. They influence: By default, echopype uses the following for calibration: -- EK60 and EK80: Environmental parameters saved with the raw data files. +- EK60 and EK80: Environmental parameters saved with the raw data files. + For EK60, temperature, salinity and pressure are entered by the instrument + operator but only calculated sound speed is saved into the raw file. - AZFP: Salinity and pressure provided by the user, and temperature recorded at the instrument. From 49ea0070166db231a0b18922a91fb956d446f237 Mon Sep 17 00:00:00 2001 From: Emilio Mayorga Date: Mon, 5 Jul 2021 17:30:29 -0400 Subject: [PATCH 2/6] Restore location_time encoding assignment (#393) * docs: Data processing page, added text about environmental parameters in EK60; fixed small mistake * Add location_time back to encodings dict; refactor DEFAULT_ENCODINGS entries to refer to a new DEFAULT_TIME_ENCODING DICT --- docs/source/contributing.rst | 2 +- echopype/convert/set_groups_base.py | 57 ++++++++--------------------- 2 files changed, 16 insertions(+), 43 deletions(-) diff --git a/docs/source/contributing.rst b/docs/source/contributing.rst index 85520ba83..d143282fe 100644 --- a/docs/source/contributing.rst +++ b/docs/source/contributing.rst @@ -43,7 +43,7 @@ Create a `conda `_ environment for echopype development .. code-block:: bash - conda install -c conda-forge -n echopype --yes python=3.8 --file requirements.txt --file requirements-dev.txt + conda create -c conda-forge -n echopype --yes python=3.8 --file requirements.txt --file requirements-dev.txt conda activate echopype # ipykernel is recommended, in order to use with JupyterLab and IPython # to aid with development. We recommend you install JupyterLab separately diff --git a/echopype/convert/set_groups_base.py b/echopype/convert/set_groups_base.py index 891594983..d89552044 100644 --- a/echopype/convert/set_groups_base.py +++ b/echopype/convert/set_groups_base.py @@ -15,49 +15,22 @@ DEFAULT_CHUNK_SIZE = {"range_bin": 25000, "ping_time": 2500} +DEFAULT_TIME_ENCODING = { + "units": "seconds since 1900-01-01T00:00:00+00:00", + "calendar": "gregorian", + "_FillValue": np.nan, + "dtype": np.dtype("float64"), +} + DEFAULT_ENCODINGS = { - "ping_time": { - "units": "seconds since 1900-01-01T00:00:00+00:00", - "calendar": "gregorian", - "_FillValue": np.nan, - "dtype": np.dtype("float64"), - }, - "ping_time_burst": { - "units": "seconds since 1900-01-01T00:00:00+00:00", - "calendar": "gregorian", - "_FillValue": np.nan, - "dtype": np.dtype("float64"), - }, - "ping_time_average": { - "units": "seconds since 1900-01-01T00:00:00+00:00", - "calendar": "gregorian", - "_FillValue": np.nan, - "dtype": np.dtype("float64"), - }, - "ping_time_echosounder": { - "units": "seconds since 1900-01-01T00:00:00+00:00", - "calendar": "gregorian", - "_FillValue": np.nan, - "dtype": np.dtype("float64"), - }, - "ping_time_echosounder_raw": { - "units": "seconds since 1900-01-01T00:00:00+00:00", - "calendar": "gregorian", - "_FillValue": np.nan, - "dtype": np.dtype("float64"), - }, - "ping_time_echosounder_raw_transmit": { - "units": "seconds since 1900-01-01T00:00:00+00:00", - "calendar": "gregorian", - "_FillValue": np.nan, - "dtype": np.dtype("float64"), - }, - "mru_time": { - "units": "seconds since 1900-01-01T00:00:00+00:00", - "calendar": "gregorian", - "_FillValue": np.nan, - "dtype": np.dtype("float64"), - }, + "ping_time": DEFAULT_TIME_ENCODING, + "ping_time_burst": DEFAULT_TIME_ENCODING, + "ping_time_average": DEFAULT_TIME_ENCODING, + "ping_time_echosounder": DEFAULT_TIME_ENCODING, + "ping_time_echosounder_raw": DEFAULT_TIME_ENCODING, + "ping_time_echosounder_raw_transmit": DEFAULT_TIME_ENCODING, + "location_time": DEFAULT_TIME_ENCODING, + "mru_time": DEFAULT_TIME_ENCODING, } From 781bfbbec9e971745562469f96ca27e601dd0490 Mon Sep 17 00:00:00 2001 From: imranmaj <49664304+imranmaj@users.noreply.github.com> Date: Sat, 17 Jul 2021 12:42:49 -0700 Subject: [PATCH 3/6] AD2CP Parsing Improvements (#388) * Add length check to _encode_dataarray * Use set_encodings, rename time_* to ping_time_* * Perform length check earlier in _encode_dataarray * Fix Beam_complex group being saved as Beam * Add time encoding conversion test for AD2CP * Change group hasattr check to None check (all groups are initialized as None even if they don't exist) * Add specific encoding for data * Index variables by their individual ping time variables instead of overall ping_time * Autoformatter * Use allclose instead of isclose * Remove velocity_echosounder * Fix AD2CP tests to match time index changes * Move echoounder raw data samples to Vendor group * Autoformatter * Fix spikes in echosounder data * Rename Real/Imaginary samples to I/Q samples * Encode AD2CP sonar group * Fix sonar_type and add sonar_firmware_version in sonar group * Format sonar_firmware_version as a string * Update AD2CP sonar group sonar_type value * Update sonar_type AD2CP value * Index echosounder data by ping_time_echosounder instead of ping_time, remove amplitude_echosounder, rename echosounder to amplitude_echosounder --- echopype/convert/api.py | 15 +- echopype/convert/parse_ad2cp.py | 740 +++++++++++++-------------- echopype/convert/set_groups_ad2cp.py | 87 ++-- echopype/convert/set_groups_azfp.py | 3 - echopype/convert/set_groups_base.py | 5 - echopype/convert/set_groups_ek60.py | 3 - echopype/convert/set_groups_ek80.py | 3 - echopype/echodata/convention/1.0.yml | 8 +- echopype/echodata/echodata.py | 1 - echopype/tests/test_convert_ad2cp.py | 74 +-- 10 files changed, 428 insertions(+), 511 deletions(-) diff --git a/echopype/convert/api.py b/echopype/convert/api.py index a1229fd90..817301be2 100644 --- a/echopype/convert/api.py +++ b/echopype/convert/api.py @@ -279,17 +279,6 @@ def _save_groups_to_file(echodata, output_path, engine, compress=True): compression_settings=COMPRESSION_SETTINGS[engine] if compress else None, ) - # Beam Complex Group: only AD2CP has Beam Complex - if hasattr(echodata, "beam_complex") and echodata.beam_complex is not None: - io.save_file( - echodata.beam_complex, - path=output_path, - mode="a", - engine=engine, - group="Beam_complex", - compression_settings=COMPRESSION_SETTINGS[engine] if compress else None, - ) - # Platform group io.save_file( echodata.platform, # TODO: chunking necessary? location_time and mru_time (EK80) only @@ -301,7 +290,7 @@ def _save_groups_to_file(echodata, output_path, engine, compress=True): ) # Platform/NMEA group: some sonar model does not produce NMEA data - if hasattr(echodata, "nmea"): + if echodata.nmea is not None: io.save_file( echodata.nmea, # TODO: chunking necessary? path=output_path, @@ -547,8 +536,6 @@ def open_raw( echodata.beam, echodata.beam_power = setgrouper.set_beam() else: echodata.beam = setgrouper.set_beam() - if sonar_model == "AD2CP": - echodata.beam_complex = setgrouper.set_beam_complex() echodata.vendor = setgrouper.set_vendor() return echodata diff --git a/echopype/convert/parse_ad2cp.py b/echopype/convert/parse_ad2cp.py index 1abaeb287..68f62a58f 100644 --- a/echopype/convert/parse_ad2cp.py +++ b/echopype/convert/parse_ad2cp.py @@ -111,6 +111,11 @@ class Dimension(Enum): """ TIME = "ping_time" + TIME_AVERAGE = "ping_time_average" + TIME_BURST = "ping_time_burst" + TIME_ECHOSOUNDER = "ping_time_echosounder" + TIME_ECHOSOUNDER_RAW = "ping_time_echosounder_raw" + TIME_ECHOSOUNDER_RAW_TRANSMIT = "ping_time_echosounder_raw_transmit" BEAM = "beam" RANGE_BIN_BURST = "range_bin_burst" RANGE_BIN_AVERAGE = "range_bin_average" @@ -297,7 +302,10 @@ def parse_config(data: str) -> Dict[str, Dict[str, Any]]: result[tokens[0]] = line_dict return result - def get_pulse_compressed(self): + def get_firmware_version(self) -> Optional[Dict[str, Any]]: + return self.config.get("GETHW") + + def get_pulse_compressed(self) -> int: for i in range(1, 3 + 1): if "GETECHO" in self.config and self.config["GETECHO"][f"PULSECOMP{i}"] > 0: return i @@ -539,20 +547,23 @@ def _parse(value: bytes, data_type: DataType) -> Any: elif data_type == DataType.STRING: return value.decode("utf-8") elif data_type == DataType.SIGNED_INTEGER: - return int.from_bytes(value, byteorder="little", signed=True) + return np.int64(int.from_bytes(value, byteorder="little", signed=True)) elif data_type == DataType.UNSIGNED_INTEGER: - return int.from_bytes(value, byteorder="little", signed=False) + return np.int64(int.from_bytes(value, byteorder="little", signed=False)) # elif data_type == DataType.UNSIGNED_LONG: # return struct.unpack(" 0: packets = merge_attrs(packets) - return xr.concat(packets, dim="ping_time", combine_attrs="override") + return xr.combine_by_coords( + packets, + data_vars="minimal", + coords="minimal", + combine_attrs="override", + ) else: return None @@ -201,14 +206,15 @@ def set_beam(self) -> xr.Dataset: "velocity_scaling": self.ds.get("velocity_scaling"), "velocity_burst": self.ds.get("velocity_data_burst"), "velocity_average": self.ds.get("velocity_data_average"), - "velocity_echosounder": self.ds.get("velocity_data_echosounder"), + # "velocity_echosounder": self.ds.get("velocity_data_echosounder"), "amplitude_burst": self.ds.get("amplitude_data_burst"), "amplitude_average": self.ds.get("amplitude_data_average"), - "amplitude_echosounder": self.ds.get("amplitude_data_echosounder"), + # "amplitude_echosounder": self.ds.get("amplitude_data_echosounder"), "correlation_burst": self.ds.get("correlation_data_burst"), "correlation_average": self.ds.get("correlation_data_average"), "correlation_echosounder": self.ds.get("correlation_data_echosounder"), - "echosounder": self.ds.get("echosounder_data"), + # "echosounder": self.ds.get("echosounder_data"), + "amplitude_echosounder": self.ds.get("echosounder_data"), "figure_of_merit": self.ds.get("figure_of_merit_data"), "altimeter_distance": self.ds.get("altimeter_distance"), "altimeter_quality": self.ds.get("altimeter_quality"), @@ -316,18 +322,34 @@ def set_vendor(self) -> xr.Dataset: "std_dev_roll": self.ds.get("std_dev_roll"), "std_dev_heading": self.ds.get("std_dev_heading"), "std_dev_pressure": self.ds.get("std_dev_pressure"), + "echosounder_raw_samples_i": self.ds.get("echosounder_raw_samples_i"), + "echosounder_raw_samples_q": self.ds.get("echosounder_raw_samples_q"), + "echosounder_raw_transmit_samples_i": self.ds.get( + "echosounder_raw_transmit_samples_i" + ), + "echosounder_raw_transmit_samples_q": self.ds.get( + "echosounder_raw_transmit_samples_q" + ), + "echosounder_raw_beam": self.ds.get("echosounder_raw_beam"), + "echosounder_raw_echogram": self.ds.get("echosounder_raw_echogram"), }, coords={ "ping_time": self.ds.get("ping_time"), "ping_time_burst": self.ds.get("ping_time_burst"), "ping_time_average": self.ds.get("ping_time_average"), "ping_time_echosounder": self.ds.get("ping_time_echosounder"), + "ping_time_echosounder_raw": self.ds.get("ping_time_echosounder_raw"), + "ping_time_echosounder_raw_transmit": self.ds.get( + "ping_time_echosounder_raw_transmit" + ), + "sample": self.ds.get("sample"), + "sample_transmit": self.ds.get("sample_transmit"), "beam": self.ds.get("beam"), "range_bin_average": self.ds.get("range_bin_average"), "range_bin_burst": self.ds.get("range_bin_burst"), "range_bin_echosounder": self.ds.get("range_bin_echosounder"), }, - attrs=attrs, + attrs={**attrs, "pulse_compressed": self.pulse_compressed}, ) ds = ds.reindex( { @@ -346,32 +368,23 @@ def set_vendor(self) -> xr.Dataset: return set_encodings(ds) - def set_beam_complex(self) -> xr.Dataset: + def set_sonar(self) -> xr.Dataset: ds = xr.Dataset( - data_vars={ - "echosounder_raw_samples_r": self.ds.get("echosounder_raw_samples_r"), - "echosounder_raw_samples_i": self.ds.get("echosounder_raw_samples_i"), - "echosounder_raw_transmit_samples_r": self.ds.get( - "echosounder_raw_transmit_samples_r" - ), - "echosounder_raw_transmit_samples_i": self.ds.get( - "echosounder_raw_transmit_samples_i" - ), - "echosounder_raw_beam": self.ds.get("echosounder_raw_beam"), - "echosounder_raw_echogram": self.ds.get("echosounder_raw_echogram"), - }, - coords={ - "ping_time": self.ds.get("ping_time"), - "ping_time_echosounder_raw": self.ds.get("ping_time_echosounder_raw"), - "ping_time_echosounder_raw_transmit": self.ds.get( - "ping_time_echosounder_raw_transmit" - ), - "sample": self.ds.get("sample"), - "sample_transmit": self.ds.get("sample_transmit"), - }, - attrs={"pulse_compressed": self.pulse_compressed}, + attrs={ + "sonar_manufacturer": "Nortek", + "sonar_model": "AD2CP", + "sonar_serial_number": "", + "sonar_software_name": "", + "sonar_software_version": "", + "sonar_firmware_version": "", + "sonar_type": "acoustic Doppler current profiler (ADCP)", + } ) - return set_encodings(ds) - - def set_sonar(self) -> xr.Dataset: - return xr.Dataset() + if "serial_number" in self.ds: + ds.attrs["sonar_serial_number"] = int(self.ds["serial_number"].data[0]) + firmware_version = self.parser_obj.get_firmware_version() + if firmware_version is not None: + ds.attrs["sonar_firmware_version"] = ", ".join( + [f"{k}:{v}" for k, v in firmware_version.items()] + ) + return ds diff --git a/echopype/convert/set_groups_azfp.py b/echopype/convert/set_groups_azfp.py index 4ee51f335..e9a9ed8d6 100644 --- a/echopype/convert/set_groups_azfp.py +++ b/echopype/convert/set_groups_azfp.py @@ -195,9 +195,6 @@ def set_beam(self) -> xr.Dataset: ) return set_encodings(ds) - def set_beam_complex(self) -> xr.Dataset: - return xr.Dataset() - def set_vendor(self) -> xr.Dataset: """Set the Vendor-specific group.""" unpacked_data = self.parser_obj.unpacked_data diff --git a/echopype/convert/set_groups_base.py b/echopype/convert/set_groups_base.py index d89552044..7972567fa 100644 --- a/echopype/convert/set_groups_base.py +++ b/echopype/convert/set_groups_base.py @@ -151,11 +151,6 @@ def set_beam(self) -> xr.Dataset: """Set the Beam group.""" raise NotImplementedError - @abc.abstractmethod - def set_beam_complex(self) -> xr.Dataset: - """Set the Beam_complex group""" - raise NotImplementedError - @abc.abstractmethod def set_platform(self) -> xr.Dataset: """Set the Platform group.""" diff --git a/echopype/convert/set_groups_ek60.py b/echopype/convert/set_groups_ek60.py index b005e91ad..840837277 100644 --- a/echopype/convert/set_groups_ek60.py +++ b/echopype/convert/set_groups_ek60.py @@ -572,9 +572,6 @@ def set_beam(self) -> xr.Dataset: return set_encodings(ds) - def set_beam_complex(self) -> xr.Dataset: - return xr.Dataset() - def set_vendor(self) -> xr.Dataset: # Retrieve pulse length and sa correction config = self.parser_obj.config_datagram["transceivers"] diff --git a/echopype/convert/set_groups_ek80.py b/echopype/convert/set_groups_ek80.py index 927f41f24..97e17cb10 100644 --- a/echopype/convert/set_groups_ek80.py +++ b/echopype/convert/set_groups_ek80.py @@ -653,9 +653,6 @@ def merge_save(ds_combine, ds_type, group_name): return [ds_beam, ds_beam_power] - def set_beam_complex(self) -> xr.Dataset: - return xr.Dataset() - def set_vendor(self) -> xr.Dataset: """Set the Vendor-specific group.""" config = self.parser_obj.config_datagram["configuration"] diff --git a/echopype/echodata/convention/1.0.yml b/echopype/echodata/convention/1.0.yml index 504e552bc..72f7b5fe0 100644 --- a/echopype/echodata/convention/1.0.yml +++ b/echopype/echodata/convention/1.0.yml @@ -20,6 +20,10 @@ groups: name: Platform description: contains information about the platform on which the sonar is installed. ep_group: Platform + nmea: + name: NMEA + description: contains information specific to the NMEA protocol + ep_group: Nmea provenance: name: Provenance description: contains metadata about how the SONAR-netCDF4 version of the data were obtained. @@ -32,10 +36,6 @@ groups: name: Beam description: contains backscatter data and other beam or channel-specific data. ep_group: Beam - beam_complex: - name: Beam Complex - description: contains raw AD2CP data samples from the echosounder mode. - ep_group: Beam_complex beam_power: name: Beam Power description: contains backscatter data (power-only) and other beam or channel-specific data. diff --git a/echopype/echodata/echodata.py b/echopype/echodata/echodata.py index eef995cc5..e18d9ea0c 100644 --- a/echopype/echodata/echodata.py +++ b/echopype/echodata/echodata.py @@ -126,7 +126,6 @@ def _load_file(self, raw_path): """Lazy load Top-level, Beam, Environment, and Vendor groups from raw file.""" for group, value in self.__group_map.items(): # EK80 data may have a Beam_power group if both complex and power data exist. - # ADCP data adds a Beam_complex group ds = None try: ds = self._load_group( diff --git a/echopype/tests/test_convert_ad2cp.py b/echopype/tests/test_convert_ad2cp.py index 70e9e86d1..51a1258a1 100644 --- a/echopype/tests/test_convert_ad2cp.py +++ b/echopype/tests/test_convert_ad2cp.py @@ -83,7 +83,7 @@ def test_raw_output(): if base.attrs[f"Instrument_echo_pulseComp{i}"]: pulse_compressed = i break - assert echodata.beam_complex.attrs["pulse_compressed"] == pulse_compressed + assert echodata.vendor.attrs["pulse_compressed"] == pulse_compressed base.close() # check raw data transmit samples @@ -100,51 +100,31 @@ def test_raw_output(): group="Data/RawEcho1_1000kHzTx", ) if "090" in filepath.parts: - assert np.isclose( - echodata.beam_complex["echosounder_raw_transmit_samples_r"] - .sel( - ping_time=echodata.beam_complex[ - "ping_time_echosounder_raw_transmit" - ] - ) + assert np.allclose( + echodata.vendor["echosounder_raw_transmit_samples_i"] .data.flatten(), base["DataI"].data.flatten(), atol=ABSOLUTE_TOLERANCE, - ).all() - assert np.isclose( - echodata.beam_complex["echosounder_raw_transmit_samples_i"] - .sel( - ping_time=echodata.beam_complex[ - "ping_time_echosounder_raw_transmit" - ] - ) + ) + assert np.allclose( + echodata.vendor["echosounder_raw_transmit_samples_q"] .data.flatten(), base["DataQ"].data.flatten(), atol=ABSOLUTE_TOLERANCE, - ).all() + ) else: - assert np.isclose( - echodata.beam_complex["echosounder_raw_transmit_samples_r"] - .sel( - ping_time=echodata.beam_complex[ - "ping_time_echosounder_raw_transmit" - ] - ) + assert np.allclose( + echodata.vendor["echosounder_raw_transmit_samples_i"] .data.flatten(), base["Data_I"].data.flatten(), atol=ABSOLUTE_TOLERANCE, - ).all() - assert np.isclose( - echodata.beam_complex["echosounder_raw_transmit_samples_i"] - .sel( - ping_time=echodata.beam_complex[ - "ping_time_echosounder_raw_transmit" - ] - ) + ) + assert np.allclose( + echodata.vendor["echosounder_raw_transmit_samples_q"] .data.flatten(), base["Data_Q"].data.flatten(), atol=ABSOLUTE_TOLERANCE, - ).all() + ) base.close() # check raw data samples @@ -152,34 +132,30 @@ def test_raw_output(): str(ocean_contour_converted_data_path), group="Data/RawEcho1_1000kHz" ) if "090" in filepath.parts: - assert np.isclose( - echodata.beam_complex["echosounder_raw_samples_r"] - .sel(ping_time=echodata.beam_complex["ping_time_echosounder_raw"]) + assert np.allclose( + echodata.vendor["echosounder_raw_samples_i"] .data.flatten(), base["DataI"].data.flatten(), atol=ABSOLUTE_TOLERANCE, - ).all() - assert np.isclose( - echodata.beam_complex["echosounder_raw_samples_i"] - .sel(ping_time=echodata.beam_complex["ping_time_echosounder_raw"]) + ) + assert np.allclose( + echodata.vendor["echosounder_raw_samples_q"] .data.flatten(), base["DataQ"].data.flatten(), atol=ABSOLUTE_TOLERANCE, - ).all() + ) else: # note the transpose - assert np.isclose( - echodata.beam_complex["echosounder_raw_samples_r"] - .sel(ping_time=echodata.beam_complex["ping_time_echosounder_raw"]) + assert np.allclose( + echodata.vendor["echosounder_raw_samples_i"] .data.flatten(), base["Data_I"].data.T.flatten(), atol=ABSOLUTE_TOLERANCE, - ).all() - assert np.isclose( - echodata.beam_complex["echosounder_raw_samples_i"] - .sel(ping_time=echodata.beam_complex["ping_time_echosounder_raw"]) + ) + assert np.allclose( + echodata.vendor["echosounder_raw_samples_q"] .data.flatten(), base["Data_Q"].data.T.flatten(), atol=ABSOLUTE_TOLERANCE, - ).all() + ) base.close() From 7d68d52a044177f3656d689120aeed364fa4d092 Mon Sep 17 00:00:00 2001 From: Emilio Mayorga Date: Sat, 17 Jul 2021 12:56:21 -0700 Subject: [PATCH 4/6] Fixed Platform/NMEA group and its location_time encoding handling; test_convert_time_encodings ek60 and ek80 tests are now passing --- echopype/convert/set_groups_base.py | 4 +--- echopype/echodata/convention/1.0.yml | 4 ++-- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/echopype/convert/set_groups_base.py b/echopype/convert/set_groups_base.py index 7972567fa..74616608a 100644 --- a/echopype/convert/set_groups_base.py +++ b/echopype/convert/set_groups_base.py @@ -184,17 +184,15 @@ def set_nmea(self) -> xr.Dataset: time, { "axis": "T", - "calendar": "gregorian", "long_name": "Timestamps for NMEA datagrams", "standard_name": "time", - "units": "seconds since 1900-01-01 00:00:00Z", }, ) }, attrs={"description": "All NMEA sensor datagrams"}, ) - return ds + return set_encodings(ds) @abc.abstractmethod def set_vendor(self) -> xr.Dataset: diff --git a/echopype/echodata/convention/1.0.yml b/echopype/echodata/convention/1.0.yml index 72f7b5fe0..abb553ad3 100644 --- a/echopype/echodata/convention/1.0.yml +++ b/echopype/echodata/convention/1.0.yml @@ -21,9 +21,9 @@ groups: description: contains information about the platform on which the sonar is installed. ep_group: Platform nmea: - name: NMEA + name: Platform/NMEA description: contains information specific to the NMEA protocol - ep_group: Nmea + ep_group: Platform/NMEA provenance: name: Provenance description: contains metadata about how the SONAR-netCDF4 version of the data were obtained. From a10cf91ee6840588f511dd7e32a29fddb04ac9da Mon Sep 17 00:00:00 2001 From: Wu-Jung Lee Date: Sun, 18 Jul 2021 10:42:27 -0700 Subject: [PATCH 5/6] add nmea to expected_repr in test_echodata.py (#396) --- echopype/echodata/convention/1.0.yml | 2 +- echopype/tests/test_echodata.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/echopype/echodata/convention/1.0.yml b/echopype/echodata/convention/1.0.yml index abb553ad3..de85bdd0a 100644 --- a/echopype/echodata/convention/1.0.yml +++ b/echopype/echodata/convention/1.0.yml @@ -22,7 +22,7 @@ groups: ep_group: Platform nmea: name: Platform/NMEA - description: contains information specific to the NMEA protocol + description: contains information specific to the NMEA protocol. ep_group: Platform/NMEA provenance: name: Provenance diff --git a/echopype/tests/test_echodata.py b/echopype/tests/test_echodata.py index 229eb46e6..d341f3142 100644 --- a/echopype/tests/test_echodata.py +++ b/echopype/tests/test_echodata.py @@ -73,6 +73,7 @@ def test_repr(self): > top: (Top-level) contains metadata about the SONAR-netCDF4 file format. > environment: (Environment) contains information relevant to acoustic propagation through water. > platform: (Platform) contains information about the platform on which the sonar is installed. + > nmea: (Platform/NMEA) contains information specific to the NMEA protocol. > provenance: (Provenance) contains metadata about how the SONAR-netCDF4 version of the data were obtained. > sonar: (Sonar) contains specific metadata for the sonar system. > beam: (Beam) contains backscatter data and other beam or channel-specific data. From e48f8f392894397c2151547a5c181a8bf43ae190 Mon Sep 17 00:00:00 2001 From: Wu-Jung Lee Date: Sun, 18 Jul 2021 12:21:34 -0700 Subject: [PATCH 6/6] Add v0.5.2 release notes to docs what's new (#399) * add v0.5.2 release notes to docs what's new * add #396 to list * fix inline code styling * fix nested list --- docs/source/whats-new.rst | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/docs/source/whats-new.rst b/docs/source/whats-new.rst index 1d23e51ba..3a0e26eff 100644 --- a/docs/source/whats-new.rst +++ b/docs/source/whats-new.rst @@ -4,20 +4,43 @@ What's new See `GitHub releases page `_ for the complete history. +v0.5.2 (2021 Jul 18) +-------------------- + +Overview +~~~~~~~~ + +This is a minor release that addresses issues related to time encoding for data variables related to platform locations and data conversion/encoding for AD2CP data files. + +Bug fixes and improvements +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Fixed the ``location_time`` encoding in the ``Platform`` group for latitude and longitude data variables (#393) +- Fixed the ``location_time`` encoding in the ``Platform/NMEA`` group (#395) +- Updated ``EchoData`` repr to show ``Platform/NMEA`` (#395, #396) +- Improved AD2CP data parsing and conversion (#388) + + - Cleaned up organization of data from different sampling modes and their corresponding time coordinates + - Fixed parsing issues that generated spikes in parsed echosounder mode amplitude data + - Removed the ``Beam_complex`` group and put raw IQ samples in the ``Vendor`` group per convention requirements + - Populated the ``Sonar`` group with AD2CP information + + + v0.5.1 (2021 Jun 16) -------------------- Overview ~~~~~~~~ -This is a minor release that addresses a couple of issues from the last major version (0.5.0) +This is a minor release that addresses a couple of issues from the last major version (0.5.0) and improves code maintenance and testing procedures. New features ~~~~~~~~~~~~ -- Added experimental functions to detect and correct ``ping_time`` reversals. +- Added experimental functions to detect and correct ``ping_time`` reversals. See `qc` subpackage (#297) @@ -160,16 +183,23 @@ New features ~~~~~~~~~~~~ - Add EK80 support: + - File conversion from EK80 `.raw` files to netCDF and zarr formats - "Simple" calibration to frequency-average Sv based on pulse compression output is implemented but needs to be thoroughly tested. + - Rename subpackage `echopype.model` to `echopype.process` + - The new name better describes the subpackage's function to process data for further analysis - Also rename class `EchoData` to `Process` to mirror the structure in `Convert` better. - Importing using the old names will be deprecated in the next release. + - Overhaul converting multiple files with `combine_opt=True` + - If target format is netCDF, temporary files will be created and finally combined to a single netCDF. This is due to current restriction that xarray does not allow simply appending new data to an existing file. - If target format is zarr, data in each file are unpacked and appended to the same output file. + - Allow reading Zarr into `Process` in addition to netCDF: thanks @lsetiawan! + - Add a logo! Bug fixes