Skip to content

feat: handle zip lcms#212

Draft
baolanlequang wants to merge 57 commits intodevelopmentfrom
211-process-lcms-data
Draft

feat: handle zip lcms#212
baolanlequang wants to merge 57 commits intodevelopmentfrom
211-process-lcms-data

Conversation

@baolanlequang
Copy link
Member

No description provided.

* Fix conditional header generation for NMR JCAMP export

* Add tests to ensure that empty headers are not added to nmr files + Add a test file with empty headers
…ter (#220)

* Fix data normalization and improve handling of runs in JcampMSConverter

* Add unit test for JCAMP single point handling in the last line

* Update requirements.txt to use specific branch of nmrglue

* Change nmrglue version for unit test in CI
@PiTrem PiTrem changed the title feat WIP: handle zip lcms feat: handle zip lcms Nov 28, 2024
dependabot bot and others added 14 commits November 28, 2024 10:43
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.3 to 3.0.6.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@3.0.3...3.0.6)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.4 to 3.1.6.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](pallets/jinja@3.1.4...3.1.6)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
…MP-DX files (#225)

ensure the data block from  `__gen_ms_spectra` ends  with a new-line

so that  '##END NTUPLES=MASS SPECTRUM' is on a separate line in MS JCAMP-DX files

and spectra viewer can handle threshold properly
Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 22.0.0 to 23.0.0.
- [Release notes](https://github.com/benoitc/gunicorn/releases)
- [Commits](benoitc/gunicorn@22.0.0...23.0.0)

---
updated-dependencies:
- dependency-name: gunicorn
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
- Add `--simAsSpectra` flag to the msconvert command to handle SIM data properly.
- Update tests to cover ESI SIM RAW file conversion, ensuring future compatibility.
- Update CI workflow to start the required Docker container (`msconvert_docker`) and set correct permissions on the temporary directory (`chem_spectra/tmp`) for successful test execution.
Fix IR preview image margin regression
@harivyasi harivyasi requested a review from Copilot May 28, 2025 19:35
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces support for LCMS (Liquid Chromatography–Mass Spectrometry) data by adding a new converter (LCMSBaseConverter), composer (LCMSComposer), integrating them into the transformer model and API endpoint, and providing unit tests to validate the end-to-end flow.

  • Added tests/lib/converter/lcms/test_lcms_converter.py to verify raw CSV reading into structured data.
  • Added tests/lib/composer/test_lcms_composer.py to ensure JCAMP-DX generation for TIC, UV-VIS, and m/z spectra.
  • Updated TransformerModel, transform_api.py, and new composer/converter modules to wire LCMS support into the existing pipeline.

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/lib/converter/lcms/test_lcms_converter.py New unit tests for LCMSBaseConverter
tests/lib/composer/test_lcms_composer.py New unit tests for LCMSComposer
chem_spectra/model/transformer.py Integrated LCMS detection and dispatch in TransformerModel
chem_spectra/lib/converter/lcms/base.py Implemented LCMSBaseConverter for reading LCMS CSV files
chem_spectra/lib/composer/lcms.py Implemented LCMSComposer for JCAMP-DX output
chem_spectra/controller/transform_api.py Extended API to handle LCMSComposer responses
Comments suppressed due to low confidence (4)

tests/lib/converter/lcms/test_lcms_converter.py:5

  • There's a typo in the alias: LCMSConveter should be LCMSConverter to match the class name.
from chem_spectra.lib.converter.lcms.base import LCMSBaseConverter as LCMSConveter

tests/lib/composer/test_lcms_composer.py:92

  • This test function duplicates the name of the positive m/z test defined at line 77; rename one to test_lcms_composer_mz_spectra_negative to avoid collision.
def test_lcms_composer_mz_spectra_positive(zip_file):

chem_spectra/controller/transform_api.py:83

  • [nitpick] The spc_type for LCMSComposer is set to 'hplc', which may be confusing. Consider using 'lcms' to reflect the actual data type.
rsp.headers['X-Extra-Info-JSON'] = json.dumps({'spc_type': 'hplc', 'invalid_molfile': invalid_molfile})

chem_spectra/lib/converter/lcms/base.py:31

  • There's a typo in the variable name tic_postive_data; it should be tic_positive_data to maintain consistency and avoid confusion.
tic_postive_data = data_frame.to_dict(orient='list')

]
for idx, _ in enumerate(xs):
my_content = '{}, {};\n'.format(xs[idx], ys[idx])
msspc += my_content
Copy link

Copilot AI May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concatenating a string into a list using += will split it into characters. Use msspc.append(my_content) or msspc += [my_content] instead.

Suggested change
msspc += my_content
msspc.append(my_content)

Copilot uses AI. Check for mistakes.
'##ORIGIN=\n',
'##OWNER=\n',
'##SPECTROMETER/DATA SYSTEM=\n',
'##$CSCATEGORY=UVVIS SPECTRUM\n',
Copy link

Copilot AI May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The m/z spectra composer mistakenly includes the UVVIS category header. Replace this line with the appropriate category variable or remove it to avoid conflicting headers.

Suggested change
'##$CSCATEGORY=UVVIS SPECTRUM\n',

Copilot uses AI. Check for mistakes.
dependabot bot and others added 11 commits July 29, 2025 21:56
Bumps [requests](https://github.com/psf/requests) from 2.32.2 to 2.32.4.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.2...v2.32.4)

---
updated-dependencies:
- dependency-name: requests
  dependency-version: 2.32.4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…240)

If the first .jdx inside a BagIt archive had no XY data, JcampNIConverter raised a KeyError.
ChemSpectra stopped processing and returned a 500 error, even when valid files followed. (Issue #238)

- Wraps the creation of JcampNIConverter in a try/except.
- Files that miss both XYDATA and PEAKTABLE / XYPOINTS are skipped and logged.
- Processing continues with the remaining (valid) files.

=>
- ChemSpectra now imports BagIt archives even if some .jdx files are empty.
- Users see the valid spectra; empty files are silently ignored (a warning is logged on the backend).
Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.19 to 2.5.0.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.19...2.5.0)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.5.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… add devcontainer (#232)

- remove unncessary and old files
- update install instructions
- add a devcontainer

- Introduce pyproject.toml for build configuration 
  - Move dependency management from requirements.txt to pyproject.toml
  - Adopte loose versioning (e.g., `flask==3.*` instead of fixed patch versions) for core packages to ease maintenance and allow minor/patch upgrades automatically
  - Add project metadata: name, version, description, license, URLs, keywords, and Python version requirement (>=3.12)
  - Define build backend as setuptools
  - Replace `nmrglue` editable install with pinned commit `4005640`
  - Ensure version pins for Python 3.12 compatibility


 - Updated dependencies in requirements.txt:
  • Flask: 2.2.5 → 3.1.1  
  • Gunicorn: 20.0.0 → 23.0.0  
  • idna: 3.7 → 3.10  
  • itsdangerous: 2.0 → 2.2.0  
  • Jinja2: 3.1.2 → 3.1.6  
  • MarkupSafe: 2.1.2 → 3.0.2  
  • matplotlib: 3.7.3 → 3.10.5  
  • netCDF4: 1.5.3 → 1.7.2  
  • numpy: 1.22.4 → 2.3.2  
  • pandas: 2.0.3 → 2.3.1  
  • Pillow: 10.3.0 → 11.3.0  
  • pluggy: 0.12.0 → 1.6.0  
  • pymzml: 2.5.2 → 2.5.11  
  • pyopenms: 2.6.0 → 3.4.0  
  • pyparsing: 2.4.2 → 3.2.3  
  • pytest: 7.2.0 → 8.4.1  
  • python-dateutil: 2.8.2 → 2.9.0.post0  
  • pytz: 2023.3 → 2025.2  
  • rdkit: 2023.9.1 → 2025.3.5  
  • regex: 2019.4.9 → 2025.7.34  
  • scipy: 1.7.3 → 1.16.1  
  • six: 1.11.0 → 1.17.0  
  • urllib3: 1.x/2.x mix → 2.5.0  
  • Werkzeug: 3.0.6 → 3.1.3
* add dockerfile

* add files for dev2deploy process
Handle incomplete Bruker pdata folders: skip missing 1r/2rr and fall back to FID

skip Bruker processed dirs without binaries instead of raising error
Nicolass67 and others added 19 commits October 7, 2025 09:47
remove the fallback to Bruker 'ser' files in search_brucker_binary.

Only  1D data with 'fid' is currently supported by the spectra client. If a 'ser' file (2D) is found, it is now
skipped and the function returns False. This prevents accidental generation
of 1D spectra from unsupported 2D data.
* chore: Bump version from 1.3.1 to 1.3.2

* chore: bump dependencies patches

(also update wrong nmrglue ref since #248)
* improved devcontainer setup
- now uses a default non-`root`
- dev dependecies are separted out
- uses docker cache in build process

* update readme
- to reflect recent changes

* ci: improve cache
- do not remove cache after building

* rename `req.txt` to avoid confusion
- remove exec permission on config.py file

* include dev deps for GH Action test

* update GH action runners
* refactor: remove unused NMRIUM generation code and clean up related files

* fix: improve error handling in NMRiumDataConverter for spectrum data retrieval
- LCMS files creation
- BinaryParser Integration
- Peak and Integration Save
This reverts commit 61c5d7c.
…poser, update dependencies to Flask 2.x and Gunicorn 22.x, and remove obsolete LCMS files. Introduce new chemotion converter profiles for LCMS.
@Nicolass67 Nicolass67 force-pushed the 211-process-lcms-data branch from fa8eb12 to 0d8f16a Compare February 4, 2026 14:55
Nicolass67 and others added 10 commits February 4, 2026 16:43
Co-authored-by: Cursor <cursoragent@cursor.com>
…t helper functions in share.py for improved filename handling
…ms functions, and clean up related code in chemotion_converter_lcms.py to streamline LCMS processing.
…params functions, and implement data extraction for MS peaks in chemotion_converter_lcms.py to enhance LCMS processing.
…by refining active MS threshold handling and implementing conditional coloring for MS peaks based on the threshold. Default to 5% when the threshold is missing or uninitialized.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants