Release v1.8.0

RELEASE v1.8.0 CHANGELOG

[1.8.0] - 2025-02-03

Added

• Full integration of extra_config support, enabling structured overriding and validation of configurations on top of the base configuration.
• New topic_config argument in the add-extra-config module to improve scoping and selective configuration removal.
• Three default initial_config.yaml files for project-specific initialization:
  • relecov
  • mepram
  • EQA2026
• Automatic loading of initial_config-relecov.yaml before executing test workflows.
• Extended documentation for add-extra-config, including improved descriptions of required_conf and new parameters.
• Minor improvements in linting, CI configuration, and dependency handling.

Changed

• Refactored configuration management by migrating project-specific settings from configuration.json to dedicated initial_config.yaml files.
• Updated build_schema to recursively traverse nested properties for more complete and consistent schema generation.
• Modified internal functions to consistently pass extra_config=True where required.
• Reworked project name extraction logic to ensure compatibility with the new initial_config structure.
• Improved schema title handling and whitespace validation for more robust API and metadata validation.
• Updated README documentation to reflect the new configuration architecture and extra_config usage.

Fixed

• Fixed compatibility issues in multiple modules (download, validate, read-lab-metadata) when using the new extra_config schema structure.
• Resolved CI workflow issues to ensure proper loading of extra_config and required environment variables during test execution.
• Corrected inconsistencies in JSON schema example generation, header ordering, and value typing.

Removed

• Removed project-specific configuration entries from configuration.json, consolidating them into project-specific initial_config.yaml files to eliminate duplication and simplify the base configuration.

Release v1.7.4

🔔 Release v1.7.4

This release introduces important improvements in metadata parsing, schema generation, and result summarization, with a particular focus on supporting complex/nested fields across the metadata lifecycle.
It also improves robustness in the download workflow and updates project requirements to consistently target Python ≥ 3.10.

⚙️ Major Workflow Changes

🧬 Metadata Parsing & Utilities

• Added metadata_nextstrain_parser.py to pipeline_utils, enabling structured parsing of Nextstrain-related metadata for downstream processing.
  #822
• Extended read_lab_metadata to support complex and nested fields, ensuring correct ingestion of enriched laboratory metadata.
  #832

📐 Schema Builder Enhancements

• Adapted schema builder Excel generation to properly handle nested and complex fields, improving consistency between schema definitions and generated templates.
  #826

📊 Results & Reporting

• Updated create_summary_tables.py to generate separate result outputs depending on the pathogen, improving clarity and usability for epidemiological reporting.
  #841

---

🛠 Fixes and Stability

• Fixed compression handling in the download module to correctly reflect compressed files in the log summary report.
• Removed legacy organism ontology mappings, aligning the workflow with the current metadata model.
  #831

---

📦 Requirements & Infrastructure

• Updated GitHub Actions: python_lint now runs using Python 3.10.
• Updated README.md to correctly reflect Python ≥ 3.10 as the minimum supported version.
  #839

---

Release v1.7.3

🔔 Release v1.7.3

This mini-release focuses on enhancing database update flexibility and addressing a critical issue in long table identifier parsing.
It extends the capabilities of the update_database module and ensures improved consistency in sample ID handling across Relecov workflows.

⚙️ Key Updates

🚀 Database Update Modes

• Extended update_database to support Full Update Modes (1–4), enabling selective updates of specific data types and platforms.
  This improvement simplifies maintenance and integration workflows for data synchronization across Relecov environments.
  #820

🧬 Metadata Handling Fix

• Fixed an issue in long_table where unique ID selection failed when the sequencing_sample_id contained underscores (_).
  This ensures consistent mapping between sequencing and unique sample identifiers during long table generation.
  #818

---

👉 Release v1.7.3

Release v1.7.2

🔔 Release v1.7.2

This release focuses on improving data synchronization with iSkyLIMS, expanding schema compatibility, and reinforcing the stability of key modules like upload_database, wrapper, read-bioinfo-metadata, and upload-results.
It also introduces new analytics scripts and enhances logging, ensuring smoother integration with institutional systems and better traceability of Relecov submissions.

⚙️ Major Workflow Changes

🧩 iSkyLIMS Integration

• Introduced a new function in upload_database.py to feed iSkyLIMS POST data with lab_requests fields, ensuring complete synchronization between laboratory requests and database uploads.
• Added a new iSkyLIMS API endpoint to retrieve mappings for lab_request fields, configurable via configuration.json.
  #803

🧬 Metadata & Schema Enhancements

• Updated relecov_schema to versions v3.2.1 and v3.2.3, introducing support for Midnight amplicon panel versions and improving validation flexibility.
• Normalized long-table sample identifiers to consistently use unique_sample_id within read-bioinfo-metadata.
  #802, #807, #813

📤 Upload & Reporting Modules

• Added input_directories support to upload-results, improving compatibility with batch processing pipelines.
• Enhanced mail error handling and improved logging consistency for the mail module.
  #808, #814

📊 Analytics & Institutional Data

• Added a UCCT status analytics script to monitor submission coverage and trends.
• Updated upload logic to send collecting_institution_code_1 to the Relecov Platform, ensuring institutional traceability in every record.
• Expanded laboratory_adress.json with new laboratories to improve metadata completeness.
  #806, #813

---

🛠 Fixes and Stability

• Fixed incorrect messages and KeyError during uploads to iSkyLIMS.
• Prevented upload of corrupted files to the SFTP in wrapper, improving reliability of upload workflows.
• Fixed create_summary_tables.py to properly generate CSV variant files for epidemiological reporting.
  #803, #808, #809

---

🧹 Cleanups & Minor Improvements

• General improvements in exception handling and mail logging consistency across modules.
• Minor refactoring and JSON metadata updates for better maintainability.

---

👉 Release v1.7.2

Release v1.7.1

Release v1.7.1

This release focuses on improving stability, robustness, and correctness across key modules. It addresses several issues related to database uploads, API responses, pipeline-manager checks, and wrapper outputs, ensuring smoother integration in production workflows.

---

⚙️ Fixes and Stability Improvements

🗄️ Database & Upload Module

• Removed obsolete sample_fingerprint usage in the upload-database module to align with the new unique_sample_id approach.
  #790

🌐 REST API Consistency

• Fixed data completeness issues in rest_api.standardize_response.
• Corrected list handling logic in rest_api.py, preventing unexpected errors.
  #791, #792

📥 Download & Pipeline Manager

• Fixed handling of invalid folders during download, ensuring invalid samples are properly managed.
• Corrected R1 check logic in pipeline-manager when processing single-end FASTQs.
  #792

📑 Wrapper & Reporting

• Fixed missing and empty report.xlsx files after wrapper execution, ensuring reports are consistently generated.
  #798

---

👥 Credits

• Alejandro Bernabeu
• Pablo Mata

Release v1.7.0

🔔 Release v1.7.0

This release introduces major improvements in sample traceability, institutional metadata standardization (REGCESS integration), and pipeline modularization.
Key modules like pipeline_manager, read-bioinfo-metadata, validate, upload_database, and build-schema have been refactored or extended to improve interoperability, maintainability, and alignment with epidemiological workflows.

⚙️ Major Workflow Changes

📤 Pipeline Manager Integration

• Added support for uploading samples to relecov-platform from pipeline_manager, storing the returned unique_sample_id in json_data.
• FASTQ and analysis files are now renamed with unique_sample_id while retaining sequencing_sample_id for traceability.
• Added fingerprint generation utility to create stable sample identifiers.
• Introduced parameter skip_upload_db to allow skipping DB upload during testing.
• Changed exit behavior to raise exceptions instead of exit(0).
  #737, #738, #774

🧬 Metadata & Schema Enhancements

• Support for using unique_sample_id as key identifier in read_bioinfo_metadata, with fallback to combined ID format.
• Refactored read-bioinfo-metadata for structure, readability, and maintainability.
• Renamed add_bioinfo_files_path → map_and_extract_bioinfo_paths for clarity.
• Enhanced validate_samplenames logic to support combined identifier format.
• Updated relecov_schema to v3.2.0 with new Genomic Typing fields.
  #738, #775, #777, #783

🏥 Institutional Metadata Standardization

• Integrated REGCESS as the main institutional metadata source, merging CCN and CODCNH datasets, enriching with geolocation, updating schema & validation logic.
• Updated build-schema to generate hospital dropdowns using hospital-city-CCN mappings.
• Standardized collecting institution codes as collecting_institution_code_[1,2].
• Updated hospital DB generation: merge REGCESS + CNH, enrich/normalize metadata, auto-geolocate cities, generate all JSON outputs.
  #726, #749, #781

📑 Validation & Upload Improvements

• upload_database now accepts json_data as path or list of dicts and returns API upload results.
• Reordered initialization in validate and upload_database to follow consistent structure.
• Included project name in email subject for upload_results.
  #692, #774, #780

📊 Epidemiological Data Handling

• Updated create_summary_tables.py to organize data by epidemiological season.
  #772

---

🛠 Fixes and Stability

• Fixed geolocation cities file and improved city normalization.
• Fixed issues with submitting institutions metadata.
• Fixed potential inconsistency when no folder name is found in metadata (warn & skip).
• Fixed broken platform config references (platform → platform-data → platform-params).
• Improved robustness and internal consistency of pipeline_utils and viralrecon.py with annotations, docstrings, and cleanup.
  #774, #775, #781

---

🧹 Cleanups & Removals

• Removed unused ResultsUpload class and stale variables (self.json_file).
• Cleaned up unused imports.
  #774

---

🙌 Credits

• Sara Monzon
• Victor Lopez
• Sarai Varona
• Pablo Mata
• Alejandro Bernabeu

---

👉 Release v1.7.0

Release v1.6.2

🔔 Release v1.6.2

This patch release tightens up the relecov-tools 1.6 series with targeted bug fixes, improved configuration handling, and the removal of now-obsolete registry code.

🛠 Fixes

• Correctly handle an empty "target_folders": "" setting in the download module.
• Make pipeline_manager locate valided.json reliably.
• Eliminate triple-counting of folder-only samples in log_summary.json.

⚙️ Changed

• Revise main and default config_json to adopt the new merged-config layout (commands + params).
• Pass the wrapper’s --output_dir flag down to download so all modules respect the same output path.
• Clarify the error message shown when output_dir is missing in extra_config.

🗑 Removed

• All registry-related functionality has been excised from the code base.

Pull Requests

• Introduced new validation workflow to discard samples in platform db by @Shettland in #721
• Updated create_summary_tables.py to organize data jointly by @victor5lm in #722
• Updated schema and configuration by @svarona in #724
• Refactor configuration.json structure by grouping fields by module usage by @Aberdur in #725
• Moved sample_id_ontology used by validate to configuration.json by @Shettland in #733
• Fixed download param override and target_folders malfunctioning by @Shettland in #736
• Standardized RestApi response dictionaries with an aux function by @Shettland in #741
• Fix detection of missing label in JSON-Schema properties by @Aberdur in #743
• Minor fix left in PR #741. data is directly included in api response by @Shettland in #744
• Fix wrapper file-name lookup and prevent duplicate upload of invalid sample by @Aberdur in #748
• pre-release 1.6.1 by @Aberdur in #753
• Fix CHANGELOG to its corresponding version by @Aberdur in #755
• Fix "target_folders": "" handling in download module by @Aberdur in #760
• minor fix in pipeline_manager so it finds valided json by @saramonzon in #764
• Update __main__ & config_json for new merged-config structure (commands + params) by @Aberdur in #765
• Improve path handling & logging across wrapper, download, and validation by @Aberdur in #766
• Refactor registry use and relecov-tools validate by @saramonzon in #767
• Release 1.6.2 by @saramonzon in #771

Full Changelog: v1.6.0...v1.6.2

Release v1.6.1

Release v1.6.1 – Smarter API, Database-Aware Validation & Config Refactor

2025-07-28

This minor-version release focuses on API consistency, database checks during validation, and a cleaner configuration layout while delivering several quality-of-life fixes.

---

🚀 Enhancements

Area	Highlight	PR
REST API	rest_api.get_request() now accepts credentials and query params, plus fresh doc-strings for every public method.	#721
Validation workflow	New database-aware path: with --check_db, validate discards samples already present in the Relecov platform.	#721
Ion Torrent support	Schema + configuration updated to recognise Ion Torrent read metrics.	#724
Summary tables	create_summary_tables.py now groups data more intuitively for downstream QC.	#722
Configuration refactor	configuration.json fields are now grouped by module for easier maintenance.	#725
Uniform API responses	All rest_api methods pass through standardize_response() for predictable output.	#741

---

🛠 Fixes

• Download – Correct target-folder logic and stop overriding CLI args with config defaults. #736
• Schema validation – Properly detects missing label in JSON-Schema properties. #743
• Wrapper – Restored correct filename lookup for read-lab-metadata output and prevents double-upload of invalid samples. #748

---

🔄 Changes

• sample_id_ontology moved from code to configuration.json for user customisation. #733
• Removed default RELECOV/ sub-folder assumption in upload-results and download CLI entry-points. #736
• upload_database and related callers updated to consume the new standardised REST API responses. #741
• All rest_api helpers now return parsed JSON under data, when available. #744

---

⬆️ Upgrade Notes

1. Config update – Merge the new module-grouped structure into your custom configuration.json.
2. Validation flag – Add --check_db to relecov-tools validate if you want automatic exclusion of already-uploaded samples.
3. Ion Torrent users – Verify your metadata complies with the new schema fields before running pipelines.

Enjoy the smoother workflows!

Release v1.6.0

🔔 Release v1.6.0

This release includes substantial changes to the validation and metadata processing workflows, improving robustness, modularity, and end-to-end traceability. Key modules like validate, read-bioinfo-metadata, download, build-schema, and the wrapper have been somewhat refactored and extended to support a more scalable and error-resilient system.

---

⚙️ Major Workflow Changes

• 🔄 Redesigned Validation Workflow
  The validate module now supports automated uploading of invalid samples and reports directly to the SFTP server. This replaces previous manual or wrapper-based logic, centralizing responsibility in the validation stage.
  #708, #715

• 📤 Wrapper Refactor
  Upload logic for invalid files has been removed from the wrapper. It now delegates entirely to the validate module, simplifying wrapper responsibilities and avoiding redundancy.
  #708

• 📑 Metadata Validation Refactor
  The read-bioinfo-metadata module was overhauled to:

  • Include both valid and invalid samples in log_summary
  • Replace exec/eval with importlib for safe dynamic loading
  • Add support for the --soft_validation flag
  • Improve robustness and modularity of long table handling
    #673, #708, #713

• 📐 SchemaBuilder & build-schema Enhancements

  • Added support for --excel_template input
  • Improved schema versioning logic
  • Removed unnecessary ontology metadata from generated Excel
    #693, #698

• 📊 Validation Infrastructure Overhaul

  • validate_instances() is now a static method and accepts custom validators to easy outer integration
  • Utility logic like summarize_errors, unique_id_generator, and parameter checks are now standalone functions
  • get_sample_id_field() added as a schema inspection utility
    #684

• 🛠Download module validation process revamped

  • Download module now only skips those samples that cannot be traced by any means (no possible ID nor file).
  • All the invalid files are now downloaded so their metadata can be processed and validated prior to send-mail

---

🧪 Test Improvements

• Added new test cases for the validation pipeline in tests/test_validate.py
• Integrated software_versions.yml for proper test environment setup
• Test data updated to support the new upload workflow
  #708, #713

---

🛠 Fixes and Stability

• Improved error messaging and exception handling in sftp_client, validation, and config parsing
• Fixed issues in:
  • logs_to_excel filename generation
  • download_only mode behavior
  • Validation error message counts
  • Sample skipping logic in wrapper
• Ensured metadata validation still occurs for folders with invalid samples
• Replaced sys.exit() usage with proper ValueError exceptions
  #672, #673, #684, #705, #707, #713

---

🧹 Cleanups & Minor Improvements

• Enforced Python 3.8+ as the minimum version
• Replaced deprecated `pkg

Release v1.5.5

🔖 Release v1.5.5.

This release focuses on improving the consistency and reliability of metadata processing across the toolchain. The primary goal is name homogenization across metadata files, schema validation, and processing modules to reduce ambiguity and ensure compatibility with public standards.

✨ Highlights

🧩 Name Homogenization

• Unified naming conventions across metadata fields in templates (lab, bioinfo) and schema definitions, along with module parameters and variables.
• Normalized sample ID and patient-related field names for consistency with GISAID and ENA submission formats.
• Removed redundant or inconsistent field aliases in the JSON schema.

✅ Validation Enhancements

• Strengthened validation rules in read-bioinfo-metadata and validate commands to reflect updated naming standards.
• Improved error messaging for invalid or missing fields now aligned with standardized field names.

🧪 Template and Schema Updates

• Updated schema to v3.1.1, incorporating normalized field names and new validations for several fields like pcr_ct values.
• Re-generated metadata templates using the unified schema.

🔧 Stability Improvements

• Fixed issues where mismatched names previously led to silent failures or incorrect validations.
• Improved resilience of update-db and other CLI tools under incomplete metadata conditions.