Develop merge for 2.3.3 release

.github/workflows/python_lint.yml

@@ -26,7 +26,7 @@ jobs:
         id: file_check
         uses: tj-actions/changed-files@v44
         with:
-          sha: ${{ github.event.pull_request.head.sha }}
+          sha: ${{ github.event.pull_request.head.sha || github.sha }}
           files: |
              **.py
       - name: Run flake8
@@ -49,7 +49,7 @@ jobs:
           id: file_check
           uses: tj-actions/changed-files@v44
           with:
-            sha: ${{ github.event.pull_request.head.sha }}
+            sha: ${{ github.event.pull_request.head.sha || github.sha }}
             files: '**.py'
         - name: Check code lints with Black
           if: steps.file_check.outputs.any_changed == 'true'

CHANGELOG.md

@@ -4,6 +4,65 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.3.3] - 2026-06-22 : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.3.3
+
+### Credits
+
+- [Enrique Sapena](https://github.com/ESapenaVentura)
+- [Sarai Varona](https://github.com/svarona)
+- [Magdalena Matito](https://github.com/magdasmat)
+- [Pau Pascual](https://github.com/PauPascualMas)
+- [Juan Ledesma](https://github.com/juanledesma78)
+- [Sandra Sánchez](https://github.com/sandrasnchz)
+- [Victor Lopez](https://github.com/victor5lm)
+
+### Template fixes and updates
+
+- Added parallelization to IRMA's 99-stats summary creation. [#648](https://github.com/BU-ISCIII/buisciii-tools/pull/648)
+- Fixed major error while filtering IRMA's vf files. [#649](https://github.com/BU-ISCIII/buisciii-tools/pull/649)
+- Fixed wgstrio 99-stats lablog to prevent OutOfMemoryError on WGS samples. [#654](https://github.com/BU-ISCIII/buisciii-tools/pull/654)
+- Fixed wgstrio folder template structure [#655](https://github.com/BU-ISCIII/buisciii-tools/pull/655)
+- Updated recommended Nextflow version in taxprofiler lablog [#655](https://github.com/BU-ISCIII/buisciii-tools/pull/655)
+- Fixing AF filtering and results symlinks in wgstrio template [#659](https://github.com/BU-ISCIII/buisciii-tools/pull/659)
+- Fixed column selection, now selects MAX_AF in wgstrio template [#660](https://github.com/BU-ISCIII/buisciii-tools/pull/660)
+- Fixed QSslSocket and wkhtmltopdf errors in bioinfo-doc PDF generation [#662](https://github.com/BU-ISCIII/buisciii-tools/pull/662)
+- Make VEP header generation dynamic and remove static template dependency [#664] (https://github.com/BU-ISCIII/buisciii-tools/pull/664)
+- Updated blast database with new assemblies downloaded in March 2026 [#665](https://github.com/BU-ISCIII/buisciii-tools/pull/665)
+- Added numeric sample ID validation for samplesheet.csv in assembly lablog [#667](https://github.com/BU-ISCIII/buisciii-tools/pull/667)
+- Updated `wgstrio`, `exometrio` and `rnaseq` templates to generate OmicsExplorer-compatible outputs. [#668](https://github.com/BU-ISCIII/buisciii-tools/pull/668)
+- Updated blast database location to refgenie directory [#683](https://github.com/BU-ISCIII/buisciii-tools/pull/683)
+- Add trim_poly_x fastp arg to assembly config [#685](https://github.com/BU-ISCIII/buisciii-tools/pull/685)
+- Updated A__N1 and A__N2 subtype results in flu_type_refs.txt [#687](https://github.com/BU-ISCIII/buisciii-tools/pull/687)
+- Updated new paths to refgenie databases in databasesheet.csv of taxprofiler [#689](https://github.com/BU-ISCIII/buisciii-tools/pull/689)
+- Updated viralrecon's config file [#691](https://github.com/BU-ISCIII/buisciii-tools/pull/691).
+- Updated the services.json file's structure, so that an auxiliary cleaning file is used by the clean module if applicable [#692](https://github.com/BU-ISCIII/buisciii-tools/pull/692).
+- Optimized annotation wgs & exome [#695](https://github.com/BU-ISCIII/buisciii-tools/pull/695)
+- Updated several software versions [#696](https://github.com/BU-ISCIII/buisciii-tools/pull/696)
+- Updated nextclade checks for IRMA and viralrecon [#697](https://github.com/BU-ISCIII/buisciii-tools/pull/697)
+- Enabled micromamba env activation when launching a script [#699](https://github.com/BU-ISCIII/buisciii-tools/pull/699)
+- Updated kaiju conf for taxprofiler [#701](https://github.com/BU-ISCIII/buisciii-tools/pull/701)
+
+### Modules
+
+#### Implementation
+
+- Added DROP as a new service, including automatic input generation, module-specific adaptations and standardized result processing. [#668](https://github.com/BU-ISCIII/buisciii-tools/pull/668)
+- Created viralrecon_clean.py, an auxiliary file to clean unnecessary files from viralrecon service [#692](https://github.com/BU-ISCIII/buisciii-tools/pull/692).
+
+#### Added enhancements
+
+- Improved clean module so that an aux cleaning file is employed for specific services if applicable [#692](https://github.com/BU-ISCIII/buisciii-tools/pull/692).
+- Improved HTTP error handling in drylab_api module and fixed fix-permissions so that it does not break given insufficient permissions [#694](https://github.com/BU-ISCIII/buisciii-tools/pull/694).
+- Implemented date filtering for research in archive, improved logging and error handling [#700](https://github.com/BU-ISCIII/buisciii-tools/pull/700)
+
+#### Fixes
+
+#### Changed
+
+#### Removed
+
+### Requirements
+
 ## [2.3.2] - 2026-03-31 : https://github.com/BU-ISCIII/buisciii-tools/releases/tag/2.3.2
 
 ### Credits

buisciii/__main__.py

@@ -66,7 +66,7 @@ def run_buisciii():
         highlight=False,
     )
 
-    __version__ = "2.3.2"
+    __version__ = "2.3.3"
     stderr.print(
         "[grey39]    BUISCIII-tools version {}".format(__version__), highlight=False
     )
@@ -998,7 +998,7 @@ def fix_permissions(ctx, input_directory):
             else:
                 log.exception(f"EXCEPTION FOUND: {e}")
                 stderr.print(f"EXCEPTION FOUND: {e}")
-                sys.exit(1)
+                continue
 
 
 if __name__ == "__main__":

buisciii/archive.py

@@ -5,6 +5,7 @@
 import shutil
 import sys
 from math import pow
+from datetime import datetime
 
 import rich
 import sysrsync
@@ -159,13 +160,11 @@ def __init__(
             and (service_id is None)
             and (services_file is None)
         ):
-            if self.ser_type == "services_and_colaborations":
+            if self.ser_type in ["services_and_colaborations", "research"]:
                 prompt_response = buisciii.utils.prompt_selection(
                     "Search services by date, or by service ID?",
                     ["Search by date", "Service ID"],
                 )
-            else:
-                prompt_response = "Service ID"
 
             log.info("Services chosen by: " + prompt_response)
             if prompt_response == "Search by date":
@@ -229,37 +228,84 @@ def __init__(
         if (self.date_from is not None) and (self.date_until is not None):
             stderr.print("Asking our trusty API about selected services")
             log.info("Asking our trusty API about selected services")
-            try:
-                for service in rest_api.get_request(
-                    request_info="services",
-                    safe=True,
-                    state="delivered",
-                    date_from=str(self.date_from),
-                    date_until=str(self.date_until),
-                ):
-                    self.services[service["service_request_number"]] = {
-                        key: value for key, value in dictionary_template.items()
-                    }
-                    self.services[service["service_request_number"]][
-                        "found_in_system"
-                    ] = True
-                    self.services[service["service_request_number"]][
-                        "delivery_date"
-                    ] = service["service_delivered_date"]
-
-                log.info(f"Services found in the time interval: {len(self.services)}")
-                log.info(
-                    "Names of the services found in said interval:"
-                    f"{','.join([service for service in self.services.keys()])}"
-                )
-            except TypeError:
-                stderr.print(
-                    "Could not connect to the API (wrong password?)", style="red"
+
+            if self.ser_type == "research":
+                research_base = os.path.join(
+                    conf.get_configuration("global")["data_path"],
+                    self.ser_type,
                 )
-                log.error(
-                    "ERROR: Connection to the API was not successful. Possible reasons: wrong API password or bad connection."
+                stderr.print(f"Scanning directory: {research_base}")
+                log.info(f"Scanning research directory: {research_base}")
+
+                date_from_ts = datetime.strptime(
+                    str(self.date_from), "%Y-%m-%d"
+                ).timestamp()
+                date_until_ts = datetime.strptime(
+                    str(self.date_until), "%Y-%m-%d"
+                ).timestamp()
+
+                try:
+                    for entry in os.scandir(research_base):
+                        if entry.is_dir(follow_symlinks=False):
+                            mtime = entry.stat(follow_symlinks=False).st_mtime
+                            if date_from_ts <= mtime <= date_until_ts:
+                                self.services[entry.name] = {
+                                    key: value
+                                    for key, value in dictionary_template.items()
+                                }
+                                self.services[entry.name]["found_in_system"] = True
+                                self.services[entry.name]["delivery_date"] = (
+                                    datetime.fromtimestamp(mtime).strftime("%Y-%m-%d")
+                                )
+                except FileNotFoundError:
+                    stderr.print(f"[red]Directory not found: {research_base}")
+                    log.error(f"Research base directory not found: {research_base}")
+                    raise
+
+                log.info(
+                    f"Research folders found in time interval: {len(self.services)}"
                 )
-                raise
+            else:
+                try:
+                    for service in rest_api.get_request(
+                        request_info="services",
+                        safe=True,
+                        state="delivered",
+                        date_from=str(self.date_from),
+                        date_until=str(self.date_until),
+                    ):
+                        self.services[service["service_request_number"]] = {
+                            key: value for key, value in dictionary_template.items()
+                        }
+                        self.services[service["service_request_number"]][
+                            "found_in_system"
+                        ] = True
+                        self.services[service["service_request_number"]][
+                            "delivery_date"
+                        ] = service["service_delivered_date"]
+
+                    log.info(
+                        f"Services found in the time interval ({self.date_from} - {self.date_until}): {len(self.services)}"
+                    )
+                    log.info(
+                        "Names of the services found in said interval:"
+                        f"{','.join([service for service in self.services.keys()])}"
+                    )
+                    stderr.print(
+                        f"Services found in the time interval ({self.date_from} - {self.date_until}): {len(self.services)}"
+                    )
+                    stderr.print(
+                        "Names of the services found in said interval:"
+                        f"{','.join([service for service in self.services.keys()])}"
+                    )
+                except TypeError:
+                    stderr.print(
+                        "Could not connect to the API (wrong password?)", style="red"
+                    )
+                    log.error(
+                        "ERROR: Connection to the API was not successful. Possible reasons: wrong API password or bad connection."
+                    )
+                    raise
 
         for service in self.services.keys():
             stderr.print(service)
@@ -301,12 +347,12 @@ def __init__(
                 self.services[service]["archived_path"] = os.path.join(
                     conf.get_configuration("global")["archived_path"],
                     self.ser_type,
-                    service_id,
+                    service,
                 )
                 self.services[service]["non_archived_path"] = os.path.join(
                     conf.get_configuration("global")["data_path"],
                     self.ser_type,
-                    service_id,
+                    service,
                 )
 
             self.services[service]["found"] = []
@@ -966,15 +1012,15 @@ def delete_non_archived_dirs(self):
         for service in self.services.keys():
             if not os.path.exists(self.services[service]["non_archived_path"]):
                 stderr.print(
-                    f"Service {self.services[service]['non_archived_path'].split('/')[-1]}"
-                    "has already been removed from"
-                    f"{'/'.join(self.services[service]['non_archived_path'].split('/')[:-1])[:-1]}."
+                    f"Service {self.services[service]['non_archived_path'].split('/')[-1]} "
+                    "has already been removed from "
+                    f"{'/'.join(self.services[service]['non_archived_path'].split('/')[:-1])}. "
                     "Nothing to delete so skipping.\n"
                 )
                 log.info(
                     f"Service {self.services[service]['non_archived_path'].split('/')[-1]} "
                     "has already been removed from"
-                    f"{'/'.join(self.services[service]['non_archived_path'].split('/')[:-1])[:-1]}."
+                    f"{'/'.join(self.services[service]['non_archived_path'].split('/')[:-1])}. "
                     "Nothing to delete so skipping.\n"
                 )
                 continue
@@ -990,14 +1036,23 @@ def delete_non_archived_dirs(self):
                     )
                 else:
                     stderr.print(
-                        f"Found archived path for service {self.services[service]['archived_path'].split('/')[-1]}."
+                        f"Found archived path for service {self.services[service]['archived_path'].split('/')[-1]}. "
                         "It is safe to delete this non_archived service. Deleting.\n"
                     )
                     log.info(
-                        f"Found archived path for service {self.services[service]['archived_path'].split('/')[-1]}."
+                        f"Found archived path for service {self.services[service]['archived_path'].split('/')[-1]}. "
                         "It is safe to delete this non_archived service. Deleting.\n"
                     )
-                    shutil.rmtree(self.services[service]["non_archived_path"])
+                    try:
+                        shutil.rmtree(self.services[service]["non_archived_path"])
+                    except Exception as e:
+                        stderr.print(
+                            f"Could not delete {self.services[service]['non_archived_path']}: {e}. Skipping.\n"
+                        )
+                        log.error(
+                            f"Could not delete {self.services[service]['non_archived_path']}: {e}. Skipping.\n"
+                        )
+                        continue
         return
 
     def generate_tsv_table(self, filename):

buisciii/assets/reports/md/sarek.md

@@ -1223,7 +1223,6 @@ You can find information on how to interpret results from VEP toolkit [here](htt
 
 [![GitHub release](https://img.shields.io/github/release/exomiser/Exomiser.svg)](https://github.com/exomiser/Exomiser/releases)
 [![CircleCI](https://circleci.com/gh/exomiser/Exomiser/tree/development.svg?style=shield)](https://circleci.com/gh/exomiser/Exomiser/tree/development)
-[![Codacy Badge](https://api.codacy.com/project/badge/Grade/b518a9448b5b4889a40b3dc660ef585a)](https://www.codacy.com/app/monarch-initiative/Exomiser?utm_source=github.com&utm_medium=referral&utm_content=exomiser/Exomiser&utm_campaign=Badge_Grade)
 [![Documentation](https://readthedocs.org/projects/exomiser/badge/?version=latest)](http://exomiser.readthedocs.io/en/latest)
 #### Overview: