From a2f45f0506efed8026b591942f8037b2d4004f57 Mon Sep 17 00:00:00 2001 From: "Alan B. Christie" <29806285+alanbchristie@users.noreply.github.com> Date: Wed, 13 Mar 2024 10:54:37 +0100 Subject: [PATCH] Latest staging code (#562) * Some changes to cset_upload.py to allow site observation short codes (#527) * stashing * fix: cset_upload.py updated to allow new-style site observation codes NB! this probably still won't work! I suspect the file I was given is broken and I cannot test it further * stashing * stashing * Short code prefix and tooltip to backend Target loader now reads short code prefix and tooltip from meta_aligner.yaml. Tooltip is saved to Experiment model. TODO: make tooltip available via API * Prefix tooltip now serverd by api/site_observation * stashing * Site observation groups for shortcodes now by experiment * feat: download structure fixed TODO: add all the yamls * All yaml files added to download * New format to download zip (issue 1326) (#530) * stashing * stashing * feat: download structure fixed TODO: add all the yamls * All yaml files added to download * cset_upload.py: lhs_pdb renamed to ref_pdb * Renamed canon- and conf site tags * Adds support for key-based SSH connections (#534) * Centralised environment variables (#529) * refactor: Restructured settings.py * docs: Minor tweaks * refactor: Move security and infection config to settings * refactor: b/e & f/e/ tags now in settings (also fixed f/e tag value) * refactor: Move Neo4j config to settings * refactor: More variables into settings * refactor: Moved remaining config * docs: Adds configuration guide as comments * docs: Variable prefix now 'stack_' not 'stack_env_' --------- Co-authored-by: Alan Christie * feat: Adds support for private keys on SSH tunnel * fix: Fixes key-based logic --------- Co-authored-by: Alan Christie * build(deps): bump cryptography from 42.0.0 to 42.0.2 (#533) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.0 to 42.0.2. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/42.0.0...42.0.2) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * docs: Updates documentation (#536) Co-authored-by: Alan Christie * build(deps): bump django from 3.2.20 to 3.2.24 (#535) Bumps [django](https://github.com/django/django) from 3.2.20 to 3.2.24. - [Commits](https://github.com/django/django/compare/3.2.20...3.2.24) --- updated-dependencies: - dependency-name: django dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: reverting wrong changes * fix: reverting wrong changes (#538) * stashing * add site observation's ligand sdf to aligned_files * fix: custom pdb now downloadable * fix: increased loglevel to error on unexpected exceptions block * fix: Discourse service check now checks API key before creating a service (#544) Co-authored-by: Alan Christie * build(deps): bump cryptography from 42.0.2 to 42.0.4 (#539) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.2 to 42.0.4. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/42.0.2...42.0.4) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * metadata.csv populated Started working on issue 1355 as well, it's too tightly coupled. Some work remaining re that: - when tag created in UI, make sure upload_name attribute is populated * upload_name automatically pouplated when creating tags in UI Only populated on creation, updates won't touch it * changes to api/download_structures - apo_file, bound_file, sdf_info and smiles_info merged into all_aligned_structures - added pdb_info field NB! download_structures was requred to provide ligand_pdb as well. This wasn't tracked previously, so I added field to SiteObservation model. Meaning there's a migration and on stack deployment data needs to be wiped and reuploaded * don't download neighbourhoods.yaml unless trans_matrix_info is checked * fixed error handling (errors.csv) and not returning combined sdf * fix: Added parsing directives to DownloadStructuresserializer * Consecutive numbering of observations under canon site * SiteObservatdion.tag split to tag and tag_prefix (1361) * fix: crystallographic_files folders in download now sans suffix (#550) * fix: tag names underdand prefix in download's metadata.csv * fix: return all proteins listed in api/download_structures * fix: fixed 'All structures' option not working in download dialog * Migrations for new file fields * Issue 1326 - mol and smiles added to download bundle NB! not prodction/staging ready, still contains a hack for testing because XCA doesn't provide all the attributes. * Target loader should handle empty code_prefix and tooltip 'Should' because haven't tested yet with real data * Column 'Downloaded' to metadata.csv in downloads * fix: restore 'upload_name' in site obvs tags to prefix-tag format * Removed ligand_smiles workaround All necessary files are now tracked by the database and returned in download. * fix: Add force_error_display to connection functions (default False) (#559) Co-authored-by: Alan Christie * Align production with staging (#555) (#560) * Some changes to cset_upload.py to allow site observation short codes (#527) * stashing * fix: cset_upload.py updated to allow new-style site observation codes NB! this probably still won't work! I suspect the file I was given is broken and I cannot test it further * stashing * stashing * Short code prefix and tooltip to backend Target loader now reads short code prefix and tooltip from meta_aligner.yaml. Tooltip is saved to Experiment model. TODO: make tooltip available via API * Prefix tooltip now serverd by api/site_observation * stashing * Site observation groups for shortcodes now by experiment * feat: download structure fixed TODO: add all the yamls * All yaml files added to download * New format to download zip (issue 1326) (#530) * stashing * stashing * feat: download structure fixed TODO: add all the yamls * All yaml files added to download * cset_upload.py: lhs_pdb renamed to ref_pdb * Renamed canon- and conf site tags * Adds support for key-based SSH connections (#534) * Centralised environment variables (#529) * refactor: Restructured settings.py * docs: Minor tweaks * refactor: Move security and infection config to settings * refactor: b/e & f/e/ tags now in settings (also fixed f/e tag value) * refactor: Move Neo4j config to settings * refactor: More variables into settings * refactor: Moved remaining config * docs: Adds configuration guide as comments * docs: Variable prefix now 'stack_' not 'stack_env_' --------- * feat: Adds support for private keys on SSH tunnel * fix: Fixes key-based logic --------- * build(deps): bump cryptography from 42.0.0 to 42.0.2 (#533) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.0 to 42.0.2. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/42.0.0...42.0.2) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... * docs: Updates documentation (#536) * build(deps): bump django from 3.2.20 to 3.2.24 (#535) Bumps [django](https://github.com/django/django) from 3.2.20 to 3.2.24. - [Commits](https://github.com/django/django/compare/3.2.20...3.2.24) --- updated-dependencies: - dependency-name: django dependency-type: direct:production ... * fix: reverting wrong changes * fix: reverting wrong changes (#538) * stashing * add site observation's ligand sdf to aligned_files * fix: custom pdb now downloadable * fix: increased loglevel to error on unexpected exceptions block * fix: Discourse service check now checks API key before creating a service (#544) * build(deps): bump cryptography from 42.0.2 to 42.0.4 (#539) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.2 to 42.0.4. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/42.0.2...42.0.4) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... * metadata.csv populated Started working on issue 1355 as well, it's too tightly coupled. Some work remaining re that: - when tag created in UI, make sure upload_name attribute is populated * upload_name automatically pouplated when creating tags in UI Only populated on creation, updates won't touch it * changes to api/download_structures - apo_file, bound_file, sdf_info and smiles_info merged into all_aligned_structures - added pdb_info field NB! download_structures was requred to provide ligand_pdb as well. This wasn't tracked previously, so I added field to SiteObservation model. Meaning there's a migration and on stack deployment data needs to be wiped and reuploaded * don't download neighbourhoods.yaml unless trans_matrix_info is checked * fixed error handling (errors.csv) and not returning combined sdf * fix: Added parsing directives to DownloadStructuresserializer * Consecutive numbering of observations under canon site * SiteObservatdion.tag split to tag and tag_prefix (1361) * fix: crystallographic_files folders in download now sans suffix (#550) * fix: tag names underdand prefix in download's metadata.csv * fix: return all proteins listed in api/download_structures * fix: fixed 'All structures' option not working in download dialog * Migrations for new file fields * Issue 1326 - mol and smiles added to download bundle NB! not prodction/staging ready, still contains a hack for testing because XCA doesn't provide all the attributes. * Target loader should handle empty code_prefix and tooltip 'Should' because haven't tested yet with real data * Column 'Downloaded' to metadata.csv in downloads * fix: restore 'upload_name' in site obvs tags to prefix-tag format * Removed ligand_smiles workaround All necessary files are now tracked by the database and returned in download. * fix: Add force_error_display to connection functions (default False) (#559) --------- Signed-off-by: dependabot[bot] Co-authored-by: Kalev Takkis Co-authored-by: Warren Thompson Co-authored-by: Alan Christie Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Warren Thompson * Add DISABLE_RESTRICT_PROPOSALS_TO_MEMBERSHIP (#561) * feat: Add DISABLE_RESTRICT_PROPOSALS_TO_MEMBERSHIP * style: Minor log tweak --------- Co-authored-by: Alan Christie --------- Signed-off-by: dependabot[bot] Co-authored-by: Kalev Takkis Co-authored-by: Warren Thompson Co-authored-by: Alan Christie Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Warren Thompson --- api/security.py | 8 ++++++-- fragalysis/settings.py | 11 +++++++++++ viewer/views.py | 2 +- 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/api/security.py b/api/security.py index 12d4a987..4c2afc19 100644 --- a/api/security.py +++ b/api/security.py @@ -354,9 +354,10 @@ def get_proposals_for_user(self, user, restrict_to_membership=False): proposals = set() ispyb_user = settings.ISPYB_USER logger.debug( - "ispyb_user=%s restrict_to_membership=%s", + "ispyb_user=%s restrict_to_membership=%s (DISABLE_RESTRICT_PROPOSALS_TO_MEMBERSHIP=%s)", ispyb_user, restrict_to_membership, + settings.DISABLE_RESTRICT_PROPOSALS_TO_MEMBERSHIP, ) if ispyb_user: if user.is_authenticated: @@ -368,7 +369,10 @@ def get_proposals_for_user(self, user, restrict_to_membership=False): # We have all the proposals where the user has authority. # Add open/public proposals? - if not restrict_to_membership: + if ( + not restrict_to_membership + or settings.DISABLE_RESTRICT_PROPOSALS_TO_MEMBERSHIP + ): proposals.update(self._get_open_proposals()) # Return the set() as a list() diff --git a/fragalysis/settings.py b/fragalysis/settings.py index 633487f6..cfef3208 100644 --- a/fragalysis/settings.py +++ b/fragalysis/settings.py @@ -464,6 +464,17 @@ COMPUTED_SET_MEDIA_DIRECTORY: str = "computed_set_data" +# The following (part of m2ms-1385) is used to prevent the +# 'restrict-to-membership' check in security.py - something that is designed to prevent +# uploading to public proposals unless the user is explicitly part of the proposal +# (according to ISPyB). This variable is used to defeat this test for situations +# when ISPyB is unavailable. It is not permitted when the DEPLOYMENT_MODE +# is 'PRODUCTION +DISABLE_RESTRICT_PROPOSALS_TO_MEMBERSHIP: bool = False +if os.environ.get("DISABLE_RESTRICT_PROPOSALS_TO_MEMBERSHIP") == "True": + assert DEPLOYMENT_MODE != "PRODUCTION" + DISABLE_RESTRICT_PROPOSALS_TO_MEMBERSHIP = True + # Discourse settings for API calls to Discourse Platform. DISCOURSE_PARENT_CATEGORY: str = "Fragalysis targets" DISCOURSE_USER: str = "fragalysis" diff --git a/viewer/views.py b/viewer/views.py index 1d1600e9..91f91fbb 100644 --- a/viewer/views.py +++ b/viewer/views.py @@ -1579,7 +1579,7 @@ def create(self, request, *args, **kwargs): contact_email=contact_email, user_id=request.user.pk, ) - logger.info("+ UploadTargetExperiments.create got Celery id %s", task.task_id) + logger.info("+ UploadTargetExperiments.create got Celery id %s", task.task_id) url = reverse('viewer:task_status', kwargs={'task_id': task.task_id}) # as it launches task, I think 202 is more appropriate