sync w/ master (#609)

* email fix for score_plugins workflow (#581) * Emails passed to jenkins * Update .github/workflows/score_new_plugins.yml Co-authored-by: Katherine Fairchild <[email protected]> * fix deprecated set output --------- Co-authored-by: Katherine Fairchild <[email protected]> * add eBarlow_Vanilla to models (#587) Co-authored-by: AutoJenkins <[email protected]> * add eBarlow_lmda_01 to models (#588) Co-authored-by: AutoJenkins <[email protected]> * add cornet_s_0_0_0 to models (#589) Co-authored-by: AutoJenkins <[email protected]> * add unet_ethan to models (#590) Co-authored-by: AutoJenkins <[email protected]> * add unet_ethan to models (#591) Co-authored-by: AutoJenkins <[email protected]> * Revert "add unet_ethan to models (#591)" (#594) This reverts commit c89059e. * Revert "add unet_ethan to models (#590)" (#595) This reverts commit 9a8774a. * Transfer plugin information by artifact for scoring runs (#592) * upload and download json artifact between jobs * fix testing changes * brain-score.org (user:514) | Add new plugin(s): models: [cornet_s_0_0_0 cornet_s_0_1_0] | (public:False) (#603) * add cornet_s_0_0_0 to models * add cornet_s_0_1_0 to models --------- Co-authored-by: AutoJenkins <[email protected]> * Add microsaccades: interface and default model helper (#492) * integrate brain-score/model-tools#75 to brain-score/vision * add exception for when temp file write fails * fix error with tf temp file management * add bandaid to a DataAssembly index problem * move microsaccades to the model side * update comments * remove indexing bug * fix bug with activations.shape * Apply suggestions from code review Co-authored-by: Martin Schrimpf <[email protected]> * Delete brainscore_vision/data/scialom2024/__init__.py * address review changes * remove needless import * fix bug with temporary file handling test * assume number_of_trials=1 and require_variance=False when getting stored activations * fix bug with access to ActivationsExtractorHelper.set_visual_degrees * move extractor calls to ModelCommitment generic * add check for whether activations_model exists * fix bug with TestVisualDegrees * add link to BrainModel issue * remove shifts from stimulus set packaging * change link signatures * change microsaccade call signature * microsaccades are now computed on both a pixel and degree basis * Apply suggestions from code review Co-authored-by: Martin Schrimpf <[email protected]> * fix outdated comments, type hints, etc. * change function call to reduce repetition * add kwargs to microsaccade helpers * refactor microsaccade usage to their own class to improve readability * refactor microsaccade coords into MicrosaccadeHelper * refactor microsaccade building * change the way MultiIndex is set * fix tf/pytorch/keras bug with image shape calculation * cv2 reshaping in translate * add test for exact microsaccades * fix microsaccade indexing * rename test to be in line with current naming * add require_variance to _from_paths_stored * reduce unnecessarily long test times by reducing the number of trials tests run for, while keeping the test the same --------- Co-authored-by: Martin Schrimpf <[email protected]> * Use MajajHong2015public.IT instead of mock data (#585) * Changed _MockBenchmark to new format * revert ceiler back to martin's suggestion * implemented MockBenchmark to use MJ2015 Public --------- Co-authored-by: Katherine Fairchild <[email protected]> Co-authored-by: AutoJenkins <[email protected]> Co-authored-by: Ben Lonnqvist <[email protected]> Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: Michael Ferguson <[email protected]>
brain-score · Mar 8, 2024 · 1b7d352 · 1b7d352
1 parent a379e1d
commit 1b7d352
Show file tree

Hide file tree

Showing 24 changed files with 1,202 additions and 123 deletions.
diff --git a/.github/workflows/score_new_plugins.yml b/.github/workflows/score_new_plugins.yml
@@ -20,12 +20,11 @@ permissions: write-all
 
 jobs:
 
-  changes_models_or_benchmarks:
-    name: Check if PR makes changes to /models or /benchmarks
+  process_submission:
+    name: If triggering PR alters /models or /benchmarks, initiates scoring for relevant plugins
     if: github.event.pull_request.merged == true
     runs-on: ubuntu-latest
     outputs:
-      PLUGIN_INFO: ${{ steps.getpluginfo.outputs.PLUGIN_INFO }}
       RUN_SCORING: ${{ steps.scoringneeded.outputs.RUN_SCORING }}
     steps:
       - name: Check out repository code
@@ -37,6 +36,13 @@ jobs:
         uses: actions/setup-python@v4
         with:
           python-version: 3.7
+
+      - name: Configure AWS Credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-east-1
 
       - name: Installing package dependencies
         run: |
@@ -53,54 +59,71 @@ jobs:
         id: getpluginfo
         run: |
           echo "PLUGIN_INFO='$(python -c 'from brainscore_core.plugin_management.parse_plugin_changes import get_scoring_info; get_scoring_info("${{ env.CHANGED_FILES }}", "brainscore_vision")')'"  >> $GITHUB_OUTPUT
-      
+
       - name: Check if scoring needed
         id: scoringneeded
         run: |
           echo "RUN_SCORING=$(jq -r '.run_score' <<< ${{ steps.getpluginfo.outputs.PLUGIN_INFO }})" >> $GITHUB_OUTPUT
-
-  get_submitter_info:
-    name: Get PR author email and (if web submission) Brain-Score user ID
-    runs-on: ubuntu-latest
-    needs: [changes_models_or_benchmarks]
-    if: needs.changes_models_or_benchmarks.outputs.RUN_SCORING == 'True'
-    env:
-      PLUGIN_INFO: ${{ needs.changes_models_or_benchmarks.outputs.PLUGIN_INFO }}
-    outputs:
-      PLUGIN_INFO: ${{ steps.add_email_to_pluginfo.outputs.PLUGIN_INFO }}
-    steps:
-      - name: Parse user ID from PR title and add to PLUGIN_INFO (WEB ONLY where we don't have access to the GitHub user)
-        id: add_uid_to_pluginfo
-        if: contains(github.event.pull_request.labels.*.name, 'automerge-web')
+          
+      - name: Find PR author email for non-web submissions
+        if: "!contains(github.event.pull_request.labels.*.name, 'automerge-web') && steps.scoringneeded.outputs.RUN_SCORING == 'True'"
+        uses: evvanErb/[email protected]
+        id: getemail
+        with:
+          github-username: ${{github.event.pull_request.user.login}} 
+          token: ${{ secrets.GITHUB_TOKEN }}  # Including token enables most reliable way to get a user's email
+      - name: Update PLUGIN_INFO for non-web submissions
+        if: "!contains(github.event.pull_request.labels.*.name, 'automerge-web') && steps.scoringneeded.outputs.RUN_SCORING == 'True'"
+        id: non_automerge_web
+        run: |
+          echo "The PR author email is ${{ steps.getemail.outputs.email }}"
+          echo "PLUGIN_INFO=$(<<<${{ steps.getpluginfo.outputs.PLUGIN_INFO }} tr -d "'"  | jq -c '. + {email: "${{ steps.getemail.outputs.email }}"}')" >> $GITHUB_ENV
+          
+      - name: Update PLUGIN_INFO for automerge-web (find uid, public v. private, and bs email)
+        if: contains(github.event.pull_request.labels.*.name, 'automerge-web') && steps.scoringneeded.outputs.RUN_SCORING == 'True'
+        id: automerge_web
         run: |
           BS_UID="$(echo '${{ github.event.pull_request.title }}' | sed -E 's/.*\(user:([^)]+)\).*/\1/')"
           BS_PUBLIC="$(echo '${{ github.event.pull_request.title }}' | sed -E 's/.*\(public:([^)]+)\).*/\1/')"
-          echo "The Brain-Score user ID is $BS_UID"
-          echo "PLUGIN_INFO=$(<<<$PLUGIN_INFO tr -d "'" | jq -c ". + {user_id: \"$BS_UID\", public: \"$BS_PUBLIC\"}")" >> $GITHUB_ENV
+          USER_EMAIL=$(python -c "from brainscore_core.submission.database import email_from_uid; from brainscore_core.submission.endpoints import UserManager; user_manager=UserManager(db_secret='${{ secrets.BSC_DATABASESECRET }}'); print(email_from_uid($BS_UID))")
+          echo "::add-mask::$USER_EMAIL" # Mask the USER_EMAIL
+          echo "PLUGIN_INFO=$(<<<${{ steps.getpluginfo.outputs.PLUGIN_INFO }} tr -d "'" | jq -c ". + {user_id: \"$BS_UID\", public: \"$BS_PUBLIC\", email: \"$USER_EMAIL\"}")" >> $GITHUB_ENV
 
-      - name: Get PR author email from GitHub username
-        id: getemail
-        uses: evvanErb/[email protected]
-        with:
-          github-username: ${{github.event.pull_request.user.login}} # PR author's username
-          token: ${{ secrets.GITHUB_TOKEN }} # Including token enables most reliable way to get a user's email
-      - name: Add PR author email to PLUGIN_INFO
-        id: add_email_to_pluginfo
+      - name: Write PLUGIN_INFO to a json file
         run: |
-          echo "The PR author email is ${{ steps.getemail.outputs.email }}"
-          echo "PLUGIN_INFO=$(<<<$PLUGIN_INFO tr -d "'"  | jq -c '. + {author_email: "${{ steps.getemail.outputs.email }}"}')" >> $GITHUB_OUTPUT
+          echo "$PLUGIN_INFO" > plugin-info.json
+      
+      - name: Upload PLUGIN_INFO as an artifact
+        uses: actions/upload-artifact@v2
+        with:
+          name: plugin-info
+          path: plugin-info.json
+
 
   run_scoring:
     name: Score plugins
     runs-on: ubuntu-latest
-    needs: [changes_models_or_benchmarks, get_submitter_info]
-    if: needs.changes_models_or_benchmarks.outputs.RUN_SCORING == 'True'
+    needs: [process_submission]
+    if: needs.process_submission.outputs.RUN_SCORING == 'True'
     env:
-      PLUGIN_INFO: ${{ needs.get_submitter_info.outputs.PLUGIN_INFO }}
       JENKINS_USER: ${{ secrets.JENKINS_USER }}
       JENKINS_TOKEN: ${{ secrets.JENKINS_TOKEN }}
       JENKINS_TRIGGER: ${{ secrets.JENKINS_TRIGGER }}
     steps:
+
+      - name: Download PLUGIN_INFO artifact
+        uses: actions/download-artifact@v2
+        with:
+          name: plugin-info
+          path: artifact-directory
+
+      - name: Set PLUGIN_INFO as an environment variable
+        run: |
+          PLUGIN_INFO=$(cat artifact-directory/plugin-info.json)
+          USER_EMAIL=$(echo "$PLUGIN_INFO" | jq -r '.email')
+          echo "::add-mask::$USER_EMAIL"  # add a mask when bringing email back from artifact
+          echo "PLUGIN_INFO=${PLUGIN_INFO}" >> $GITHUB_ENV
+        
       - name: Add domain, public, competition, and model_type to PLUGIN_INFO
         run: |
           echo "PLUGIN_INFO=$(<<<$PLUGIN_INFO tr -d "'"  | jq -c '. + {domain: "vision", competition: "None", model_type: "Brain_Model"}')" >> $GITHUB_ENV