Add microsaccades to Brain-Score core #492

benlonnqvist · 2024-01-10T14:13:23Z

In this PR we add microsaccades to Brain-Score core. Integration of model_tools PR #75 to Brain-Score 2.0.

Conceptual Problem: We need variance in model responses to static stimuli.

Conceptual Solution: Microsaccades are a prevalent phenomenon in vision and visual experiments. They are involuntary movements of the eye at very small scales. Microsaccades are not (and often can not due to their small extent) be controlled for in experiments, and are thus implicitly baked into the variance in primate responses. As such, microsaccades are a phenomenon that is either explicitly or implicitly believed to be unimportant for behavioral/neural outcomes, and thus a good candidate for getting variance in responses to static stimuli. (e.g. Rolfs 2009, Vis. Res.; Haddad & Steinmann 1973, Vis. Res.)

Technical problem: The current Brain-Score interface does not support microsaccades.

Technical solution: the ActivationsExtractorHelper will be changed. The from_paths method now takes a path, loads the image from the path, does pixel position jittering to the image, and saves the jittered image into a temporary new path. The new path is passed to the downstream functions to get the model activations.

To accomplish this, the from_paths method will now receive a dictionary of tags. This dictionary of tags comes from a call to the look_at method, where number_of_trials is replaced with a new dictionary. This new dictionary then traverses the following path:

run_activations
A call to activations_model
In practice, this calls a TensorFlowWrapper/PytorchWrapper/etc.’s self._extractor class’s __call__() method in its __call__() method. (I.e., PytorchWrapper.__call__() => self._extractor.__call__() )
This extractor’s call method is ActivationsExtractorHelper.__call__()
Thus we pass the new dictionary to ActivationsExtractorHelper.__call__() where it will be parsed.

TODO list:

Implement microsaccades
Implement unit tests for microsaccades

Other Brain-Score benchmark PRs depend on this PR (#365).

Huge thanks to Hannes Mehrer (@hanme) who implemented microsaccades, which I've only adapted for the generalized case here.

benlonnqvist · 2024-01-15T09:15:37Z

It seems that the Jenkins build is skipping tests on this PR, which is not the intended case: http://braintree.mit.edu:8080/job/unittest_brainscore/1516/parsed_console/ shows the log:

fatal: ambiguous argument 'origin/vision_microsaccades': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
...
MODIFIES_PLUGIN: False
PLUGIN_ONLY: True
Changed files:  
Only plugins modified. Will skip tests.

Unsure as to why this happens, and whether it's something I can fix as a part of the PR? The intended case certainly is for tests to run on this PR. Any help would be much appreciated!

kvfairchild · 2024-01-15T13:29:32Z

Hi @benlonnqvist - thanks for flagging this; it looks like you've caught an issue with the way we're handling forked PRs in Jenkins. This should raise an assertion error in the Travis PR build, but that isn't able to run while there are unresolved conflicts with master. I should be able to push a fix later today and will let you know!

benlonnqvist · 2024-01-15T13:40:16Z

Thank you @kvfairchild! Much appreciated.

kvfairchild · 2024-01-16T01:52:05Z

Hi @benlonnqvist - can you try again now?

benlonnqvist · 2024-01-16T13:58:30Z

Hi @kvfairchild, tests seem to be working now, thanks again!

benlonnqvist · 2024-01-18T17:43:09Z

Hi! I think there are some remaining tests that fail on Jenkins that I don't think have anything to do with the PR itself - although I could be wrong. Perhaps it might be possible to have a quick look as to what might be going wrong with the few remaining tests and whether they have anything to do with this PR specifically? Thanks!

brainscore_vision/model_helpers/activations/core.py

benlonnqvist · 2024-02-06T14:34:23Z

Hi, it seems that Jenkins is now crashing on building the PR altogether (see here). I don't think this is related to the PR - is there something I can do to resolve the issue potentially?

mschrimpf · 2024-02-07T10:07:14Z

re-trigger jenkins

…t/brain-score into vision_microsaccades

mschrimpf

in addition to in-code comments, please delete brainscore_vision/data/scialom2024/__init__.py

brainscore_vision/model_helpers/activations/core.py

brainscore_vision/model_helpers/brain_transformation/neural.py

tests/test_model_helpers/activations/test___init__.py

Co-authored-by: Martin Schrimpf <[email protected]>

mschrimpf

Looks good once comments are addressed and tests pass. Especially adding unit tests for the exact values are essential imo.

brainscore_vision/model_helpers/activations/core.py

brainscore_vision/model_helpers/brain_transformation/neural.py

tests/test_model_helpers/activations/test___init__.py

Co-authored-by: Martin Schrimpf <[email protected]>

… tests run for, while keeping the test the same

mschrimpf · 2024-03-07T09:14:27Z

All tests on Jenkins (and Travis) are green. Travis reports a failure because it gets killed (likely runtime, unrelated to this PR)

* email fix for score_plugins workflow (#581) * Emails passed to jenkins * Update .github/workflows/score_new_plugins.yml Co-authored-by: Katherine Fairchild <[email protected]> * fix deprecated set output --------- Co-authored-by: Katherine Fairchild <[email protected]> * add eBarlow_Vanilla to models (#587) Co-authored-by: AutoJenkins <[email protected]> * add eBarlow_lmda_01 to models (#588) Co-authored-by: AutoJenkins <[email protected]> * add cornet_s_0_0_0 to models (#589) Co-authored-by: AutoJenkins <[email protected]> * add unet_ethan to models (#590) Co-authored-by: AutoJenkins <[email protected]> * add unet_ethan to models (#591) Co-authored-by: AutoJenkins <[email protected]> * Revert "add unet_ethan to models (#591)" (#594) This reverts commit c89059e. * Revert "add unet_ethan to models (#590)" (#595) This reverts commit 9a8774a. * Transfer plugin information by artifact for scoring runs (#592) * upload and download json artifact between jobs * fix testing changes * brain-score.org (user:514) | Add new plugin(s): models: [cornet_s_0_0_0 cornet_s_0_1_0] | (public:False) (#603) * add cornet_s_0_0_0 to models * add cornet_s_0_1_0 to models --------- Co-authored-by: AutoJenkins <[email protected]> * Add microsaccades: interface and default model helper (#492) * integrate brain-score/model-tools#75 to brain-score/vision * add exception for when temp file write fails * fix error with tf temp file management * add bandaid to a DataAssembly index problem * move microsaccades to the model side * update comments * remove indexing bug * fix bug with activations.shape * Apply suggestions from code review Co-authored-by: Martin Schrimpf <[email protected]> * Delete brainscore_vision/data/scialom2024/__init__.py * address review changes * remove needless import * fix bug with temporary file handling test * assume number_of_trials=1 and require_variance=False when getting stored activations * fix bug with access to ActivationsExtractorHelper.set_visual_degrees * move extractor calls to ModelCommitment generic * add check for whether activations_model exists * fix bug with TestVisualDegrees * add link to BrainModel issue * remove shifts from stimulus set packaging * change link signatures * change microsaccade call signature * microsaccades are now computed on both a pixel and degree basis * Apply suggestions from code review Co-authored-by: Martin Schrimpf <[email protected]> * fix outdated comments, type hints, etc. * change function call to reduce repetition * add kwargs to microsaccade helpers * refactor microsaccade usage to their own class to improve readability * refactor microsaccade coords into MicrosaccadeHelper * refactor microsaccade building * change the way MultiIndex is set * fix tf/pytorch/keras bug with image shape calculation * cv2 reshaping in translate * add test for exact microsaccades * fix microsaccade indexing * rename test to be in line with current naming * add require_variance to _from_paths_stored * reduce unnecessarily long test times by reducing the number of trials tests run for, while keeping the test the same --------- Co-authored-by: Martin Schrimpf <[email protected]> * Use MajajHong2015public.IT instead of mock data (#585) * Changed _MockBenchmark to new format * revert ceiler back to martin's suggestion * implemented MockBenchmark to use MJ2015 Public --------- Co-authored-by: Katherine Fairchild <[email protected]> Co-authored-by: AutoJenkins <[email protected]> Co-authored-by: Ben Lonnqvist <[email protected]> Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: Michael Ferguson <[email protected]>

* add callable microsaccades * add test to catch errors in stimulus set meta attachment when using microsaccades * mark new test as memory intense * Apply suggestions from code review Co-authored-by: Martin Schrimpf <[email protected]> * add require_variance to the signature of look_at for all behavioral arbiters (required since the generic look_at no longer takes kwargs but specifically require_variance) --------- Co-authored-by: Martin Schrimpf <[email protected]>

benlonnqvist added 3 commits January 10, 2024 14:54

integrate �brain-score/model-tools#75 to brain-score/vision

4690626

Merge branch 'brain-score:master' into vision_microsaccades

26ed57a

retrigger

1fee011

retrigger

26313aa

benlonnqvist added 3 commits January 17, 2024 15:54

add exception for when temp file write fails

71ba207

fix error with tf temp file management

3ea9a9c

add bandaid to a DataAssembly index problem

3203e7b

retrigger

a2e0a15

mschrimpf reviewed Jan 26, 2024

View reviewed changes

brainscore_vision/model_helpers/activations/core.py Outdated Show resolved Hide resolved

benlonnqvist added 3 commits February 6, 2024 14:04

move microsaccades to the model side

c1abd95

update comments

e690b4e

Merge branch 'brain-score:master' into vision_microsaccades

5ad9863

mschrimpf closed this Feb 7, 2024

mschrimpf reopened this Feb 7, 2024

benlonnqvist added 3 commits February 7, 2024 17:18

remove indexing bug

e05d815

Merge branch 'vision_microsaccades' of https://github.com/benlonnqvis…

903d3f5

…t/brain-score into vision_microsaccades

fix bug with activations.shape

629884e

mschrimpf reviewed Feb 11, 2024

View reviewed changes

benlonnqvist and others added 4 commits February 12, 2024 11:11

Apply suggestions from code review

0226495

Co-authored-by: Martin Schrimpf <[email protected]>

Delete brainscore_vision/data/scialom2024/__init__.py

307735d

address review changes

e13391a

remove needless import

e79fbd2

benlonnqvist closed this Feb 20, 2024

benlonnqvist reopened this Feb 20, 2024

benlonnqvist closed this Feb 22, 2024

benlonnqvist reopened this Feb 22, 2024

mschrimpf approved these changes Feb 29, 2024

View reviewed changes

benlonnqvist and others added 15 commits February 29, 2024 13:40

Apply suggestions from code review

926778c

Co-authored-by: Martin Schrimpf <[email protected]>

fix outdated comments, type hints, etc.

f7a0340

change function call to reduce repetition

5a8881d

add kwargs to microsaccade helpers

2655ecf

refactor microsaccade usage to their own class to improve readability

232f4d7

refactor microsaccade coords into MicrosaccadeHelper

06e4b4c

refactor microsaccade building

18e48ae

change the way MultiIndex is set

954b4cc

fix tf/pytorch/keras bug with image shape calculation

52596ac

cv2 reshaping in translate

9245cb4

add test for exact microsaccades

ead4cee

fix microsaccade indexing

63990cf

rename test to be in line with current naming

453a2a6

add require_variance to _from_paths_stored

6e92878

reduce unnecessarily long test times by reducing the number of trials…

9a67b15

… tests run for, while keeping the test the same

benlonnqvist closed this Mar 6, 2024

benlonnqvist reopened this Mar 6, 2024

Merge branch 'brain-score:master' into vision_microsaccades

0dad5e5

mschrimpf merged commit b177ad8 into brain-score:master Mar 7, 2024
3 of 4 checks passed

benlonnqvist mentioned this pull request Mar 18, 2024

require_variance missing from the signature of look_at #651

Open

benlonnqvist deleted the vision_microsaccades branch June 12, 2024 09:04

benlonnqvist mentioned this pull request Jun 14, 2024

Add callable microsaccades as intended by #492 #917

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add microsaccades to Brain-Score core #492

Add microsaccades to Brain-Score core #492

benlonnqvist commented Jan 10, 2024 •

edited

Loading

benlonnqvist commented Jan 15, 2024

kvfairchild commented Jan 15, 2024

benlonnqvist commented Jan 15, 2024

kvfairchild commented Jan 16, 2024

benlonnqvist commented Jan 16, 2024

benlonnqvist commented Jan 18, 2024

benlonnqvist commented Feb 6, 2024

mschrimpf commented Feb 7, 2024

mschrimpf left a comment

mschrimpf left a comment •

edited

Loading

mschrimpf commented Mar 7, 2024

Add microsaccades to Brain-Score core #492

Add microsaccades to Brain-Score core #492

Conversation

benlonnqvist commented Jan 10, 2024 • edited Loading

benlonnqvist commented Jan 15, 2024

kvfairchild commented Jan 15, 2024

benlonnqvist commented Jan 15, 2024

kvfairchild commented Jan 16, 2024

benlonnqvist commented Jan 16, 2024

benlonnqvist commented Jan 18, 2024

benlonnqvist commented Feb 6, 2024

mschrimpf commented Feb 7, 2024

mschrimpf left a comment

Choose a reason for hiding this comment

mschrimpf left a comment • edited Loading

Choose a reason for hiding this comment

mschrimpf commented Mar 7, 2024

benlonnqvist commented Jan 10, 2024 •

edited

Loading

mschrimpf left a comment •

edited

Loading