Issues with bids.internal.get_metadata for derivatives

[bmdeen]

I'm encountering issues using bids.internal.get_metadata to extract metadata for derivative files. Specifically, I have a directory of Freesurfer-derived surface files, each with their own json file, including:

sub-01_hemi-R_space-fsnative_den-native_thickness.shape.gii
sub-01_space-fsnative_den-native_thickness.dscalar.nii
sub-01_hemi-R_space-fsnative_den-native_midthickness.surf.gii

The first file is a GIFTI file specifying cortical thickness; the second file is a CIFTI file combining cortical thickness measures from left and right hemispheres; the third is a representation of the right cortical surface (mid-thickness).

When I use bids.internal.get_metadata (default pattern) on the first file, information from the json metadata for files 2 and 3 is included, as if all of these json files apply to the first file.

The file sub-01_space-fsnative_den-native_thickness.json is included because its key/value pairs are a subset of the first file's. So get_metadata is properly applying the bids standard here, but the standard behaves weirdly in this case. Not sure how to address this.

The file sub-01_hemi-R_space-fsnative_den-native_midthickness.json is included because the file matches the pattern ^.*thickness\.json$, and its key-value pairs are the same as the first file's. Here, the bids standard is not being properly applied - the midthickness json data should not be included.

[Remi-Gau]

Hey @bmdeen,

get_metadata is one of the thing that is on my plate for the end of this week (See PR #172 ). So I will add an extra portion of "thing to check" with this. 😄

And there has been a couple of issues that have been needing fixing for the metadata aspect since like FOR freaking EVER.

Just to know: you are just using the get_metadata function as a "standalone" and not through the call to layout, correct?

Also you are using the master branch version of bids-matlab, or the dev branch?

Last question: do you know if there is a dataset available somewhere where I could test that on? Otherwise no worry I will try to create some dummy files to test that on.

[bmdeen]

Sounds good, thanks.

I'm using get_metadata as a standalone, from the master branch.

I put some of the data I'm having issues with in a folder here: https://www.dropbox.com/sh/7dw1x0h0txwv5fu/AACzARARw1MWFHCoxdlNnoWla?dl=0

[Remi-Gau]

OK so here are the tests I would set up. They indeed fail.

    data_dir = fullfile(fileparts(mfilename('fullpath')), 'data', 'SurfaceData');
    
    file = fullfile(data_dir, 'sub-06_hemi-R_space-individual_den-native_thickness.shape.gii');
    side_car = fullfile(data_dir, 'sub-06_hemi-R_space-individual_den-native_thickness.json');
    metadata = bids.internal.get_metadata(file);
    expected_metadata = bids.util.jsondecode(side_car);
    assertEqual(metadata, expected_metadata)
    
    file = fullfile(data_dir, 'sub-06_hemi-R_space-individual_den-native_midthickness.surf.gii');
    side_car = fullfile(data_dir, 'sub-06_hemi-R_space-individual_den-native_midthickness.json');
    metadata = bids.internal.get_metadata(file);
    expected_metadata = bids.util.jsondecode(side_car);
    assertEqual(metadata, expected_metadata)    
    
    file = fullfile(data_dir, 'sub-06_space-individual_den-native_thickness.dscalar.nii');
    side_car = fullfile(data_dir, 'sub-06_space-individual_den-native_thickness.json');
    metadata = bids.internal.get_metadata(file);
    expected_metadata = bids.util.jsondecode(side_car);
    assertEqual(metadata, expected_metadata)  

The fix is to change the default pattern to this

pattern = '^.*_%s\\.json$'

instead of

pattern = '^.*%s\\.json$'

Otherwise, the suffixes thickness and midthickness are not distinguishable.

This makes the default a bit less general but given that _suffix.extension is the rule in the vast majority of cases, I am tempted to go with it.

Will open a PR for this.

[bmdeen]

Makes sense. You could also check for files that have no underscores (i.e. just suffix.extension) and use a different pattern in this case, to include those json files.

[Remi-Gau]

Makes sense. You could also check for files that have no underscores (i.e. just suffix.extension) and use a different pattern in this case, to include those json files.

ah yeah.
thanks!!! not sure why I did not think of that.