Downloading Data: 414 Error from Dataverse

[jbteves]

Hi @jsheunis ! I'm looking forward to checking out this data. When I got to download the whole project from dataverse, however, I can only download the files in piecemeal. If I try to select all files, I get the following error:

414 Request - URI Too Long
The requested URL's length exceeds the capacity limit for this server

As a novice to dataverse, perhaps I'm missing something obvious here for how to download the whole data set. I'd like to avoid downloading in piecemeal because then I'll have to manually merge many directories, which seems counter to your having put it into BIDS format for me. Are there any workarounds that you're aware of, or alternative ways to access the data?

[jsheunis]

Hey @jbteves! Thanks for the interest and for logging the issue. I'm really sorry about this issue with downloading the data. I've replicated it on my side as well, and have now contacted the DataverseNL admins to see if they have a workaround for it.

One option I will try in the mean time is to see if I can write a script to do a bulk download via the API. Dataverse allows you to create an account and then an API token in order to interact with datasets via the API. One possible challenge with this is the fact that this dataset requires the user to agree to the terms of the custom Data Use Agreement, and I'm not sure how that is handled via the API. I'll find out.

A last option (although I first want to see if the above ones work) would be to have you sign the agreement manually and then I could send you a secure download link for the dataset. This would be a short term fix, since it's not scaleable long term.

[jbteves]

Alright, that sounds good, please keep me updated! I will happily test your API token approach once you have it.
I would hope they have something like CLI anaconda installs, where you're just dropped into an "agree" prompt.

[jsheunis]

@jbteves The issue has been logged with DataverseNL. They've responded and said it's a known issue with the version that they have currently installed, and is fixed in an upgrade, but they don't intend on upgrading soon. They are now checking whether they can patch it up in the current version, and I should have an answer by latest on Monday.

[jsheunis]

In the mean time I can try and store the data securely elsewhere and can send a secure download link once its ready. Will contact you via email for that.

[jbteves]

Awesome to hear that it's slated to be fixed. Sure, please feel free to e-mail me, I'd love to take a look.

[jsheunis]

An update: The bug in the version of Dataverse that DataverseNL is running has been fixed in a newer version. Unfortunately it cannot be patched in their current version and they will have to upgrade. DataverseNL should be upgraded to this newer version somewhere in Q1/Q2 of 2021. Until then, the plan is to store the data securely elsewhere, with private download links to be sent to anyone requesting access to the data (after signing the DUA). Unless noted otherwise on the README of this repository, data access requests will still be via the DataverseNL dataset repository, although downloading the data will happen (after signing of the DUA) via a separate link.