Write a MFDB data import tutorial

[bthe]

The tutorial for mfdb is currently incomplete. Would be useful to discuss how one would import various types of data, namely:

• Survey data. Here we can use currently available DATRAS example as a starting point
• Landings data.
• Stomach data
• Custom taxonomies

Any further ideas? @vbartolino do you have scripts that link to publicly available stomach data?

A really out there example would be to import the IWC's commercial catch data base.

[vbartolino]

good idea to use DATRAS as an example for the survey as many surveys coordinated within the ICES network are stored under that common format.

About the stomachs, I do not think there's a common format in ICES which makes difficult to provide a generic importing procedure. I've prepared the cod stomachs from my local files, but given that they have been uploaded on the ICES database approx. as they are on my local, I should be able to provide an example pointing there.

Sad reality is that for many shared stocks in ICES we do not have access to disaggregated biological catch data. Annual N@age and mean individual W@age per stock are generally available, but with some luck and digging into reports one can in several cases compile them by quarter and subdivision (it would be very useful if ICES could have them already into a database). Anyway, although quite aggregated, in my experience it is still very useful to have them in mfdb by subdivision and quarter.

STECF database has useful landing statistics by fleet, country, area, and if I recall also quarter, but I never tried to include them in mfdb. Note, STECF has also effort data of the main EU fleets...

Have you considered an example where we store some hydrographic data such temperature? We could point to https://www.copernicus.eu/en

[bthe]

good idea to use DATRAS as an example for the survey as many surveys coordinated within the ICES network are stored under that common format.

Yes, that is what I have already started with, but I of course have little knowledge of the data. Mind having a look at the example and make sure I don't leave anything useful out?

As to stomach data it looks like we can also import data from DAPSTOM (https://www.cefas.co.uk/data-and-publications/fish-stomach-records/ ) or do you have a better example?

Sad reality is that for many shared stocks in ICES we do not have access to disaggregated biological catch data.

Yes, and for the most part it is a pain to access the raw data from the various countries that participate in a particular fishery. We here in Iceland are quite lucky in that regard, having only a single source for most data types. However I hope that the DATRAS tutorial will people a good idea how to import the commercial samples.

Landings data from ICES (https://www.ices.dk/data/dataset-collections/Pages/Fish-catch-and-stock-assessment.aspx ) are a bit of pain though, I have typically spread the landings from statlant evenly to the months of the year.

STECF database has useful landing statistics by fleet, country, area, and if I recall also quarter, but I never tried to include them in mfdb. Note, STECF has also effort data of the main EU fleets...

That would be very interesting, is STECF data publicly available?

Have you considered an example where we store some hydrographic data such temperature? We could point to https://www.copernicus.eu/en

This would be a good idea but I must admit, I have no experience with working with hydrographic data. Do you think mfdb can digest these data sets properly and is there something we need to add for it to work?

[vbartolino]

good idea to use DATRAS ...

Yes, that is what I have already started with, but I of course have little knowledge of the data. Mind having a look at the example and make sure I don't leave anything useful out?

Sure

As to stomach data it looks like we can also import data from DAPSTOM (https://www.cefas.co.uk/data-and-publications/fish-stomach-records/ ) or do you have a better example?

Alternative would be the stomach database at ICES https://www.ices.dk/data/data-portals/Pages/Fish-stomach.aspx

Sad reality is that for many shared stocks in ICES we do not have access to disaggregated biological catch data.

Yes, and for the most part it is a pain to access the raw data from the various countries that participate in a particular fishery. We here in Iceland are quite lucky in that regard, having only a single source for most data types. However I hope that the DATRAS tutorial will people a good idea how to import the commercial samples.

Agree, the procedure will be very similar no need to repeate

Landings data from ICES (https://www.ices.dk/data/dataset-collections/Pages/Fish-catch-and-stock-assessment.aspx ) are a bit of pain though, I have typically spread the landings from statlant evenly to the months of the year.

STECF database has useful landing statistics by fleet, country, area, and if I recall also quarter, but I never tried to include them in mfdb. Note, STECF has also effort data of the main EU fleets...
That would be very interesting, is STECF data publicly available?

Yes, it seems to contain landings in biomass, value of landings and effort
https://stecf.jrc.ec.europa.eu/dd/fdi

Have you considered an example where we store some hydrographic data such temperature? We could point to https://www.copernicus.eu/en

This would be a good idea but I must admit, I have no experience with working with hydrographic data. Do you think mfdb can digest these data sets properly and is there something we need to add for it to work?

In the past I've tried to store in mfdb variables from an hydrographic model (salinity, temperature, oxygen) already calculated by month and ICES rectangles as separate survey indices for the different depth layers of my interest. I would be happy to try