Overall tracking towards implementation

[znicholls]

The problem

There is a paper which outlines how to go from CMIP6 variable names and other info, to new 'branded' variable names: https://docs.google.com/document/d/19jzecgymgiiEsTDzaaqeLP6pTvLT-NzCMaq-wu-QoOc/edit?pli=1&tab=t.0

Appendix F is the bit we care about.

There is also this short note which provides further clarification: https://docs.google.com/document/d/1P-CudQtIq5dwAmU1kTeXCEzyclduEJobJDixGf7E5qc/edit?tab=t.0

The doc above links to this spreadsheet: https://docs.google.com/spreadsheets/d/1_y7OnhSvWc_Cjwnf8dSZcvuVC9BQuSQTPYRWYM1Q9Oc/edit?gid=201745125#gid=201745125

Updated below based on #4 (comment)

From my quick glance of the spreadsheet and the docs, I would suggest we:

1. define some (around 6) tests by hand based on our reading of the docs and tables that are there
2. use the spreadsheet to define the rest of our tests
3. for a given value of variable name (column BG), cell methods (let's use the new cell methods, column T) and dimensions (let's use the corrected dimensions, column AZ), make sure that we can produce the expected branded variable label based on the CMIP6 root name (column BI)
- more information on allowed cell method values here: https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#cell-methods
5. then go back and try and this with raw CMIP6 information, which will mean we need to add some corrections layer to go from raw CMIP6 data to the corrected information I suspect
- the cell methods in the data request are in the cell methods tab of https://airtable.com/appUDtj7V6Zirj7RS/shrlZUy53O8YSasKj/tbllIG82q5b8YMHw0/viwQcEEUFubTqjs6F
- the dimensions in the data request are in the dimensions tab of https://airtable.com/appUDtj7V6Zirj7RS/shrlZUy53O8YSasKj/tbllIG82q5b8YMHw0/viwQcEEUFubTqjs6F
6. then try and add the new variable root mapping (i.e. use column BK as the target output rater than column BI)

Based on #4 (comment)

1. define some (around 6) tests by hand based on our reading of the docs and tables that are there
2. use the spreadsheet to define the rest of our tests
   3. for a given value of variable name (column BY), cell methods (use the new cell methods, column T) and dimensions (use the corrected dimensions, column AZ), make sure that we can produce the expected branded variable label based on the corrected root name (column BK)
   - more information on allowed cell method values here: https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#cell-methods
   - more information on allowed dimensions here: https://github.com/PCMDI/cmip6-cmor-tables/blob/main/Tables/CMIP6_coordinate.json
3. then try and do the reverse (start with branded variable, go back to components)
4. then go back and try and this with raw CMIP6 information, which will mean we need to add some corrections layer to go from raw CMIP6 data to the corrected information

Definition of "done"

• build a tool that, given the CMIP6 variable, cell methods and dimensions, produces a branded variable name
• also support the reverse functionality of the main feature: from a branded variable label, extract and display its decomposed components

Additional context

There may be some labels missing apparently, see the screenshot below

[znicholls]

@taylor13 here is the issue we're working in. Connecting you here with @NiklasSchwind @eahoegner @TessaM97

[ltroussellier]

Hi,

I’ll probably be the main user of this at the moment, along with Dan (@wolfiex).

Besides the main feature:

Build a tool that, given the CMIP6 variable, cell methods, and dimensions, produces a branded variable name.

The ideal scenario would also include:

An easy way to explore all possible options for each component of the branded variable. For example, being able to see all possible "cell methods," etc.
The reverse functionality of the main feature: from a branded variable label, extract and display its decomposed components.

[znicholls]

super thanks for getting in touch @ltroussellier !

An easy way to explore all possible options for each component of the branded variable. For example, being able to see all possible "cell methods," etc.

As a check, do we know this in advance? I thought this was handled by the cf conventions (https://cfconventions.org/Data/cf-conventions/cf-conventions-1.12/cf-conventions.html#cell-methods) so here we'd just need to be able to parse them, we wouldn't need to define them ourselves.

We can definitely think about how to try and extract the list of components we know about. I guess I'm just not sure that it will be exhaustive (there are cell method components that we don't need to support because they don't impact the branded variable name, e.g. stuff like (where the cell is snow covered)).

The reverse functionality of the main feature: from a branded variable label, extract and display its decomposed components.

Nice, added to the list

[znicholls]

@taylor13 a question for you (or anyone else who knows), where is the list of dimension names (like is there somewhere that defines the difference between time1, time2 and time3)?

[ltroussellier]

We can definitely think about how to try and extract the list of components we know about. I guess I'm just not sure that it will be exhaustive (there are cell method components that we don't need to support because they don't impact the branded variable name, e.g. stuff like (where the cell is snow covered)).

Yes, exactly. At the moment, I don’t think we need more than the ones we already know about (e.g., those in the documents you shared). I didn’t mean to add more to the to-do list—just wanted to share a "nice-to-have" idea to provide direction for the developer if it’s feasible.

[znicholls]

Ok cool makes sense, thanks

[eahoegner]

@taylor13 a question for you (or anyone else who knows), where is the list of dimension names (like is there somewhere that defines the difference between time1, time2 and time3)?

found in Appendix C (no complete list, just clarifies the different times):

For CMIP variables that are a function of time, there are four kinds of temporal sampling that have been needed. In the list of a variable's dimensions, one of the following names are used to indicate the sampling:
"time", if the variable represents a mean over each reported time interval
"time1", if the variable is reported synoptically at each time
"time2", if the variable represents a multi-year climatology of monthly means
"time3", if the variable represents an hourly climatology for each month.

[bethdingley]

@taylor13 a question for you (or anyone else who knows), where is the list of dimension names (like is there somewhere that defines the difference between time1, time2 and time3)?

There's also a list with definitions in the DR here

[znichollscr]

An easy way to explore all possible options for each component of the branded variable. For example, being able to see all possible "cell methods," etc.

@ltroussellier do the cell methods and coordinates and dimensions tabs of https://airtable.com/appUDtj7V6Zirj7RS/shrlZUy53O8YSasKj/tbljH6pVfVBryqWYq/viweLduZWYl5VakxF have the information you were thinking of?

@bethdingley which way around does this go? Does the data request define the cell methods and dimensions or are the cell methods and dimensions defined upstream (e.g. by the CF-conventions) and then used by the data request?

[bethdingley]

As far as I'm aware at least some of it comes from CF:

1. In the Coordinates and Dimensions tab, you can see where things are defined via CF, if they have a CF name in the 'CF Standard Name' column. E.g. for the ones tagged as area_type, CF have defined the area_type options (listed in column 'Value (Scalar or String)'). Then a coordinate/dimension is defined for the Data Request pointing to that CF coordinate. I.e. the coordinate name 'typebare' is defined by the Data Request to point to the CF area_type bare_ground.
2. I think cell methods are also from CF but I'm less sure

Perhaps @martinjuckes can confirm

[ltroussellier]

@ltroussellier do the cell methods and coordinates and dimensions tabs of https://airtable.com/appUDtj7V6Zirj7RS/shrlZUy53O8YSasKj/tbljH6pVfVBryqWYq/viweLduZWYl5VakxF have the information you were thinking of?

Yes, in my view, the sheets that @bethdingley provided are a summary of the Airtable, specifically focusing on the branded variable, and they seem to be the most relevant input for this project. However, as you dig deeper into it, I believe @bethdingley will be a great resource to help with further insights!

@bethdingley which way around does this go? Does the data request define the cell methods and dimensions or are the cell methods and dimensions defined upstream (e.g. by the CF-conventions) and then used by the data request?

Karl or Dan will be able to provide a much clearer response to this than I can. This project idea is just to not duplicate effort and help building the CV (e.g the variable register). Some of the dimension and cell-method were defined by the DR, but it wasn't his primary focus. Some could potentially be defined by CF, but I believe it will ultimately be included in the CV pointing wherever it is relevant thanks to the new framework, regardless of its origin. The DR, and others (soft or human) could then make use of it from the CV. Those questions will need to be officially addressed by the CV Task Team.

[znicholls]

Those questions will need to be officially addressed by the CV Task Team

Ah ok cool. So we should do our best here to create the best thing we can but be aware that the final home is still moving around a bit?

[taylor13]

I've had other urgent matters to attend to today, but I'll provide some brief feedback on the above questions and suggestions below. Let's not get too caught up with urgency (which I think we all are quite aware of) at the cost of causing too many future problems for us and our users. Remember that this whole exercise primarily aims to advance our understanding of climate. We do not expect the science to change dramatically between now and the next IPCC report. Most penetrating and innovative studies that the IPCC relies on will be based on individual models and CMIP6 results. There will, of course, be lots of graphs based on CMIP7 results, but we can't expect these to differ much from the past. All that being said, I think we should spend our full energy aiming to put in place all the essential CVs and variable register as soon as possible.

As Laurent mentioned, he will be the primary initial user of much of what we put in place; My colleagues at PCMDI and I will be the other primary early user because the CVs and variable register are essential for creating the CMOR tables needed by most modeling groups to write data in compliance with CMIP7 specs, and we are responsible for that. Since Laurent and I are co-chairs of the CV task team, you can count on us to work with the CV task team to do our best to put everything in place as soon as we can.

Now to my quick feedback on the comments above:

also support the reverse functionality of the main feature: from a branded variable label, extract and display its decomposed components

This will best be done by putting in place the variable register described in the draft journal article: https://docs.google.com/document/d/19jzecgymgiiEsTDzaaqeLP6pTvLT-NzCMaq-wu-QoOc/edit?tab=t.0. Although the register might be constructed following the JSON-LD approach, the information will be equivalent to what is shown in Appendix A of the draft journal article. We can use the register to simply look up all the attributes associated with a branded variable.

where is the list of dimension names (like is there somewhere that defines the difference between time1, time2 and time3)?

The list of CMIP6 dimensions, which has more than a 90% overlap with CMIP7, are found in this table: https://github.com/PCMDI/cmip6-cmor-tables/blob/main/Tables/CMIP6_coordinate.json. This table can be taken as the preliminary reference source for the coordinates and their attributes. The information may also be available in the air-tables Beth linked us to, but I've only quickly reviewed those, so not sure they contain everything we need.

Regarding testing the little code to generate branded variable names and testing it against CMIP6 variables:

for a given value of variable name (column BG)

For our purposes, the only information I'll vouch for now (without further review of the google sheets) is found in columns BY: root name, AZ: dimensions, T: cell_methods, and BK: branded variable name. Some of the information includes corrections to the CMIP6 root names (given in column BG, which should not be used), cell names, and dimensions. Column BK, giving the corrected CMIP6 branded variable names, is also not quite consistent with the latest version of the the branded variable construction rules. I describe the main differences in https://docs.google.com/document/d/1P-CudQtIq5dwAmU1kTeXCEzyclduEJobJDixGf7E5qc/edit?tab=t.0 . If you produce branded variables with the python code being written and display them against the strings in column BK, I can tell you if the two are consistent.

make sure that we can produce the expected branded variable label based on the CMIP6 root name (column BI)

Column Bl in the spread sheet was produced based on old information; the column to check against is BK.

3. then go back and try and this with raw CMIP6 information,

I'm not sure what the point of that would be. Can you explain?
Here's some background: The point of the google sheet was to identify errors and inconsistencies and incomplete descriptions found in the CMIP6 data request, which I hope are being addressed in CMIP7. When there are incomplete descriptions or errors, it is impossible to correctly create a branding suffix. That's part of its value; it forces us to pay attention to details. The corrections that need to be made if problems are found can only be made by human intervention (I'm quite sure).

Given the above, I think steps 3 and 4 in your proposed work might be replaced with ingesting from CMIP7 the new root names, dimensions, and cell_methods. I'll be happy to review the results and attempt to identify any problems. I'm confident that testing the python code on CMIP6 data will be sufficient to apply it to the new data request. Any problems will not be due to the python code, but rather with the input data, which will need to be corrected.

which way around does this go? Does the data request define the cell methods and dimensions or are the cell methods and dimensions defined upstream (e.g. by the CF-conventions) and then used by the data request?

Scientists gets to decide on how the data should be sampled and reported. Thus, the scientists specify:

1. the region sampled (global or some limited portion of the globe)
2. the sampling frequency (e.g., monthly, daily, hourly, etc.)
3. the method used for temporal sampling (e.g., a mean over an interval, an instantaneous value, a climatological value)
4. the horizontal sampling (e.g., a field carried on a horizontal grid, zonal means, site data, means over an area)
5. the vertical sampling (e.g., at a single height, depth, or layer, or on model standard levels or model levels)
6. the portion of a region that is "unmasked" (e.g., unmasked where "sea", where "land", where "ice sheets")

The specifications determine the dimensions and the cell_methods. The names of dimensions are under control of our CV task team (without any restrictions imposed by the CF conventions). The format of a cell_method is defined by the CF conventions. And there is controlled vocabulary included in the CF conventions that enables the text of cell_methods to be interpretated. Included are such terms as "mean" and "point" (indicating whether there is averaging or not), and the names of area_types (see http://cfconventions.org/Data/area-type-table/current/build/area-type-table.html).

[znichollscr]

Thanks @taylor13, helpful clarifications. To make sure I've understood (and to try and summarise maybe for others)

This will best be done by putting in place the variable register described in the draft journal article

Ok great. If it's easy to build in the reverse functionality, we'll build it into the python too (can always be used as an extra check of the register). If not, we'll just rely on the register.

The list of CMIP6 dimensions, which has more than a 90% overlap with CMIP7, are found in this table: PCMDI/cmip6-cmor-tables@main/Tables/CMIP6_coordinate.json

Great thank you

For our purposes, the only information I'll vouch for now (without further review of the google sheets) is found in columns BY: root name, AZ: dimensions, T: cell_methods, and BK: branded variable name

Ok great, good to understand. We'll use the columns you mention then, supplemented by docs.google.com/document/d/1P-CudQtIq5dwAmU1kTeXCEzyclduEJobJDixGf7E5qc/edit?tab=t.0 then write out what we produce and BK side by side so you can review.

I'm not sure what the point of that would be. Can you explain?

Provide an automated tool that can translate CMIP6 data (with its inconsistencies) into 'corrected' CMIP7 data. As a user, this would be super helpful so I can e.g. do analysis that compares CMIP6 and CMIP7 without having to write that mapping myself.

[taylor13]

Sounds like a plan.

Regarding

Provide an automated tool that can translate CMIP6 data (with its inconsistencies) into 'corrected' CMIP7 data. As a user, this would be super helpful so I can e.g. do analysis that compares CMIP6 and CMIP7 without having to write that mapping myself.

Part of my plan for the variable register is to record "usage" information that would translate the new branded variable tags into the compound "variable name + table" (e.g. "Amon.ta") labels that are being used in the CMIP7 data request and were used to uniquely identify variables in earlier MIP phases. An alternative would build a translator drawing information from the vocabulary and each MIP phase and linking the two. Maybe that's closer to your tool. I don't think we can do that until the structure and format of the variable register has been agreed. For CMIP6, the linkage information can probably be extracted from my google spreadsheet, but some work will need to be done to identify corresponding variables in the CMIP5 and CMIP3 lists of variables.