Delete messages

[sosna]

Handling incremental deletion of messages in SDMX-ML is described in Section 3A, Part IV, page 69.

The last sentence in that section looks sub-optimal. It reads as follows: "Finally, to delete a data attribute or observation value it is recommended that the value to be deleted be supplied; however, it is only required that any valid value be provided."

This looks sub-optimal considering that, in order to delete a particular attribute value, all I need to know is the attribute ID and the key of the element to which that attribute is attached. This simple, logical way is how SDMX-EDI works by the way. So, why, in SDMX-ML, are we asked to supply the attribute value? Worse, why is it OK to supply "any valid value", which is even more confusing?

For structure specific message, the XML specification allows empty attribute values (e.g. CONF_STATUS=""), so there is no technical reason why the attribute value must be provided.

For generic messages, the current syntax is as follows:

<generic:Value id="BIS_TOPIC" value="ABBA"/>

In that case as well, it would be sufficient in delete messages to write:

<generic:Value id="BIS_TOPIC" />

The only reason we could think of is that the schema generated for structure specific messages would need to be dependent on the action, i.e. there would be one schema for delete messages and one schema for the other action types. This can easily be addressed in the RESTful API though, by adding an action parameter to schema queries.

Maybe that this could be addressed within the scope of SDMX 3.0?

[dosse]

I agree with you that deletions are a "sub-optimal". This topics links also to the discussions on handling of missing/not-provided values in SDMX-ML/JSON messages with actions "REPLACE" and "INFO", as well as in SDMX-CSV messages.
Supporting different schema types might not be sufficient to solve this for generic SDMX-ML messages, because a message can contain several datasets with each a different action (e.g. as a result of includeHistory). Another solution would be to be less strict with the presence of values (as done for structure-specific messages).

[sosna]

Thanks, @dosse!

I believe that having multiple datasets in the response will not be a problem. The issue is similar with dimensionAtObservation: There as a well, the schema will vary depending on how the data are packaged (as time series, flat or cross-sectional messages).

The way this was addressed for dimensionAtObservation was to make the "packaging information" part of the URN, for example:

urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=ESTAT:NA_MAIN(1.10):ObsLevelDim:TIME_PERIOD

The URN is used as key to the schema and so, multiple datasets (in the same message) may very well reference different schemas.

The same could be achieved in this case, as long as the action is made part of the URN.

What do you think?

[dosse]

@sosna Sounds good!

[dosse]

Hi @sosna, since the generic message type has been deprecated, can we now close this ticket?

[dosse]

Hi @sosna, could we now close this ticket, please?

[agent96]

This is either a change to the technical notes on how to generate a schema (allow empty values) - however it would be inconsistent with the behavior of CSV which requires a value.

Alternatively this could be coupled with the ticket on reporting null values:
sdmx-twg/sdmx-csv#27

If a definition was defined in all formats which is the same - then the schema can allow this for attributes in addition to the standard list of allowable content - and the generated schema does not have to be action dependent.

The benefit of using a reserved term for missing is that the user does not need to process the data to discover the action in order to generate the correct schema.

[sosna]

@dosse: Sorry, I just saw your question about closing the ticket now :(.

And I'm not sure I understand the request to close the ticket actually. Yes, the examples use the Generic format, a now deprecated format. But these were just examples and the issue still remains with the other formats?

[dosse]

@sosna @agent96 What if instead of "any value" to be provided, the requirement would be to provide an "intentionally missing value", which we already have defined (but maybe not reflected in the messages yet)? Then this value is already allowed within the normal schema and no different delete schema would be required.

[sosna]

As indicated in another discussion (sdmx-twg/sdmx-rest#216 (comment)), the delete schema is not about which special char to use. I'm copying the link here, before we forget the issue was discussed in 2 places :-)