GH-47838: [C++][Parquet] Set Variant specification version to 1 to align with the variant spec

[aihuaxu]

Rationale for this change

According to the Variant specification, the specification_version field must be set to 1 to indicate Variant encoding version 1. Currently, this field defaults to 0, which violates the specification. Parquet readers that strictly enforce specification version validation will fail to read files containing Variant types.

What changes are included in this PR?

The change includes defaulting the specification version to 1.

Are these changes tested?

The change is covered by unit test.

Are there any user-facing changes?

The Parquet files produced the variant logical type annotation VARIANT(1).

Schema:
message schema {
  optional group V (VARIANT(1)) = 1 {
    required binary metadata;
    required binary value;
  }
}

• GitHub Issue: [C++][Parquet] Variant specification version should be set 1 to align with the variant spec rather than 0 #47838

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[aihuaxu]

@wgtmac Can you take a look?

[github-actions]

⚠️ GitHub issue #47838 has been automatically assigned in GitHub to PR creator.

[wgtmac]

+1

I'm fine with the current change for now. Eventually we may need to add setter and getter to VariantLogicalType for specification_version and validating its value.

cc @pitrou @raulcd

[pitrou]

Parquet readers that strictly enforce specification version validation will fail to read files containing Variant types.

Why is it not the case for our own reader?

[aihuaxu]

Parquet readers that strictly enforce specification version validation will fail to read files containing Variant types.

Why is it not the case for our own reader?

You mean Parquet C++ reader? Currently Parquet C++ reader hasn't implemented Variant read/write yet. The only thing it added is to support Variant logical type and we are not doing the check. Currently Variant spec version is version 1 (and this is the only version) and the other readers may/may not add the check.

[aihuaxu]

@wgtmac and @pitrou I added specification_version to VariantLogicalType. Please take another look. Thanks.

[raulcd]

You mean Parquet C++ reader? Currently Parquet C++ reader hasn't implemented Variant read/write yet. The only thing it added is to support Variant logical type and we are not doing the check. Currently Variant spec version is version 1 (and this is the only version) and the other readers may/may not add the check.

Sorry, because I might be missing something obvious, I am not too familiar with this part of the codebase, but if we haven't implemented Parquet C++ Variant write yet, I am not sure I understand how can a user would be able to create Variant files with a logical type annotation that will be incorrect with Parquet C++ if we release without this fix.

[aihuaxu]

You mean Parquet C++ reader? Currently Parquet C++ reader hasn't implemented Variant read/write yet. The only thing it added is to support Variant logical type and we are not doing the check. Currently Variant spec version is version 1 (and this is the only version) and the other readers may/may not add the check.

Sorry, because I might be missing something obvious, I am not too familiar with this part of the codebase, but if we haven't implemented Parquet C++ Variant write yet, I am not sure I understand how can a user would be able to create Variant files with a logical type annotation that will be incorrect with Parquet C++ if we release without this fix.

The engines will implement the reader/writer parts but will use the variant type defined in Arrow Parquet. That would cause the engines to write incorrect annotation. That's what I'm seeing internally.