Skip to content

[FEA][Java][JNI] Expose Parquet VARIANT DECIMAL decoding #23819

Description

@nartal1

Is your feature request related to a problem? Please describe.

After libcudf adds Parquet VARIANT DECIMAL decoding - #23817, cudf-spark still needs a public cuDF Java/JNI path to request it. The current JNI methods pass only a native type ID; JNI cannot construct a decimal cudf::data_type without its scale.

Without this binding, try_variant_get(v, '$.amount', 'decimal(18,2)') must remain on CPU even though libcudf can decode the payload.

Describe the solution you'd like

Extend ai.rapids.cudf.VariantUtils and its cuDF-owned JNI bridge to support the finalized libcudf DECIMAL API. Target-decimal calls should retain the existing public shape:

DType decimal = DType.create(DType.DTypeEnum.DECIMAL64, -2);

try (ColumnVector value =
         VariantUtils.castVariantValue(valueBytes, decimal);
     ColumnVector extracted =
         VariantUtils.extractVariantField(variantStruct, path, decimal)) {
  // consume value and extracted
}

Pass both type ID and scale through JNI and construct the complete native cudf::data_type. A cuDF scale of -2 represents two fractional digits. Spark declared precision remains a cudf-spark responsibility.

If the native issue exposes a lossless mixed-scale source representation for DECIMAL-to-other-target conversion, add the corresponding owning Java result—for example, a table of unscaled values and per-row scale—rather than discarding that information in JNI.

Follow the existing VariantUtils conventions for argument validation, device selection, default stream/current memory resource, exception translation, ownership and caller-managed ColumnVector.close().

Describe alternatives you've considered

Additional context

Definition of done:

  • Java and JNI tests cover Variant source scales through each encoding's maximum precision: DECIMAL4 0–9, DECIMAL8 0–18, and DECIMAL16 0–38.
  • Tests cover translation to the corresponding cuDF scale, mixed per-row source scales, width boundaries, rescaling/rounding, overflow, mixed types, null/type mismatch, slices, empty/all-null inputs, and invalid arguments.
  • Any lossless source-decoding result preserves row alignment, unscaled values, scale, ownership and cleanup semantics.
  • Java results and exceptions match the native API.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions