Is your feature request related to a problem? Please describe.
After libcudf adds Parquet VARIANT DECIMAL decoding - #23817, cudf-spark still needs a public cuDF Java/JNI path to request it. The current JNI methods pass only a native type ID; JNI cannot construct a decimal cudf::data_type without its scale.
Without this binding, try_variant_get(v, '$.amount', 'decimal(18,2)') must remain on CPU even though libcudf can decode the payload.
Describe the solution you'd like
Extend ai.rapids.cudf.VariantUtils and its cuDF-owned JNI bridge to support the finalized libcudf DECIMAL API. Target-decimal calls should retain the existing public shape:
DType decimal = DType.create(DType.DTypeEnum.DECIMAL64, -2);
try (ColumnVector value =
VariantUtils.castVariantValue(valueBytes, decimal);
ColumnVector extracted =
VariantUtils.extractVariantField(variantStruct, path, decimal)) {
// consume value and extracted
}
Pass both type ID and scale through JNI and construct the complete native cudf::data_type. A cuDF scale of -2 represents two fractional digits. Spark declared precision remains a cudf-spark responsibility.
If the native issue exposes a lossless mixed-scale source representation for DECIMAL-to-other-target conversion, add the corresponding owning Java result—for example, a table of unscaled values and per-row scale—rather than discarding that information in JNI.
Follow the existing VariantUtils conventions for argument validation, device selection, default stream/current memory resource, exception translation, ownership and caller-managed ColumnVector.close().
Describe alternatives you've considered
Additional context
Definition of done:
- Java and JNI tests cover Variant source scales through each encoding's maximum precision: DECIMAL4 0–9, DECIMAL8 0–18, and DECIMAL16 0–38.
- Tests cover translation to the corresponding cuDF scale, mixed per-row source scales, width boundaries, rescaling/rounding, overflow, mixed types, null/type mismatch, slices, empty/all-null inputs, and invalid arguments.
- Any lossless source-decoding result preserves row alignment, unscaled values, scale, ownership and cleanup semantics.
- Java results and exceptions match the native API.
Is your feature request related to a problem? Please describe.
After libcudf adds Parquet VARIANT DECIMAL decoding - #23817, cudf-spark still needs a public cuDF Java/JNI path to request it. The current JNI methods pass only a native type ID; JNI cannot construct a decimal
cudf::data_typewithout its scale.Without this binding,
try_variant_get(v, '$.amount', 'decimal(18,2)')must remain on CPU even though libcudf can decode the payload.Describe the solution you'd like
Extend
ai.rapids.cudf.VariantUtilsand its cuDF-owned JNI bridge to support the finalized libcudf DECIMAL API. Target-decimal calls should retain the existing public shape:Pass both type ID and scale through JNI and construct the complete native
cudf::data_type. A cuDF scale of-2represents two fractional digits. Spark declared precision remains a cudf-spark responsibility.If the native issue exposes a lossless mixed-scale source representation for DECIMAL-to-other-target conversion, add the corresponding owning Java result—for example, a table of unscaled values and per-row scale—rather than discarding that information in JNI.
Follow the existing
VariantUtilsconventions for argument validation, device selection, default stream/current memory resource, exception translation, ownership and caller-managedColumnVector.close().Describe alternatives you've considered
Additional context
Definition of done: