GH-49420: [C++][Gandiva] Fix castVARCHAR memory allocation and len<=0 handling

[dmitry-chirkov-dremio]

Rationale for this change

The castVARCHAR functions in Gandiva have memory allocation inefficiencies and missing edge case handling. See GH-49420 for details.

What changes are included in this PR?

Functional fixes:

• bool: Remove unused 5-byte arena allocation; return string literal directly
• int32/int64: Add handling for len=0 (return empty string) and len<0 (set error)

Memory allocation optimizations:

• int32/int64: Allocate fixed small buffer (11/20 bytes) directly in arena, use optimized digit-pair conversion writing right-to-left, then memmove to align. Returns min(len, actual_size) bytes.
• date64: Allocate only min(len, 10) bytes upfront (output is always "YYYY-MM-DD")
• float32/float64: Allocate only min(len, 24) bytes upfront (max output length)

Code cleanup:

• Extract common code into helper macros to reduce duplication

Are these changes tested?

Yes. Added tests for len=0 and len<0 edge cases for int64, date64, float32, float64, and bool types. All existing Gandiva tests pass. Adhoc performance benchmarking was performed both via direct expression evaluation as well as via query execution via Dremio.

Are there any user-facing changes?

No. Users will see reduced memory usage and proper error messages for invalid len parameter values.
Note: Error messages for negative len remain different between precompiled ("Output buffer length can't be negative") and interpreted ("Buffer length cannot be negative") code paths, preserving existing behavior.

• GitHub Issue: [C++][Gandiva] Fix castVARCHAR memory inefficiencies, unused bool allocation, and missing len<=0 handling #49420

[dmitry-chirkov-dremio]

Let me go through first-timer's hurdles (like pre-commit clang format failures).
GH-49347 for 'aws/core/utils/pagination/Paginator.h' file not found

[dmitry-chirkov-dremio]

Pushed clang-format fixes

[kou]

@lriggs @akravchukdremio @xxlaykxx You may want to review this.

[dmitry-chirkov-dremio]

@lriggs @akravchukdremio @xxlaykxx You may want to review this.

https://github.com/telemenar is reviewing offline. Second commit is based on their feedback.

[dmitry-chirkov-dremio]

Performance Benchmark Results

Benchmarks run on Apple M3, 5 repetitions each. Tests exercise castVARCHAR for Int32/Int64 with various len parameters. We see small yet noticeable improvements for int conversion. Date conversion's performance remained as is.

Test data:

• Int32 / Int64: Full-range random integers (1M rows)
• Int32Small / Int64Small: 2-digit integers 10-99 (1M rows)

Int32 (Full-Range Random)

len	Original (μs)	Optimized (μs)	Δ
1	16,416	16,243	-1.1% ✅
11	18,436	18,235	-1.1% ✅
100	19,043	18,455	-3.1% ✅
65536	18,844	18,013	-4.4% ✅

Int32 (Small 2-Digit)

len	Original (μs)	Optimized (μs)	Δ
1	11,710	10,853	-7.3% ✅
11	12,217	11,751	-3.8% ✅
100	12,208	12,182	-0.2% ✅
65536	12,207	11,614	-4.9% ✅

Int64 (Full-Range Random)

len	Original (μs)	Optimized (μs)	Δ
1	17,198	17,016	-1.1% ✅
20	19,369	18,922	-2.3% ✅
100	19,223	18,749	-2.5% ✅
65536	19,484	18,744	-3.8% ✅

Int64 (Small 2-Digit)

len	Original (μs)	Optimized (μs)	Δ
1	11,345	10,708	-5.6% ✅
20	12,306	11,736	-4.6% ✅
100	12,237	11,545	-5.7% ✅
65536	12,147	11,528	-5.1% ✅

Memory savings: Allocates 11-20 bytes per integer instead of up to 65,536 bytes - significant for engines that tend to use VARCHAR(65536)

p.s. updated to match 891c1ca

[telemenar]

There is a allocation pattern change for int that might be unexpected when len < max_len previously we would limit the allocation to len but now we always allocate at max_len.

I think this is fine though given the stated performance impact as there are a number of mitigating factors:

• max_len = 11|20 bytes for int32|int64 so the allocation size is capped and not huge.
• the arena used for this allocation is batch scoped. Which means that any overallocation is limited both the the processing time of a batch and to the number of rows/columns in a batch.
• in my experience it is rare that len < 11|20 bytes in most schemas I've seen in practice.
• out_len which is what is used by the caller is correctly limited by len so callers shouldn't ever touch the part of the buffer exceeding len

[dmitry-chirkov-dremio]

@telemenar

now we always allocate at max_len.
I reverified - going with out_len allocation introduces a measurable performance penalty (5-10%!) and I don't want to chase that optimization for ~10 byte/value savings. Some other day! 😄

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit b9274bc.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 30 possible false positives for unstable benchmarks that are known to sometimes produce them.