Support SQLAlchemy ARRAY partial UPDATE and slice resizing

[laughingman7743]

WHAT

Compile SQLAlchemy ARRAY element and inclusive-slice assignments into a single Athena Iceberg UPDATE, including ORM updates.
Rebuild nested arrays, extend with NULL padding, and resize slices while preserving WHERE clauses and other column assignments.
Preserve explicit value types, bind expressions, callable bindings, binary values, decorated ARRAYs, and decimal precision checks.

Keep the update expression, assignment/index types, and SQL generation classes in array.py, reusing the ARRAY type inspector and value processor.
Separate element and slice rendering, and share their prefix and NULL-padding SQL generation.
Group test_array.py tests by the corresponding implementation classes; keep runtime coverage in the existing SQLAlchemy suite.

Empty slice replacements delete, reversed ranges insert, NULL destinations act as empty arrays, and starts beyond the end pad with NULL.
Only one partial assignment per ARRAY column is accepted; combining it with a whole-column assignment or using a nonfinal slice is rejected.
Use deterministic index, boundary, and replacement expressions because compilation can repeat them. Padding follows Athena's native repeat() limits.

WHY

Athena does not accept ARRAY subscripts as UPDATE targets.
Rebuilding the column within the UPDATE enables indexed assignment without a client-side read/modify/write cycle.
Prefix/replacement/suffix concatenation also supports large existing arrays and scalar-subquery replacements.

Closes #755.
#774 and #775 are merged; this PR targets master.

TEST

Current head: 8f3f237ade70c8d70ed1a748a6562692722d664a on master.

• just format and just lint pass; the focused command below passes 257 type/compiler cases without AWS queries.
• All 18 current-head CI checks pass, including PyAthena, SQLAlchemy sync/async on Python 3.10–3.14, lint, and documentation build.
• The master rebase and follow-up refactor each passed two focused self-reviews and an independent Claude Code follow-up (claude-opus-5-5, first-party max, effort medium).

uv run --env-file .env pytest --noconftest \
  tests/pyathena/sqlalchemy/test_types.py \
  tests/pyathena/sqlalchemy/test_array.py \
  tests/pyathena/sqlalchemy/test_map.py \
  tests/pyathena/sqlalchemy/test_struct.py \
  tests/pyathena/sqlalchemy/test_temporal.py \
  tests/pyathena/sqlalchemy/test_compiler.py -q

[laughingman7743]

Self-review round one — implementation behavior: CLEAN after the rebase/refactor repairs. Full initial scope: d3b296c..4dac6f5, all six changed files. Checked statement cloning/ordered assignments, ORM target identity and conflicts, cached and callable bindings, native/decorated/generic ARRAY types, nested rebuilding, NULL/empty/inclusive resize boundaries, scalar subqueries, and formatter transport for binary/decimal values.

The update expression, assignment/index types, and SQL renderer now own their responsibilities in array.py; tests are grouped in TestArrayUpdateCompiler. Confirmed and repaired during adaptation: decorated ARRAY bind indices were rejected by a raw isinstance check; precision-free Numeric partial binds bypassed the parent precision guard; detached target columns could produce an attribute error. Existing shared inspector/processor contracts are reused, and all 156 inherited methods in the affected test files retain their ASTs.

Validation: just format/lint, 243 pure type/compiler cases, six synchronous real-Athena update cases plus the affected decimal repair case, and six asynchronous update cases pass using existing fixtures and -n 1. The existing integration cases cover resize/delete/insert, padding, NULL destinations/replacements, nested cached indices, typed bindings, binary/decimal values, ORM, a 10,001-element array and scalar subqueries. SQLAlchemy 2.0.0 compile smoke passes. Volatile SQL expressions and native repeat() limits remain explicitly documented constraints; no performance guarantee is inferred. Current full CI is pending.

[laughingman7743]

Self-review round one repair: CLEAN. Repair scope: 4dac6f5..6f2bebe, with the same merge base d3b296c and a patch-series comparison. Checked all four repair files and direct type/compiler/cache callers. The only production logic change applies the existing precision guard to SQL-expression assignments too; a probe against the frozen old snapshot confirms that the old version emitted bare DECIMAL, while new element/slice expression cases reject it. Test responsibilities now map to TestArrayUpdate, TestArrayAssignmentType, TestArrayWriteIndexType and TestArrayUpdateCompiler. Fresh compile rejection, bind/literal processor rejection and cached literal server guards are independently exercised. 255 pure cases pass; both affected synchronous scenarios pass (the Timestamp case after fixing a test-only wildcard-import collision), and the two-case asynchronous repair run passes. All 156 inherited method ASTs remain unchanged. No further actionable implementation issue found.

[laughingman7743]

Repair scope: 6f2bebebca87aefc05ccf6bfa798da263169269b..9d260111e10600355c8b9b1b23e4b7e5f67698bf; frozen PR merge base d3b296cdfa63dff4b5cd06b4949324a810b19684. The initial full review remains recorded above. Round one follow-up: CLEAN. Traced the private require_precision rename through CAST and recursive ARRAY/MAP/STRUCT/decorator callers; defaults and generated types remain equivalent. Restored compiled-index processor attachment assertions and retained both bound-decimal and SQL-expression runtime assignments. Test classes now follow implementation responsibilities and source order. just format, just lint, 257 pure cases, and the affected real-Athena case in both sync and async modes pass. All 156 inherited test methods remain unchanged.

[laughingman7743]

Rebase follow-up: d3b296cdfa63dff4b5cd06b4949324a810b19684..9d260111e10600355c8b9b1b23e4b7e5f67698bf compared with fbcf8b88d845eb7577d5c481fad0fb265e832ebb..a9c8c18330413bf47e1bb0d9fa47a84d13bb3c06. Initial full reviews and repair follow-ups remain recorded above. Round one: CLEAN. Audited all six PR paths against the updated parent. Git object IDs for the complete pyathena and tests trees, pyproject.toml and uv.lock are identical before/after the rebase. Range-diff shows eight equivalent commits and one documentation-context difference; no ARRAY write behavior or regression test changed. The merged read-expression, direct cursor and partial-update sections remain coherent in the rendered HTML. Format/lint, 257 focused type/compiler cases, Markdown lint and current-source Sphinx HTML build pass; these focused cases do not query AWS. Current CI will exercise the full runtime suite.

[laughingman7743]

Self-review round two — claims, caller compatibility, and AWS operations: CLEAN. Full initial inventory at d3b296c..4dac6f5 includes all six changed files, PR description, new class docstrings, and partial-update documentation.

Traced the one-statement claim through visit_update and the generated whole-column expression: the rewrite is local compilation and adds no metadata fetch, client-side read/modify/write, retry, or account configuration. Existing WHERE/ordered assignments and target table identity remain intact. Checked every documented resize boundary against the integration case matrix, including NULL destinations, empty deletes, reversed insertion, out-of-range padding and invalid replacement rejection. The long-array/scalar-subquery claim is limited to the measured 10,001-element and scalar-subquery cases; repeat limits and deterministic expression requirements remain explicit. The example now identifies an existing Iceberg table and reflects its ARRAY column rather than using an undefined table variable.

Checked typed and callable bindings, decorators, binary/decimal transport and the parent precision rule, plus SQLAlchemy 2.0.0 compile compatibility. Test classes/fixtures retain the existing conventions and all 156 inherited method ASTs are preserved. Validation claims distinguish six synchronous cases before the decimal guard plus the affected repair case, current six asynchronous cases, 243 pure cases, and the current-source HTML build (154 warnings). The standard multiversion build targets master/tags and is not substituted for the current-source build. Full current CI and independent review remain pending. No additional actionable claim or operational issue found.

[laughingman7743]

Self-review round two repair: CLEAN. Repair scope: 4dac6f5..6f2bebe, with the same merge base d3b296c and a patch-series comparison. The docs now require declared decimal precision for expression assignments and explain validation-phase/exception differences with cached statements. Target-type casts remain deliberate: the proposed microsecond-loss premise was not applicable to Athena's documented millisecond read/write support (https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html), and synchronous/asynchronous tests preserve 456 ms Timestamp values and 4.56 decimal expression results. The new cache test observes DBAPIError from both invalid-index and NULL-slice guards and verifies unchanged data. Earlier long-array setup already passed on real Athena, so the unverified bigint INSERT concern is not treated as a defect. The current-source doc rebuild passes (56 incremental Sphinx warnings). Current aggregate CI and independent follow-up remain pending.

[laughingman7743]

Repair scope: 6f2bebebca87aefc05ccf6bfa798da263169269b..9d260111e10600355c8b9b1b23e4b7e5f67698bf; frozen PR merge base d3b296cdfa63dff4b5cd06b4949324a810b19684. The initial full review remains recorded above. Round two follow-up: CLEAN. Audited the revised precision and exception claims: Numeric precision is required, scale is needed to preserve fractional values; compilation/binding/server validation may expose CompileError/StatementError/DBAPIError. The private keyword has no remaining old callers. SQLAlchemy 2.0.0 compilation smoke, Markdown lint and current-source HTML build pass. Bound decimal 1.23 and expression result 4.56 were both verified through sync and async connections. Full current-head CI is pending; these targeted runs are not represented as a completed full matrix.

[laughingman7743]

Rebase follow-up: d3b296cdfa63dff4b5cd06b4949324a810b19684..9d260111e10600355c8b9b1b23e4b7e5f67698bf compared with fbcf8b88d845eb7577d5c481fad0fb265e832ebb..a9c8c18330413bf47e1bb0d9fa47a84d13bb3c06. Initial full reviews and repair follow-ups remain recorded above. Round two: CLEAN. Checked the revised PR body against source and evidence: the full current source/tests are identical to the previously reviewed head, while old real-Athena and CI claims are labeled prior-revision evidence. Closes #755 is present; automatic closure is conditional on eventual merge to the default branch. The new contribution template is followed with WHAT/WHY/TEST and a concrete validation record; all six changed files already exist, so no new-file header applies. Updated CI admission permits the same-repository branch. Current-head CI is pending, and 154 Sphinx warnings belong to the successful local build.

[laughingman7743]

Independent review (relayed): FINDINGS. Claude claude-fable-5-1 / verified first-party Max / effort high; session dd66cd12-7dbd-4fb8-9c53-2b72b047522c. Full static review of d3b296c..4dac6f5 covered all six changed files, direct compiler/formatter/type/ORM/cache paths and dependency sources. Read/Glob/Grep only; no execution, edits, network, PR discussion or memory access. Actual result collected; snapshot and worktree unchanged. The reviewer flagged unconditional RHS casts as a possible precision loss and noted that Numeric() SQL-expression assignments bypass the bound-value guard. The generated bare DECIMAL cast is confirmed by compilation; the repair will require declared decimal precision for SQL-expression assignments as well as bound values. The proposed wholesale removal of target-type casts is not applied on the unverified timestamp premise; supported timestamp precision will be checked in a real update. The timestamp(6) premise is not established for Athena: AWS explicitly documents millisecond precision for both Iceberg reads and writes (https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html). We will not claim microsecond preservation from Trino assumptions.

The reviewer also questioned whether the 10,001-element setup INSERT can run. That scenario already passed in both synchronous and asynchronous real-Athena tests, so it is not a confirmed failure. No general sequence-size or unlimited-array claim is made.

[laughingman7743]

Disposition at 6f2bebe: the SQL-expression decimal guard gap is repaired; both element and slice expression cases now require declared precision. Target-type casts are retained rather than removed on the unverified timestamp premise. Real sync/async UPDATE tests preserve millisecond timestamps and decimal expression values within Athena's supported contract. No microsecond-preservation claim is made. The long-array INSERT and UPDATE already passed in the initial sync/async suite; no setup change was needed. Both self-review perspectives were reapplied; independent follow-up is pending.

[laughingman7743]

Independent narrow follow-up (relayed): FINDINGS, all low severity, no blocking defect; the reviewer confirmed the production repair is correct. Claude claude-fable-5-1 / Max / high, session 451eaea0-a1a9-470c-9154-e0551c594988, statically reviewed only 4dac6f5..6f2bebe, its range comparison, focused public excerpts, and the AWS contract excerpt (four files, about 32 KB). No other files were accessed; package and worktree were unchanged during review.

The reviewer confirmed the decimal guard, millisecond fixtures, and literal-cache tests, which would fail on a cache miss. Remaining items: (1) moving processor assertions lost verification that compilation attaches the index validator; (2) the expression decimal case replaced the prior bound-element runtime case; (3) callable compiler checks and a leftover 2-D test setup should be placed more precisely, with new classes ordered by responsibility; (4) precision-only validation and its helper/error wording should not imply enforced scale or binds-only use. Restoring both coverage paths and clarifying the internal keyword, error and documentation; both author review perspectives and a further bounded follow-up will be collected. The reviewer did not execute the Timestamp case; author runtime verification remains separately recorded.

[laughingman7743]

Relayed independent follow-up: CLEAN. Claude claude-fable-5-1, first-party max profile, effort high; session 6cbb9514-16b5-4e73-bd9e-9574d0f754e7. Static review of 6f2bebebca87aefc05ccf6bfa798da263169269b..9d260111e10600355c8b9b1b23e4b7e5f67698bf with merge base d3b296cdfa63dff4b5cd06b4949324a810b19684, using only four small public diff/context files. No edits, commands, tests, network, PR discussion, memory or delegation were allowed. The actual result was collected; review package hashes and the clean PR worktree were unchanged.

Covered the precision-keyword rename and callers, revised exception/precision documentation, test class responsibilities, compiler processor wiring, and the restored decimal round-trip. No blocking findings. The reviewer noted an optional split of the callable index/slice compile test at test_array.py:795-806; retained together because both cases exercise compiler handling of callable bindings, and the existing compile-success assertion remains intentional.

Package limits were checked locally: no implicit_bind or old decimal binds references remain in source/tests; precision assertions match the stable message substring; the suite declares Numeric(8, 2) at tests/sqlalchemy/test_suite.py:267. The affected bound-decimal and expression test passed against Athena in sync (37.03s) and async (37.62s) modes. These executions are author validation, not reviewer execution. Full current-head CI remains pending.

[laughingman7743]

Final validation for 9d260111e10600355c8b9b1b23e4b7e5f67698bf: all 18 current-head checks succeeded. Test run completed the ordinary, synchronous SQLAlchemy and asynchronous SQLAlchemy jobs on Python 3.10–3.14; documentation lint/build and benchmark tooling also succeeded. The Python 3.12 synchronous suite reports 536 passed and 766 skipped under suite capability conditions. Both self-review rounds and the collected independent follow-up cover this head. Published/local/reviewed heads match, the worktree is clean, and GitHub reports MERGEABLE. Marking this PR Ready; no merge performed.

[laughingman7743]

Independent post-merge rebase follow-up (Claude Code claude-opus-5-5, verified first-party max, effort medium; session e9de26ce-6322-4516-a906-71b788549205): CLEAN for fbcf8b88d845eb7577d5c481fad0fb265e832ebb..a9c8c18330413bf47e1bb0d9fa47a84d13bb3c06 versus e083e8caa759a232f8a1a0b60d1cf086080ee841..e58f4e020d8a58a097e374063cf33233940c0b6f. The reviewer inspected a bounded three-file public package: redacted range comparison, exact base/head tree and patch hashes, and focused source/test/docs excerpts. All nine patches, both base trees, both head trees, and both full PR diffs match. No merge-introduced ARRAY behavior, documentation, dependency, or in-tree CI change was found. This was static, read-only review; unchanged implementation and runtime behavior were not re-reviewed. The reviewer noted that external branch-protection/runner state is outside the package. The package and PR worktree were unchanged. Current-head CI is checked separately and remains in progress.

[laughingman7743]

Self-review round 1 of the ARRAY compiler simplification: CLEAN for repair delta e58f4e020d8a58a097e374063cf33233940c0b6f..8f3f237ade70c8d70ed1a748a6562692722d664a (PR merge-base e083e8caa759a232f8a1a0b60d1cf086080ee841). Reviewed the sole changed file, pyathena/sqlalchemy/array.py:593-655, against the element, nested-index, slice, NULL-padding, step-validation, and SQL-expression cases in the existing tests. _rebuild still normalizes the array/type once; it dispatches to element or slice rendering with the same arguments. remaining_path is path[1:], so the recursive case and base case are unchanged. _prefix_and_padding contains the same SQL fragments previously duplicated in each branch. No changed public API, SQL operator, or error condition was found. just format, just lint, and 257 focused no-AWS type/compiler cases pass on this head. Current-head AWS CI is pending.

[laughingman7743]

Self-review round 2 of e58f4e020d8a58a097e374063cf33233940c0b6f..8f3f237ade70c8d70ed1a748a6562692722d664a (PR merge-base e083e8caa759a232f8a1a0b60d1cf086080ee841), focused on claims, callers, and operations: CLEAN. The updated PR text accurately describes separate element/slice rendering and shared prefix/padding generation. The source diff changes one private compiler class only; existing callers, documentation examples, bind/literal processors, index validation, and AWS query count are unaffected. The previous revision passed 18/18 CI checks, while this new head is explicitly marked pending; no old run is represented as current validation. Re-read the deterministic-expression and repeat() limits in docs/sqlalchemy.md; both still apply because the generated SQL terms are unchanged. No additional tests that mirror this extraction are needed; the existing compiler cases exercise both branches. Independent follow-up and current-head CI remain pending.

[laughingman7743]

Independent refactor follow-up (Claude Code claude-opus-5-5, verified first-party max, effort medium; session ac3901f2-dfc7-4969-80ea-0a6a55859b6a): CLEAN for e58f4e020d8a58a097e374063cf33233940c0b6f..8f3f237ade70c8d70ed1a748a6562692722d664a. The reviewer read the exact one-file patch and focused ARRAY compiler, caller-test, and documentation excerpts. It found byte-equivalent SQL fragments, unchanged evaluation and validation order, equivalent recursive path handling, no expression-cache state change, and preserved documented slice/index boundaries. This was a static, read-only review with no tests or builds; unchanged implementation and helper bodies were outside scope. Separately, just lint and 257 focused no-AWS tests pass on this head. Current-head AWS CI remains in progress. The public package and PR worktree were unchanged.

[laughingman7743]

Final readiness verification for 8f3f237ade70c8d70ed1a748a6562692722d664a on master: all 18 current-head checks completed successfully. Test workflow 35808717094 passes PyAthena and synchronous/asynchronous SQLAlchemy jobs on Python 3.10–3.14; Docs Lint 35808716915 and Benchmark tooling 35808716911 also completed successfully. Locally, just format, just lint, and 257 focused no-AWS type/compiler cases pass. The master-rebase and compiler-simplification deltas each received two focused self-reviews and a collected independent Claude Code claude-opus-5-5 / first-party max / effort medium CLEAN follow-up. Published and local heads match; the worktree is clean, GitHub reports MERGEABLE, and Closes #755 is retained in the PR body. Marking Ready; no merge is performed.

[laughingman7743]

Independent review (relayed): the new tests are all in TestArrayUpdateCompiler, but rewrite validation, assignment processors, index processors, and SQL generation are separate implementation responsibilities. This does not fully meet the requested class correspondence. Split them into TestArrayUpdate, TestArrayAssignmentType, TestArrayWriteIndexType, and TestArrayUpdateCompiler while retaining the existing assertions and common table factory.

[laughingman7743]

Reorganized at 6f2bebe into TestArrayUpdate (rewrite/identity/order/cache), TestArrayAssignmentType (typed value transport), TestArrayWriteIndexType (bind/literal validation), and TestArrayUpdateCompiler (generated SQL/boundaries/precision). A shared pure table factory replaces the old class-local factory. Split the mixed slice test and moved index-processor assertions to their corresponding class; retained all inherited tests. 255 type/compiler cases pass.

[laughingman7743]

Independent review (relayed): fresh compilation and a compiled-cache hit can reject invalid literal indices or NULL slice replacements through different exception paths. SQL runtime guards protect the data, but literal-to-literal cache reuse is not directly pinned. The existing test here already covers named replacement bindings switching to None; add literal index/replacement cache cases, assert the server guard failures and unchanged data, and describe the validation-phase distinction. Non-blocking limits noted in the review remain: positive bounds rule out slice(1, 0), expression-based normalized indices are checked by Athena, and repeated SQL can reach native query-size limits sooner for deeply nested/large replacements. The generalized processor result type guard has no current non-ARRAY caller and is not a behavior defect.

[laughingman7743]

Added literal-to-literal compiled-cache coverage at 6f2bebe. It warms an element assignment before index 0 and a slice replacement before None, asserts the specific Athena guard failures as DBAPIError, and checks that both failed updates leave the array unchanged. Both synchronous and asynchronous execution confirm this behavior. Existing fresh-compilation and named-binding cases remain covered; the docs explain why validation phase and exception class may differ.