Skip to content

fix(solparq): match S3 restore columns by name, not position - #68

Open
picard8472 wants to merge 1 commit into
mainfrom
fix/solparq-s3-restore-column-name-match
Open

fix(solparq): match S3 restore columns by name, not position#68
picard8472 wants to merge 1 commit into
mainfrom
fix/solparq-s3-restore-column-name-match

Conversation

@picard8472

Copy link
Copy Markdown
Contributor

Problem

Fixes #62.

The solparq S3 restore path used INSERT INTO dest SELECT * FROM s3(...), which maps columns by position. The local restore path (INSERT INTO dest FORMAT Parquet) maps by name. So if a destination table's column order ever differs from the archived file's, local restores fine but S3 silently writes into the wrong columns (or errors on a type mismatch).

For an archive format meant to outlive schema changes, that drift will happen eventually. (Original review comment.)

Fix

Before the restore INSERT, read the archived file's own column order via DESCRIBE TABLE s3(...) (ClickHouse only reads the Parquet footer for this — cheap), then emit an explicit, backtick-quoted column list on both sides:

INSERT INTO dest (`col1`, `col2`, ...) SELECT `col1`, `col2`, ... FROM s3(...)

This makes the S3 path match columns by name, mirroring the local path. A file column absent from the destination now surfaces as an explicit error rather than a silent mis-column; a destination column absent from the file takes its default, exactly as the local FORMAT Parquet reader already does.

Changes

  • describe_s3_columns on the ClickHouse HTTP client + a query_text helper that keeps the response body.
  • build_s3_restore_sql now takes the archived column list and emits the explicit (cols) SELECT cols form.
  • S3Object struct groups the object URL + credentials (shared by describe and restore; also keeps build_s3_restore_sql under clippy's arg limit).
  • quote_identifier / parse_describe_columns helpers.
  • Updated module docs and crates/superbank/README.md.

Testing

  • cargo test -p superbank --locked — all pass (new unit tests for parse_describe_columns, quote_identifier, and the name-matched SQL shape).
  • cargo fmt --all -- --check and cargo clippy -p superbank --all-targets --locked -- -D warnings — clean.

🤖 Generated with Claude Code

The S3 restore used `INSERT INTO dest SELECT * FROM s3(...)`, which maps
columns by position, while the local restore (`FORMAT Parquet`) maps by
name. If a destination table's column order ever drifts from the archived
file's, the local path restores fine but the S3 path silently writes into
the wrong columns (or errors on a type mismatch).

For an archive format meant to outlive schema changes, that drift will
happen eventually. Read the archived file's own column order via
`DESCRIBE TABLE s3(...)` and emit an explicit, backtick-quoted column list
on both sides — `INSERT INTO dest (cols) SELECT cols FROM s3(...)` — so the
S3 path matches by name, mirroring the local path.

Fixes #62

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 restore matches columns by position, local restore by name — silent mis-column on schema drift

1 participant