Skip to content

fix: isolate unsupported column types during schema discovery - #33

Merged
hjianbo merged 3 commits into
emqx:mainfrom
hjianbo:fix/schema-json2-fallback
Oct 26, 2026
Merged

hjianbo merged 3 commits into
emqx:mainfrom
hjianbo:fix/schema-json2-fallback

Conversation

@hjianbo

@hjianbo hjianbo commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Row inserts currently discard the server schema when any column has an unsupported type, even if that column is omitted from the input. Falling back to inference loses the declared types of supported columns. For example, a table containing both JSON2 and DECIMAL rejects an ordinary JSON binary because it is inferred as STRING.

Separate supported schema columns from unsupported column metadata. Supported columns retain their server-declared types, independently of unrelated unsupported columns. This applies to all supported types; JSON2 is one regression case.

  • Omitted unsupported columns are left to database defaults and constraints.
  • Explicitly supplying an unsupported column, including nil or undefined, rejects the whole batch before sending it. Omitting a field is distinct from requesting SQL NULL.
  • Bulk writes still require a fully supported schema. This change does not add codecs for unsupported types.
  • Only GreptimeDB's structured TableNotFound error (4001) permits local inference. Other schema-query and decoding errors are returned directly.
  • Metadata queries quote and escape table identifiers. Names containing hyphens, mixed case or dots refer to the same raw table name used by gRPC; database selection comes from the client configuration.

Validation against real GreptimeDB v1.2.1 on OTP 27:

  • Full Common Test: 103/103 passed, including TCP, TLS, authentication and type suites. Regression coverage includes supported-column isolation using JSON2, mixed/new-column batches, async writes, whole-batch rejection, bulk rejection and authentication-error propagation. Identifier regressions cover existing tables, auto-creation and subsequent writes, mixed-row defaults through SHOW CREATE TABLE, and server rejection of invalid names containing double quotes.
  • Additional manual probes verified INT32 boundaries, server defaults for omitted DECIMAL columns, NOT NULL enforcement without partial rows, default changes and mixed-batch last-row ordering. Explicit nil rejection was compared with the old inference path, which silently omitted that field and applied its default.
  • Rust tests: 16/16 passed; build, formatting, cargo check, CI clippy flags, TOML formatting and cargo-deny checks passed.

Extra input-column scanning is limited to tables with unsupported types.

Type codec and query-result coverage gaps (including DECIMAL and VECTOR) are tracked separately in #34 and remain outside this PR.

Skip unused unsupported columns in row inserts and restrict schema inference to confirmed missing tables. Add real-database regression coverage for writes and error propagation.
@hjianbo hjianbo changed the title fix: preserve JSON2 encoding with unrelated unsupported columns fix: isolate unsupported column types during schema discovery Oct 21, 2026
Use general schema and test names, and document unsupported-column isolation and explicit NULL rejection.
@hjianbo
hjianbo marked this pull request as ready for review October 21, 2026 08:18
@hjianbo
hjianbo requested review from JimMoen and a balanced review from Copilot and removed request for Copilot October 21, 2026 10:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Preserve raw gRPC table names across metadata lookups and escape SQL identifiers. Cover existing and auto-created special names, mixed-row defaults, and authentication error propagation against GreptimeDB.
@hjianbo
hjianbo merged commit 17c5865 into emqx:main Oct 26, 2026
2 checks passed
@hjianbo
hjianbo deleted the fix/schema-json2-fallback branch October 26, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants