Skip to content

fix: resolve database and schema qualified temporary tables as Snowflake does - #472

Open
THEtheChad wants to merge 10 commits into
tekumara:mainfrom
THEtheChad:fix/schema-temp-tables
Open

THEtheChad wants to merge 10 commits into
tekumara:mainfrom
THEtheChad:fix/schema-temp-tables

Conversation

@THEtheChad

@THEtheChad THEtheChad commented Oct 26, 2026 •

Copy link
Copy Markdown

Fixes #458.

dbt's native unit tests create qualified temporary result tables, for example:

create or replace temporary table "DB"."S"."MY_MODEL__dbt_tmp" as (select 1 as x);
select * from "DB"."S"."MY_MODEL__dbt_tmp";

Creation failed with DuckDB's TEMPORARY table names can *only* use the "temp" catalog. Qualified reads, inserts and drops also failed after unqualified creation. DuckDB stores temporary tables in temp.main; Snowflake associates them with a database and schema, where they take precedence over permanent tables with the same name (docs).

The new per-connection TemporaryTables class owns qualification, schema validation and the existing OID registry extracted from TableStages. Both SQL resolution and table-stage resolution now use the same registry. It runs after unquoted identifiers are uppercased, validates the original schema before stripping CREATE qualifiers, and records ownership only after successful execution. Qualified table and column references matching a live temporary table are rewritten to temp.main. Permanent CREATE targets and non-table DDL targets are preserved.

Tests cover qualified creation, reads, INSERT, UPDATE, DELETE, MERGE, DROP IF EXISTS, replacement, permanent-table shadowing, aliases, CTEs, subqueries, joins, quoted mixed-case names, missing schemas, and a same-named sequence. A server test connects two sessions to one shared DuckDB instance and verifies that their same-named temporary tables remain isolated. The exact dbt sequence above was also run directly and returned [(1,)].

Remaining limitations:

  • COPY INTO S.T targeting a temporary table still fails, as on main. COPY targets retain their Snowflake names for table-stage and load-history resolution.
  • SHOW TABLES, DESCRIBE TABLE and information_schema.tables still omit temporary tables, as on main.
  • DuckDB's single temporary namespace still allows an unqualified reference to find a temporary table after switching schemas, and prevents same-named temporary tables in different schemas of one session. Fixing those requires a separate naming strategy.
  • Schema-qualified wildcards such as DB.S.T.* still hit DuckDB's parser limitation; T.* works. This also affects permanent tables on main.
  • This change does not implement the separate case-resolution work in fix: resolve table, schema and database names with Snowflake's case rules #453.

Validation of the first two commits: 558 passed, 2 xfailed; ruff check, ruff format --check and pyright pass. The 25 focused cases pass here; against v0.12.1 / 3f65f02, 23 fail and two guards pass (other-schema resolution and sequence isolation).

Checked against a live Snowflake account (10.38): create or replace temporary table <db>.<schema>.qtemp as select 1 as x and a qualified select return (1,) there and here.

Follow-up: a schema's temporary tables go with it

Snowflake drops a schema's temporary tables with the schema, including other sessions' tables, on DROP SCHEMA, CREATE OR REPLACE SCHEMA and DROP DATABASE. Here they stayed in DuckDB's temp catalog, so their names stayed taken. A differential check against live Snowflake found this. The later commits fix it, along with what the fix turned up:

  • The registry records each temporary table's owning schema by OID. Before each statement, a sweep drops temporary tables whose schema is gone or was replaced. The sweep is skipped inside an aborted transaction.
  • CREATE TEMPORARY TABLE drops foreign key constraints, so a dropped schema can't be blocked by them. Snowflake doesn't enforce foreign keys either. Permanent tables' foreign keys are unchanged.
  • CREATE TEMPORARY TABLE ... CLONE stays temporary, in the resolved schema.
  • Owner resolution waits until after a DDL statement's implicit commit, and TemporaryTables.validate() keeps CREATE OR REPLACE from destroying a table when its target schema is missing.
  • The catalog is read only when a statement names a live temporary table, so unrelated statements behave as on main.
  • The cursor tracks BEGIN/COMMIT/ROLLBACK in either autocommit mode, and only once the statement succeeds. A failed BEGIN therefore can't leave a later INSERT committed when it should have rolled back.

Checked against live Snowflake: DROP SCHEMA, CREATE OR REPLACE SCHEMA and DROP DATABASE remove that schema's temporary tables from every session. BEGIN; DROP SCHEMA ...; ROLLBACK doesn't bring them back, because DDL commits.

Differences from Snowflake that remain:

  • Snowflake validates a temporary table's foreign keys at CREATE: a missing table is 002003, and a non-key column is 001456. Here they aren't validated.
  • With autocommit off, after a statement fails inside a transaction, DuckDB's transaction stays aborted. A statement that names a live temporary table with a database or schema qualifier then raises DuckDB's raw TransactionException, which the server returns as a 500. That belongs with fix: a statement that fails to compile no longer aborts the transaction #470, which turns the aborted state into a connector error.

Validation: 648 passed, 2 xfailed; ruff check, ruff format --check, pyright and typos pass.

🤖 Generated with Claude Code

THEtheChad and others added 2 commits October 26, 2026 16:03
…ake does

Creating a temporary table with a qualified name, as dbt's unit tests do
with `create or replace temporary table "DB"."S"."MODEL__dbt_tmp" as (...)`,
failed with DuckDB's "TEMPORARY table names can *only* use the "temp"
catalog". A temporary table created unqualified couldn't be read, written
or dropped by its qualified name either, because DuckDB keeps every
temporary table in its temp catalog rather than in the database and schema
the name gives.

In Snowflake a temporary table belongs to the database and schema it was
created in, and takes precedence over a table of the same name there
(https://docs.snowflake.com/en/user-guide/tables-temp-transient).

A new temporary_table transform creates temporary tables unqualified and
records the schema they belong to, using the per-session registry
TableStages already keeps for temporary table stages. A qualified table
reference that names a live temporary table in its own schema is pointed
at DuckDB's temp catalog. Other references, including CREATE targets and
names in other schemas, are left alone. COPY INTO keeps resolving its
target by Snowflake name, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Share temporary-table identity through a dedicated per-connection registry. Validate schemas before removing CREATE qualifiers, rewrite qualified columns, and leave non-table DDL targets intact. Add SQL and shared-server session regression coverage.

Co-Authored-By: GPT-6 Astra <noreply@openai.com>
THEtheChad added a commit to THEtheChad/fakesnow that referenced this pull request Oct 26, 2026
# Conflicts:
#	fakesnow/conn.py
#	fakesnow/cursor.py
#	fakesnow/transforms/stage.py
#	tests/test_server.py
THEtheChad and others added 8 commits October 29, 2026 12:42
A temporary table created in a schema survived DROP SCHEMA, so it could
still be read by its qualified name, including after the schema was
recreated, and its name couldn't be reused there. DuckDB keeps temporary
tables in temp.main, which DROP SCHEMA ... CASCADE doesn't touch, and the
registry kept the dropped schema as their owner.

In Snowflake dropping or replacing a schema drops its temporary tables,
whichever session runs it. Register each temporary table with the OID of
the schema it was created in. Before each statement, drop the session's
temporary tables whose schema is gone or has a new OID, referencing
tables first since DuckDB enforces foreign keys, and resolve qualified
references only while the OID matches. This covers drops by other
sessions, and DuckDB's ROLLBACK, which restores the schema's OID along
with it.

Registrations are no longer pruned when a table is dropped, because
ROLLBACK can restore it under the same OID; dropped OIDs aren't reused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed transactions

A temporary table referenced by a foreign key from another schema
survived DROP SCHEMA, because DuckDB won't drop a referenced table, so
it could still be read and its name couldn't be reused. Snowflake
accepts foreign keys but doesn't enforce them, so drop them from
CREATE TEMPORARY TABLE. No foreign key can then involve a temporary
table, so the dependents-first ordering is no longer needed.

DDL after a failed statement in a transaction raised DuckDB's "Current
transaction is aborted" once a temporary table had been created,
because the sweep queries the catalog before each statement. The sweep
now leaves an aborted transaction to the next statement, so ROLLBACK,
COMMIT, and DDL without qualified table references, behave as they do
without temporary tables. Later commits handle CREATE TEMPORARY TABLE
and qualified references.

CREATE TEMPORARY TABLE now drops its foreign keys, so for temporary
tables they are:
- not validated, a deviation: a missing referenced table or non-key
  referenced columns are accepted, where Snowflake rejects them with
  002003 or 001456
- not enforced, as in Snowflake, where fakesnow previously enforced them
  through DuckDB
Permanent tables are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CREATE TEMPORARY TABLE ... CLONE created a permanent table in the
current schema when its target was qualified, because lowering CLONE to
CREATE TABLE AS SELECT dropped both the temporary property and the
schema the table was resolved to. So with current schema DB1.S1,
CREATE TEMPORARY TABLE DB1.S2.C CLONE DB1.S1.P created a permanent
DB1.S1.C that other sessions could read, DB1.S2.C didn't exist, and
dropping S2 left it behind. Keep both, so the clone is a temporary
table registered to S2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A qualified foreign key in CREATE TABLE was resolved like any other
qualified reference, so a temporary table sharing the referenced name
captured it. With a permanent DB1.S1.PARENT and a temporary PARENT,
CREATE TABLE CHILD (ID INT REFERENCES DB1.S1.PARENT (ID)) was rewritten
to reference temp.main.PARENT, which DuckDB rejects as a cross-catalog
foreign key. Resolving it also read the catalog, so after a failed
statement outside autocommit the CREATE raised "Current transaction is
aborted" instead of reaching the DDL handler, whenever any temporary
table existed.

CREATE TEMPORARY TABLE already drops its foreign keys, so any left are
a permanent table's, and reference permanent tables. Leave them as
written, without reading the catalog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Outside autocommit, DDL ends a transaction a failed statement aborted
(see tekumara#470), so CREATE TEMPORARY TABLE RECOVERY (ID INT) succeeds there.
Since temporary tables resolve their schema, it instead raised "Current
transaction is aborted", because the schema is looked up before the
statement runs, even with no temporary tables.

When the aborted transaction rejects the lookup, look the schema up
again once the DDL has ended the transaction, just before the table is
created, and drop orphaned temporary tables then too. If the schema
doesn't exist, raise the usual error without running the statement, so
CREATE OR REPLACE never replaces an existing table first. Otherwise the
table is registered, resolves by its qualified name and is dropped with
its schema. MERGE, which first replaces a temporary table of
candidates, takes the same path.

A qualified CREATE TEMPORARY TABLE takes the same path, so it's created
too. With autocommit on, the transaction is still aborted when the
statement runs, so the lookup fails as the statement itself would.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BEGIN with autocommit on, then autocommit(False), INSERT and ROLLBACK
raised "cannot start a transaction within a transaction": BEGIN was
tracked only outside autocommit, so the INSERT tried to start an
implicit transaction inside the one BEGIN had opened.

Track BEGIN, COMMIT and ROLLBACK whatever the autocommit mode, so DML
then joins the open transaction and ROLLBACK undoes it. COMMIT and
ROLLBACK clear _in_transaction before they run, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Outside autocommit, DDL ends a transaction a failed statement aborted
(see tekumara#470), so CREATE TABLE RECOVERY AS SELECT * FROM DB1.S1.P succeeds
there. Once any temporary table had been created, even an unrelated
one, since dropped, it instead raised "Current transaction is aborted",
because each qualified reference read the catalog to check for a
temporary table before the statement ran. The same went for temporary
CTAS, CLONE and ALTER.

Keep the names of the session's registered temporary tables, added as
each is created and refreshed from the catalog before each statement,
which picks up renames, drops and ROLLBACK. A qualified reference to
any other name can't be a temporary table, so it's left as written,
without reading the catalog. A reference whose name does match a
temporary table, such as DB1.S1.P while a temporary P exists, still
reads it, so it still fails in an aborted transaction; tekumara#470 owns that.

While a transaction is open the names only grow, because ending it can
restore a temporary table dropped or renamed in it. Otherwise, after
BEGIN, DROP TABLE T and a failed statement, DROP TABLE DB1.S1.T outside
autocommit skipped T, and once its implicit commit had rolled back the
transaction, restoring the temporary T, dropped the permanent T.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A BEGIN that DuckDB rejected, eg: BEGIN NAME probe, still marked the
transaction active. With autocommit then off, the stale flag skipped
the implicit BEGIN, so DML committed as it ran and a later ROLLBACK
undid nothing. Through the server too, with array bindings.

Update _in_transaction once BEGIN, COMMIT or ROLLBACK succeeds, and
leave it as it was when one fails. A failed COMMIT or ROLLBACK cleared
it while DuckDB's transaction stayed open; DuckDB aborts that
transaction, so nothing observable changes, but the flag now matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
THEtheChad added a commit to THEtheChad/fakesnow that referenced this pull request Oct 30, 2026
… with it (tekumara#472)

Conflicts resolved against tekumara#470 (a failed statement aborts an explicit transaction):
cursor.py keeps tekumara#472's tracking of BEGIN in either autocommit mode and its
temporary_tables.validate(), but leaves COMMIT and ROLLBACK to end the transaction only
once they succeed, as tekumara#470 does. temporary_tables.py keeps tekumara#472's registry (table OID ->
catalog, schema, schema OID) and its _names guard, with fork main's quoted temp.main
identifiers; owners() returns the catalog and schema from the new registry. The tests
are both sides' additions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
THEtheChad added a commit to THEtheChad/fakesnow that referenced this pull request Oct 30, 2026
…ROLLBACK

Since tekumara#470, a failed statement inside an explicit transaction aborts it and fakesnow
refuses the DDL that tekumara#472's tests ran next, rather than letting it end the transaction.
Each test now expects that refusal, rolls back, and then checks the temporary table's
state and name resolution as before. The MERGE is refused by DuckDB's own error, as
other DML in an aborted transaction is, because resolving its table names reads the
catalog first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
THEtheChad added a commit to THEtheChad/fakesnow that referenced this pull request Oct 30, 2026
…h it

tekumara#472 makes fakesnow drop a schema's temporary tables with it, as Snowflake does, so a
case that drops another schema leaves no temporary table behind to report. The test of
another schema's temporary table outliving later cases keeps that schema, and the test
of a failed drop by the runner's fallback keeps fakesnow's old behaviour to reach it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
THEtheChad added a commit to THEtheChad/fakesnow that referenced this pull request Oct 30, 2026
…wflake

tekumara#472 fixed the bug that ddl.drop-schema-drops-temporary-table expected, so it now
expects fakesnow to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
THEtheChad added a commit to THEtheChad/fakesnow that referenced this pull request Oct 30, 2026
…mara#472)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@THEtheChad

Copy link
Copy Markdown
Author

Merge order: this PR is easiest to land last. It overlaps three other open PRs:

Once those are in, I'll rebase this onto main.

The rebase onto #470 is more than a textual merge. #470 refuses every statement in a transaction that a failed statement aborted, until ROLLBACK. Six tests here create, replace or merge into a temporary table after a failed statement with autocommit off:

  • test_temporary_table_create_after_failed_statement[unqualified|qualified]
  • test_temporary_table_create_after_failed_statement_missing_schema
  • test_temporary_table_replace_after_failed_statement_missing_schema[replace|if_not_exists]
  • test_temporary_table_merge_after_failed_statement

After the rebase they'll expect #470's refusal, then run ROLLBACK, then make the checks they make now. In cursor.py, the result keeps #470's aborted-transaction check, and records BEGIN, COMMIT and ROLLBACK only once they succeed, as this PR does.

I've already done that merge, together with every other open PR of mine, in my fork's main: 1274 passed, 1 xfailed. The rebase will reuse it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

# Schema-qualified temporary tables fail, blocking native dbt unit tests

1 participant