Repository navigation
fix: resolve database and schema qualified temporary tables as Snowflake does - #472
THEtheChad wants to merge 10 commits into
Conversation
…ake does Creating a temporary table with a qualified name, as dbt's unit tests do with `create or replace temporary table "DB"."S"."MODEL__dbt_tmp" as (...)`, failed with DuckDB's "TEMPORARY table names can *only* use the "temp" catalog". A temporary table created unqualified couldn't be read, written or dropped by its qualified name either, because DuckDB keeps every temporary table in its temp catalog rather than in the database and schema the name gives. In Snowflake a temporary table belongs to the database and schema it was created in, and takes precedence over a table of the same name there (https://docs.snowflake.com/en/user-guide/tables-temp-transient). A new temporary_table transform creates temporary tables unqualified and records the schema they belong to, using the per-session registry TableStages already keeps for temporary table stages. A qualified table reference that names a live temporary table in its own schema is pointed at DuckDB's temp catalog. Other references, including CREATE targets and names in other schemas, are left alone. COPY INTO keeps resolving its target by Snowflake name, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Share temporary-table identity through a dedicated per-connection registry. Validate schemas before removing CREATE qualifiers, rewrite qualified columns, and leave non-table DDL targets intact. Add SQL and shared-server session regression coverage. Co-Authored-By: GPT-6 Astra <noreply@openai.com>
# Conflicts: # fakesnow/conn.py # fakesnow/cursor.py # fakesnow/transforms/stage.py # tests/test_server.py
A temporary table created in a schema survived DROP SCHEMA, so it could still be read by its qualified name, including after the schema was recreated, and its name couldn't be reused there. DuckDB keeps temporary tables in temp.main, which DROP SCHEMA ... CASCADE doesn't touch, and the registry kept the dropped schema as their owner. In Snowflake dropping or replacing a schema drops its temporary tables, whichever session runs it. Register each temporary table with the OID of the schema it was created in. Before each statement, drop the session's temporary tables whose schema is gone or has a new OID, referencing tables first since DuckDB enforces foreign keys, and resolve qualified references only while the OID matches. This covers drops by other sessions, and DuckDB's ROLLBACK, which restores the schema's OID along with it. Registrations are no longer pruned when a table is dropped, because ROLLBACK can restore it under the same OID; dropped OIDs aren't reused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed transactions A temporary table referenced by a foreign key from another schema survived DROP SCHEMA, because DuckDB won't drop a referenced table, so it could still be read and its name couldn't be reused. Snowflake accepts foreign keys but doesn't enforce them, so drop them from CREATE TEMPORARY TABLE. No foreign key can then involve a temporary table, so the dependents-first ordering is no longer needed. DDL after a failed statement in a transaction raised DuckDB's "Current transaction is aborted" once a temporary table had been created, because the sweep queries the catalog before each statement. The sweep now leaves an aborted transaction to the next statement, so ROLLBACK, COMMIT, and DDL without qualified table references, behave as they do without temporary tables. Later commits handle CREATE TEMPORARY TABLE and qualified references. CREATE TEMPORARY TABLE now drops its foreign keys, so for temporary tables they are: - not validated, a deviation: a missing referenced table or non-key referenced columns are accepted, where Snowflake rejects them with 002003 or 001456 - not enforced, as in Snowflake, where fakesnow previously enforced them through DuckDB Permanent tables are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CREATE TEMPORARY TABLE ... CLONE created a permanent table in the current schema when its target was qualified, because lowering CLONE to CREATE TABLE AS SELECT dropped both the temporary property and the schema the table was resolved to. So with current schema DB1.S1, CREATE TEMPORARY TABLE DB1.S2.C CLONE DB1.S1.P created a permanent DB1.S1.C that other sessions could read, DB1.S2.C didn't exist, and dropping S2 left it behind. Keep both, so the clone is a temporary table registered to S2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A qualified foreign key in CREATE TABLE was resolved like any other qualified reference, so a temporary table sharing the referenced name captured it. With a permanent DB1.S1.PARENT and a temporary PARENT, CREATE TABLE CHILD (ID INT REFERENCES DB1.S1.PARENT (ID)) was rewritten to reference temp.main.PARENT, which DuckDB rejects as a cross-catalog foreign key. Resolving it also read the catalog, so after a failed statement outside autocommit the CREATE raised "Current transaction is aborted" instead of reaching the DDL handler, whenever any temporary table existed. CREATE TEMPORARY TABLE already drops its foreign keys, so any left are a permanent table's, and reference permanent tables. Leave them as written, without reading the catalog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Outside autocommit, DDL ends a transaction a failed statement aborted (see tekumara#470), so CREATE TEMPORARY TABLE RECOVERY (ID INT) succeeds there. Since temporary tables resolve their schema, it instead raised "Current transaction is aborted", because the schema is looked up before the statement runs, even with no temporary tables. When the aborted transaction rejects the lookup, look the schema up again once the DDL has ended the transaction, just before the table is created, and drop orphaned temporary tables then too. If the schema doesn't exist, raise the usual error without running the statement, so CREATE OR REPLACE never replaces an existing table first. Otherwise the table is registered, resolves by its qualified name and is dropped with its schema. MERGE, which first replaces a temporary table of candidates, takes the same path. A qualified CREATE TEMPORARY TABLE takes the same path, so it's created too. With autocommit on, the transaction is still aborted when the statement runs, so the lookup fails as the statement itself would. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BEGIN with autocommit on, then autocommit(False), INSERT and ROLLBACK raised "cannot start a transaction within a transaction": BEGIN was tracked only outside autocommit, so the INSERT tried to start an implicit transaction inside the one BEGIN had opened. Track BEGIN, COMMIT and ROLLBACK whatever the autocommit mode, so DML then joins the open transaction and ROLLBACK undoes it. COMMIT and ROLLBACK clear _in_transaction before they run, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Outside autocommit, DDL ends a transaction a failed statement aborted (see tekumara#470), so CREATE TABLE RECOVERY AS SELECT * FROM DB1.S1.P succeeds there. Once any temporary table had been created, even an unrelated one, since dropped, it instead raised "Current transaction is aborted", because each qualified reference read the catalog to check for a temporary table before the statement ran. The same went for temporary CTAS, CLONE and ALTER. Keep the names of the session's registered temporary tables, added as each is created and refreshed from the catalog before each statement, which picks up renames, drops and ROLLBACK. A qualified reference to any other name can't be a temporary table, so it's left as written, without reading the catalog. A reference whose name does match a temporary table, such as DB1.S1.P while a temporary P exists, still reads it, so it still fails in an aborted transaction; tekumara#470 owns that. While a transaction is open the names only grow, because ending it can restore a temporary table dropped or renamed in it. Otherwise, after BEGIN, DROP TABLE T and a failed statement, DROP TABLE DB1.S1.T outside autocommit skipped T, and once its implicit commit had rolled back the transaction, restoring the temporary T, dropped the permanent T. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A BEGIN that DuckDB rejected, eg: BEGIN NAME probe, still marked the transaction active. With autocommit then off, the stale flag skipped the implicit BEGIN, so DML committed as it ran and a later ROLLBACK undid nothing. Through the server too, with array bindings. Update _in_transaction once BEGIN, COMMIT or ROLLBACK succeeds, and leave it as it was when one fails. A failed COMMIT or ROLLBACK cleared it while DuckDB's transaction stayed open; DuckDB aborts that transaction, so nothing observable changes, but the flag now matches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… with it (tekumara#472) Conflicts resolved against tekumara#470 (a failed statement aborts an explicit transaction): cursor.py keeps tekumara#472's tracking of BEGIN in either autocommit mode and its temporary_tables.validate(), but leaves COMMIT and ROLLBACK to end the transaction only once they succeed, as tekumara#470 does. temporary_tables.py keeps tekumara#472's registry (table OID -> catalog, schema, schema OID) and its _names guard, with fork main's quoted temp.main identifiers; owners() returns the catalog and schema from the new registry. The tests are both sides' additions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ROLLBACK Since tekumara#470, a failed statement inside an explicit transaction aborts it and fakesnow refuses the DDL that tekumara#472's tests ran next, rather than letting it end the transaction. Each test now expects that refusal, rolls back, and then checks the temporary table's state and name resolution as before. The MERGE is refused by DuckDB's own error, as other DML in an aborted transaction is, because resolving its table names reads the catalog first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h it tekumara#472 makes fakesnow drop a schema's temporary tables with it, as Snowflake does, so a case that drops another schema leaves no temporary table behind to report. The test of another schema's temporary table outliving later cases keeps that schema, and the test of a failed drop by the runner's fallback keeps fakesnow's old behaviour to reach it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wflake tekumara#472 fixed the bug that ddl.drop-schema-drops-temporary-table expected, so it now expects fakesnow to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mara#472) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merge order: this PR is easiest to land last. It overlaps three other open PRs:
Once those are in, I'll rebase this onto The rebase onto #470 is more than a textual merge. #470 refuses every statement in a transaction that a failed statement aborted, until
After the rebase they'll expect #470's refusal, then run I've already done that merge, together with every other open PR of mine, in my fork's |
Fixes #458.
dbt's native unit tests create qualified temporary result tables, for example:
Creation failed with DuckDB's
TEMPORARY table names can *only* use the "temp" catalog. Qualified reads, inserts and drops also failed after unqualified creation. DuckDB stores temporary tables intemp.main; Snowflake associates them with a database and schema, where they take precedence over permanent tables with the same name (docs).The new per-connection
TemporaryTablesclass owns qualification, schema validation and the existing OID registry extracted fromTableStages. Both SQL resolution and table-stage resolution now use the same registry. It runs after unquoted identifiers are uppercased, validates the original schema before stripping CREATE qualifiers, and records ownership only after successful execution. Qualified table and column references matching a live temporary table are rewritten totemp.main. Permanent CREATE targets and non-table DDL targets are preserved.Tests cover qualified creation, reads, INSERT, UPDATE, DELETE, MERGE, DROP IF EXISTS, replacement, permanent-table shadowing, aliases, CTEs, subqueries, joins, quoted mixed-case names, missing schemas, and a same-named sequence. A server test connects two sessions to one shared DuckDB instance and verifies that their same-named temporary tables remain isolated. The exact dbt sequence above was also run directly and returned
[(1,)].Remaining limitations:
COPY INTO S.Ttargeting a temporary table still fails, as on main. COPY targets retain their Snowflake names for table-stage and load-history resolution.SHOW TABLES,DESCRIBE TABLEandinformation_schema.tablesstill omit temporary tables, as on main.DB.S.T.*still hit DuckDB's parser limitation;T.*works. This also affects permanent tables on main.Validation of the first two commits: 558 passed, 2 xfailed; ruff check, ruff format --check and pyright pass. The 25 focused cases pass here; against v0.12.1 /
3f65f02, 23 fail and two guards pass (other-schema resolution and sequence isolation).Checked against a live Snowflake account (10.38):
create or replace temporary table <db>.<schema>.qtemp as select 1 as xand a qualifiedselectreturn(1,)there and here.Follow-up: a schema's temporary tables go with it
Snowflake drops a schema's temporary tables with the schema, including other sessions' tables, on
DROP SCHEMA,CREATE OR REPLACE SCHEMAandDROP DATABASE. Here they stayed in DuckDB'stempcatalog, so their names stayed taken. A differential check against live Snowflake found this. The later commits fix it, along with what the fix turned up:CREATE TEMPORARY TABLEdrops foreign key constraints, so a dropped schema can't be blocked by them. Snowflake doesn't enforce foreign keys either. Permanent tables' foreign keys are unchanged.CREATE TEMPORARY TABLE ... CLONEstays temporary, in the resolved schema.TemporaryTables.validate()keepsCREATE OR REPLACEfrom destroying a table when its target schema is missing.BEGIN/COMMIT/ROLLBACKin either autocommit mode, and only once the statement succeeds. A failedBEGINtherefore can't leave a laterINSERTcommitted when it should have rolled back.Checked against live Snowflake:
DROP SCHEMA,CREATE OR REPLACE SCHEMAandDROP DATABASEremove that schema's temporary tables from every session.BEGIN; DROP SCHEMA ...; ROLLBACKdoesn't bring them back, because DDL commits.Differences from Snowflake that remain:
CREATE: a missing table is002003, and a non-key column is001456. Here they aren't validated.TransactionException, which the server returns as a 500. That belongs with fix: a statement that fails to compile no longer aborts the transaction #470, which turns the aborted state into a connector error.Validation: 648 passed, 2 xfailed; ruff check, ruff format --check, pyright and typos pass.
🤖 Generated with Claude Code