Read the dialect's own queries through an API cursor and resolve federated absence from information_schema

[laughingman7743]

WHAT

Two reflection defects with one cause: the SQLAlchemy dialect parses the rows of queries it issues itself, and it ran those queries through whatever cursor class the connection was configured with, or decided absence from an error message it could not read.

get_view_definition() on DataFrame dialects (#785). SHOW CREATE VIEW returns the definition one row per line. Measured against a view whose formatted DDL contains a blank line:

dialect	before
awsathena+rest	correct, 16 lines
awsathena+pandas	TypeError: sequence item 6: expected str instance, float found
awsathena+arrow	15 lines — the blank line silently dropped
awsathena+polars	TypeError: sequence item 6: expected str instance, NoneType found

The dialect's own queries — SHOW CREATE VIEW and the information_schema.columns fallback — now open a plain API cursor through _internal_cursor(), with the default converter pinned, whatever cursor_class or unload the connection carries. The async adapter accepts a cursor class and maps Cursor to AioCursor. Under unload=true the fallback also stops writing a Parquet UNLOAD to S3 for a five-row metadata lookup.

A missing view is a query Athena runs and fails, raised without an underlying API error, and only that maps to NoSuchTableError. A failed GetQueryResults call propagates, including the first page that execute() prefetches. So does a DatabaseError from a failed StartQueryExecution.

has_table() on Lambda federated catalogs (#781 / #783 item 1). Measured against a live AthenaDynamoDBConnector catalog, a missing table comes back as:

MetadataException: Failed to invoke lambda function due to ... Requested resource not found
(Service: DynamoDb, Status Code: 400, Request ID: ...)

There is no (Service: AmazonDataCatalog; ...; Error Code: EntityNotFoundException; ...) envelope, so since #777 has_table() raised OperationalError instead of returning False, and create_all(checkfirst=True) failed against such a catalog. The wording is the connector's, so recognizing this string would not generalize.

A MetadataException that survives unwrapping now goes to the existing information_schema fallback, but only outside AwsDataCatalog. The fallback runs against the same catalog through the same connector: 0 rows means absent, a failing connector fails the query too, so an outage is not reported as absence.

Glue is excluded on purpose. It reports missing tables and permission failures in a recognized envelope, so an unrecognized message there has an unknown cause. information_schema filters by Lake Formation instead of failing, so answering from it would report a table the caller cannot see as absent — the false absence #777 exists to prevent. _is_fallback_error() holds that boundary. An S3 Tables catalog is not AwsDataCatalog, but it was measured to return the AmazonDataCatalog / EntityNotFoundException envelope, so _lookup_table maps it before the fallback is reached and its behavior is unchanged.

WHY

#781 asked whether #777's error handling and fallback hold on federated catalogs. Verified live against a temporary DynamoDB connector catalog; results are recorded on #781:

• Item 1 — missing-table detection: broken as suspected. Repaired here.
• Item 2 — information_schema.columns in a federated catalog: works. Same type names as GetTableMetadata, lowercase names, and 0 rows rather than an error for a missing table or schema. That last property is what makes the item 1 repair sound.
• Item 3 — ListTableMetadata Expression over 128 characters: works. A 140-character name matched with the lowercase regex _lookup_table sends, the original case, and a prefix regex.

get_view_definition() was found while comparing both reflection paths after #784 and shares the root cause of the arrow comment defect repaired in #782. #782 normalized values at one column; SHOW CREATE VIEW loses whole rows on arrow, so no value-level normalization could recover them.

Behavior changes

• Reflection no longer runs through a cursor_class set for instrumentation. The dialect's own queries always use the API cursor.
• Inspector.get_view_definition() can now raise a raw pyathena.error.* instead of a SQLAlchemy-wrapped DBAPIError, because it no longer goes through Connection.scalars(). The information_schema fallback has behaved this way since Fix metadata reflection throttling and reuse listed metadata #777.
• In a non-Glue catalog, an unrecognized MetadataException from GetTableMetadata is no longer retried. information_schema is asked immediately instead, as for throttling. _without_throttling_retries is renamed _without_fallback_retries; the excluded code set is unchanged.
• The compliance-suite case test_metadata_errors_do_not_establish_absence[None] now uses a permission message. It still asserts propagation, which is correct because that suite runs against Glue. The federated case has no catalog in CI and is covered by TestAthenaDialect::test_unrecognized_metadata_error_asks_information_schema.

Validation

• just format, just lint (ruff, mypy, cfn-lint), just docs lint pass.
• TestAthenaDialect (no AWS): 11 passed.
• Local live runs: tests/pyathena/sqlalchemy/ + tests/pyathena/aio/sqlalchemy/ — 446 passed at the rebased head. just test sqla — 510 passed; just test sqla-async — 509 passed. Both suite runs were before the last review repair and rebase, which CI covers.
• Regression coverage fails without the fix. Reverting _internal_cursor() to the user's cursor fails test_get_view_definition_across_cursor_types for pandas, pandas+unload and arrow. That test compares the full reflected definition with the rows Athena returned through an API cursor, and skips only if Athena formats the view without a blank line.
• Federated catalog, live: has_table('this_table_does_not_exist') changed from OperationalError to False, has_table on an existing table is True, and get_columns on the 140-character name returns the right columns.
• CI at 8e90677 passed every job: lint, docs build, offline, and 5 each of test, test-sqla and test-sqla-async.
• Reviews: /code-review (7 findings), a claims audit, an independent Codex review (3 findings), and two independent follow-ups. Every actionable finding was repaired, including two regressions this branch had introduced: result-paging failures reported as a missing view, and the comment guard dropped although a cursor_kwargs converter can still produce NaN. Records follow inline.

Limits

• Still unverified on federated catalogs: extra_info = 'partition key', because a DynamoDB backend has no partition columns, and case-preserving connectors, because the DynamoDB connector lowercases names in both ListTableMetadata and information_schema. Both need a Hive or similar connector.
• A converter given in cursor_kwargs still reaches the internal cursor, because Connection.cursor() applies cursor_kwargs last. The isinstance(comment, str) guard keeps that from producing NaN comments.
• get_view_definition() still reports a CANCELLED query, or one FAILED for a reason other than a missing view, as absence. This predates the branch: master mapped every OperationalError to NoSuchTableError, and this branch removes only API failures from that. The cursor does not keep the failed query's state, and a missing view reports ErrorCategory 1 / ErrorType 1502, which does not single it out.
• get_table_comment() and get_table_options() still have no throttling fallback. Tracked in get_table_comment and get_table_options have no throttling fallback #786.
• The temporary connector stack, Athena data catalog and DynamoDB tables are deleted after this merges.

Closes #781, closes #783, closes #785. Refs #777, #782, #784, #786

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one — implementation behavior (/code-review, high). Base 15325e172714e2e809dd5c597e8babb2c51a6fac (rebased; reviewed originally at d7e1dcd..580cf7a), head 205c9b14d4e152b39c810609455852f22d38512d. Surfaces: _get_columns fallback eligibility, _internal_cursor, _columns_from_information_schema, get_view_definition, the async adapter, the retry-policy derivation, and both test trees.

Result: FINDINGS, repaired.

1. High, repaired in 82d5648. MetadataException in the fallback set turned unparsed permission failures into false absence. _get_error_code unwraps only the AmazonDataCatalog envelope, and information_schema filters by Lake Formation instead of erroring, so it returns 0 rows for a table the caller cannot see. The fallback is now gated on the catalog not being AwsDataCatalog, in _answerable_from_information_schema (line 388).
2. Medium, disproven by measurement. "The fallback lowercases, so a mixed-case federated table is missed." The DynamoDB connector reports PyAthenaFederatedProbeMixedCase as pyathenafederatedprobemixedcase in both ListTableMetadata and information_schema. No change. A case-preserving connector remains unverified (see PR Limits).
3. Medium, repaired by test. A failed StartQueryExecution raises DatabaseError, the parent of OperationalError, so it is not caught. That is correct: a request that never ran is not a missing view. test_get_view_definition_propagates_a_failed_request pins it.
4. Low, repaired in 82d5648. A connection-level converter would reach the internal cursor, so the default converter is now pinned. The follow-up is under the independent review below.
5. Low, repaired in a3fbf4d. The propagation case for an unrecognized Glue error had been removed. It is restored with a permission message.
6. Low, repaired, then superseded by the independent review. The blank-line assertion depended on Athena's formatter.

Also found myself: the async adapter did not accept a positional cursor class, and could not drive a synchronous Cursor anyway. just test sqla-async failed 3 cases on it. Repaired in c61c361 by mapping Cursor to AioCursor.

[laughingman7743]

Renamed _answerable_from_information_schema to _is_fallback_error in 8e90677, to match _FALLBACK_ERROR_CODES and _without_fallback_retries. The docstring now also says the catalog, not only the code, decides the result. No behavior change.

[laughingman7743]

Self-review round two — claims and operational behavior. Base 15325e172714e2e809dd5c597e8babb2c51a6fac, head 205c9b14d4e152b39c810609455852f22d38512d. Claims audited: this docs paragraph, the docstrings of _internal_cursor and _answerable_from_information_schema, the commit messages, and the PR body.

Result: FINDINGS, corrected.

1. The docs still said any MetadataException without an envelope goes to the fallback, after the code had been narrowed to non-Glue catalogs. Corrected in 8ecd6bf. The paragraph now states the catalog boundary and the Lake Formation reason for it.
2. The claim "only outside AwsDataCatalog" was not yet checked against S3 Tables, which is outside AwsDataCatalog but is Glue-federated. Measured live: a missing S3 Tables table returns Entity Not Found (Service: AmazonDataCatalog; ...; Error Code: EntityNotFoundException; ...), so _lookup_table maps it before the fallback. S3 Tables behavior is unchanged. The PR body says so.
3. get_view_definition against a missing view was measured live on rest and pandas: NoSuchTableError on both. The stub test for that path cites this.

Operator view: the fallback adds one information_schema query per unrecognized federated failure, instead of that failure's retry ladder. Internal queries no longer UNLOAD, which removes an S3 write and adds no query.

Deferred and stated in the PR: extra_info partition marking and case-preserving connectors on federated catalogs remain unverified. The raw pyathena.error.* from get_view_definition is not SQLAlchemy-wrapped. A cursor_kwargs converter still reaches the internal cursor.

[laughingman7743]

Independent review — relayed result. Reviewer: Codex, a different model from the author, through the repository's Codex rescue agent. Read-only static inspection, with no edits, tests or GitHub access. Reviewed d7e1dcd..5f87ed3; the rebased equivalent of that head is 8ecd6bf. Surfaces the reviewer read: all changed files, plus util.py, connection.py, common.py, cursor.py, converter.py and result_set.py, the async connection/cursor/adapter paths, and the pandas/arrow/polars cursor and result-set paths.

Result: FINDINGS, 3 items, all repaired in 205c9b1.

1. P2. fetchall() sat inside the OperationalError -> NoSuchTableError handler. A GetQueryResults page that exhausted its retries (result_set.py:370) reported an existing view as absent. This was a regression introduced by this branch: before it, iteration was outside the handler. Now only execute() is inside the handler.
2. P2. Connection.cursor() applies cursor_kwargs after the explicit arguments, so a cursor_kwargs["converter"] still overrides the pinned one. Dropping the isinstance(comment, str) guard had therefore reopened the NaN-comment path. The guard is restored at line 482, and the stub test has its NaN row back.
3. P2. The blank-line test could skip on the defect itself. Losing every blank row left CREATE VIEW and UNION ALL, so both assertions passed and the skip blamed Athena's formatting. The test now compares the full definition against rows fetched through an API cursor (line 949), and skips only when those rows have no blank line.

A follow-up on 8ecd6bf..205c9b1 has been requested and will be recorded in reply.

[laughingman7743]

Independent follow-ups on the repairs (same Codex reviewer, read-only, static inspection).

Follow-up 1, 8ecd6bf..205c9b1: item 2 (comment guard) CONFIRMED-RESOLVED. Two new FINDINGS, both repaired in 185f582:

• Cursor.execute() prefetches the first result page (result_set.py _pre_fetch), so narrowing the handler to execute() still caught a failed first GetQueryResults as a missing view. Measured live: a missing view is a query Athena runs and fails (View not found or not a valid presto view: ...), raised as OperationalError with __cause__ is None. An API failure is raised from its ClientError. Only the cause-less form maps to NoSuchTableError now. My earlier missing-view stub had modelled a rejected StartQueryExecution, which actually raises DatabaseError and never reaches this handler. It is replaced, and a failed-first-page test is added.
• The live test's baseline is read through the same API-cursor path as the fix, so a loss on both sides would go unnoticed. The independent CREATE VIEW / UNION ALL content checks are restored alongside the full comparison.

Follow-up 2, 205c9b1..185f582: stubs and live test CONFIRMED-RESOLVED. One remaining FINDING, recorded rather than fixed in bde5075: a CANCELLED query, or one FAILED for a reason other than a missing view, also raises a cause-less OperationalError and is still reported as absence. The reviewer confirmed this mapping predates the branch. Master mapped every OperationalError to NoSuchTableError; this branch only removed API failures from it. Distinguishing further would need a GetQueryExecution call on the failure path. I measured one: a missing view reports ErrorCategory 1 (SYSTEM) and ErrorType 1502, which does not single it out, so keying on it would be a guess. The comment now states the limit instead of implying a precise test, and the PR body lists it. No has-cause-but-failed-query path was found, sync or async.

Live after the repairs: a missing view gives NoSuchTableError on rest and pandas, and test_get_view_definition_across_cursor_types passes 5 of 5.

[laughingman7743]

Final independent follow-up, 185f582..8e90677 (same Codex reviewer, read-only): CONFIRMED-RESOLVED. There is no behavior change. The only edits are the rename, the docstring, and the new limit comment in get_view_definition(), and the decision logic of _is_fallback_error is unchanged. git grep finds no stale reference to the old name in tracked code, tests or docs. The new comment and docstring match the code: a failed or cancelled query raises a cause-less OperationalError and is read as absence, while API-layer failures keep their cause and propagate. The ErrorCategory/ErrorType measurement was taken as given, not re-derived.