Skip to content

to_sql misses an existing table when the name has uppercase letters #799

Description

@laughingman7743

Problem

pyathena.pandas.util.to_sql() does not find an existing table whose name was given with uppercase letters, so if_exists="fail" does not fail and data is appended twice.

to_sql checks existence with:

SELECT table_name FROM information_schema.tables
WHERE table_schema = '{schema}' AND table_name = '{name}'

It passes schema and name through unchanged. Athena stores names in lowercase and reports them in lowercase in information_schema, so name="MyTable" never matches the existing mytable.

Reproduced against live Athena on master (3c69daf):

to_sql(df, "PyAthenaToSqlCase…", conn, location, schema="default", if_exists="fail")  # creates the table
to_sql(df, "PyAthenaToSqlCase…", conn, location, schema="default", if_exists="fail")  # no error
# SELECT count(*) -> 2   (one row written twice)
  • if_exists="fail" writes Parquet into the existing table's location and runs CREATE EXTERNAL TABLE IF NOT EXISTS, which is a no-op, so the rows are appended instead of raising.
  • if_exists="replace" follows the same path: the existing table is not dropped and its S3 objects are not deleted, so it appends too. This follows from the code and was not reproduced separately.

The existing test_to_sql uses a lowercase generated name, so it cannot see this.

Same query, same defect class

  • The check runs through conn.cursor() without disabling query result reuse. A connection with result_reuse_enable=True can get a cached "no rows" answer from before the table existed, and the existence check fails the same way. Fix metadata reflection throttling and reuse listed metadata #777 turned result reuse off for the dialect's information_schema fallback for this reason.
  • schema and name are interpolated into string literals without escaping '.

Proposed fix

Lowercase and quote-escape both identifiers in the existence check, the way _columns_from_information_schema does in the SQLAlchemy dialect. Execute the check with result_reuse_enable=False. Add a regression test that runs to_sql twice with a mixed-case name: once expecting if_exists="fail" to raise, and once expecting if_exists="replace" to leave one copy of the rows.

Out of scope

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions