Skip to content

chore: remove sqlparse #33564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

chore: remove sqlparse #33564

wants to merge 1 commit into from
+493 −3,139

Conversation

betodealmeida
Copy link
Member

@betodealmeida betodealmeida commented May 22, 2025 •
edited
Loading

SUMMARY

Part of SIP-117, stacked on:

Remove sqlparse from Superset.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@korbit-ai Korbit AI
Copy link

korbit-ai bot commented May 22, 2025

Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment /korbit-review.

Your admin can change your review schedule in the Korbit Console

@github-actions github-actions bot added risk:db-migration PRs that require a DB migration api Related to the REST API labels May 22, 2025
@michael-s-molina michael-s-molina added review:checkpoint Last PR reviewed during the daily review standup review:draft labels May 22, 2025
@rusackas rusackas removed the review:checkpoint Last PR reviewed during the daily review standup label May 27, 2025
@betodealmeida betodealmeida force-pushed the sqlglot-adhoc-subquery branch 14 times, most recently from 7d53164 to 0ff7dc3 Compare May 29, 2025 02:32
@betodealmeida betodealmeida force-pushed the remove-sql_parse branch 6 times, most recently from c315f16 to 4bc09be Compare May 29, 2025 14:09
betodealmeida
betodealmeida commented May 29, 2025
View reviewed changes
superset/db_engine_specs/base.py
Comment on lines -1770 to -1777
if not cls.allows_sql_comments:
query = sql_parse.strip_comments_from_sql(query, engine=cls.engine)
disallowed_functions = current_app.config["DISALLOWED_SQL_FUNCTIONS"].get(
cls.engine, set()
)
if sql_parse.check_sql_functions_exist(query, disallowed_functions, cls.engine):
raise DisallowedSQLFunction(disallowed_functions)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved this logic to the caller, to prevent DB engine specs from defining their own execute methods that skip this validation.

@betodealmeida betodealmeida marked this pull request as ready for review May 29, 2025 14:13
@korbit-ai Korbit AI
Copy link

korbit-ai bot commented May 29, 2025

Korbit doesn't automatically review large (3000+ lines changed) pull requests such as this one. If you want me to review anyway, use /korbit-review.

@betodealmeida betodealmeida force-pushed the remove-sql_parse branch 3 times, most recently from b123b54 to 094e4b9 Compare May 29, 2025 17:00
betodealmeida
betodealmeida commented May 29, 2025
View reviewed changes
tests/unit_tests/db_engine_specs/test_elasticsearch.py
@@ -93,21 +92,3 @@ def test_opendistro_sqla_column_label(original: str, expected: str) -> None:
from superset.db_engine_specs.elasticsearch import OpenDistroEngineSpec

assert OpenDistroEngineSpec.make_label_compatible(original) == expected


def test_opendistro_strip_comments() -> None:
Copy link
Member Author

@betodealmeida betodealmeida May 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is no longer responsibility of the DB engine spec.

betodealmeida
betodealmeida commented May 29, 2025
View reviewed changes
superset/models/helpers.py
@@ -1950,15 +1946,6 @@ def get_sqla_query( # pylint: disable=too-many-arguments,too-many-locals,too-ma
if extras:
where = extras.get("where")
if where:
try:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved this to the query object, since we need it for sanitize_clause.

@betodealmeida betodealmeida mentioned this pull request May 30, 2025
Open
9 tasks
@rusackas rusackas requested a review from villebro May 30, 2025 16:59
mistercrunch
mistercrunch reviewed May 30, 2025
View reviewed changes
superset/utils/rls.py Outdated
from superset.sql.parse import Table


def get_predicates_for_table(
Copy link
Member

@mistercrunch mistercrunch May 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering if in some cases the caller already has a handle on the table / database object and could save a db roundtrip (?) Maybe we need a command / DAO for this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we're always coming from a query, so there are no dataset handles — in fact, a query could have multiple tables, so we'd need dataset handles for all of them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, make sense, so potentially multiple round trips, one for each table ... probably fine as I'm guessing it was this way before.

@betodealmeida betodealmeida force-pushed the sqlglot-adhoc-subquery branch 2 times, most recently from 9a3a034 to acd467b Compare May 30, 2025 21:41
Base automatically changed from sqlglot-adhoc-subquery to master May 30, 2025 22:09
@betodealmeida betodealmeida force-pushed the remove-sql_parse branch 6 times, most recently from 2788837 to 2ca634b Compare May 31, 2025 00:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mistercrunch mistercrunch mistercrunch approved these changes

@michael-s-molina michael-s-molina Awaiting requested review from michael-s-molina michael-s-molina is a code owner

@eschutho eschutho Awaiting requested review from eschutho eschutho is a code owner

@villebro villebro Awaiting requested review from villebro

Assignees
No one assigned
Labels
api Related to the REST API preset-io risk:db-migration PRs that require a DB migration size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@betodealmeida @mistercrunch @rusackas @michael-s-molina