Skip to content

feat: add CSVQ support for rule parsing #3595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jun 18, 2025
Merged

Conversation

encima
Copy link
Member

@encima encima commented May 21, 2025

What kind of change does this PR introduce?

Feature

What is the current behavior?

Currently, inspect reports are quite modular and provide output where column titles can vary

What is the new behavior?

  • Refined queries to output in a similar format and reduced/combined work to minimise the number of output files produced (or commands needed)
  • Support custom validation rules in supabase/config.toml
[[experimental.inspect.rules]]
query = "SELECT LISTAGG(stmt, ',') AS match FROM `locks.csv` WHERE age > '00:02:00'"
name = "No old locks"
pass = "✔"
fail = "There is at least one lock older than 2 minutes"

Additional context

CSVQ has been added as a library to provide a table printout of some rules that are/are not met by the CSV files produced by report

image

@encima encima requested a review from a team as a code owner May 21, 2025 13:53
@encima encima force-pushed the feat/inspect-rules branch from 80c020e to e266458 Compare May 26, 2025 09:54
@encima encima force-pushed the feat/inspect-rules branch from b9644ed to 9f4a035 Compare May 29, 2025 12:45
@encima encima force-pushed the feat/inspect-rules branch 3 times, most recently from 976759f to 287e217 Compare June 11, 2025 07:23
@coveralls
Copy link

Pull Request Test Coverage Report for Build 15578639723

Details

  • 41 of 132 (31.06%) changed or added relevant lines in 11 files are covered.
  • 6 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-0.5%) to 54.614%

Changes Missing Coverage Covered Lines Changed/Added Lines %
internal/inspect/report.go 7 11 63.64%
internal/inspect/db_stats/db_stats.go 13 19 68.42%
cmd/inspect.go 0 81 0.0%
Files with Coverage Reduction New Missed Lines %
cmd/inspect.go 1 0.0%
internal/gen/keys/keys.go 5 12.9%
Totals Coverage Status
Change from base Build 15552166588: -0.5%
Covered Lines: 5877
Relevant Lines: 10761

💛 - Coveralls

@encima encima force-pushed the feat/inspect-rules branch 2 times, most recently from 0555e38 to 4bbfacf Compare June 12, 2025 06:25
added csvq driver and normalised column names across queries
combine report queries to creats rollup for tables, indexes, roles and db overall
@encima encima force-pushed the feat/inspect-rules branch from 4bbfacf to a9f8459 Compare June 12, 2025 06:28
@encima encima requested a review from sweatybridge June 13, 2025 07:50
@@ -45,10 +44,10 @@ func Run(ctx context.Context, config pgconn.Config, fsys afero.Fs, options ...fu
return err
}

table := "|Schema|Table|Last Vacuum|Last Auto Vacuum|Row count|Dead row count|Expect autovacuum?\n|-|-|-|-|-|-|-|\n"
table := "|Name|Last Vacuum|Last Auto Vacuum|Row count|Dead row count|Expect autovacuum?\n|-|-|-|-|-|-|\n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
table := "|Name|Last Vacuum|Last Auto Vacuum|Row count|Dead row count|Expect autovacuum?\n|-|-|-|-|-|-|\n"
table := "|Table|Last Vacuum|Last Auto Vacuum|Row count|Dead row count|Expect autovacuum?\n|-|-|-|-|-|-|\n"

For clarity and consistency with unused_indexes.

(interval '1 millisecond' * (
COALESCE((row_to_json(s)->>'blk_read_time')::numeric, 0) +
COALESCE((row_to_json(s)->>'blk_write_time')::numeric, 0)
))::text AS sync_io_time,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we converting to json then back to number?

@sweatybridge sweatybridge changed the title Refine Inspect queries and add CSVQ support for rule parsing feat: refine inspect queries and add CSVQ support for rule parsing Jun 18, 2025
@sweatybridge sweatybridge changed the title feat: refine inspect queries and add CSVQ support for rule parsing feat: add CSVQ support for rule parsing Jun 18, 2025
@sweatybridge sweatybridge merged commit 0f1e20f into develop Jun 18, 2025
13 of 14 checks passed
@sweatybridge sweatybridge deleted the feat/inspect-rules branch June 18, 2025 06:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants