Skip to content

perf: partition management DDL during startup can block under load #31

Description

@TheSentinel454

Problem

ensure_future_partitions(3) runs at startup (main.rs:200) and issues CREATE TABLE IF NOT EXISTS ... PARTITION OF DDL for each missing monthly partition across 2 tables (events and delivery_log).

Issues

  1. DDL during startup: CREATE TABLE ... PARTITION OF takes an ACCESS EXCLUSIVE lock on the parent table. If other pods are serving queries, the DDL blocks behind them AND blocks all subsequent readers.

  2. No lock timeout on partition DDL: partition creation uses a bare sqlx query with no statement or lock timeout. If another long-running query holds a conflicting lock, partition creation blocks indefinitely.

  3. Sequential partition creation: partitions are created one at a time in a loop (partition.rs:18-53). If 6 partitions need creating (3 months x 2 tables), that is 6 sequential DDL statements.

  4. Catch-all partition overlap: when a _p_future catch-all partition already covers the range, the overlap error (42P17) is swallowed with an info log. This means partition management never actually creates new partitions until the catch-all is detached — which there is no code for. The catch-all partition will accumulate all future data in one unpartitioned bucket.

  5. No partition pruning validation: there are no metrics or checks confirming that query plans actually use partition pruning.

Proposed changes

  1. Move partition management to a separate pre-deploy step or background task.
  2. Add SET lock_timeout before partition DDL to cap wait time.
  3. Add a partition-count metric polled in the metrics loop.
  4. Document or implement catch-all partition lifecycle (detach/split as time passes).

Priority

Medium — partition DDL is infrequent but each occurrence is a deploy-time risk. The catch-all accumulation is a slow-burn correctness issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions