GC improvements: GC only on a single node and add a missing index in PG #2159

josephschorr · 2024-12-09T20:46:07Z

Have GC lock so that it only runs on a single node at a time
Add a missing index in the Postgres datastore for GC

This should reduce datastore CPU pressure

vroldanbet · 2024-12-10T11:21:51Z

internal/datastore/postgres/common/locks.go

+
+// RunWithLocksClient runs the provided function with a pglock.Client. Should only be used
+// for migrations.
+func RunWithLocksClient(conn *pgx.Conn, runner func(client *pglock.Client) error) error {


Make it very clear this is only for migrations.

Suggested change

func RunWithLocksClient(conn *pgx.Conn, runner func(client *pglock.Client) error) error {

func RunWithLocksClientForMigrations(conn *pgx.Conn, runner func(client *pglock.Client) error) error {

vroldanbet · 2024-12-10T11:27:10Z

internal/datastore/postgres/common/locks.go

+	client, err := pglock.UnsafeNew(db,
+		pglock.WithCustomTable(locksTableName),
+		pglock.WithLeaseDuration(timeout),
+		pglock.WithHeartbeatFrequency(heartbeatFrequency),


The default gcTimeout is 1m, meaning the lease will last 1 minute. The heartbeat is 2 seconds by default. If folks change the gcTimeout via SpiceDB configuration, the heartbeat won't scale with it. WithHeartBeatFrequency docs indicates it should never be greater than half of the timeout.

// WithHeartbeatFrequency defines the frequency of the heartbeats. Heartbeats
// should have no more than half of the duration of the lease.

Given that user-provided configuration could violate this, I propose we use a fraction of the timeout: e.g. 1/3.

You should also use max(heartbeatFrequency, timeout/3) to ensure we never go to zero or into a very frequent rate.

vroldanbet · 2024-12-10T11:32:34Z

internal/datastore/postgres/gc.go

+
+func (pgd *pgDatastore) LockGCRun(ctx context.Context, timeout time.Duration, gcRun func(context.Context) error) (bool, error) {
+	if pgd.gcInterval < lockMinimumInterval {
+		return true, gcRun(ctx)


it seems like with a gcInterval less than 30 seconds, locks will be bypassed. If customer reconfigures SpiceDB GC, they could end up regressing the singleflighted GC runs, overload the datastore, and cause an incident. Seems like dangerous behaviour we should prevent.

I don't think this should be compared against lockMinimumInterval, but against timeout, which is what's used for the lock duration. If that's the case, we can validate this during application bootstrap, instead of failing later on, when the service is considered healthy. I vaguely recall we already had such safeguards, better double check.

vroldanbet · 2024-12-10T11:35:54Z

internal/datastore/postgres/gc.go

+		// Run the GC process under the lock.
+		currentTimestampData, err := time.Now().UTC().MarshalBinary()
+		if err != nil {
+			return fmt.Errorf("failed to marshal current timestamp: %w", err)
+		}


it does not seem like this is being used to drive the lock. If the reason for this is jumping into the postgres instance and troubleshooting the locks, then please clarify this with a comment accordingly.

vroldanbet · 2024-12-10T11:37:36Z

internal/datastore/common/gc.go

+	// If the implementation does not support locking, it should just execute
+	// the function and return true.
+	// If GC was run within the last interval, the function should return false.
+	LockGCRun(ctx context.Context, timeout time.Duration, gcRun func(context.Context) error) (bool, error)


I don't see any new tests added - please add one to pg integration test suite, simulating multiple GC instances running at once.

vroldanbet · 2024-12-10T11:39:36Z

internal/datastore/postgres/migrations/zz_migration.0022_add_gc_lock_table.go

+	if err := DatabaseMigrations.Register("add-gc-lock-table", "add-expiration-support",
+		func(ctx context.Context, conn *pgx.Conn) error {
+			return common.RunWithLocksClient(conn, func(client *pglock.Client) error {
+				return client.TryCreateTable()


Makes me uncomfortable being at the mercy of a third-party library migration code. How do we know bumping the library does not completely change the definition of the table and cause subsequent migrations to fail?

vroldanbet · 2024-12-10T11:41:04Z

internal/datastore/postgres/migrations/zz_migration.0023_add_missing_gc_index.go

+	"github.com/jackc/pgx/v5"
+)
+
+const addGCIndexForRelationTupleTransaction = `CREATE INDEX CONCURRENTLY 


Please add documentation to the PR body on why this was added, the previous explain, and the new explain.

Added a comment into the code

josephschorr · 2024-12-10T21:11:21Z

Redesigned to use the native locks as discussed

1) Have GC lock so that it only runs on a single node at a time 2) Add a missing index in the Postgres datastore for GC This should reduce datastore CPU pressure

ecordell

LGTM, though need to resolve the test failures

vroldanbet

Changes look good to me, BUT:

Please note this only solves a part of the problem: it prevents multiple nodes from GC'ing simultaneously, which will avoid contention and spikes, but it does not elect a leader node who is the only one that should be running GC.

For example, if 100 nodes are doing GC and have their timers sufficiently skewed, it's possible to have 100 nodes GC'ing one after the other (this is an extreme, worst-case scenario). It won't happen at the same time, but it is still an inefficient use of datastore compute and can cause a sustained load on the datastore, proportional to the number of nodes.

internal/datastore/mysql/locks.go

internal/datastore/postgres/migrations/zz_migration.0022_add_missing_gc_index.go

internal/datastore/mysql/locks.go

internal/datastore/postgres/locks.go

josephschorr · 2024-12-11T17:05:09Z

For example, if 100 nodes are doing GC and have their timers sufficiently skewed, it's possible to have 100 nodes GC'ing one after the other (this is an extreme, worst-case scenario). It won't happen at the same time, but it is still an inefficient use of datastore compute and can cause a sustained load on the datastore, proportional to the number of nodes.

That's true, but following the first GC, the next N GCs should more or less no-op. I considered adding a "last time GCed", but that would require a new table

This is necessary because GC takes an exclusive lock now

josephschorr · 2024-12-11T17:18:52Z

Updated

josephschorr requested review from vroldanbet and a team as code owners December 9, 2024 20:46

github-actions bot added area/datastore Affects the storage system area/dependencies Affects dependencies area/tooling Affects the dev or user toolchain (e.g. tests, ci, build tools) labels Dec 9, 2024

josephschorr changed the title ~~Have Postgres GC lock so that it only runs on a single node at a time~~ Postgres GC improvements: single node GC and a new index Dec 9, 2024

josephschorr force-pushed the postgres-gc-lock branch 4 times, most recently from a96f3fb to 62550af Compare December 9, 2024 22:00

vroldanbet reviewed Dec 10, 2024

View reviewed changes

josephschorr force-pushed the postgres-gc-lock branch from 62550af to 9fcc5c1 Compare December 10, 2024 21:10

josephschorr changed the title ~~Postgres GC improvements: single node GC and a new index~~ GC improvements: GC only on a single node and add a missing index in PG Dec 10, 2024

Garbage Collection improvements in MySQL and Postgres

c5d52b1

1) Have GC lock so that it only runs on a single node at a time 2) Add a missing index in the Postgres datastore for GC This should reduce datastore CPU pressure

josephschorr force-pushed the postgres-gc-lock branch from 9fcc5c1 to c5d52b1 Compare December 10, 2024 21:12

ecordell previously approved these changes Dec 10, 2024

View reviewed changes

josephschorr dismissed ecordell’s stale review via 58c7602 December 10, 2024 21:58

josephschorr force-pushed the postgres-gc-lock branch 4 times, most recently from 405bec5 to b7d77b7 Compare December 10, 2024 23:48

vroldanbet reviewed Dec 11, 2024

View reviewed changes

Move GC tests in PG out of parallel running

4f1ed5f

This is necessary because GC takes an exclusive lock now

josephschorr force-pushed the postgres-gc-lock branch from b7d77b7 to 4f1ed5f Compare December 11, 2024 17:18

vroldanbet previously approved these changes Dec 11, 2024

View reviewed changes

vroldanbet added this pull request to the merge queue Dec 11, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 11, 2024

vroldanbet added this pull request to the merge queue Dec 11, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 11, 2024

josephschorr added this pull request to the merge queue Dec 11, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 11, 2024

josephschorr added this pull request to the merge queue Dec 11, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 11, 2024

Increase timeout on GC check in GC test

6e584aa

josephschorr dismissed vroldanbet’s stale review via 6e584aa December 11, 2024 21:40

ecordell approved these changes Dec 11, 2024

View reviewed changes

josephschorr added this pull request to the merge queue Dec 12, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 12, 2024

josephschorr added this pull request to the merge queue Dec 12, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 12, 2024

josephschorr added this pull request to the merge queue Dec 12, 2024

Merged via the queue into authzed:main with commit d50ec46 Dec 12, 2024
22 checks passed

josephschorr deleted the postgres-gc-lock branch December 12, 2024 17:07

github-actions bot locked and limited conversation to collaborators Dec 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GC improvements: GC only on a single node and add a missing index in PG #2159

GC improvements: GC only on a single node and add a missing index in PG #2159

josephschorr commented Dec 9, 2024 •

edited

Loading

vroldanbet Dec 10, 2024

vroldanbet Dec 10, 2024

vroldanbet Dec 10, 2024

vroldanbet Dec 10, 2024

vroldanbet Dec 10, 2024

vroldanbet Dec 10, 2024

vroldanbet Dec 10, 2024

josephschorr Dec 10, 2024

josephschorr commented Dec 10, 2024

ecordell left a comment

vroldanbet left a comment

josephschorr commented Dec 11, 2024

josephschorr commented Dec 11, 2024

	func RunWithLocksClient(conn pgx.Conn, runner func(client pglock.Client) error) error {
	func RunWithLocksClientForMigrations(conn pgx.Conn, runner func(client pglock.Client) error) error {

GC improvements: GC only on a single node and add a missing index in PG #2159

GC improvements: GC only on a single node and add a missing index in PG #2159

Conversation

josephschorr commented Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josephschorr commented Dec 10, 2024

ecordell left a comment

Choose a reason for hiding this comment

vroldanbet left a comment

Choose a reason for hiding this comment

josephschorr commented Dec 11, 2024

josephschorr commented Dec 11, 2024

josephschorr commented Dec 9, 2024 •

edited

Loading