Skip to content

fix(db): seed postgresql-custom idempotently instead of empty-dir check - #250

Open
ymichaelson wants to merge 1 commit into
supabase-community:mainfrom
ymichaelson:fix/pgsodium-seed-idempotent
Open

fix(db): seed postgresql-custom idempotently instead of empty-dir check#250
ymichaelson wants to merge 1 commit into
supabase-community:mainfrom
ymichaelson:fix/pgsodium-seed-idempotent

Conversation

@ymichaelson

Copy link
Copy Markdown

What kind of change does this PR introduce?

Bug fix

What is the current behavior?

The init-pgsodium container copies default /etc/postgresql-custom config
files only when the pgsodium volume is empty ([ -z "$(ls -A)" ]). On
block-storage volumes (OpenEBS LVM, AWS EBS, GCE PD, Ceph RBD, ...) the
ext4/xfs filesystem ships a lost+found directory, so the volume is never
truly empty on first use. The copy is skipped, the config files are missing,
and postgres fails to start:

FATAL: configuration file "/etc/postgresql/postgresql.conf" contains errors
could not open configuration file "/etc/postgresql-custom/wal-g.conf": No such file or directory

Result: db StatefulSet in CrashLoopBackOff on any block-backed StorageClass.

Fixes #248.

What is the new behavior?

Replace the emptiness check with an idempotent, no-clobber copy:

cp -an /etc/postgresql-custom/. /mnt/pgsodium/ 2>/dev/null || true

Additional context

Verified cp -an idempotency locally: a runtime-modified file is preserved,
a missing file is re-seeded, and a lost+found directory has no effect.
helm lint passes and templates render. Chart version bumped 0.7.2 -> 0.7.3.

The init-pgsodium container gated the copy of default postgresql-custom
config files on the pgsodium volume being empty ([ -z "$(ls -A)" ]).
On block-storage volumes (EBS, GCE PD, Ceph RBD, OpenEBS LVM, ...) the
filesystem is formatted ext4/xfs and ships a lost+found directory, so the
volume is never truly empty on first use. The copy was skipped, leaving
/etc/postgresql-custom/*.conf missing and postgres failing to start with
"configuration file postgresql.conf contains errors" (CrashLoopBackOff).

Replace the emptiness check with an idempotent no-clobber copy (cp -an):
it seeds any missing config files on every start without overwriting
existing ones, preserving runtime-generated pgsodium keys (the reason the
persistent volume was introduced in supabase-community#197).
@luizfelmach

Copy link
Copy Markdown
Contributor

Hi @ymichaelson, thanks for the PR. The idea of replacing the empty-dir check with an idempotent copy is definitely the right direction for block-storage volumes.

I wanted to let you know that the test / test (pull_request) CI check is currently failing after ~6 minutes. The chart-testing install step fails because the db StatefulSet ends up in CrashLoopBackOff.

I reproduced it locally and the root cause is that cp -an behaves differently (and silently skips the copy) in BusyBox, which is what the supabase/postgres image uses:

$ docker run --rm --entrypoint /bin/sh supabase/postgres:17.6.1.136 -c 'cp --version'
cp: unrecognized option: version
BusyBox v1.37.0
$ docker run --rm -v vol:/mnt/pgsodium supabase/postgres:17.6.1.136 -c 'cp -an /etc/postgresql-custom/. /mnt/pgsodium/ && ls /mnt/pgsodium'
# exit 0, but nothing is copied

So the init-pgsodium container finishes successfully but leaves /mnt/pgsodium empty. Then the postgres container mounts that empty volume at /etc/postgresql-custom and fails with:

could not open configuration file "/etc/postgresql-custom/wal-g.conf": No such file or directory
FATAL: configuration file "/etc/postgresql/postgresql.conf" contains errors

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

db pod CrashLoopBackOff on block storage: pgsodium config copy skipped due to lost+found

2 participants