Shared index registry is keyed by index OID only: databases cloned with `CREATE DATABASE … TEMPLATE` share index state (wrong-page reads, memtable corruption, a server crash under load)

[ifeulner]

Environment

• pg_textsearch v1.4.0 (tag, commit 7a93250), built from source
• PostgreSQL 18 (pgvector/pgvector:pg18 base image), shared_preload_libraries = 'pg_textsearch'
• Linux arm64 (container)

Summary

The global registry that maps an index to its shared state lives in cluster-wide shared memory, but its key is the index OID alone. OIDs are only unique within one database. A database created with CREATE DATABASE b TEMPLATE a gets a copy of a's catalogs, so its BM25 indexes have the same OIDs as those in a. Backends in a and b then look up the same TpSharedIndexState and write into one memtable, although the index relations underneath are different. Under concurrent writes this leads to reads of the wrong index pages and to memtable corruption, and in our test suite it crashed a backend under load, which put the whole cluster into crash recovery.

Where in the code (v1.4.0)

• src/index/registry.h:35-39 — TpRegistryEntry { Oid index_oid; /* Hash key - must be first */ dsa_pointer shared_state_dp; }
• src/index/registry.c:84 — params->key_size = sizeof(Oid); and the hash at :45 covers sizeof(Oid) only
• src/index/registry.c:138 — the registry is created with ShmemInitStruct, i.e. shared by all databases in the cluster
• src/index/state.c:184 — tp_get_local_index_state(Oid index_oid); registration at :415/:527 passes only index_oid
• MyDatabaseId isn't referenced anywhere in src/

#485 describes the same keying from the per-backend side (TpLocalIndexState cached per OID). That cache on its own is safe across databases, because a backend is connected to one database only. The shared registry is where databases collide.

What we observed

We give each parallel test worker its own database, cloned from one migrated template (a common test-isolation pattern). The databases contain BM25 indexes on two tables. Results with N workers writing concurrently, one database each, in one cluster:

Workers	Result
1	no errors
4	index corruption errors, queries fail
8	the same errors, then a backend crash and cluster-wide crash recovery (the database system is in recovery mode)

Errors, verbatim:

ERROR:  pg_textsearch memtable record at block 1, offset 1184 extends past free_offset
ERROR:  pg_textsearch fragment head page 1 has n_records=86 (must be 1)
ERROR:  pg_textsearch fragment head page 1 has n_records=92 (must be 1)

We didn't keep the server log from the 8-worker run, so we can't attach the signal or stack trace of the crashed backend. We can rerun and add it if that helps.

When every worker database is migrated separately instead of cloned, the indexes get distinct OIDs and the problem goes away. We reran the same suite at the same concurrency (4 and 8 workers) with separately migrated databases: every test passed, and the server log had no pg_textsearch errors, no crash and no recovery. The only thing that changed between the failing and the passing runs is how the databases were created.

Reproduction

Reproduces on every run within seconds.

-- as superuser, pg_textsearch in shared_preload_libraries
CREATE DATABASE tpl;
\c tpl
CREATE EXTENSION pg_textsearch;
CREATE TABLE docs (id bigserial PRIMARY KEY, content text NOT NULL);
CREATE INDEX docs_bm25 ON docs USING bm25 (content) WITH (text_config = 'english');
\c postgres
CREATE DATABASE a TEMPLATE tpl;
CREATE DATABASE b TEMPLATE tpl;

-- same OID in both databases:
\c a
SELECT 'docs_bm25'::regclass::oid;
\c b
SELECT 'docs_bm25'::regclass::oid;

Then write to both databases at the same time, e.g.:

cat > ins.sql <<'EOF'
INSERT INTO docs (content)
SELECT md5(random()::text) || ' ' || md5(random()::text) FROM generate_series(1, 50);
SELECT id FROM docs ORDER BY content <@> 'abc' LIMIT 10;
EOF
pgbench -n -c 4 -T 60 -f ins.sql a &
pgbench -n -c 4 -T 60 -f ins.sql b &
wait

Expected: both runs succeed.

Actual: both clones report the same index OID, one database runs fine, and every client in the other aborts immediately:

oid in a: 139571
oid in b: 139571

== a
number of transactions actually processed: 0
pgbench: error: client 0 script 0 aborted in command 1 query 0: ERROR:  could not read blocks 5..5 in file "base/139572/139571": read only 0 of 8192 bytes
pgbench: error: client 1 script 0 aborted in command 1 query 0: ERROR:  could not read blocks 6..6 in file "base/139572/139571": read only 0 of 8192 bytes
pgbench: error: client 2 script 0 aborted in command 1 query 0: ERROR:  could not read blocks 7..7 in file "base/139572/139571": read only 0 of 8192 bytes
pgbench: error: client 3 script 0 aborted in command 1 query 0: ERROR:  could not read blocks 6..6 in file "base/139572/139571": read only 0 of 8192 bytes

== b
number of transactions actually processed: 42944
number of failed transactions: 0 (0.000%)

The backends in a follow shared state that describes b's index and try to read pages that don't exist in a's own file.

Control. The same script with a and b created via plain CREATE DATABASE (not from the template) gives distinct OIDs (139656 / 139724), and both runs succeed: 39,273 and 39,411 transactions, 0 failed. The only variable is template cloning.

Suggested fix

Key the registry by (MyDatabaseId, index_oid), a 2×Oid key in the dshash and in the registry functions, and clean up a database's entries on DROP DATABASE. That also makes the registry safe against OID collisions between unrelated databases after OID wraparound, which is unlikely but possible.

Workaround

Don't create databases that contain BM25 indexes via CREATE DATABASE … TEMPLATE if they'll be written in the same cluster. Create and migrate each database separately so the indexes get distinct OIDs.