feat: use a single shared parity-db instance for all chain data

[ozgb]

Overview

Uses the Custom database type to initialize a shared instance of ParityDb with custom config.

Alternative to #1273 to allow for GC implementation plus column compression

Requires an update to midnight-ledger: midnightntwrk/midnight-ledger#321

• UPDATE: This is included in ledger-8.1.0-rc.1 Update to ledger 8.1 #1296
• UPDATE 2: There is another change required to get this working for shared parity-db instances: feat: allow shared parity_db backend through generic Deref midnight-ledger#430
  • After discussion with @tkerber , we're leaving this change for the next release (ledger 8.2)

This is a breaking node binary update - it will require updated nodes to re-sync from genesis.

Resolves: https://github.com/midnightntwrk/midnight-security/issues/66

🗹 TODO before merging

• Ready
• Requires feat: allow shared parity_db backend through generic Deref midnight-ledger#430
• Waiting for feat: ledger 8.1 #1301
• Requires merge and release of feat: allow initialization of ParityDb backend using an existing database instance midnight-ledger#321

📌 Submission Checklist

• Changes are backward-compatible (or flagged if breaking)
• Pull request description explains why the change is needed
• Self-reviewed the diff
• I have included a change file, or skipped for this reason:
• If the changes introduce a new feature, I have bumped the node minor version
• Update documentation (if relevant)
• Updated AGENTS.md if build commands, architecture, or workflows changed
• No new todos introduced

🧪 Testing Evidence

Please describe any additional testing aside from CI:

• Additional tests are provided (if possible)

🔱 Fork Strategy

• Node Runtime Update
• Node Client Update
• Other:
• N/A

Links