Skip to content

Backend: Verify soft-delete and data retention actually prevent deleted data from being served #1350

Description

@pope-h

Background

The platform has substantial data-protection machinery: migrations/039_soft_delete_data_retention.sql, and routes for adminDataRetention.ts, adminErasure.ts, tenantErasure.ts, userErasure.ts, and tenantDataExport.ts. A recent merge (PR #1290) addressed PII handling.

Soft deletion introduces a specific hazard: the row is still there, so every read path must exclude it. Correctness depends on nothing forgetting.

Why This Matters

An erasure request is a legal commitment. Soft-delete means the data physically remains, so a single query that omits the deleted-at filter silently continues serving records the platform has told a user — and possibly a regulator — were erased. It fails open, produces no error, and is invisible until someone notices their data still visible after deletion.

The same class of bug affects ordinary deletion: a tenant who removes a document expects it gone, not merely hidden from one view.

Scope

  1. Inventory every table with a soft-delete column and every read path touching them — routes, repositories, admin views, exports, search indexes, and reports.
  2. Verify each read path excludes soft-deleted records. Prefer a mechanism that is correct by default (a repository-level filter or a database view) over relying on every future query remembering.
  3. Verify erasure covers derived and denormalised data — search indexes, caches, aggregates, outbox payloads, and logs — not just the primary row.
  4. Verify data export excludes erased records and does not leak another user's data.
  5. Verify retention jobs actually run and hard-delete on schedule, and that they cannot delete records still under a legal or financial hold. Financial records typically must be retained even after an account erasure — confirm the intended policy and make sure the implementation matches it.
  6. Add tests proving a soft-deleted record is unreachable via every enumerated read path.
  7. Document the retention policy per data category, if it is not already written down.

Acceptance Criteria

  • A complete inventory of soft-deletable tables and their read paths appears in the PR.
  • No read path serves a soft-deleted record — proven by test, per path.
  • Erasure covers derived data, not just primary rows.
  • Exports exclude erased records.
  • Retention jobs are verified to run and to respect holds.
  • Retention policy is documented per data category.
  • npm run lint and npm run test:ci pass.

Out of Scope

  • Changing the retention policy itself (document what exists; raise a separate issue to change it).
  • Frontend erasure UI.

CI — this PR must pass the Backend (lint, test) job

Extracted from .github/workflows/ci.yml. These are the exact steps CI runs, in order, from the backend/ directory on Node 22. Run them locally and get them green before pushing.

cd backend
npm ci                      # the backend uses npm — NOT pnpm
npm run lint                # eslint .
npm run test:ci             # vitest run (network-dependent suites excluded)
npm run openapi:validate    # redocly lint docs/openapi.yml

A red job blocks the PR. If you add or change an endpoint, update backend/docs/openapi.yml in the same PR — openapi:validate is a hard gate.

Metadata

Metadata

Assignees

Labels

Stellar WaveIssues in the Stellar wave programbackendBackend/API worksecuritySecurity hardening

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions