feat: prepare Didit KYC and KYB migration

[aledefra]

Summary

• replace provider-specific onboarding with a provider-neutral verification session API
• add Didit KYC/KYB session creation, signed durable webhook ingestion, authoritative reconciliation, retries, dead-letter handling, and health probes
• preserve approved/final-rejected Sumsub records while safely resetting every other legacy application for Didit
• retain restricted grandfathered Sumsub monitoring only for revocation/suspension
• add production-safe configuration, CI coverage, a dry-run-by-default cutover CLI, and an operator runbook

Why

Ratio1 is migrating individual KYC and company KYB onboarding from Sumsub to Didit without overlapping providers or weakening approval, invoicing, AML, country, or access-control gates.

The backend remains configured for Sumsub by default. Merging this PR does not activate Didit or modify production data.

Companion

• dApp PR: feat: migrate KYC and KYB UX to Didit ratio1-dapp#65
• Notion task: https://app.notion.com/p/38ed59ab6657805dadfbda77dc44a877

Data and rollout safety

• schema additions are backward-compatible
• provider switch defaults to sumsub
• cutover apply requires a maintenance window and exact reviewed counts
• only approved and finalRejected legacy KYCs are preserved
• reset-cohort stale identity/invoicing projections are removed transactionally
• no production database write is performed by this PR
• production workflow versions, secrets, webhook URL, age-of-majority defaults, privacy/retention approval, backup, and acceptance checks remain explicit activation gates

Verification

• focused config/service/handler/storage/cutover tests
• PostgreSQL integration and contention tests
• race tests
• go vet
• go build ./...
• runtime package compile gate
• JSON, formatting, whitespace, and secret scans
• independent critic review

Local Didit Sandbox E2E

Verified the full local chain with a disposable PostgreSQL 15 database, local backend, local dApp, and a real SIWE-authenticated test account:

1. the local dApp created a Didit individual-verification session through this backend
2. Didit preserved the immutable internal KYC UUID in vendor_data
3. Didit sent signed status.updated webhooks through a temporary HTTPS tunnel; the backend returned HTTP 202
4. duplicate HTTP deliveries were accepted idempotently and produced one processed event per provider event ID
5. reconciliation fetched Didit's authoritative current decision and projected the local session/KYC state
6. dApp polling observed the persisted transition and displayed the under-review state

The browser-automated camera could not satisfy Didit's liveness check, so a Sandbox approval was simulated in the Didit console to exercise the terminal webhook path. Since the decision lacked required liveness, proof-of-address, and questionnaire evidence, the backend correctly failed closed to onHold with approval_evidence_missing; it did not create an approved identity/invoicing projection.

No production database or production webhook was touched. The temporary tunnel, local services, disposable database, generated credentials, and test artifacts were cleaned up, and the Didit Sandbox webhook destination was restored to its previous disabled state.

[Copilot]

Pull request overview

This PR prepares the backend for migrating KYC/KYB onboarding from Sumsub to Didit by introducing a provider-neutral verification session + webhook persistence model, adding Didit-specific validation/risk logic, and wiring new endpoints/health probes and CI coverage while keeping Sumsub as the default provider.

Changes:

• Add verification persistence primitives (sessions, durable webhook events, notifications) plus migration preflight for unique KYC emails.
• Introduce Didit webhook signature verification/canonicalization, decline/risk classification, and policy/config plumbing.
• Add new verification + health HTTP endpoints, background worker lifecycle wiring, and a dry-run-by-default cutover CLI with runbook and CI workflow.

Reviewed changes

Copilot reviewed 69 out of 70 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
storage/verificationWorkflowStorer_test.go	Adds storage-level workflow tests for locking, idempotency, projection application, and dead-letter behavior.
storage/verificationStorer.go	Implements persistence APIs for verification sessions and durable webhook events.
storage/verificationStorer_test.go	Adds unit tests for verification session/event persistence and idempotency.
storage/kycStorer.go	Makes KYC upsert concurrency-safe with conflict detection and explicit field assignments.
storage/kycStorer_test.go	Adds tests for KYC upsert semantics, uniqueness migration preflight, and concurrency.
storage/init_test.go	Introduces isolated-schema PostgreSQL harness for storage tests (opt-in by env).
storage/healthStorer.go	Adds DB ping helper for readiness probes.
storage/conn.go	Adds KYC email uniqueness preflight and migrates new verification tables.
storage/allocationStorer_test.go	Ensures storage DB harness is required and adds missing UserInfo setup for test.
storage/accountStorer.go	Adds transactional account update + KYC creation helper.
storage/accountKycStorer_test.go	Adds tests ensuring transaction rollback/commit behavior for account+KYC updates.
service/verificationWorker_test.go	Adds tests for worker logic (retry backoff, terminal mapping, inactive worker).
service/verificationSessionIntegration_test.go	Adds PostgreSQL integration tests for concurrent Didit retry session creation and Sumsub monitoring recovery.
service/verificationService_test.go	Adds tests for session response validation, Didit URL allowlist, mapping/policy behaviors, and monitoring rules.
service/testdata/didit/error_rate_limited_429.json	Adds Didit API error fixture (429).
service/testdata/didit/error_invalid_workflow_400.json	Adds Didit API error fixture (400).
service/testdata/didit/decision_user_declined_minimum_age.json	Adds Didit decision fixture for minimum-age decline.
service/testdata/didit/decision_user_declined_expired_document.json	Adds Didit decision fixture for expired document decline.
service/testdata/didit/decision_user_approved.json	Adds Didit decision fixture for approved user.
service/testdata/didit/decision_user_approved_incomplete.json	Adds Didit decision fixture for incomplete approval payload.
service/testdata/didit/decision_business_declined_blocked_country.json	Adds Didit decision fixture for blocked-country KYB decline.
service/testdata/didit/decision_business_approved.json	Adds Didit decision fixture for approved business.
service/testdata/didit/create_session_201.json	Adds Didit create-session success fixture.
service/sumsubService.go	Adds provider ownership helpers and revoke-only grandfathered monitoring processing.
service/sumsubMonitoringPolicy_test.go	Adds tests asserting revoke-only monitoring behavior and ownership constraints.
service/diditWebhookValidation_test.go	Adds tests for Didit webhook envelope + family/kind validation rules.
service/diditSignature.go	Implements Didit signature verification (v2/raw/simple), envelope decode, and canonical JSON serialization.
service/diditSignature_test.go	Adds signature vectors and negative tests for Didit signature verification.
service/diditRisk.go	Implements Didit decline classification policy (reason codes, warnings, AML signals, country restrictions).
service/diditRisk_test.go	Adds tests for decline classification, projections, and restricted country set.
service/diditPolicy_test.go	Adds tests for building runtime Didit KYC/KYB policy sets and mapping constraints.
service/diditClient_sandbox_test.go	Adds opt-in live sandbox contract test for Didit client.
service/accountService.go	Uses transactional account update + KYC creation during email confirmation.
proxy/server.go	Wires VerificationService into server lifecycle, starts worker on Run, stops on Shutdown.
proxy/handlers/verification.go	Adds provider-neutral verification session endpoint and Didit webhook ingestion endpoint.
proxy/handlers/verification_test.go	Adds test ensuring oversized Didit webhooks are rejected before service execution.
proxy/handlers/sumsub.go	Adds Sumsub disablement when inactive, provider-ownership checks, and durable monitoring persistence with body limits.
proxy/handlers/sumsub_test.go	Adds test ensuring oversized Sumsub webhooks are rejected early.
proxy/handlers/health.go	Adds /health/live and /health/ready endpoints with DB ping + provider reporting.
model/verification.go	Adds GORM models/constants for verification sessions, webhook events, and notifications.
model/verification_test.go	Adds schema/index tests for verification model unique indexes.
model/kyc.go	Adds verification_provider ownership, unique email constraint, and cutover preservation helper.
model/kyc_test.go	Adds tests for provider-cutover preservation helper.
model/didit.go	Adds Didit API models (sessions/decisions/entities/features/warnings).
go.mod	Adds dependency for country handling (github.com/biter777/countries).
go.sum	Updates dependency checksums accordingly.
docs/didit-migration-runbook.md	Adds detailed operator runbook for cutover invariants, dry-run/apply, and safety gates.
deployScripts/deploy_testnet.py	Adds verification provider + Didit env vars to deployment template.
deployScripts/deploy_mainnet.py	Adds verification provider + Didit env vars to deployment template.
deployScripts/deploy_devnet.py	Adds verification provider + Didit env vars to deployment template.
config/didit_verification_config_test.go	Adds tests for Didit env override parsing/validation behavior.
config/config.testnet.json	Adds Verification/Didit config blocks for testnet defaults.
config/config.mainnet.json	Adds Verification/Didit config blocks for mainnet defaults (inactive Didit IDs by default).
config/config.go	Adds Verification and Didit config structs plus env override parsing.
config/config.devnet.json	Adds Verification/Didit config blocks and questionnaire IDs for devnet defaults.
cmd/verification-cutover/main.go	Adds dry-run-by-default cutover CLI entry point.
cmd/verification-cutover/cutover_test.go	Adds unit tests for cutover option + aggregate validation and DATABASE_LINK parsing.
cmd/verification-cutover/cutover_integration_test.go	Adds PostgreSQL integration test for apply transaction invariants.
cmd/main.go	Updates graceful shutdown wiring to stop verification worker and trap SIGTERM.
.github/workflows/test.yml	Adds CI workflow with PostgreSQL service and targeted/race test coverage for new verification paths.
.env.example	Documents new verification provider + Didit environment variables.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.