[Feature]: Harden and document native project planning and project knowledge

[roomote-community]

Problem

The complete project system needs staged rollout, recovery, security, observability, evaluations, and user documentation before broad release.

Proposed behavior

Add independent feature flags, opt-in conversion/backfill, backup/restore/deletion/reindex support, operational telemetry, notification batching, adversarial authorization/security tests, restart/race/provider-failure coverage, accessibility/responsive validation, and public docs.

Reuse existing telemetry, deployment settings, backup/restore, docs, package validation, and visual-proof workflows.

Acceptance criteria

• Features enable incrementally and disable without corrupting current task behavior.
• Backup/restore and deletion handle canonical and projected knowledge.
• Adversarial tests show no cross-project/user/repository leakage.
• Coordinator recovers from duplicate events and restarts.
• Documentation matches actual defaults, permissions, costs, and degraded modes.
• Main end-to-end scenarios pass in CI and browser proof.

Dependencies

• [Feature]: Add Manual, Guided, Full auto, and Custom project execution policies #1045
• [Feature]: Let Roomote recommend and fill project execution settings #1046
• [Feature]: Mirror Roomote projects to GitHub Project V2, issues, and milestones #1047
• [Feature]: Add deterministic project knowledge boundaries and cited full-text search #1048
• [Feature]: Add hybrid semantic search to project knowledge #1049
• [Feature]: Add cited historical summaries and legacy repository stabilization plans #1050

[roomote-community]

I've analyzed this issue and here's my proposed implementation plan:

1. After [Feature]: Add Manual, Guided, Full auto, and Custom project execution policies #1045-[Feature]: Add cited historical summaries and legacy repository stabilization plans #1050 land, add separate, default-off project-system flags in packages/feature-flags/src/config.ts for project UI/manual execution, coordination, GitHub mirroring, lexical knowledge, and semantic knowledge. Expose them through the existing admin controls in apps/web/src/trpc/commands/feature-flags/; disabling a flag must stop new work while preserving existing tasks, canonical records, and manual read access.
2. Add explicit, idempotent conversion and backfill jobs for existing plan artifacts and knowledge sources. Persist checkpoints and per-source outcomes so operators can opt in, resume safely, inspect failures, and retry without duplicating projects, events, chunks, or embeddings.
3. Extend the project services introduced by the dependency issues with lifecycle operations for archive/delete, export, restore reconciliation, and reindex. Treat PostgreSQL project/knowledge records as canonical; treat full-text indexes, embeddings, and pgvector rows as rebuildable projections. Tombstone unauthorized or deleted sources before asynchronous cleanup, and remove associated artifact objects using the existing task-artifact deletion pattern.
4. Integrate project data with the existing encrypted deployment backup in deploy/host/roomote and deploy/host/tests/backup-restore.integration.sh: verify canonical PostgreSQL rows and local MinIO content survive fresh-host restore, validate references before re-enabling coordinators, and enqueue projection reindexing after restore. Document that external S3-compatible storage still requires provider-level backup.
5. Harden coordination and notifications in apps/bullmq: use durable project events and stable job/idempotency keys, bounded retry classes, leases/checkpoints, and a scheduled recovery sweep so duplicate events, crashes, restarts, and provider outages cannot duplicate task launches or merges. Aggregate routine item activity into project-level batches while delivering blocking questions, approvals, and terminal failures immediately.
6. Add operational signals through the existing BullMQ Sentry hooks and @roomote/telemetry/server, without repository names, source text, citations, or other identifying content. Cover queue depth/age, dispatch and merge outcomes, retry exhaustion, recovery actions, indexing freshness and coverage, degraded lexical-only operation, embedding cost, mirror lag, and cleanup/reindex failures.
7. Add adversarial database/service tests for cross-project, cross-user, and cross-repository access across every read, search, citation, export, restore, and background-job path. Add deterministic tests for duplicate delivery, restart recovery, dispatch/merge races, deletion during indexing, embedding-model migration, unavailable pgvector, and source-control/inference failures.
8. Add CI coverage for package tests, the fresh-host backup/restore job in .github/workflows/CI.yml, project end-to-end scenarios, retrieval evaluations, and delegated browser proof covering responsive layouts, keyboard/focus behavior, permissions, degraded states, and the main manual/guided/full-auto flows.
9. Add public project documentation under apps/docs/ and register it in apps/docs/docs.json, covering setup, roles and permissions, execution policies, feature defaults, approvals, knowledge boundaries and citations, indexing/embedding costs, degraded modes, notifications, backup/restore/reindex/delete procedures, and rollback/disable behavior. Validate with pnpm --filter @roomote/docs check.

This stages rollout independently, keeps canonical data recoverable, makes derived knowledge safely rebuildable, and verifies that automation cannot weaken existing task behavior or authorization boundaries.

Please tag @roomote if you'd like me to implement this, or reply with feedback on the plan.