Skip to content

Integrate PRs #5, #6, and #7: hardening and pairwise evaluation - #8

Merged
VamikaSinghal merged 25 commits into
mainfrom
integration/cortex-launch
Aug 2, 2026
Merged

Integrate PRs #5, #6, and #7: hardening and pairwise evaluation#8
VamikaSinghal merged 25 commits into
mainfrom
integration/cortex-launch

Conversation

@VamikaSinghal

@VamikaSinghal VamikaSinghal commented Aug 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

This is the single integration pull request for PRs #5, #6, and #7, merged in that order onto integration/cortex-launch.

It combines:

The original branches remain unchanged. This PR is the only convergence point targeting main.

Combined changes

  • Adds reproducible Python 3.12 setup, root make workflows, examples, contributor documentation, issue templates, and SDK/plugin release gates.
  • Hardens connectors, credential handling, redirects, exports, backups, database concurrency, dependency auditing, hosted readiness, and deployment defaults.
  • Moves the archived prototype from repository root into legacy/.
  • Adds the pairwise evaluation domain, repository, ranking, preflight, admission receipts, execution checkpoints, owner/scalar study tooling, retention, encrypted artifacts, benchmarks, and integration documentation.
  • Adds the authenticated pairwise preflight endpoint to both FastAPI and the standalone runtime.
  • Keeps remote pairwise production execution intentionally unavailable pending the documented empirical and deployment gates.

Overlap and conflict resolution

Overlaps were reviewed across CI, setup, architecture/site documentation, README, configuration, connectors, API entrypoints, storage, vault backups, readiness tests, reranking tests, and release-manifest validation.

Resolved conflicts:

  • SETUP.md: kept the make setup/check/test workflow and pytest-only temporary vault/database isolation guidance.
  • site/index.html: retained signed/notarized release information with truthful local-first beta limitations.
  • CI: combined hash-locked dependency audits, release validation, pairwise checks, and realistic Python-suite timeout headroom.
  • README.md: preserved concise onboarding while incorporating pairwise architecture and limitations; removed duplicated merge content.
  • FastAPI and standalone entrypoints: preserved hardened export/session errors alongside pairwise preflight routing.
  • storage.py: combined embedding fingerprints, shared database coordination, configurable embeddings, conditional vault creation, and redirect hardening.
  • Readiness and update-manifest validation: retained HTTPS-only artifacts, digest/size validation, and the combined readiness assertions.
  • vault.py: fixed a semantic auto-merge regression where the pairwise backup security manifest was not forwarded into the locked backup helper.
  • Documentation: removed stale feature-branch links and aligned architecture, roadmap, and contributor guidance with the integrated legacy/ layout.

Validation

  • Backend: 2,467 passed, 7 skipped, 523 subtests passed
  • Python SDK: 29 passed
  • TypeScript SDK: 26 passed; type-check, build, and package dry-run passed
  • OpenClaw plugin: 8 passed; type-check, build, and release gate passed
  • Obsidian plugin: type-check and production build passed
  • Public examples and standalone-runtime smoke tests passed
  • Markdown links, documentation currency, distribution site, and update manifest passed
  • Retrieval, routing, adapter, context-pack, token-calibration, deterministic agent-task, delivery, adaptation, connector, beta-smoke, ops-readiness, and App Store gates passed
  • Deterministic pairwise benchmark: pair accuracy 1.0, repeat agreement 1.0, swap agreement 1.0, position bias 0.0, invalid rate 0.0
  • Bandit: no medium/high findings across 169,463 scanned lines
  • pip-audit: no known vulnerabilities in contributor, hosted, archived, or packaged-runtime dependency sets
  • macOS release runtime: bundled Python, Model2Vec, sqlite-vec, runtime smoke, build, and strict code-sign verification passed
  • GitHub Actions: all seven jobs passed

No repository-wide Python style linter or mypy gate is currently configured; Python compilation, tests, security analysis, and the application-specific lint gates passed.

Deployment and migration notes

  • New pairwise tables and lightweight schema upgrades are applied through database/repository initialization.
  • Existing plaintext pairwise reports require the documented preview → apply → audit → backup-audit → finalize migration.
  • Pre-cutover backups and external snapshots must be migrated or removed before pairwise storage is considered production-safe.
  • Pairwise retention must be added to the deployment scheduler; it is not scheduled automatically.
  • Hosted deployments must install the updated locked dependencies and restart API/worker services.
  • Admission receipts require a distinct secret of at least 32 bytes.
  • Encrypted report access requires CORTEX_KEK or CORTEX_KEK_FILE.
  • Review the new legal approval, account verification, encrypted-credential, export-limit, password-concurrency, and pairwise budget settings before deployment.

Remaining production gates

The pairwise subsystem is suitable for continued experimental use, not end-user production routing. Production enablement still requires owner-label calibration, repeated real-provider measurements, explicit consent/provider policy, quotas and spend limits, complete job lifecycle APIs, scheduled retention, and verified legacy backup cleanup.

Merge policy

Do not merge this pull request until Vamika explicitly approves it.

VamikaSinghal and others added 24 commits July 24, 2026 13:57
@VamikaSinghal
feat: add pairwise digital-twin evaluation
aa06715
@VamikaSinghal
feat: productionize pairwise digital-twin evaluation
6d67743
@VamikaSinghal
Merge docs/repo-refresh: bring doc refresh + build-52 bump onto the t…
d6ca35d 
…win-eval branch

Reconciles the two diverged checkouts so the README rewrite has a single base.
No file overlap between the branches; merge is clean.
@roshantaneja @claude
docs: correct four claims that contradict the code or the rest of the…
8ef7b89 
… docs

site/index.html contradicted itself on the same page: the Distribution
paragraph said "ad-hoc signed ... not yet notarized" while the install steps
directly below, the README, and site/downloads/latest.json all say Developer
ID signed and notarized by Apple. Three sources agree, so the paragraph was
the stale one.

SETUP.md documented `unittest discover` as the test command. CI deliberately
overrides that with pytest, because backend/tests/conftest.py -- which only
pytest loads -- pins CORTEX_DB_PATH/CORTEX_VAULT_PATH to a throwaway temp dir
at import time. Under `unittest discover` the suite binds to the default
backend/data/Cortex.vault instead, persists across runs, and writes into the
working tree. Matches SETUP.md to CI and says why.

The CI comment justifying that choice claimed conftest.py "provides fixtures".
It declares none; the import-time env isolation is the whole job, and it
matters because backend.app.main builds a process-singleton store on first
import. Corrected so the next reader doesn't inherit the wrong reason.

docs/ARCHITECTURE.md hand-copied the MCP tool list and had drifted to 42
stale entries against 110 in TOOLS -- query_memory and expand, which
docs/CMP_PROTOCOL.md references, were both absent. Replaced with the core
ten, the group structure, and a pointer to mcp_tools.py as source of truth,
rather than re-copying a list that will drift again. Also documents the
tool-surface mechanism (Cursor caps at 40, ChatGPT at 128) and that surface
is advertisement only, never authorization. Every tool name cited was
verified present in mcp_tools.py.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M12nsqf2Mw9cvEBeRiDBv1
@roshantaneja @claude
docs: add CLAUDE.md with the architecture facts that need multiple fi…
0b31fe2 
…les to learn

Onboarding guidance for Claude Code sessions, focused on what is not
discoverable from any single file: the two-server split and its parity
footgun, vault-as-truth vs SQLite-as-disposable-index, the review approval
gate, the CMP/SMP invariants, and the hosted control plane that is built but
dormant in the local product.

The load-bearing one: semantic near-dup, rerank, MMR and entity boost are all
gated on provider != "hash", and hash is exactly what CI runs, since
backend/requirements.txt does not install model2vec. A green CI run proves
the deterministic skeleton, not the semantic system -- rerank_eval.py exits 0
with status=skipped when the model is absent, and only release packaging
enforces it via CORTEX_REQUIRE_MODEL=1.

Also records the two cross-cutting conventions that surprise people: the
Python tests that assert on the text of Swift/HTML/CSS sources, and the
docs-currency phrase gate.

Drop this commit if the repo would rather not carry an agent-guidance file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M12nsqf2Mw9cvEBeRiDBv1
@VamikaSinghal @claude
docs: rewrite README against verified repo state
f631b4b 
Corrects the build flow, architecture description, and capability claims
against the code, CI, and shipped Info.plist.

Build/test:
- pip install backend/requirements.txt + pytest (runtime-requirements.txt
  has neither FastAPI nor pytest, so the documented flow could not work)
- Python 3.12, not 3.11+ (3.9 fails collection with 16 errors; 3.12
  collects 2408 cleanly)
- add dev_backend.sh, standalone_server, adaptation_eval; point at ci.yml

Architecture:
- the bundled engine is stdlib http.server (backend/app/standalone_server.py),
  not FastAPI; FastAPI is the hosted plane in backend/app/main.py
- relabel the Engine badge accordingly

Operating model, corrected against source:
- consolidation is never auto-scheduled in the app and is maintenance-gated
  off, so drop "runs while you are away"
- personalized-PageRank and multi-hop are opt-in, not the default Ask path
- the knapsack engages only when a model is named
- the delta channel never re-announces; the pack still carries content
- intent is a fallback retriever, not an always-on fused signal
- packs are sha256-addressed when pinned
- fix the seven layer names to match MEMORY_LAYERS

Removed unverifiable claims: "No account needed" (Info.plist ships
CortexRequireAccount=true), "nothing is remembered until you approve"
(auto_approve defaults true for user imports), "any MCP client",
the LLM condenser, and automatic update installation.

Added: current limitations, the four undocumented shipped surfaces
(extension, Obsidian plugin, OpenClaw plugin, SDKs), and a repo layout
that labels the root-level prototype as legacy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VamikaSinghal
docs: align repository guidance with shipping product
600e23a
@VamikaSinghal
chore: harden open-source readiness
8eedd5c
@VamikaSinghal
Harden Cortex for open-source review
7f27dc4
@VamikaSinghal
Harden release, docs, and developer experience
5aa2e94
@VamikaSinghal
Fix pairwise CI release and security gates
78fcdeb
@VamikaSinghal
Make release CI platform-safe and resilient
1cb883a
@VamikaSinghal
Make archived dependency audit platform-safe
58a7e0a
@VamikaSinghal
Mark archived macOS dependency correctly
6fd6d9b
@VamikaSinghal
Mark archived macOS dependency correctly
5d428b6
@VamikaSinghal
Fix pairwise integration regressions
45468f0
@VamikaSinghal
Give pairwise CI realistic test headroom
61fa6f2
@VamikaSinghal
Validate remote release artifacts in readiness CI
f2910c3
@VamikaSinghal
Merge PR #5: correct route and documentation claims
18c3fbb
@VamikaSinghal
Merge PR #6: harden Cortex for open-source launch
ad6fc40 
# Conflicts:
#	SETUP.md
#	site/index.html
@VamikaSinghal
Merge PR #7: productionize pairwise digital-twin evaluation
7c69ea5 
# Conflicts:
#	.github/workflows/ci.yml
#	README.md
#	backend/app/main.py
#	backend/app/standalone_server.py
#	backend/app/storage.py
#	backend/tests/test_ops_readiness_check.py
#	scripts/validate_update_manifest.py
@VamikaSinghal
Fix integrated backup and documentation contracts
89ea13a
@VamikaSinghal
Remove duplicated integration copy from README
43377f3
@VamikaSinghal
Align contributor docs with integrated repository layout
267a188
@VamikaSinghal VamikaSinghal changed the title Integrate repository hardening and pairwise evaluation Integrate PRs #5, #6, and #7: hardening and pairwise evaluation Aug 2, 2026
@VamikaSinghal
VamikaSinghal marked this pull request as ready for review August 2, 2026 11:23
@VamikaSinghal
VamikaSinghal merged commit 81ad843 into main Aug 2, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@VamikaSinghal @roshantaneja