Productionize pairwise digital-twin evaluation - #7
Merged
Conversation
…win-eval branch Reconciles the two diverged checkouts so the README rewrite has a single base. No file overlap between the branches; merge is clean.
Corrects the build flow, architecture description, and capability claims against the code, CI, and shipped Info.plist. Build/test: - pip install backend/requirements.txt + pytest (runtime-requirements.txt has neither FastAPI nor pytest, so the documented flow could not work) - Python 3.12, not 3.11+ (3.9 fails collection with 16 errors; 3.12 collects 2408 cleanly) - add dev_backend.sh, standalone_server, adaptation_eval; point at ci.yml Architecture: - the bundled engine is stdlib http.server (backend/app/standalone_server.py), not FastAPI; FastAPI is the hosted plane in backend/app/main.py - relabel the Engine badge accordingly Operating model, corrected against source: - consolidation is never auto-scheduled in the app and is maintenance-gated off, so drop "runs while you are away" - personalized-PageRank and multi-hop are opt-in, not the default Ask path - the knapsack engages only when a model is named - the delta channel never re-announces; the pack still carries content - intent is a fallback retriever, not an always-on fused signal - packs are sha256-addressed when pinned - fix the seven layer names to match MEMORY_LAYERS Removed unverifiable claims: "No account needed" (Info.plist ships CortexRequireAccount=true), "nothing is remembered until you approve" (auto_approve defaults true for user imports), "any MCP client", the LLM condenser, and automatic update installation. Added: current limitations, the four undocumented shipped surfaces (extension, Obsidian plugin, OpenClaw plugin, SDKs), and a repo layout that labels the root-level prototype as legacy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Why
Pairwise evaluation gives Cortex a more stable and auditable way to compare whether candidate responses match a digital twin than independent scalar scoring.
Verification
main; this draft PR does not merge or deploy anything