feat(pkg)!: v0.4.0 — installable package, new backends, scenarios, API server

[Neal006]

Summary

Release-sized PR that makes MemoryLens pip-installable and closes every remaining v0.4 issue except #28 (HuggingFace, intentionally skipped).

Packaging (#30)

• The package was unpublishable before this PR: build-backend = "setuptools.backends.legacy:build" is not a valid backend (sdist builds failed), and the wheel shipped four generic top-level packages (memory, utils, evaluation, simulator) that would collide with other installs. Everything now lives under a single memorylens package.
• pip install memorylens → memorylens CLI (memorylens.cli:main). python main.py still works.
• Lean core deps (numpy, sentence-transformers, python-dotenv, networkx) + [dashboard] [server] [faiss] [groq] [openai] [anthropic] [all] [dev] extras.
• CI: Python 3.10–3.13 on Linux, plus Windows and macOS, plus a build job (python -m build, twine check, wheel-install smoke test). Verified locally: wheel installs in a fresh venv, entry point runs.

Features

• GraphMemory (feat: GraphMemory — knowledge-graph backend using NetworkX #20) — NetworkX knowledge graph, in-place fact updates
• FAISSMemory (Qdrant / FAISS vector DB backend: production-grade RAG for LLM memory decay benchmark #14) — FAISS vector index as optional extra with a clear missing-dep error
• contradiction_score (feat: contradiction_score — detect when context surfaces both old and new fact values #21) — detects old+new fact values co-occurring in retrieved context
• Scenario framework (feat: BaseScenario — abstract interface for all domain benchmark scenarios #22) + support (feat: CustomerSupportScenario — 8-fact support-ticket domain benchmark #23) and medical (feat: MedicalScenario — 8-fact patient-consultation domain benchmark #24) scenarios; --scenario/--list-scenarios
• FastAPI server (feat: FastAPI server — async REST API exposing the MemoryLens benchmark pipeline #25) — job-based POST /v1/benchmarks, poll for results
• Dashboard Run History tab (feat: History tab — compare past benchmark runs side-by-side in the dashboard #27) — compare past runs side-by-side

Fixes

• Cascading cold-tier recall regression: the newest-first cold-summary merge introduced with the drift fix truncated away the oldest facts — cascading recall at T=100 had collapsed from ~75% to ~8% (bisected against 3a654df). Reverted to oldest-first merging (stale values are already patched in place) and stopped appending empty summaries. Regression tests added; recall at T=200 is now 100% with drift still 0.0.
• CSV experiment logger crashed on every run since has_llm_eval was added (bool indexed as dict).

Docs (#29)

• README rewritten with reproduced numbers (100-turn and 200-turn multi-seed runs on this exact code) and explicit caveats where the templated benchmark favours extraction backends.
• Removed: dead paper/memorylens_paper.md links (file never existed), "the only evaluation framework" superlatives, stale results, fabricated ₹-cost projections.

Tests

38 integration tests (16 new), all passing without an API key on Python 3.13/Windows locally.

Line count exceeds the usual 400-line guideline because this is a single atomic package-namespace migration; the restructure touches every import.

Closes #14, #20, #21, #22, #23, #24, #25, #27, #29, #30, #31

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This release-sized PR migrates MemoryLens into an installable memorylens Python package and ships v0.4.0 feature additions (new memory backends, scenario framework, contradiction metric, FastAPI server, dashboard history) alongside expanded CI and integration test coverage.

Changes:

• Package namespace migration to memorylens with a new memorylens CLI entry point, plus PyPI-ready build metadata and optional extras.
• New capabilities: GraphMemory + FAISSMemory backends, scenario registry (default/edtech/support/medical), contradiction metric wired through benchmark/logging/dashboard, and a job-based FastAPI server.
• CI and tests expanded (Py 3.10–3.13 matrix, OS coverage, build/wheel smoke test; many new integration tests).

Reviewed changes

Copilot reviewed 35 out of 52 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utils/init.py	(Unchanged content shown) package init placeholder.
tests/test_pipeline.py	Updates imports for new package namespace; adds integration/regression tests for v0.4 features.
tests/test_imports.py	Updates import smoke test to validate v0.4 registries/exports.
ROADMAP.md	Updates roadmap status to reflect v0.4 shipped scope and v0.5 next items.
requirements.txt	Refreshes dev requirements and adds new deps (networkx/fastapi/uvicorn).
README.md	Major README refresh: install/extras, new backends/scenarios, results tables, API/dashboard usage.
quick_demo.py	Updates imports and refreshes cost projection output formatting.
pyproject.toml	Fixes build backend; sets v0.4.0 metadata; defines extras; configures package discovery and pytest.
memorylens/utils/providers.py	Updates usage docs to reflect new import path.
memorylens/utils/llm.py	Adds Groq-backed LLM helper wrapper used by LLM compression/judging.
memorylens/utils/embeddings.py	Adds local embedding utilities (sentence-transformers wrapper) under package namespace.
memorylens/utils/init.py	Package init placeholder.
memorylens/simulator/scenarios/medical.py	Adds medical scenario definition and filler turns.
memorylens/simulator/scenarios/edtech.py	Wraps edtech constants into a Scenario object; updates imports.
memorylens/simulator/scenarios/customer_support.py	Adds customer-support scenario (named support) and filler turns.
memorylens/simulator/scenarios/base.py	Introduces Scenario dataclass and validation helper.
memorylens/simulator/scenarios/init.py	Adds scenario registry + getters/listing.
memorylens/simulator/personas.py	Adds persona pool under new namespace for multi-seed runs.
memorylens/simulator/facts.py	Adds Fact dataclass + BENCHMARK_FACTS under new namespace.
memorylens/simulator/conversation.py	Adds conversation generator with optional scenario filler turns.
memorylens/simulator/init.py	Package init placeholder.
memorylens/memory/vector_faiss.py	Adds FAISS-backed vector memory backend with optional dependency guard.
memorylens/memory/summary.py	Updates LLM compression import to new namespace.
memorylens/memory/rag.py	Updates embeddings import path to new namespace.
memorylens/memory/rag_chunked.py	Updates embeddings import path to new namespace.
memorylens/memory/naive.py	Adds naive memory backend under new namespace.
memorylens/memory/graph.py	Adds GraphMemory backend using NetworkX.
memorylens/memory/entity.py	Adds EntityMemory backend + shared regex extractor.
memorylens/memory/decay.py	Adds decay registry/functions under new namespace.
memorylens/memory/cascading.py	Fixes cold-tier merge regression behavior and empty-summary appends; updates embeddings import path.
memorylens/memory/base.py	Adds BaseMemory ABC and shared token_count helper.
memorylens/memory/init.py	Package init placeholder.
memorylens/evaluation/stats.py	Adds aggregation + forgetting-curve fitting utilities.
memorylens/evaluation/metrics.py	Updates imports; adds contradiction_score metric; updates provider helper imports.
memorylens/evaluation/logger.py	Updates CSV logging to handle new metric column and rotate schema mismatches.
memorylens/evaluation/llm_judge.py	Updates imports to new namespace for judging pipeline.
memorylens/evaluation/benchmark.py	Registers new backends; wires contradiction into checkpoints and display dict; updates imports.
memorylens/evaluation/init.py	Package init placeholder.
memorylens/cli.py	New primary CLI implementation (used by console script).
memorylens/api.py	Adds FastAPI server with async job execution via threads and status polling endpoints.
memorylens/init.py	Exposes version and top-level benchmark API + registries.
main.py	Backward-compatible shim delegating to memorylens.cli.
docs/why-memory-evaluation-matters.md	Doc wording updates for accuracy/clarity.
docs/comparison-with-existing-tools.md	Updates comparison table to reflect 8 backends.
docs/adding-a-new-backend.md	Updates paths/examples to new package namespace; documents shipped backends.
demo_results.json	Refreshes demo payload schema/metrics for v0.4 outputs.
dashboard.py	Adds new backends/colors, contradiction KPI, and run history comparison tab; updates imports.
CONTRIBUTING.md	Updates install/test commands and repository structure for v0.4 package layout.
CITATION.cff	Updates abstract/version/date for v0.4 release metadata.
CHANGELOG.md	Adds v0.4.0 changelog entry and clarifies removed/stale references.
.gitignore	Adds new benchmark output artifacts to ignore.
.github/workflows/ci.yml	Expands CI matrix (3.10–3.13 + OS coverage) and adds build/wheel validation job.

Comments suppressed due to low confidence (2)

memorylens/evaluation/metrics.py:70

• contradiction_score() has the same two failure modes as temporal_drift_score(): updated_at=0 is treated as no update, and new_val becomes "" when updated_value is None, which will always match and can yield false contradictions. Validate updated_at is not None and require updated_value to be present before computing.
  memorylens/evaluation/benchmark.py:185
• drift_facts (defined just above) is filtered with a truthiness check on updated_at, which will incorrectly exclude updates at turn 0. It also doesn’t exclude facts with updated_at set but missing updated_value, which can make contradiction/drift calculations misleading.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.