feat(pkg)!: v0.4.0 — installable package, new backends, scenarios, API server

.github/workflows/ci.yml

@@ -8,10 +8,17 @@ on:
 
 jobs:
   test:
-    runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11"]
+        os: [ubuntu-latest]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+        include:
+          - os: windows-latest
+            python-version: "3.12"
+          - os: macos-latest
+            python-version: "3.12"
+    runs-on: ${{ matrix.os }}
 
     steps:
       - uses: actions/checkout@v4
@@ -20,25 +27,42 @@ jobs:
         uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
+          cache: pip
 
-      - name: Cache pip
+      - name: Cache embedding model
         uses: actions/cache@v4
         with:
-          path: ~/.cache/pip
-          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
-          restore-keys: ${{ runner.os }}-pip-
+          path: ~/.cache/huggingface
+          key: hf-all-MiniLM-L6-v2-${{ runner.os }}
 
-      - name: Install dependencies
-        run: pip install --upgrade pip && pip install -r requirements.txt
+      - name: Install package
+        run: pip install -e ".[server,dev]"
 
       - name: Import smoke test
         env:
           TRANSFORMERS_NO_TF: "1"
           USE_TF: "0"
         run: python tests/test_imports.py
 
-      - name: Pipeline integration tests
+      - name: Test suite
         env:
           TRANSFORMERS_NO_TF: "1"
           USE_TF: "0"
-        run: python tests/test_pipeline.py
+        run: pytest tests -q
+
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Build sdist and wheel
+        run: |
+          pip install build twine
+          python -m build
+          twine check dist/*
+      - name: Install from wheel and run CLI
+        run: |
+          pip install dist/*.whl
+          memorylens --help

.gitignore

@@ -22,3 +22,8 @@ results.json
 experiment_logs/memorylens.db
 *.log
 .streamlit/secrets.toml
+benchmark_v04.json
+demo_venv/
+benchmark_v04_200.json
+*.db
+experiment_logs/runs_summary*.csv

CHANGELOG.md

@@ -5,19 +5,58 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ---
 
-## [Unreleased]
+## [0.4.0] — 2026-07-04
 
 ### Added
-
-- SQLite persistent storage (`utils/storage.py`) — queryable database replacing flat JSON/CSV logs
-- Migration script (`utils/migrate_legacy_logs.py`) — one-shot import of existing JSON logs into SQLite
-- `Storage.compare_runs()` — cross-run recall comparison API
-- `log_run()` now writes to SQLite alongside existing JSON/CSV output (backward compatible)
-- `list_runs()` queries SQLite first, falls back to filesystem scan
+- **PyPI-ready packaging** — all code now lives under a single `memorylens` package
+  (`memorylens.memory`, `memorylens.simulator`, `memorylens.evaluation`, `memorylens.utils`).
+  Fixed an invalid `build-backend` that made the sdist unbuildable, added the
+  `memorylens` console command (`memorylens.cli:main`), and split dependencies into a
+  lean core plus `[dashboard]`, `[server]`, `[faiss]`, `[groq]`, `[openai]`,
+  `[anthropic]`, `[all]`, `[dev]` extras. `python main.py` still works.
+- **GraphMemory** (`graph`) — NetworkX knowledge-graph backend; fact updates replace
+  edges in place so stale values cannot survive (#20)
+- **FAISSMemory** (`faiss`) — FAISS `IndexFlatIP` vector backend as an optional extra (#14)
+- **contradiction_score** — flags contexts that surface both the old and new value of an
+  updated fact; wired into checkpoints, CSV logs, and the dashboard (#21)
+- **Scenario framework** — `Scenario` dataclass + registry (#22) with three domain
+  scenarios: `edtech` (#18), `support` (#23), `medical` (#24); `--scenario` and
+  `--list-scenarios` CLI flags
+- **FastAPI server** — `uvicorn memorylens.api:app`; job-based POST `/v1/benchmarks`,
+  GET `/v1/backends`, `/v1/scenarios`, `/health` (#25)
+- **Dashboard Run History tab** — overlay Recall@T curves from past `experiment_logs/`
+  runs and compare final metrics side-by-side (#27)
+- **SQLite persistent storage** (`memorylens/utils/storage.py`) — queryable database
+  alongside the JSON/CSV logs; `log_run()` writes to it, `list_runs()` queries it
+  first, `Storage.compare_runs()` compares recall across runs, and
+  `python -m memorylens.utils.migrate_legacy_logs` imports legacy JSON logs
+  (#26, contributed by @Sugaria0427)
+- 23 new integration tests (45 total): GraphMemory, contradiction_score, scenarios,
+  FAISS, API lifecycle, SQLite storage, and cascading regressions (#31)
+- CI matrix expanded to Python 3.10–3.13 on Linux plus Windows and macOS, with a
+  package build + `twine check` + wheel-install job (#30)
 
 ### Fixed
+- **Cascading cold-tier recall regression** — the newest-first cold-summary merge
+  introduced with the drift fix truncated away the oldest fact summaries, collapsing
+  cascading recall at T=100 from ~75% to ~8%. Merging is oldest-first again (stale
+  values are already rewritten in place by the update patcher) and empty "No key
+  facts." summaries are no longer appended to the cold tier. Regression tests added.
+- Experiment CSV logger crashed on every run since the `has_llm_eval` flag was added
+  (a bool was indexed as a dict); it now skips non-backend keys and rotates the CSV
+  when the metric schema changes
+- Benchmark results in the README were stale and did not reproduce; all tables are
+  regenerated from the current code
+
+### Removed
+- Dead references to an unpublished research paper (`paper/memorylens_paper.md` never
+  existed in the repository); unverifiable "the only framework" marketing claims;
+  fabricated ₹-cost projections in the dashboard (replaced with a clearly labelled
+  illustrative $-projection)
+
+---
 
-- `_append_csv_summary` now properly filters `has_llm_eval` from display_data (pre-existing bug where `has_llm_eval: True` caused `TypeError` when iterating display_data)
+## [0.3.0] — 2026-05-24
 
 ### Documentation — Metric Accuracy Clarifications
 
@@ -53,8 +92,7 @@ Three research-quality gaps identified and documented across README, docs/, and
   - Shows realistic 85–87% recall at T=100 vs ideal RAG's 100% — contrast is the key finding
 - Registered as `rag_chunked` backend in benchmark runner and CLI
 
-**Fix 4 — Research paper**
-- `paper/memorylens_paper.md`: 6-section academic paper with proper citations (Ebbinghaus 1885, MemGPT, RAGAS, Jost 1897, Atkinson & Shiffrin 1968), ablation tables, multi-seed results tables, and related work comparison against RAGAS, TruLens, DeepEval, MemGPT, A-MEM
+**Fix 4 — Research paper** *(never merged into the repository; stale references to it were removed in 0.4.0)*
 
 **Tests**: 10 new tests covering decay functions, ChunkedRAGMemory, stats aggregation, and persona pool structure (24 total, all passing)
 

CITATION.cff

@@ -5,21 +5,22 @@ title: "MemoryLens: A Temporal Decay Benchmark for LLM Memory Architectures"
 abstract: >
   MemoryLens is an open-source evaluation framework for measuring LLM memory decay
   — how AI memory systems forget personal facts across long conversations. It implements
-  five memory architectures (Naive, Ideal RAG, Chunked RAG, Cascading Temporal, SummaryMemory),
-  five evaluation metrics (Recall@T, Precision@K, Temporal Drift, Memory Noise Ratio,
-  Cascade Efficiency), Ebbinghaus-grounded temporal decay with ablation, multi-seed
-  statistical validation across five diverse personas, and a dual evaluation pipeline
-  (content-based + LLM answer+judge) supporting five provider backends.
+  eight memory architectures (Naive, RAG, Chunked RAG, Cascading Temporal, Summary,
+  Entity, Graph, FAISS), six evaluation metrics (Recall@T, Precision@K, Temporal Drift,
+  Contradiction, Memory Noise Ratio, Cascade Efficiency), Ebbinghaus-grounded temporal
+  decay with ablation, forgetting-curve fitting, multi-seed statistical validation,
+  four domain scenarios, and a dual evaluation pipeline (content-based + LLM
+  answer+judge) supporting five provider backends.
 authors:
-  - family-names: Srivastava
+  - family-names: Daftary
     given-names: Neal
     alias: Neal006
     orcid: ""
 repository-code: "https://github.com/Neal006/memorylens"
 url: "https://github.com/Neal006/memorylens"
 license: MIT
-version: 0.3.0
-date-released: "2026-05-22"
+version: 0.4.0
+date-released: "2026-07-04"
 keywords:
   - LLM memory
   - memory decay