Merge pull request #4 from softmaxdata/fix/core-correctness-and-mcp

Fix core correctness, storage atomicity, and MCP setup

Author: weirdindiankid

.github/workflows/ci.yml

@@ -0,0 +1,84 @@
+name: Regression and package checks
+
+on:
+  push:
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+env:
+  PIP_DISABLE_PIP_VERSION_CHECK: "1"
+  LITELLM_LOCAL_MODEL_COST_MAP: "True"
+  DO_NOT_TRACK: "1"
+
+jobs:
+  sqlite-and-package:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+      - name: Install test and build dependencies, including the MCP extra
+        run: python -m pip install -e '.[dev,mcp]' build
+      - name: Check recorded shared-core contracts
+        run: python scripts/check-shared-core.py --checkout .
+      - name: Run full SQLite suite
+        run: python scripts/check-ci.py --backend sqlite --output reports/sqlite.xml
+      - name: Run controlled retrieval regression benchmark
+        run: python -m evals.retrieval --output reports/retrieval.json
+      - name: Build sdist, then wheel; certify installed package and MCP
+        run: python scripts/check-package.py --output-dir reports/package
+      - name: Retain verification reports and package artifacts
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: sqlite-package-python-${{ matrix.python-version }}
+          path: reports/
+          retention-days: 14
+
+  postgres:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.13"]
+    services:
+      postgres:
+        image: pgvector/pgvector:pg16
+        env:
+          POSTGRES_DB: engram_test
+          POSTGRES_USER: engram_test
+          POSTGRES_PASSWORD: engram-test-only
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U engram_test -d engram_test"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 12
+    env:
+      ENGRAM_TEST_POSTGRES_DSN: postgresql://engram_test:engram-test-only@127.0.0.1:5432/engram_test
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+      - run: python -m pip install -e '.[dev,mcp]'
+      - name: Require ready pgvector and run full suite with PostgreSQL tests enabled
+        run: python scripts/check-ci.py --backend postgres --output reports/postgres.xml
+      - name: Retain PostgreSQL verification report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: postgres-python-${{ matrix.python-version }}
+          path: reports/
+          retention-days: 14

README.md

@@ -91,11 +91,9 @@ There are two storage options:
 - SQLite (local) — zero setup, tables auto-created on first run
 - PostgreSQL (local/remote) — requires creating the database first
 
-For PostgreSQL, create the database and enable pgvector before starting the server:
+For PostgreSQL, install pgvector on the database server and create the database first:
 ```sql
 CREATE DATABASE engram;
-\c engram
-CREATE EXTENSION IF NOT EXISTS vector;
 ```
 
 Then set the environment variables:
@@ -104,7 +102,14 @@ ENGRAM_STORAGE_BACKEND=postgres
 ENGRAM_POSTGRES_DSN=postgresql://user:password@host:5432/engram
 ```
 
-All tables are auto-created on first server start for both backends.
+Engram enables the `vector` extension before opening its connection pool and creates
+tables on first startup. The Docker Compose pgvector image already includes the
+extension files, so `docker compose up` works with a fresh volume.
+
+On managed PostgreSQL, an administrator may need to run
+`CREATE EXTENSION IF NOT EXISTS vector` in the Engram database before the first
+startup if the application role cannot install extensions. Existing extensions and
+data are preserved across restarts; do not delete your database volume to upgrade.
 
 ### 1. Configure the Server
 
@@ -149,15 +154,11 @@ ENGRAM_CURATOR_SLOW_PATH_MODEL=claude-haiku-4-5  # For complex conflicts
 # ─── Embedding ──────────────────────────────────────────────
 ENGRAM_EMBEDDING_MODEL=text-embedding-3-small
 
-# ─── Consolidation ──────────────────────────────────────────
-ENGRAM_CONSOLIDATION_TRIGGER=every_10_commits  # or "daily", "manual"
-ENGRAM_FAST_DECAY_RATE=0.97            # Per-day salience decay for normal bullets
-ENGRAM_SLOW_DECAY_RATE=0.995           # Per-day decay for decisions/schemas (slower)
+# Consolidation is invoked through the SDK or REST API.
+# This server does not schedule consolidation from environment variables.
 
-# ─── Auth (disabled for local dev, enable for production) ───
-ENGRAM_AUTH_ENABLED=false
-# COGNITO_USER_POOL_ID=us-west-2_xxx
-# COGNITO_APP_CLIENT_ID=xxx
+# The OSS server has no built-in account authentication.
+# Cloud account/API-key authentication belongs to the hosted edition.
 ```
 
 ### 2. Start the Server
@@ -769,33 +770,59 @@ Capacity management with pressure levels:
 
 ### MCP Server (Claude)
 
+For **Engram Cloud**, add `https://api.engram.so/mcp/sse` in Claude's
+**Settings → Connectors → Add custom connector**, then sign in to Engram and
+authorize access. See [Cloud setup](https://app.engram.so/setup).
+
+For a **local stdio bridge**, install from a checkout containing this fix using
+Python 3.11 or later. The published `engram-contextdb` 0.4.5 wheel lacks the
+module entrypoint; use the corrected source checkout until a fixed release is
+available. The distribution is `engram-contextdb`; imports remain `engram`.
+
 ```bash
-pip install engram[mcp]
-python -m engram.integrations.mcp_server
+cd /absolute/path/to/engram
+python3 -m venv .venv
+.venv/bin/python -m pip install ".[mcp]"
 ```
 
-Configure in Claude Code:
+Start your self-hosted Engram HTTP server separately as described in Quick Start.
+The MCP process connects to that server; it does not start the database API.
+
+Configure Claude Code with the absolute path to the installed interpreter:
+
 ```bash
-claude mcp add engram -- python -m engram.integrations.mcp_server
+claude mcp add --transport stdio \
+  --env ENGRAM_API_URL=http://localhost:5820 \
+  engram -- /absolute/path/to/engram/.venv/bin/python -m engram.integrations.mcp_server
 ```
 
-Or in `claude_desktop_config.json`:
+For Claude Desktop, open **Settings → Developer → Edit Config** and merge the
+`engram` entry into the existing `mcpServers` object. Keep one JSON root object:
+
 ```json
 {
   "mcpServers": {
     "engram": {
-      "command": "python",
+      "command": "/absolute/path/to/engram/.venv/bin/python",
       "args": ["-m", "engram.integrations.mcp_server"],
       "env": {
-        "ENGRAM_API_URL": "http://localhost:5820",
-        "ENGRAM_API_KEY": "eng_sk_..."
+        "ENGRAM_API_URL": "http://localhost:5820"
       }
     }
   }
 }
 ```
 
-Tools: `engram_list_contexts`, `engram_recall`, `engram_commit`, `engram_decide`, `engram_health`, `engram_create_context`
+Replace the executable path with your actual absolute path (on Windows, use the
+venv's `Scripts\\python.exe` and escape backslashes in JSON). Restart Claude
+Desktop after saving. For Cloud access through this bridge, set `ENGRAM_API_URL`
+to `https://api.engram.so` and add `ENGRAM_API_KEY` with your Engram API key to
+`env`. Self-hosted OSS has no account authentication and does not require a key.
+The bridge forwards a configured key as an HTTP Bearer token.
+
+Tools: `engram_list_contexts`, `engram_recall`, `engram_commit`, `engram_decide`,
+`engram_health`, `engram_create_context`; the original unprefixed tools remain
+available. Ask Claude to list your contexts to verify an actual API call.
 
 ### OpenAI Function Calling
 
@@ -880,14 +907,6 @@ All settings via environment variables or `.env` file:
 | `ENGRAM_VALIDITY_GATE_MODEL` | `claude-haiku-4-5` | Model used by the validity gate when enabled |
 | **Embedding** | | |
 | `ENGRAM_EMBEDDING_MODEL` | `text-embedding-3-small` | For bullet similarity search **and** worked-example retrieval over the activity ledger |
-| **Consolidation** | | |
-| `ENGRAM_CONSOLIDATION_TRIGGER` | `every_10_commits` | When to run consolidation |
-| `ENGRAM_FAST_DECAY_RATE` | `0.97` | Per-day salience decay (normal bullets) |
-| `ENGRAM_SLOW_DECAY_RATE` | `0.995` | Per-day decay (decisions/schemas) |
-| **Auth** | | |
-| `ENGRAM_AUTH_ENABLED` | `false` | Enable multi-tenant auth |
-| `COGNITO_USER_POOL_ID` | — | AWS Cognito user pool |
-| `COGNITO_APP_CLIENT_ID` | — | AWS Cognito app client |
 
 ## Development
 
@@ -969,3 +988,4 @@ please email logos@engram.so (company + use case).
 For large-scale deployments (>20,000 external users), we offer paid support
 and an enterprise agreement — contact logos@engram.so.
 
+Upgrade requirements and preserved interfaces are documented in [Compatibility and upgrade notes](docs/backward-compatibility.md).

docs/backward-compatibility.md

@@ -0,0 +1,37 @@
+# Compatibility and upgrade notes
+
+The supported upgrade target is the existing REST, MCP and Python SDK API with
+populated SQLite or PostgreSQL databases. The compatibility audit compares the
+previous Git revision with the corrected source: all 34 existing HTTP operations,
+40 schema definitions and 33 public SDK methods retain their existing fields,
+enum values, documented responses and method signatures. All 353 existing public Python definitions remain present. Database upgrades add
+columns and indexes without replacing account-independent context, memory,
+activity or receipt IDs. No developer or customer database was upgraded during
+validation.
+
+Built-in SQLite and PostgreSQL support explicit transaction-scoped storage.
+Out-of-tree storage implementations must implement `StorageBackend.transaction`
+before adopting these atomic mutation paths; unsupported implementations fail
+clearly. This is an extension-interface requirement. Existing renderers and
+completion adapters without a per-call model argument remain supported. OSS v0.5
+core memory, worked examples, validity checks and reconsolidation remain intact.
+
+Incorrect or unsafe behavior is intentionally corrected: cross-context writes
+are rejected, active capacity is checked against the actual committed state,
+failed mutations cannot leave partial audit/activity records, and unavailable or
+invalid similarity comparisons cannot justify destructive matching. Configured
+reflector models and deduplication thresholds are now honored.
+
+Historical audit rows that lack a generated target ID or required inverse state
+cannot be reliably undone. Rollback refuses those batches before a partial write
+rather than claiming success. An old content update without a saved embedding
+clears the mismatched vector on rollback; it cannot recreate that historical
+embedding. Old receipt rows cannot recover consumption history never recorded.
+See [embedding-backfill.md](embedding-backfill.md) for the optional, explicit,
+bounded repair of missing vectors using the correct historical embedding model.
+
+The distribution is `engram-contextdb`; the Python import and CLI remain `engram`.
+MCP 1.x is the supported dependency range. Installed-package checks build from an
+sdist, install the resulting wheel outside the checkout, run the CLI, and exercise
+the stdio protocol. These checks do not publish a corrected package or validate
+native third-party sign-in. All fixes remain local until released.

docs/embedding-backfill.md

@@ -0,0 +1,38 @@
+# Repair missing embeddings
+
+Use the corrected source checkout to repair existing active memories that have no embedding. This does not repeat extraction, change text, or create new memories. The current database schema uses 1536-dimensional embeddings; choose the same embedding model used for the existing vectors.
+
+Preview one context in an existing SQLite database:
+
+```bash
+.venv/bin/python -m engram.maintenance.backfill_embeddings \
+  --sqlite /absolute/path/to/engram.db \
+  --context-id CONTEXT_UUID --limit 100
+```
+
+For PostgreSQL, put the existing database DSN in an environment variable and pass its **name**:
+
+```bash
+.venv/bin/python -m engram.maintenance.backfill_embeddings \
+  --postgres-dsn-env ENGRAM_MAINTENANCE_DSN \
+  --context-id CONTEXT_UUID --limit 100
+```
+
+Preview opens the existing database read-only. It does not initialize schemas, create the vector extension, change SQLite journal mode, load `.env`, or call an embedding provider. SQLite may create WAL coordination sidecars when opening an existing WAL database; its database content and journal mode remain unchanged. There is no default database target. A missing database or context fails instead of creating one.
+
+To save embeddings, add the explicit apply and provider options to either command:
+
+```bash
+  --apply --embedding-model text-embedding-3-small \
+  --embedding-key-env ENGRAM_EMBEDDING_API_KEY
+```
+
+Set the named provider-key environment variable before running apply. Apply makes provider requests and may incur usage charges. The limit is between 1 and 1000 and bounds attempted memories per invocation; it defaults to 100. The command is an operator tool using database access, not a new unauthenticated HTTP endpoint.
+
+The JSON summary reports eligible, selected, attempted, updated, skipped, and failed counts. Exit status is 0 for a successful preview/apply, 1 for an operational or per-memory failure, and 2 for invalid command options. Errors identify a memory and a reason without printing provider messages, credentials, or memory text.
+
+Each vector is computed outside a database transaction. Before saving, the command rechecks context, content, lifecycle, and whether a vector is still missing. Concurrent edits or an already-completed repair are skipped. Only the embedding column changes; IDs, text, timestamps, metadata, counters, and audit history remain intact. Archived and inactive memories are excluded. Both backends validate the float32 representation used by PostgreSQL, rejecting empty, nonfinite, zero, underflow-to-zero, overflowing, and wrong-dimension vectors. Squared norms must also remain finite and nonzero in float32 so repaired vectors work with cosine search; vector scale is preserved.
+
+Repairs commit one memory at a time. If a later provider request fails, earlier successful repairs remain saved. Re-run the same command to repair remaining items; completed items are skipped. This tool cannot determine the model used for historical vectors, so model compatibility must be selected by the operator.
+
+Validation is isolated: `pytest tests/test_embedding_backfill.py` uses temporary SQLite databases and fake providers. Setting `ENGRAM_TEST_POSTGRES_DSN` to a disposable test admin database also exercises PostgreSQL in newly created random databases. No repair was run against an existing developer or customer database during implementation.