Skip to content

feat: Implement episodic memory layer and experiment gating with LLM resolution#341

Open
dishafaujdar wants to merge 5 commits intokarpathy:masterfrom
dishafaujdar:memory
Open

feat: Implement episodic memory layer and experiment gating with LLM resolution#341
dishafaujdar wants to merge 5 commits intokarpathy:masterfrom
dishafaujdar:memory

Conversation

@dishafaujdar
Copy link

Key Changes

  • Persistent Storage (memory.py): Added SQLite read/write tracking, hooking each run to its val_bpb and confidence limits.
  • Entry Gating (should_run_experiment): The agent now actively gates experiment proposals. It blocks exploration on high-confidence REJECT zones while allowing refinement in ACCEPT regions or low-confidence zones.
  • Geometric Retrieval: Standardized hyperparameters via Z-score scaling (compute_stats/normalize) to prevent variables like batch_size=64 and lr=0.001 from distorting distances. Added cosine_similarity to perform precise nearest-neighbor threshold checks.
  • Orchestrating Write-Backs (record_verdict): Centralizes the logic to update past experiment confidence bounds dynamically upon new evaluation completion.
  • LLM Conflict Resolution (resolve_with_llm): Built a gpt-4o wrapper to act as an explicit judge when highly similar sets yield contradicting verdicts, effectively determining the new overarching ground truth based on their underlying val_bpb scores.
  • Test Coverage (test.py): Added an autonomous :memory: DB test validating the three-case gating logic, metric bounding, and the API fallback states.
  • Documentation (memory.md): Wrote a full architectural markdown briefing outlining the workflow, geometry mapping, and quick start setup.

How to Test

  1. pip install pydantic requests
  2. Run the memory layer simulation:
python test.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant