[Docs]: Session Persistence Architecture Undocumented — Three Storage Layers with No Described Contract

[hariom888]

Summary

Description

Session state in rag-service/main.py is distributed across three distinct storage layers written by different code paths on different schedules:

Layer	File	Written by	Frequency
Global snapshot	data/sessions.json	cleanup_expired_sessions	On eviction/cleanup
Per-session metadata	data/<session_id>/session_meta.json	_flush_dirty_sessions background thread	Every SESSION_FLUSH_INTERVAL_SECONDS
Session registry	data/session_registry.json	persist_session_registry_entry	On upload + TTL update

load_sessions() reads all three layers and applies them in a specific order: sessions.json first, then session_meta.json as an authoritative overlay (chat + flashcards), with last_accessed used as a tiebreaker. None of this is documented anywhere — not in README.md, not in CONTRIBUTING.md, not in docstrings on the functions themselves.

A contributor who adds a new session field (e.g. a notes field) has no documented guidance on:

• Which layer to write it to.
• Whether to add it to _snapshot_session_for_persistence, save_sessions_unlocked, or persist_session_registry_entry.
• What happens to the field on server restart if the flush interval hasn't fired.
• How the field behaves during session recovery via _recover_session_unlocked.

This has already produced real inconsistencies: flashcards are stored in session_meta.json but not in session_registry.json; processing_progress is stored in the in-memory sessions dict but not flushed to any file; retrieval_cache is explicitly zeroed on persist but this decision is undocumented.

Affected Files

• README.md — no architecture section
• CONTRIBUTING.md — no data-layer guidance for contributors
• rag-service/main.py — load_sessions, save_sessions_unlocked, _flush_dirty_sessions, _snapshot_session_for_persistence, persist_session_registry_entry, _recover_session_unlocked

Steps to reproduce

N/A

Expected behavior

N/A

Actual behavior

N/A

Additional context

No response

[hariom888]

Hi — requesting GSSoC'26 assignment.

Technical understanding: The three-layer persistence model exists for good operational reasons — sessions.json is a cold-start bootstrap, per-session session_meta.json reduces write amplification, and session_registry.json enables fast expiry sweeps without loading all session data. But without a written contract, every contributor who touches session state risks introducing a field that silently disappears on restart (like processing_progress) or that is inconsistently available across recovery paths. The last_accessed tiebreaker in load_sessions is particularly fragile and undocumented.

Affected modules/files:

• README.md and CONTRIBUTING.md (primary targets)
• rag-service/main.py — docstrings on load_sessions, _flush_dirty_sessions, _snapshot_session_for_persistence

Proposed documentation additions:

1. README.md or a new docs/session-persistence.md: A diagram or table describing the three layers, their authority hierarchy, write triggers, and the recovery merge order.
2. CONTRIBUTING.md: A "Adding a new session field" checklist: (a) add to _snapshot_session_for_persistence for per-session persistence, (b) add to save_sessions_unlocked only if needed for cold-start bootstrap, (c) confirm field is re-hydrated in _recover_session_unlocked, (d) note whether the field should be in the registry.
3. Inline docstrings on _snapshot_session_for_persistence and _recover_session_unlocked explaining which fields each is responsible for and why retrieval_cache is explicitly excluded.

Regression/unit test considerations: No code changes — documentation only. Consider adding a test that snapshots a session, simulates a restart by calling load_sessions() from the written files, and asserts every documented persistent field is present and correct in the recovered session.

[garima-agarwall]

Hii @FireFistisDead
I would like to work on this issue.
Please assign this issue to me if possible.