[Feature]: Content-addressed /predict response cache (TTL + LRU) with reload-aware invalidation

[pavsoss]

Problem

/predict re-runs vectorization + inference + enrichment on every call, including for identical repeated inputs (common in bulk/retry traffic). There is no caching, so duplicate work scales linearly with request volume.

Proposed solution

Add an in-process, thread-safe cache keyed by the normalized input and the current serving model version, so identical inputs return instantly and a model reload can never serve stale predictions.

Scope

• A prediction-cache module (backend/predict_cache.py): a bounded TTL + LRU cache (size and TTL from settings), keyed by sha256(normalized_text) (reusing utils/text_normalizer) combined with serving_state.version and any prediction options; thread-safe.
• In /predict: look up before inference and store after; bypass on Cache-Control: no-cache or ?fresh=1; set an X-Cache: HIT|MISS response header; expose hit/miss/size (via a small /cache-stats or folded into /model-info).
• A README + OpenAPI note; regression coverage (hit/miss, TTL expiry, invalidation across a version bump, bypass header).

Why it's safe

Namespacing the key by serving_state.version means a hot-swap (#980) transparently invalidates the cache — no stale model outputs. Success/response shapes are unchanged; X-Cache is purely additive.

Acceptance criteria

• Repeated identical input yields X-Cache: HIT and an identical body.
• A model reload flips subsequent lookups to MISS (no stale results).
• The cache is bounded (evicts by size and TTL) and never grows unbounded.

Out of scope

Distributed/Redis-backed caching; caching non-idempotent endpoints.

[github-actions]

👋 Welcome back! Thanks for your continued contribution. Your issue/PR is received and waiting for maintainer review.

[github-actions]

The text outlines a proposal for implementing a thread-safe, in-process cache for the /predict endpoint to avoid redundant processing for identical inputs. The cache would use TTL and LRU mechanisms, be keyed by normalized input and model version to prevent stale results after model reloads, and support manual bypass via headers. The solution aims to improve efficiency and scalability without changing response formats, adding only a new X-Cache header for cache status and offering cache statistics via an API endpoint. Distributed caching and non-idempotent endpoints are explicitly excluded from the scope.

[pavsoss]

@Userunknown84 Please assign this to me