You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Apr 29, 2025. It is now read-only.
Caching is awesome to save on tokens, but there are few scenarios where someone might want to opt out of it.
E.g. allowing to re-generate a response if the user doesn't like it. In this case the prompt is the same as a previous request (so the response would be taken from the cache) but we actually want to obtain a fresh new result.
I would then suggest to allow to opt out of caching on a request basis. Maybe using a specific header for this?
Caching is awesome to save on tokens, but there are few scenarios where someone might want to opt out of it.
E.g. allowing to re-generate a response if the user doesn't like it. In this case the prompt is the same as a previous request (so the response would be taken from the cache) but we actually want to obtain a fresh new result.
I would then suggest to allow to opt out of caching on a request basis. Maybe using a specific header for this?