[Enhancement] LLM observability: per-request token, cost, and latency telemetry with tracing

[zeemscript]

Summary

You cannot improve what you cannot see. The maintainer says the AI "isn't good," but the service emits no per-request signal about why it is slow, expensive, or failing: no token counts, no cost, no latency breakdown, no trace of what the model actually did. Every answer-quality issue in this campaign — sampling (#ai-18), RAG (#24), tafsir (#17), caching (#27) — adds LLM calls whose cost and latency will silently balloon with nothing watching. This issue instruments the LLM layer: capture per-request input/output token usage, estimated cost, and end-to-end + model latency, tie it to a request/trace ID, and expose it so regressions and runaway spend are visible. This is the ops backbone that keeps the quality push from bankrupting the Gemini quota.

Current state

• The /chat handler logs three human strings — the truncated prompt, "Sending message to chat...", and "Chat response generated successfully" (main.py, lines 121, 142, 174) — via logging.basicConfig(level=logging.INFO) (line 14). No token counts, no timings, no cost, no correlation ID.
• The Gemini response object carries usage_metadata (prompt/candidates token counts), but the code only reads response.text (lines 153, 176) and throws the usage data away.
• There is no timing anywhere; the send_message call (lines 143–151) is not even wrapped in a timer, and (per [Bug] /chat blocks the event loop and turns every Gemini failure into a leaky 500 #7) it blocks the event loop, so latency is currently invisible and bad.
• stellar.py similarly logs only a truncated public key (line 127). There is no metrics endpoint and nothing aggregates anything.
• [Enhancement] Structured JSON logging with request IDs and prompt redaction #11 (structured JSON logging + request IDs + prompt redaction) defines the log format and correlation IDs; it does not capture token/cost/latency telemetry or expose metrics — this issue supplies the LLM-specific measurements that ride on [Enhancement] Structured JSON logging with request IDs and prompt redaction #11's request-ID plumbing.

What to build

1. Per-request LLM telemetry — wrap every model call to record: input tokens, output tokens, total tokens (from usage_metadata), model name, estimated cost (a configurable per-model price table), model latency, and end-to-end handler latency. Attach the request/trace ID from [Enhancement] Structured JSON logging with request IDs and prompt redaction #11 so telemetry joins logs. Build on the async fix ([Bug] /chat blocks the event loop and turns every Gemini failure into a leaky 500 #7) so timing is meaningful.
2. A metrics surface — expose aggregates (request count, token/cost totals and per-request percentiles, p50/p95 latency, error rate, cache hit rate if [Enhancement] Semantic response cache with embedding-similarity matching for repeated questions #27 lands) via a metrics endpoint (Prometheus-style text or a JSON summary) or structured metric log lines a collector can scrape. Keep it lightweight — this is a small Render service (render.yaml).
3. Tracing spans — emit spans for the meaningful stages of a request (intent → retrieval → generation → verification → post-processing) so a slow request can be attributed to a stage. OpenTelemetry-compatible if the dep footprint is justified; otherwise a minimal internal span logger. Don't require a heavy external backend to be useful.
4. Cost guardrails hooks — expose the per-request token/cost numbers to the response metadata (optional block) and to any budget/rate-limit logic ([Enhancement] Add service API-key auth and rate limiting to protect the Gemini quota #9, [Enhancement] Cap conversation history with a token budget to prevent unbounded prompt growth #13) so they can act on real usage rather than guesses. Coordinate with [Enhancement] Cap conversation history with a token budget to prevent unbounded prompt growth #13's token budget rather than re-counting tokens twice.
5. Redaction-aware — telemetry must not leak prompt/answer content into metrics labels or high-cardinality fields; respect [Enhancement] Structured JSON logging with request IDs and prompt redaction #11's redaction rules.
6. Tests — offline tests with a mocked client returning known usage_metadata, asserting token/cost/latency are captured correctly, the cost table math is right, spans are emitted per stage, and no content leaks into metric labels. No live API calls in CI.

Acceptance criteria

• Every model call records input/output/total tokens, model name, estimated cost, and model + end-to-end latency, correlated by the [Enhancement] Structured JSON logging with request IDs and prompt redaction #11 request/trace ID.
• A lightweight metrics surface exposes counts, token/cost totals and percentiles, latency p50/p95, and error rate (plus cache hit rate if [Enhancement] Semantic response cache with embedding-similarity matching for repeated questions #27 lands).
• Per-request stages emit spans so a slow request is attributable to a stage; no heavy external backend is required to read them.
• Token/cost numbers are available to budget/rate-limit logic ([Enhancement] Add service API-key auth and rate limiting to protect the Gemini quota #9/[Enhancement] Cap conversation history with a token budget to prevent unbounded prompt growth #13) and optionally to response metadata, reusing [Enhancement] Cap conversation history with a token budget to prevent unbounded prompt growth #13's token counting rather than duplicating it.
• No prompt/answer content leaks into metric labels or high-cardinality fields (respects [Enhancement] Structured JSON logging with request IDs and prompt redaction #11 redaction).
• Tests with a mocked client verify capture, cost math, spans, and no-leak offline; CI stays green.

Pointers

• main.py — logging setup (lines 13–15), the three log lines (121, 142, 174), the send_message call and discarded response metadata (lines 143–176).
• stellar.py — logging (line 127) if telemetry is extended to non-LLM endpoints.
• render.yaml / requirements.txt — keep added deps light; justify OpenTelemetry if used.
• Adjacent issues to compose with, not duplicate: [Enhancement] Structured JSON logging with request IDs and prompt redaction #11 (log format + request IDs + redaction — ride on it, don't reimplement), [Bug] /chat blocks the event loop and turns every Gemini failure into a leaky 500 #7 (async fix — needed for real latency), [Enhancement] Cap conversation history with a token budget to prevent unbounded prompt growth #13 (token budget — share token counting), [Enhancement] Add service API-key auth and rate limiting to protect the Gemini quota #9 (rate limiting — consume cost signal), [Enhancement] Semantic response cache with embedding-similarity matching for repeated questions #27 (cache hit-rate metric). This issue owns LLM token/cost/latency telemetry and tracing specifically.
• CI (.github/workflows/ci.yml) has no pytest step — add one; extend flake8 to new modules.
• PRs target the dev branch (see CONTRIBUTING.md).

Difficulty

Medium — the measurements themselves are tractable (the SDK already returns usage metadata), but doing it cleanly means threading a correlation ID through the request, building on the async refactor for honest latency, keeping the metrics surface light on a tiny host, and carefully not duplicating token counting (#13) or logging (#11) that neighboring issues own.

---

🏆 GrantFox OSS — Official Campaign | FWC26. Apply for this issue through the GrantFox campaign page. The maintainer assigns one contributor before work starts; unassigned PRs may not be reviewed. PRs target the dev branch. Quality bar: CI must stay green.

💬 Questions or need help? Reach the maintainers and other contributors on the DeenBridge Telegram: https://t.me/+nst9lXNj1wc4ZDE0

[mzterwalexzyy]

I would love to fix this issue, please assign

[mr-abims]

Hello maintainer, i will like to work on this, i have an extensive experience building AI systems and help improve the stucture of the codebase also

[grantfox-oss]

🦊 GrantFox — @mzterwalexzyy has been assigned to this issue as part of the Official Campaign | FWC26 campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #61)
2. Your PR will be reviewed by the Deen-Bridge maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Official Campaign | FWC26 campaign!

@mzterwalexzyy's PR #69 was approved and merged by @zeemscript.

🏆 @mzterwalexzyy: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (156 total points). Track your full progress on GrantFox.

👏 Great work, @mzterwalexzyy! Keep contributing to Deen-Bridge.