Support Triage Agent

An AI support triage agent, available as a terminal tool and an HTTP API. Retrieves relevant knowledge-base articles with a hybrid BM25 + embedding retrieval pipeline fused via Reciprocal Rank Fusion (RRF), runs a deterministic safety classifier and a retrieval-confidence gate before any generation happens, returns structured (schema-validated) responses, and writes every decision — including the ones that skip generation entirely — to a CSV audit log.

Retrieval quality is backed by numbers, not just a demo: scripts/eval_retrieval.py measures Recall@k and MRR against a plain BM25 baseline on a held-out, paraphrased query set (see Retrieval evaluation).

Built in 24 hours for the HackerRank Orchestrate hackathon; rebuilt here as a clean, tested, evaluated reference implementation.

Why this design

• BM25 implemented from scratch, no rank_bm25 / sklearn. The point is to show the retrieval math, not import it. src/bm25.py is ~90 lines implementing the standard Okapi BM25 scoring formula directly.
• Three retrieval signals fused with RRF, not a single ranker. BM25 over article bodies (lexical overlap), BM25 over titles + headings (topical match), and — when GEMINI_API_KEY is set — cosine similarity over Gemini embeddings (semantic match, useful when a ticket is phrased nothing like the source article). RRF combines rank positions, not raw scores, which sidesteps the problem of BM25 scores and cosine similarities living on completely different scales.
• Two independent gates before generation, not one. A safety category can auto-escalate a ticket (deterministic regex rules, see src/safety.py). Separately, if the top RRF score falls below a confidence threshold, the pipeline escalates rather than letting the LLM answer from a weak or irrelevant match — that decision costs nothing (no API call), and is itself deterministic and auditable.
• Structured output, not free text. Generation is constrained to a JSON schema (answer, cited_doc_id, confidence) via Gemini's responseSchema, so downstream code — the CSV log, the API response — never has to parse or guess at a model's prose.

Architecture

ticket text
     │
     ├──► safety.classify()          deterministic regex rules
     │         │
     │         ├─ CRITICAL/HIGH + auto_escalate → skip generation, escalate
     │         └─ NORMAL → continue
     │
     ├──► HybridRetriever.retrieve()
     │         │
     │         ├─ BM25 over article bodies      ─┐
     │         ├─ BM25 over titles + headings   ─┼──► RRF fusion → ranked docs
     │         └─ Gemini embeddings (optional)  ─┘
     │
     ├──► confidence gate: top RRF score < threshold? → skip generation, escalate
     │
     ├──► llm.generate_response()     grounded, structured JSON output
     │         │                       cached on (ticket_text, retrieved doc set)
     │
     └──► pipeline.write_csv()        full audit row: safety, confidence,
                                       tokens, latency, cache hits, response

Reciprocal Rank Fusion

For each candidate document d, and each retrieval signal s in which it appears at 1-indexed rank rank_s(d):

RRF(d) = Σ  1 / (k + rank_s(d))

k = 60 is the standard constant from Cormack, Clarke & Buettcher (2009) — it dampens the influence of a single very-top-ranked outlier and keeps the score stable even when a document is missing from one of the lists.

Safety classification

Rules are checked in priority order (src/safety.py); first match wins:

Category	Severity	Auto-escalates
self_harm_risk	critical	yes
security_incident	critical	yes
legal_threat	high	yes
payment_fraud	high	yes
service_outage	high	no
billing_question	normal	no
account_access	normal	no
general_inquiry (default)	normal	no

Confidence gating

Separately from safety, TriagePipeline.process() checks the top retrieval result's RRF score against DEFAULT_CONFIDENCE_THRESHOLD (0.02 by default, tunable via --confidence-threshold). Below it, the ticket escalates without an LLM call — see test_low_confidence_ticket_skips_llm_call in tests/test_pipeline.py for the exact behavior.

Project layout

src/
  bm25.py         pure-Python BM25 (Okapi) index and scorer
  corpus.py       loads markdown KB into body / title+headings views
  embeddings.py   optional Gemini embedding signal, disk-cached
  retrieval.py    HybridRetriever — BM25 x2 + embeddings, fused via RRF
  safety.py       deterministic rule-based safety classifier
  llm.py          structured, cached, latency/token-tracked generation
  pipeline.py     orchestration, confidence gating, CSV audit logging
  cli.py          terminal entrypoint (single / batch / interactive)
  api.py          optional FastAPI wrapper (POST /triage, GET /health)
data/
  knowledge_base/   sample support articles (markdown)
  eval/queries.jsonl  paraphrased query -> expected_doc_id eval set
scripts/
  eval_retrieval.py   Recall@k / MRR: BM25-only baseline vs RRF hybrid
sample_data/tickets.csv   example tickets for batch mode
tests/            pytest unit tests: BM25, RRF, safety, pipeline, eval math
Dockerfile        containerized API service

Running it

Requires Python 3.10+. The CLI has zero required runtime dependencies — BM25, RRF, and the Gemini API calls are all standard library only.

git clone https://github.com/epsilon003/support-triage-agent.git
cd support-triage-agent

# optional: set this for real LLM-generated responses.
# without it, the pipeline still runs end-to-end in mock mode.
# either export it directly, or copy .env.example to .env and fill it in —
# src/cli.py auto-loads .env with no extra dependency.
export GEMINI_API_KEY=AIza...

# single ticket
python -m src.cli --ticket "I can't log into my account, it's locked"

# batch mode — --delay adds a pause between calls to stay under free-tier rate limits
python -m src.cli --batch sample_data/tickets.csv --delay 4

# interactive REPL
python -m src.cli --interactive

# tune the confidence gate (lower = more tickets get an LLM-generated answer)
python -m src.cli --ticket "..." --confidence-threshold 0.015

Every run appends to output/triage_log.csv:

column	meaning
ticket_id / timestamp	short unique id, UTC ISO 8601
safety_category / safety_severity / matched_rule	which rule fired, if any
auto_escalated	whether generation was skipped for safety
retrieved_doc_ids / top_doc_title / top_rrf_score	retrieval trace
low_confidence	whether generation was skipped for weak retrieval
cited_doc_id / llm_confidence	from the model's structured output
prompt_tokens / response_tokens	from Gemini's usageMetadata
cache_hit	served from the response cache, no API call made
retrieval_latency_ms / generation_latency_ms	per-stage timing
response	final response text (or escalation message)

Running the API

pip install -r requirements.txt
uvicorn src.api:app --reload
# docs at http://127.0.0.1:8000/docs

curl -X POST http://127.0.0.1:8000/triage \
  -H "Content-Type: application/json" \
  -d '{"ticket_text": "I forgot my password"}'

Or with Docker:

docker build -t support-triage-agent .
docker run -p 8000:8000 -e GEMINI_API_KEY=AIza... support-triage-agent

Retrieval evaluation

python scripts/eval_retrieval.py

Compares the RRF hybrid retriever against a plain single-signal BM25 baseline on 20 held-out queries that are deliberately paraphrased — worded differently from the source articles, so lexical overlap alone can't trivially win. Sample output (body+heading signals only; add GEMINI_API_KEY to also include the embedding signal):

Retriever                          Recall@3        MRR
-------------------------------- ---------- ----------
BM25 (body only)                      0.900      0.793
RRF hybrid (body+heading)             0.950      0.846

Δ Recall@3: +0.050   Δ MRR: +0.053

--k controls the Recall@k cutoff (default 3).

Testing

pip install -r requirements.txt   # includes pytest
pip install mypy                  # optional, for type checking
pytest tests/ -v
mypy src/ scripts/ --ignore-missing-imports

22 tests cover BM25 ranking correctness, the RRF fusion formula against a manual calculation, every safety rule branch, confidence-gating behavior, and the eval harness's Recall@k / MRR math — all offline, no API key required. CI additionally boots the FastAPI app and runs the eval script against the live corpus.

What's intentionally out of scope

This is a reference implementation, not a production system:

• Safety rules are a coarse first-pass filter (a handful of regex patterns), not comprehensive coverage — the point is to demonstrate a deterministic-gate-before-generation pattern.
• No persistence layer beyond the CSV log and the two on-disk caches; no auth, no multi-tenant KB, no rate limiting on the API itself.
• The knowledge base is 8 sample articles; corpus.py will index any directory of markdown files.