Logs Analyzer

AI agent that investigates production issues by querying Grafana/Loki logs. Describe a problem in natural language — the agent clarifies your intent, searches logs iteratively, and produces structured findings.

How It Works

1. Enter a query describing the problem (e.g., "why did payment ORD-12345 fail yesterday")
2. The agent may ask clarifying questions about time range, entities, or context
3. It autonomously builds and refines LogQL queries against your Grafana/Loki instance
4. Findings are printed to stdout and saved as JSON to completed_investigations/

Setup

Prerequisites

• Go 1.25+
• Grafana instance with Loki datasource
• LLM API key (any Anthropic-compatible endpoint)

Installation

go build -o agent ./cmd/agent/

Configuration

All settings are environment variables with prefix LOGS_ANALYZER_. Create a dev.env file at project root — it is loaded automatically.

cp env.example dev.env

See env.example for all available variables with descriptions.

What You Need to Fill In

1. LLM API

The agent uses an Anthropic-compatible API endpoint. Point it at your LiteLLM proxy — create a separate API key in LiteLLM to track usage independently from other tools (e.g., Claude Code):

• LOGS_ANALYZER_LLM_BASE_URL — LiteLLM proxy URL
• LOGS_ANALYZER_LLM_API_KEY — dedicated key created for this agent

Model names should match what your LiteLLM instance exposes (e.g., claude-sonnet-4-6, claude-haiku-4-5).

2. Grafana Session Cookie

The agent authenticates to Grafana via browser session cookies:

1. Log in to Grafana in your browser
2. DevTools → Application → Cookies
3. Copy grafana_session and grafana_session_expiry values
4. Save to grafana-session.txt (see grafana-session.example.txt for format):

grafana_session=abc123...; grafana_session_expiry=xyz789...

Sessions are short-lived — refresh manually when expired.

3. Grafana Datasource Identifiers

Two IDs for your Loki datasource:

• LOGS_ANALYZER_GRAFANA_DATASOURCE_UID — Grafana → Connections → Data Sources → your Loki. UID is in the URL: .../datasources/edit/<UID>
• LOGS_ANALYZER_GRAFANA_DATASOURCE_NUMERIC_ID — Open DevTools → Network, run any query in Explore, find request to /api/ds/query — numeric ID is in the payload (datasourceId field)

4. Log Stream Selector

LOGS_ANALYZER_SEARCH_LOG_STREAM_SELECTOR is the base LogQL stream selector prepended to every query. Scopes all searches to your target services:

{namespace="production", service_name=~"my-service-.*"}

Use the same selector you'd use in Grafana Explore.

Running

./agent

Type your query and press Enter twice to submit. The agent handles the rest.

Completed Investigations

Every investigation is saved as a JSON file in completed_investigations/:

completed_investigations/investigation_2026-04-29_09-52-49.json

Each file contains:

Field	Description
Findings	List of discoveries, each with description, evidence, and confidence (0–1)
StepSummaries	One-sentence summary of what each search step found
StepCount	Total number of search iterations used
HitStepLimit	Whether the agent exhausted max steps before finishing
History	Full LLM message history (useful for debugging agent behavior)

Example finding

{
  "description": "No merchant currency events were received from processing in the last 24 hours",
  "evidence": "Three search strategies all returned zero results for ProcessingMerchantCurrencyEventConsumer over 2026-04-28 to 2026-04-29",
  "confidence": 0.95
}

Business Knowledge

Edit prompts/business_knowledge.md to teach the agent about your system — entity definitions, identifier formats, service names, and log structure. This knowledge is injected into both parsing and search stages.

Testing

# Unit tests
go test -short ./internal/...

# Grafana integration tests (requires valid session)
go test -v ./tests/grafana/...