diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8b79c243..d59098f1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -19,6 +19,15 @@ jobs: - name: Check version sync run: bash scripts/check-version-sync.sh + - name: Check plugin manifest version policy + run: bash scripts/check-plugin-manifest-versions.sh + + - name: Check agent memory symlinks + run: bash scripts/check-agent-memory-symlinks.sh + + - name: Check public identity strings + run: bash scripts/check-public-identity.sh + fmt: name: Formatting runs-on: ubuntu-latest @@ -45,7 +54,7 @@ jobs: - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 - name: Clippy - run: cargo clippy -- -D warnings + run: cargo clippy --all-targets -- -D warnings test: name: Tests diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index bf17bedf..00000000 --- a/AGENTS.md +++ /dev/null @@ -1,313 +0,0 @@ -# CLAUDE.md — cortex - -Rust binary: syslog receiver (UDP/TCP) + MCP server for homelab log intelligence. Receives RFC 3164/5424 syslog from all homelab hosts, stores in SQLite with FTS5, exposes a single `cortex` MCP tool (with action dispatch) for AI agents. - -## Commands - -```bash -cargo build # debug build -cargo build --release # release build -cargo run # run locally (reads config.toml) -cargo test # test suite -cargo clippy # lint (must pass before committing) -cargo fmt # format (enforced by CI) -docker compose up -d # production deployment -docker compose down # stop -docker compose logs -f # follow logs -docker compose build # rebuild image -cortex compose doctor # diagnose live Compose/listener ownership -cortex compose status --json # inspect canonical cortex container/project -cortex compose pull # pull image for resolved Compose project -cortex compose up # run docker compose up -d for resolved service -cortex compose restart # restart resolved service -cortex compose logs --tail 20 # bounded compose logs -cortex db status # inspect SQLite maintenance state -cortex db integrity # run SQLite integrity_check -cortex db backup # create WAL-safe SQLite backup -``` - -```bash -just dev # cargo run alias -just test # cargo test alias -just health # curl /health | jq (server must be running) -just gen-token # openssl rand -hex 32 (generate API token) -just build-plugin # release build → installs binary to bin/ (Linux; requires git lfs install) -just publish [major|minor|patch] # bump version, tag, push (triggers CI) -just generate-cli # build standalone CLI (server must be running) -``` - -## Architecture - -Key modules in `src/` (most are directories with sidecar `*_tests.rs` files): - -| Module | Purpose | -|--------|---------| -| `config.rs` | Config: `config.toml` + env vars (`CORTEX_*`, `CORTEX_*`, `CORTEX_API_*`, `CORTEX_DOCKER_*`) | -| `runtime.rs` | `RuntimeCore`: wires all subsystems, starts syslog ingest, spawns maintenance tasks | -| `app/` | Service layer: `SyslogService`, request/response models, business logic | -| `db/` | SQLite pool, FTS5 queries, maintenance (retention, storage enforcement) | -| `syslog/` | UDP + TCP listeners, RFC 3164/5424 parsing, mpsc batch writer | -| `mcp/` | RMCP Streamable HTTP server, single `cortex` tool with action dispatch | -| `api.rs` | Optional non-MCP REST API (enabled via `CORTEX_API_ENABLED=true`) | -| `docker_ingest/` | Docker container log ingestion via remote docker-socket-proxy endpoints | -| `main.rs` | Entrypoint: `serve mcp` (full server with ingest) or `mcp` (stdio query-only) | - -Tests: unit tests live in sidecar files beside their source modules (e.g. `src/db/queries_tests.rs`). Source files keep only the `#[cfg(test)] #[path = "..._tests.rs"] mod tests;` hook, so sidecar tests compile as module-local unit tests with `use super::*` access to private items. Run with `cargo test`. - -## Ports - -| Port | Protocol | Purpose | -|------|----------|---------| -| 1514 | UDP + TCP | Syslog receiver (not 514 — avoids `CAP_NET_BIND_SERVICE`) | -| 3100 | TCP | Shared HTTP listener for MCP (`POST /mcp`, `GET /health`) and OTLP HTTP ingest (`POST /v1/logs`); non-loopback OAuth-only `/v1/logs` exposure is blocked at startup unless `CORTEX_TOKEN` is set | - -## MCP Tools - -One MCP tool: **`cortex`** — dispatches by `action` argument. - -| Action | Description | -|--------|-------------| -| `search` | Full-text search (FTS5 syntax) with host/severity/app/time filters | -| `tail` | Recent N entries, optionally filtered by host/app | -| `errors` | Error/warning summary grouped by host and severity | -| `hosts` | All known hosts with first/last seen + log counts | -| `correlate` | Cross-host event correlation in a time window | -| `stats` | DB stats (total logs, logical/physical size, free disk, configured thresholds, write-block state, time range) | -| `status` | Lightweight runtime and DB health | -| `apps` | Distinct application names with log and host counts | -| `sessions` | AI transcript sessions grouped by project/tool/session/host | -| `search_sessions` | Full-text search over indexed AI transcript sessions | -| `usage_blocks` | AI transcript activity grouped into time blocks | -| `project_context` | Recent AI transcript context for a project | -| `list_ai_tools` | AI tools present in transcript metadata | -| `list_ai_projects` | AI projects present in transcript metadata | -| `source_ips` | Distinct source identifiers with hostname breakdown | -| `timeline` | Bucketed counts over time | -| `patterns` | Near-duplicate message template clusters | -| `context` | Surrounding logs around a log id or timestamp | -| `get` | One log entry by id, including raw frame | -| `ingest_rate` | Recent ingest throughput and write-block state | -| `silent_hosts` | Hosts whose last_seen is older than a threshold | -| `clock_skew` | Per-host received_at minus timestamp distribution | -| `anomalies` | Recent vs baseline volume/error comparison | -| `compare` | Side-by-side comparison of two time ranges | -| `compose_status` | Redacted Docker Compose runtime status projection | -| `compose_doctor` | Redacted Docker Compose diagnostics projection | -| `help` | Built-in usage reference | - -## Plugin Skills - -Skills available after installing the Claude Code plugin (`plugins/skills/`): - -| Command | Description | -|---------|-------------| -| `/syslog:dr` | Full health check: MCP, HTTP /health, service status, syslog port, Docker ingest, fleet drop-ins (named `dr` to avoid colliding with Claude Code's built-in `/doctor`) | -| `/syslog:deploy-dropins` | Push rsyslog forwarding configs to `fleet_hosts` via SSH (idempotent) | - -## Config - -`config.toml` at repo root for local dev. **Not copied into Docker** — the Dockerfile was cleaned up (no COPY for config.toml). In Docker, defaults + env vars apply exclusively. - -```bash -# Syslog listener -CORTEX_RECEIVER_HOST=0.0.0.0 # host only, no port -CORTEX_RECEIVER_PORT=1514 # shared by UDP + TCP -CORTEX_MAX_MESSAGE_SIZE=8192 -CORTEX_BATCH_SIZE=100 -CORTEX_FLUSH_INTERVAL=500 # ms - -# MCP server -CORTEX_HOST=0.0.0.0 -CORTEX_PORT=3100 -CORTEX_TOKEN=your-secret-token # optional; enables Bearer auth on /mcp - # (CORTEX_API_TOKEN still works, logs deprecation) -CORTEX_ALLOWED_HOSTS=myhost.local # optional; comma-separated extra Host allowlist -CORTEX_ALLOWED_ORIGINS=https://app # optional; comma-separated extra Origin allowlist - -# Storage -CORTEX_DB_PATH=data/cortex.db -CORTEX_POOL_SIZE=4 -CORTEX_RETENTION_DAYS=90 # 0 = keep forever -CORTEX_MAX_DB_SIZE_MB=1024 # 0 = disable logical DB size guard -CORTEX_RECOVERY_DB_SIZE_MB=900 # cleanup target after DB-size breach -CORTEX_MIN_FREE_DISK_MB=512 # 0 = disable free-disk guard -CORTEX_RECOVERY_FREE_DISK_MB=768 # cleanup target after free-disk breach -CORTEX_CLEANUP_INTERVAL_SECS=60 # storage-budget enforcement interval (>= 5) -CORTEX_CLEANUP_CHUNK_SIZE=1000 # rows deleted per enforcement cycle - -# OAuth / JWT auth (disabled by default — set CORTEX_AUTH_MODE=oauth to activate) -CORTEX_AUTH_MODE=bearer # bearer (default) or oauth -CORTEX_PUBLIC_URL=https://syslog.example.com # required when CORTEX_AUTH_MODE=oauth -CORTEX_GOOGLE_CLIENT_ID=... # required when CORTEX_AUTH_MODE=oauth -CORTEX_GOOGLE_CLIENT_SECRET=... # required when CORTEX_AUTH_MODE=oauth -# Paths, TTLs, allowlist → config.toml [mcp.auth] (not env vars). See docs/OAUTH.md. - -# Non-MCP REST API (disabled by default) -CORTEX_API_ENABLED=false # set true to mount /api/* endpoints -CORTEX_API_TOKEN=your-api-token # required when CORTEX_API_ENABLED=true - -# Docker container log ingestion (disabled by default) -CORTEX_DOCKER_INGEST_ENABLED=false # set true to ingest from docker-socket-proxy hosts -CORTEX_DOCKER_HOSTS=host-a,host-b # comma-separated hostnames → http://:2375 -CORTEX_DOCKER_RECONNECT_INITIAL_MS=1000 -CORTEX_DOCKER_RECONNECT_MAX_MS=60000 - -# Log verbosity (set to debug or trace for development) -RUST_LOG=info -``` - -## Key Files - -| File | Purpose | -|------|---------| -| `config.toml` | Runtime config (syslog bind, DB path, retention) | -| `docker-compose.yml` | Production deployment (ports 1514, 3100) | -| `docs/SETUP.md` | Per-host syslog forwarding (rsyslog, UniFi, ATT router, WSL) | -| `src/db/queries.rs` | All SQL queries and FTS5 search implementation | -| `src/mcp/tools.rs` | Single `cortex` tool with action dispatch | -| `config/mcporter.json` | mcporter config (HTTP transport to localhost:3100) | -| `CORTEX_DOCKER_HOSTS` env var | Docker ingest host list — comma-separated hostnames, each becomes `http://:2375` | -| `scripts/smoke-test.sh` | Live smoke test — all MCP actions via mcporter, strict PASS/FAIL | -| `scripts/backup.sh` | WAL-safe SQLite backup script (checkpoint + `.backup` method) | -| `scripts/reset-db.sh` | WAL-safe backup + destructive DB reset helper for local/dev recovery | -| `scripts/bump-version.sh` | Bump version across all version-bearing files; called by `just publish` | -| `cortex db status\|integrity\|checkpoint\|vacuum\|backup` | Direct SQLite maintenance commands for the configured DB | -| `scripts/check-version-sync.sh` | Assert all version-bearing files have the same version (used in CI) | -| `scripts/block-env-commits.sh` | Pre-commit hook that blocks commits containing env credential patterns | -| `CHANGELOG.md` | Version history; entry required per version bump | -| `.lavra/memory/recall.sh` | Query the local knowledge DB: `bash .lavra/memory/recall.sh ` | - -## Gotchas - -- **Port 1514 not 514** — avoids needing root; use iptables PREROUTING to redirect 514→1514 for devices that can't be reconfigured (see docs/SETUP.md) -- **Cargo.lock is tracked** — binary crates should commit Cargo.lock for reproducible builds (Cargo docs guidance) -- **FTS5 query syntax** — `cortex action=search` uses SQLite FTS5: `error AND nginx`, `"disk full"`, `kern OR syslog`; invalid FTS5 syntax returns a db error. **Hyphen is the FTS5 NOT operator** — to search for hyphenated terms, use phrase syntax: `"smoke-test"` not `smoke-test` -- **WAL mode** — SQLite runs in WAL mode; copying `.db`, `.db-wal`, and `.db-shm` together without a checkpoint captures potentially inconsistent state. Safe backup options: (1) run `PRAGMA wal_checkpoint(FULL);` first, then copy all three files, or (2) use `sqlite3 source.db '.backup dest.db'` which is WAL-safe and requires no manual checkpoint -- **MCP transport** — HTTP MCP runs in stateless JSON-response mode on `POST /mcp`; SSE streams (`GET /mcp` or `/sse`) are not enabled in the current server. -- **Data volume** — DB lives in `./data/` (bind mount); `*.db` is gitignored so the database files won't be committed -- **Retention purge** — `retention_days` defaults to 90; logs older than 90 days are **permanently deleted hourly** with no recovery path. Set `CORTEX_RETENTION_DAYS=0` to disable purging entirely. -- **Storage guardrail** — Logical DB size and free-disk limits are enabled by default (`1024/900 MB` DB, `512/768 MB` free disk). When thresholds are breached, the server deletes oldest logs by `received_at` until recovery targets are met. If cleanup still cannot recover enough space, the batch writer blocks new writes until storage becomes healthy again. -- **CEF hostname vs source_ip** — For UniFi CEF messages, the stored `hostname` comes from the CEF `UNIFIdeviceName` extension field (message body), **not** the syslog header. Any LAN device can spoof this value. `source_ip` is the only network-verified identity. See `src/syslog/parser.rs` for the trust boundary. -- **Batch writer failure** — If `insert_logs_batch` fails, the batch is retained for the next flush (up to 1000 entries, then discarded). A 250ms pause prevents hammering a failing DB. Persistent write failures will eventually cause data loss via the 10K-entry channel cap. The mpsc channel is in-memory only — no durable write-ahead log. -- **correlate action limit cap** — The `limit` parameter is silently capped at 999 (not 1000) because the implementation fetches `limit+1` rows to detect truncation, and `search` hard-caps at 1000. -- **Auth / trust model** — MCP endpoint is unauthenticated by default; any client reaching port 3100 has full log read access. Set `CORTEX_TOKEN` to require Bearer auth. CORS is restricted to `localhost:3100` (browser-only; curl/mcporter unaffected). If exposing via SWAG/reverse proxy, add auth at the proxy layer or set the token. See README Security section for details. -- **FTS5 phantom rows** — When logs are deleted by retention purge or storage enforcement, their FTS5 index entries persist as phantom rows in `logs_fts` until the next merge cycle. The MCP query path is unaffected (the JOIN to `logs` prunes phantoms at query time), but direct SQLite access to `logs_fts` reveals porter-stemmed tokens for deleted messages. For right-to-erasure compliance (GDPR/HIPAA), use `INSERT INTO logs_fts(logs_fts) VALUES('rebuild')` after deletion instead of the periodic incremental merge. Monitor phantom row count via `stats` action → `phantom_fts_rows`. -- **OAuth refresh token TTL** — Refresh tokens default to 8h (`refresh_token_ttl_secs = 28800`). lab-auth's default is 30 days; cortex deliberately uses 8h for the read-only homelab profile. Adjustable via `[mcp.auth].refresh_token_ttl_secs` in config.toml. -- **Stdio mode always uses LoopbackDev** — `cargo run -- mcp` (stdio query-only) always uses `AuthPolicy::LoopbackDev` regardless of config. No auth is enforced. This is intentional: the local process boundary is the trust boundary for stdio clients. -- **Docker bind-mount ownership** — `auth.db` and `auth-jwt.pem` are written by the container UID. Host-side backup scripts or file managers may need `sudo` or a sidecar copy step to read them without permission errors. - -## Testing MCP Tools - -```bash -# Full smoke test (requires server running) -bash scripts/smoke-test.sh - -# WAL-safe backup, then destructive DB reset (service should be stopped first) -bash scripts/reset-db.sh - -# Using mcporter (project config at config/mcporter.json) -mcporter list cortex --config config/mcporter.json -mcporter call --config config/mcporter.json cortex.cortex action=stats -mcporter call --config config/mcporter.json cortex.cortex action=tail n=10 -mcporter call --config config/mcporter.json cortex.cortex action=search query=error limit=5 - -# Health check -curl http://localhost:3100/health - -# Tail recent logs (raw JSON-RPC) -curl -s -X POST http://localhost:3100/mcp \ - -H "Content-Type: application/json" \ - -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"syslog","arguments":{"action":"tail","n":10}}}' - -# Search -curl -s -X POST http://localhost:3100/mcp \ - -H "Content-Type: application/json" \ - -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"syslog","arguments":{"action":"search","query":"error","limit":5}}}' - -# Stats -curl -s -X POST http://localhost:3100/mcp \ - -H "Content-Type: application/json" \ - -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"syslog","arguments":{"action":"stats"}}}' - -# stdio mode (query-only, no ingest — useful for Claude Desktop) -cargo run -- mcp -``` - -`cortex compose` commands resolve the live Compose owner before mutation. They refuse ambiguous cwd fallback, stale Compose labels, listener conflicts, and destructive `down` without `--yes`. - - - -## Beads Issue Tracker - -This project uses **bd (beads)** for issue tracking. Run `bd prime` to see full workflow context and commands. - -### Quick Reference - -```bash -bd ready # Find available work -bd show # View issue details -bd update --claim # Claim work -bd close # Complete work -``` - -### Rules - -- Use `bd` for ALL task tracking — do NOT use TodoWrite, TaskCreate, or markdown TODO lists -- Run `bd prime` for detailed command reference and session close protocol -- Use `bd remember` for persistent knowledge — do NOT use MEMORY.md files - -## Session Completion - -**When ending a work session**, you MUST complete ALL steps below. Work is NOT complete until `git push` succeeds. - -**MANDATORY WORKFLOW:** - -1. **File issues for remaining work** - Create issues for anything that needs follow-up -2. **Run quality gates** (if code changed) - Tests, linters, builds -3. **Update issue status** - Close finished work, update in-progress items -4. **PUSH TO REMOTE** - This is MANDATORY: - ```bash - git pull --rebase - bd dolt push - git push - git status # MUST show "up to date with origin" - ``` -5. **Clean up** - Clear stashes, prune remote branches -6. **Verify** - All changes committed AND pushed -7. **Hand off** - Provide context for next session - -**CRITICAL RULES:** -- Work is NOT complete until `git push` succeeds -- NEVER stop before pushing - that leaves work stranded locally -- NEVER say "ready to push when you are" - YOU must push -- If push fails, resolve and retry until it succeeds - - - -## Version Bumping - -**Every feature branch push MUST bump the version in ALL version-bearing files.** - -Bump type is determined by the commit message prefix: -- `feat!:` or `BREAKING CHANGE` → **major** (X+1.0.0) -- `feat` or `feat(...)` → **minor** (X.Y+1.0) -- Everything else (`fix`, `chore`, `refactor`, `test`, `docs`, etc.) → **patch** (X.Y.Z+1) - -**Files to update (if they exist in this repo):** -- `Cargo.toml` — `version = "X.Y.Z"` in `[package]` -- `package.json` — `"version": "X.Y.Z"` -- `pyproject.toml` — `version = "X.Y.Z"` in `[project]` -- `.claude-plugin/plugin.json` — `"version": "X.Y.Z"` -- `.codex-plugin/plugin.json` — `"version": "X.Y.Z"` -- `gemini-extension.json` — `"version": "X.Y.Z"` -- `README.md` — version badge or header -- `CHANGELOG.md` — new entry under the bumped version - -All files MUST have the same version. Never bump only one file. -CHANGELOG.md must have an entry for every version bump. - -## Plugin setup hooks - -Plugin setup is owned by the binary. Keep `scripts/plugin-setup.sh` as a thin adapter that maps `CLAUDE_PLUGIN_OPTION_*` values to environment variables, prepares appdata, ensures `cortex` is on `PATH`, and then calls `cortex setup plugin-hook "$@"`. - -`cortex setup check` is read-only, `cortex setup repair` is idempotent, and `cortex setup plugin-hook --no-repair` is audit mode. Do not add Docker Compose, systemd, or service bootstrap logic back into the hook script. diff --git a/AGENTS.md b/AGENTS.md new file mode 120000 index 00000000..681311eb --- /dev/null +++ b/AGENTS.md @@ -0,0 +1 @@ +CLAUDE.md \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index 3676896b..d327f765 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.14.2] - 2026-06-07 + +### Fixed + +- Shortened inventory graph projection write-lock scope by planning expensive + projection work in memory before applying graph mutations, and avoided the + redundant graph count query on successful projection. +- Hardened SSH-backed inventory and drop-in deploy flows with shared host + validation, strict host key checking by default, explicit TOFU opt-in, and a + shared SSH argument policy. Inventory collectors and remote Docker event + streams also share bounded concurrency and retry backoff. +- Aligned MCP argument parsing with typed request structs so unknown fields are + rejected consistently with HTTP validation, and moved action dispatch behind + the action registry. +- Moved expensive log pattern clustering out of the DB closure, bound SQL + limits instead of interpolating them on touched query paths, and reused the + freshly collected inventory snapshot for graph projection. + +### Changed + +- Rebranded current docs, plugin docs, schema metadata, mcporter examples, and + release guidance around the `cortex` tool name, `cortex:*` scopes, and the + current plugin/package naming. +- Clarified current-versus-archive documentation authority, release/version + policy, live smoke gates, security trust assumptions, OAuth non-Unix behavior, + cargo-deny exception ownership, and rmcp lower-bound intent. +- Made `CLAUDE.md` the agent-memory source of truth and restored sibling + `AGENTS.md` symlinks wherever local `CLAUDE.md` files exist. + +### CI + +- Ran clippy with `--all-targets` and added checks for agent-memory symlinks, + unversioned plugin manifests, and stale public project identity strings. + ## [1.14.1] - 2026-06-06 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 946290b5..c9ed100a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,247 +1,240 @@ -# cortex +# CLAUDE.md — cortex -## Purpose +Rust binary: syslog receiver (UDP/TCP) + MCP server for homelab log intelligence. Receives RFC 3164/5424 syslog from all homelab hosts, stores in SQLite with FTS5, exposes a single `cortex` MCP tool (with action dispatch) for AI agents. -**Syslog Intelligence for Homelabs** — Receives RFC 3164/5424 syslog from all homelab hosts (UDP/TCP), ingests Docker logs via socket proxy, stores everything in SQLite with FTS5, and exposes a comprehensive `cortex` MCP tool for AI agents. +## Commands -**Status**: Active development, Production-ready -**Version**: 1.13.2 - -## Key Files - -| File | Description | -|------|-------------| -| `src/main.rs` | Entry point — CLI initialization and server start | -| `Cargo.toml` | Rust crate definition and dependencies | -| `README.md` | Project overview, install, usage | -| `CLAUDE.md` | Dev environment rules and standard commands | -| `config.toml` | Local development configuration | -| `Justfile` | Command runner for dev, build, and test | -| `src/cli.rs` | Standalone CLI binary (`cortex` command) | -| `src/compose.rs` | Docker Compose lifecycle management | -| `src/scanner.rs` | AI transcript indexer (Claude/Codex sessions) | -| `src/doctor.rs` | Self-debugging diagnostics — binary, DB, and AI-watch health | -| `src/deploy.rs` | CLI remote deploy — provisions cortex on remote hosts | -| `src/api.rs` | HTTP API surface — all routes for CLI HTTP transport | - -## Project Structure - -``` -cortex/ -├── src/ -│ ├── main.rs # CLI entrypoint (serve mcp / mcp stdio) -│ ├── lib.rs # Library root, module declarations -│ ├── cli.rs # Standalone CLI binary -│ ├── compose.rs # Docker Compose lifecycle CLI -│ ├── scanner.rs # AI transcript indexer (Claude/Codex/Gemini) -│ ├── setup.rs # First-run setup + plugin bootstrap -│ ├── runtime.rs # RuntimeCore — wiring and lifecycle -│ ├── config.rs # Configuration (TOML + env) -│ ├── api.rs # HTTP API surface -│ ├── otlp.rs # OpenTelemetry/OTLP ingestion -│ ├── ingest.rs # Log ingestion coordinator -│ ├── ingest_metadata.rs # Ingestion metadata helpers -│ ├── ai_watch.rs # AI transcript watcher (live indexing) -│ ├── observability.rs # Tracing and metrics -│ ├── logging.rs # Service log setup -│ ├── app.rs / db.rs / mcp.rs / syslog.rs / docker_ingest.rs # Module entrypoints -│ ├── deploy.rs # CLI remote deploy -│ ├── cli/ # CLI command implementations -│ ├── compose/ # Docker Compose helpers -│ ├── setup/ # First-run setup internals -│ ├── app/ # Service Layer (business logic) -│ │ ├── service.rs # SyslogService implementation -│ │ ├── models.rs # Request/response types -│ │ ├── correlate.rs # Event correlation logic -│ │ ├── error.rs # Error types -│ │ └── time.rs # Time utilities -│ ├── db/ # Database Layer (SQLite + FTS5) -│ │ ├── pool.rs # Pool management and schema -│ │ ├── queries.rs # SQL queries and search logic -│ │ ├── analytics.rs # Stats and timeline logic -│ │ ├── ingest.rs # Log insertion logic -│ │ ├── maintenance.rs # Retention and storage guardrails -│ │ ├── notifications.rs # Push notification persistence -│ │ ├── error_signatures.rs # Error pattern signatures -│ │ ├── error_detection/ # Error detection rules + scoring -│ │ └── models.rs # DB model types -│ ├── mcp/ # MCP Server Layer -│ │ ├── tools.rs # cortex tool action dispatch -│ │ ├── routes.rs # HTTP route handlers -│ │ ├── schemas.rs # JSON Schema definitions -│ │ └── rmcp_server.rs # RMCP transport implementation -│ ├── syslog/ # Ingestion Layer -│ │ ├── parser.rs # RFC 3164/5424 parsing -│ │ ├── listener.rs # UDP/TCP listeners -│ │ ├── writer.rs # Batch writer -│ │ └── enrichment.rs # Log enrichment (legacy path) -│ ├── enrich/ # Enrichment framework — structured field extraction at ingest -│ ├── notifications/ # Push notification dispatch (Apprise, digest, rules) -│ ├── logging/ # Structured service logging -│ ├── scanner/ # AI transcript scanner -│ │ ├── claude.rs # Claude transcript parsing -│ │ ├── codex.rs # Codex transcript parsing -│ │ └── checkpoint.rs # Scan progress checkpointing -│ ├── doctor.rs # Self-debugging diagnostics (binary, DB, AI-watch) -│ └── docker_ingest/ # Docker remote ingestion -├── config/ # Deployment config templates -├── deploy/ # Host-side manifests (rsyslog, otel) -├── docs/ # Deep-dive documentation -├── plugins/ # Claude Code skills and hooks -├── scripts/ # Maintenance and CI scripts -└── tests/ # Integration tests + tests/test_live.sh smoke runner +```bash +cargo build # debug build +cargo build --release # release build +cargo run # run locally (reads config.toml) +cargo test # test suite +cargo clippy # lint (must pass before committing) +cargo fmt # format (enforced by CI) +docker compose up -d # production deployment +docker compose down # stop +docker compose logs -f # follow logs +docker compose build # rebuild image +cortex compose doctor # diagnose live Compose/listener ownership +cortex compose status --json # inspect canonical cortex container/project +cortex compose pull # pull image for resolved Compose project +cortex compose up # run docker compose up -d for resolved service +cortex compose restart # restart resolved service +cortex compose logs --tail 20 # bounded compose logs +cortex db status # inspect SQLite maintenance state +cortex db integrity # run SQLite integrity_check +cortex db backup # create WAL-safe SQLite backup ``` -## For AI Agents - -### Working In This Directory - -1. **Adding MCP Actions**: Modify `src/mcp/tools.rs` to add the action to the dispatch table. -2. **Database Changes**: Queries go in `src/db/queries.rs`. Schema and pool management in `src/db/pool.rs`. -3. **Adding Ingest Types**: Add parsers in `src/syslog/parser.rs`. -4. **Testing**: Run `just test` before committing. -5. **Building**: Run `just build` to verify compilation. - -### Architecture - -``` -Inbound Logs (UDP/TCP/Docker) - ↓ -Ingestion Layer (src/syslog/) - ↓ Parse RFC 3164/5424 → mpsc channel -Runtime Core (src/runtime.rs) - ↓ Batching → Transaction -Database Layer (src/db/) - ↓ -SQLite Database (/data/cortex.db) - ↑ -Service Layer (src/app/) - ↑ -MCP Layer (src/mcp/) - ↑ -AI Agents (Claude/Codex/Gemini) +```bash +just dev # cargo run alias +just test # cargo test alias +just health # curl /health | jq (server must be running) +just gen-token # openssl rand -hex 32 (generate API token) +just build-plugin # release build → installs binary to bin/ (Linux; requires git lfs install) +just publish [major|minor|patch] # bump version, tag, push (triggers CI) +just generate-cli # build standalone CLI (server must be running) ``` -### Key Design Patterns - -- **Action Dispatch**: Single `cortex` MCP tool dispatches to handlers via an `action` argument. -- **Sidecar Tests**: `#[cfg(test)] #[path = "..._tests.rs"] mod tests;` pattern for unit tests. -- **SQLx + SQLite**: Async SQLx for database operations with WAL mode enabled. -- **FTS5 Search**: Full-text search with BM25-like ranking for log discovery. -- **RuntimeCore Lifecycle**: Centralized management of background tasks (retention, storage guardrails). -- **Transaction Pattern**: All batch inserts use explicit SQLx transactions for atomicity. -- **Storage Guardrails**: Automated cleanup of oldest logs when DB size or disk space limits are breached. -- **Self-Debugging Surfaces**: `cortex ai doctor` checks binary-vs-container version parity, DB health, and AI-watch coordination in one command. CI-safe and idempotent. - -### Transaction Pattern (Rust/SQLx) - -All batch ingestions follow this atomic pattern: +## Architecture + +Key modules in `src/` (most are directories with sidecar `*_tests.rs` files): + +| Module | Purpose | +|--------|---------| +| `config.rs` | Config: `config.toml` + env vars (`CORTEX_*`, `CORTEX_*`, `CORTEX_API_*`, `CORTEX_DOCKER_*`) | +| `runtime.rs` | `RuntimeCore`: wires all subsystems, starts syslog ingest, spawns maintenance tasks | +| `app/` | Service layer: `SyslogService`, request/response models, business logic | +| `db/` | SQLite pool, FTS5 queries, maintenance (retention, storage enforcement) | +| `syslog/` | UDP + TCP listeners, RFC 3164/5424 parsing, mpsc batch writer | +| `mcp/` | RMCP Streamable HTTP server, single `cortex` tool with action dispatch | +| `api.rs` | Optional non-MCP REST API (enabled via `CORTEX_API_ENABLED=true`) | +| `docker_ingest/` | Docker container log ingestion via remote docker-socket-proxy endpoints | +| `main.rs` | Entrypoint: `serve mcp` (full server with ingest) or `mcp` (stdio query-only) | + +Tests: unit tests live in sidecar files beside their source modules (e.g. `src/db/queries_tests.rs`). Source files keep only the `#[cfg(test)] #[path = "..._tests.rs"] mod tests;` hook, so sidecar tests compile as module-local unit tests with `use super::*` access to private items. Run with `cargo test`. + +## Ports + +| Port | Protocol | Purpose | +|------|----------|---------| +| 1514 | UDP + TCP | Syslog receiver (not 514 — avoids `CAP_NET_BIND_SERVICE`) | +| 3100 | TCP | Shared HTTP listener for MCP (`POST /mcp`, `GET /health`) and OTLP HTTP ingest (`POST /v1/logs`); non-loopback OAuth-only `/v1/logs` exposure is blocked at startup unless `CORTEX_TOKEN` is set | + +## MCP Tools + +One MCP tool: **`cortex`** — dispatches by `action` argument. + +| Action | Description | +|--------|-------------| +| `search` | Full-text search (FTS5 syntax) with host/severity/app/time filters | +| `tail` | Recent N entries, optionally filtered by host/app | +| `errors` | Error/warning summary grouped by host and severity | +| `hosts` | All known hosts with first/last seen + log counts | +| `correlate` | Cross-host event correlation in a time window | +| `stats` | DB stats (total logs, logical/physical size, free disk, configured thresholds, write-block state, time range) | +| `status` | Lightweight runtime and DB health | +| `apps` | Distinct application names with log and host counts | +| `sessions` | AI transcript sessions grouped by project/tool/session/host | +| `search_sessions` | Full-text search over indexed AI transcript sessions | +| `usage_blocks` | AI transcript activity grouped into time blocks | +| `project_context` | Recent AI transcript context for a project | +| `list_ai_tools` | AI tools present in transcript metadata | +| `list_ai_projects` | AI projects present in transcript metadata | +| `source_ips` | Distinct source identifiers with hostname breakdown | +| `timeline` | Bucketed counts over time | +| `patterns` | Near-duplicate message template clusters | +| `context` | Surrounding logs around a log id or timestamp | +| `get` | One log entry by id, including raw frame | +| `ingest_rate` | Recent ingest throughput and write-block state | +| `silent_hosts` | Hosts whose last_seen is older than a threshold | +| `clock_skew` | Per-host received_at minus timestamp distribution | +| `anomalies` | Recent vs baseline volume/error comparison | +| `compare` | Side-by-side comparison of two time ranges | +| `compose_status` | Redacted Docker Compose runtime status projection | +| `compose_doctor` | Redacted Docker Compose diagnostics projection | +| `help` | Built-in usage reference | + +## Plugin Skills + +Skills available after installing the Claude Code plugin +(`plugins/cortex/skills//`): + +| Command | Description | +|---------|-------------| +| `/cortex-dr` | Full health check: MCP, HTTP /health, service status, syslog port, Docker ingest, fleet drop-ins (named `dr` to avoid colliding with Claude Code's built-in `/doctor`) | +| `/cortex-deploy-dropins` | Push rsyslog forwarding configs to `fleet_hosts` via SSH (idempotent) | + +## Config + +`config.toml` at repo root for local dev. **Not copied into Docker** — the Dockerfile was cleaned up (no COPY for config.toml). In Docker, defaults + env vars apply exclusively. -```rust -let mut tx = pool.begin().await?; -for log in logs { - insert_log(&mut *tx, log).await?; -} -tx.commit().await?; +```bash +# Syslog listener +CORTEX_RECEIVER_HOST=0.0.0.0 # host only, no port +CORTEX_RECEIVER_PORT=1514 # shared by UDP + TCP +CORTEX_MAX_MESSAGE_SIZE=8192 +CORTEX_BATCH_SIZE=100 +CORTEX_FLUSH_INTERVAL=500 # ms + +# MCP server +CORTEX_HOST=0.0.0.0 +CORTEX_PORT=3100 +CORTEX_TOKEN=your-secret-token # optional; enables Bearer auth on /mcp + # (CORTEX_API_TOKEN still works, logs deprecation) +CORTEX_ALLOWED_HOSTS=myhost.local # optional; comma-separated extra Host allowlist +CORTEX_ALLOWED_ORIGINS=https://app # optional; comma-separated extra Origin allowlist + +# Storage +CORTEX_DB_PATH=data/cortex.db +CORTEX_POOL_SIZE=4 +CORTEX_RETENTION_DAYS=90 # 0 = keep forever +CORTEX_MAX_DB_SIZE_MB=1024 # 0 = disable logical DB size guard +CORTEX_RECOVERY_DB_SIZE_MB=900 # cleanup target after DB-size breach +CORTEX_MIN_FREE_DISK_MB=512 # 0 = disable free-disk guard +CORTEX_RECOVERY_FREE_DISK_MB=768 # cleanup target after free-disk breach +CORTEX_CLEANUP_INTERVAL_SECS=60 # storage-budget enforcement interval (>= 5) +CORTEX_CLEANUP_CHUNK_SIZE=1000 # rows deleted per enforcement cycle + +# OAuth / JWT auth (disabled by default — set CORTEX_AUTH_MODE=oauth to activate) +CORTEX_AUTH_MODE=bearer # bearer (default) or oauth +CORTEX_PUBLIC_URL=https://cortex.example.com # required when CORTEX_AUTH_MODE=oauth +CORTEX_GOOGLE_CLIENT_ID=... # required when CORTEX_AUTH_MODE=oauth +CORTEX_GOOGLE_CLIENT_SECRET=... # required when CORTEX_AUTH_MODE=oauth +# Paths, TTLs, allowlist → config.toml [mcp.auth] (not env vars). See docs/OAUTH.md. + +# Non-MCP REST API (disabled by default) +CORTEX_API_ENABLED=false # set true to mount /api/* endpoints +CORTEX_API_TOKEN=your-api-token # required when CORTEX_API_ENABLED=true + +# Docker container log ingestion (disabled by default) +CORTEX_DOCKER_INGEST_ENABLED=false # set true to ingest from docker-socket-proxy hosts +CORTEX_DOCKER_HOSTS=host-a,host-b # comma-separated hostnames → http://:2375 +CORTEX_DOCKER_RECONNECT_INITIAL_MS=1000 +CORTEX_DOCKER_RECONNECT_MAX_MS=60000 + +# Log verbosity (set to debug or trace for development) +RUST_LOG=info ``` -### Testing - -**Prerequisite:** `just test` uses [cargo-nextest](https://nexte.st), which is **not** -bundled with the Rust toolchain. Fresh checkouts must install it once, or `just test` -fails with `error: no such subcommand: nextest`: +## Key Files -```bash -cargo install cargo-nextest --locked -``` +| File | Purpose | +|------|---------| +| `config.toml` | Runtime config (syslog bind, DB path, retention) | +| `docker-compose.yml` | Production deployment (ports 1514, 3100) | +| `docs/SETUP.md` | Per-host syslog forwarding (rsyslog, UniFi, ATT router, WSL) | +| `src/db/queries.rs` | All SQL queries and FTS5 search implementation | +| `src/mcp/tools.rs` | Single `cortex` tool with action dispatch | +| `config/mcporter.json` | mcporter config (HTTP transport to localhost:3100) | +| `CORTEX_DOCKER_HOSTS` env var | Docker ingest host list — comma-separated hostnames, each becomes `http://:2375` | +| `scripts/smoke-test.sh` | Live smoke test — all MCP actions via mcporter, strict PASS/FAIL | +| `scripts/backup.sh` | WAL-safe SQLite backup script (checkpoint + `.backup` method) | +| `scripts/reset-db.sh` | WAL-safe backup + destructive DB reset helper for local/dev recovery | +| `scripts/bump-version.sh` | Bump version across all version-bearing files; called by `just publish` | +| `cortex db status\|integrity\|checkpoint\|vacuum\|backup` | Direct SQLite maintenance commands for the configured DB | +| `scripts/check-version-sync.sh` | Assert all version-bearing files have the same version (used in CI) | +| `scripts/block-env-commits.sh` | Pre-commit hook that blocks commits containing env credential patterns | +| `CHANGELOG.md` | Version history; entry required per version bump | +| `.lavra/memory/recall.sh` | Query the local knowledge DB: `bash .lavra/memory/recall.sh ` | + +## Gotchas + +- **Port 1514 not 514** — avoids needing root; use iptables PREROUTING to redirect 514→1514 for devices that can't be reconfigured (see docs/SETUP.md) +- **Cargo.lock is tracked** — binary crates should commit Cargo.lock for reproducible builds (Cargo docs guidance) +- **FTS5 query syntax** — `cortex action=search` uses SQLite FTS5: `error AND nginx`, `"disk full"`, `kern OR syslog`; invalid FTS5 syntax returns a db error. **Hyphen is the FTS5 NOT operator** — to search for hyphenated terms, use phrase syntax: `"smoke-test"` not `smoke-test` +- **WAL mode** — SQLite runs in WAL mode; copying `.db`, `.db-wal`, and `.db-shm` together without a checkpoint captures potentially inconsistent state. Safe backup options: (1) run `PRAGMA wal_checkpoint(FULL);` first, then copy all three files, or (2) use `sqlite3 source.db '.backup dest.db'` which is WAL-safe and requires no manual checkpoint +- **MCP transport** — HTTP MCP runs in stateless JSON-response mode on `POST /mcp`; SSE streams (`GET /mcp` or `/sse`) are not enabled in the current server. +- **Data volume** — DB lives in `./data/` (bind mount); `*.db` is gitignored so the database files won't be committed +- **Retention purge** — `retention_days` defaults to 90; logs older than 90 days are **permanently deleted hourly** with no recovery path. Set `CORTEX_RETENTION_DAYS=0` to disable purging entirely. +- **Storage guardrail** — Logical DB size and free-disk limits are enabled by default (`1024/900 MB` DB, `512/768 MB` free disk). When thresholds are breached, the server deletes oldest logs by `received_at` until recovery targets are met. If cleanup still cannot recover enough space, the batch writer blocks new writes until storage becomes healthy again. +- **CEF hostname vs source_ip** — For UniFi CEF messages, the stored `hostname` comes from the CEF `UNIFIdeviceName` extension field (message body), **not** the syslog header. Any LAN device can spoof this value. `source_ip` is the only network-verified identity. See `src/syslog/parser.rs` for the trust boundary. +- **Batch writer failure** — If `insert_logs_batch` fails, the batch is retained for the next flush (up to 1000 entries, then discarded). A 250ms pause prevents hammering a failing DB. Persistent write failures will eventually cause data loss via the 10K-entry channel cap. The mpsc channel is in-memory only — no durable write-ahead log. +- **correlate action limit cap** — The `limit` parameter is silently capped at 999 (not 1000) because the implementation fetches `limit+1` rows to detect truncation, and `search` hard-caps at 1000. +- **Auth / trust model** — MCP endpoint is unauthenticated by default; any client reaching port 3100 has full log read access. Set `CORTEX_TOKEN` to require Bearer auth. CORS is restricted to `localhost:3100` (browser-only; curl/mcporter unaffected). If exposing via SWAG/reverse proxy, add auth at the proxy layer or set the token. See README Security section for details. +- **FTS5 phantom rows** — When logs are deleted by retention purge or storage enforcement, their FTS5 index entries persist as phantom rows in `logs_fts` until the next merge cycle. The MCP query path is unaffected (the JOIN to `logs` prunes phantoms at query time), but direct SQLite access to `logs_fts` reveals porter-stemmed tokens for deleted messages. For right-to-erasure compliance (GDPR/HIPAA), use `INSERT INTO logs_fts(logs_fts) VALUES('rebuild')` after deletion instead of the periodic incremental merge. Monitor phantom row count via `stats` action → `phantom_fts_rows`. +- **OAuth refresh token TTL** — Refresh tokens default to 8h (`refresh_token_ttl_secs = 28800`). lab-auth's default is 30 days; cortex deliberately uses 8h for the read-only homelab profile. Adjustable via `[mcp.auth].refresh_token_ttl_secs` in config.toml. +- **Stdio mode always uses LoopbackDev** — `cargo run -- mcp` (stdio query-only) always uses `AuthPolicy::LoopbackDev` regardless of config. No auth is enforced. This is intentional: the local process boundary is the trust boundary for stdio clients. +- **Docker bind-mount ownership** — `auth.db` and `auth-jwt.pem` are written by the container UID. Host-side backup scripts or file managers may need `sudo` or a sidecar copy step to read them without permission errors. + +## Testing MCP Tools ```bash -just test # Run all unit and integration tests (requires cargo-nextest) -just test-doc # Run doc tests (nextest does not execute these) -just test-live # Live smoke test against a running server (tests/test_live.sh) -bash scripts/smoke-test.sh # Lower-level smoke harness (used by CI; superset of test-live) +# Full smoke test (requires server running) +bash scripts/smoke-test.sh + +# WAL-safe backup, then destructive DB reset (service should be stopped first) +bash scripts/reset-db.sh + +# Using mcporter (project config at config/mcporter.json) +mcporter list cortex --config config/mcporter.json +mcporter call --config config/mcporter.json cortex.cortex action=stats +mcporter call --config config/mcporter.json cortex.cortex action=tail n=10 +mcporter call --config config/mcporter.json cortex.cortex action=search query=error limit=5 + +# Health check +curl http://localhost:3100/health + +# Tail recent logs (raw JSON-RPC) +curl -s -X POST http://localhost:3100/mcp \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"tail","n":10}}}' + +# Search +curl -s -X POST http://localhost:3100/mcp \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"search","query":"error","limit":5}}}' + +# Stats +curl -s -X POST http://localhost:3100/mcp \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"stats"}}}' + +# stdio mode (query-only, no ingest — useful for Claude Desktop) +cargo run -- mcp ``` -## CLI Commands - -| Command | Purpose | Example | -|---------|---------|---------| -| `cortex serve mcp` | Start full server with ingest | `cortex serve mcp` | -| `cortex mcp` | Start stdio query-only mode | `cortex mcp` | -| `just health` | Check server health | `just health` | -| `just dev` | Run in dev mode | `just dev` | -| `just lint` | Run clippy (strict) | `just lint` | -| `just fmt` | Format code | `just fmt` | -| `just test-live` | Run live integration tests | `just test-live` | -| `just up` / `just down` | Start/stop Docker Compose | `just up` | -| `just build` | Cargo release build | `just build` | -| `just validate-skills` | Validate plugin skill manifests | `just validate-skills` | -| `just gen-token` | Generate a random API token | `just gen-token` | -| `just build-plugin` | Copy release binary into plugins/cortex/bin/ | `just build-plugin` | -| `just publish [bump]` | Version bump + tag + push | `just publish patch` | -| `just setup` | Initialize .env from .env.example | `just setup` | -| `cortex ai doctor` | Self-debug: binary vs container version, DB health, AI-watch | `cortex ai doctor` | -| `cortex db status` | DB size, WAL, page count, drift check | `cortex db status` | -| `cortex db integrity` | SQLite integrity_check | `cortex db integrity --quick` | -| `cortex db vacuum` | Reclaim DB space | `cortex db vacuum` | -| `cortex db backup` | Backup DB to path | `cortex db backup --output /tmp/out.db` | -| `cortex setup check` | Validate config and env | `cortex setup check` | -| `cortex setup repair` | Auto-fix missing config | `cortex setup repair` | -| `cortex compose status` | Container running status | `cortex compose status` | -| `cortex compose doctor` | Full coordination diagnostics | `cortex compose doctor` | -| `cortex source-ips` | List unique source IPs with log counts | `cortex source-ips --limit 50` | -| `cortex timeline` | Log volume over time (bucketed) | `cortex timeline --bucket hour` | -| `cortex patterns` | Recurring message patterns | `cortex patterns --top-n 25` | -| `cortex ingest-rate` | Current ingest rate (logs/sec) | `cortex ingest-rate --by-host` | -| `cortex sig list` | List unaddressed error signatures | `cortex sig list` | -| `cortex sig ack HASH` | Acknowledge/suppress an error signature | `cortex sig ack ab12cd --notes "fixed"` | -| `cortex sig unack HASH` | Revoke an acknowledgement | `cortex sig unack ab12cd` | -| `cortex notify recent` | Recent notification firings | `cortex notify recent --limit 25` | -| `cortex notify test` | Send a test notification via Apprise (HTTP-only) | `cortex --http notify test --body "ping"` | - -## Diagnostics: host/container drift - -Two coordination diagnostics guard against the CLI and the container talking -to different SQLite files: - -- `data-mount` — verifies the host directory bind-mounted at `/data` matches - `CORTEX_DATA_VOLUME`. -- `ai-watch-coord` — verifies the host systemd `syslog-ai-watch.service` - resolves `CORTEX_DB_PATH` to the same canonical directory as the - container's `/data` bind. - -Where they run: - -- `cortex compose doctor` — always runs both phases. `--json` includes them - under a `coordination` array. A canonical mismatch is fatal (exit 1). -- `cortex db status --check-coord` — opt-in. Adds both phases to the JSON - payload under `coordination`. The default `cortex db status` path is - unchanged (no shell-outs). - -Both phases shell out to `docker inspect` and `systemctl --user show`, which -adds roughly 100-200ms per invocation. Within a single `compose doctor` -invocation the results are cached so each shell-out fires only once. - -Status semantics for these phases: - -- `ok` — canonical paths match. -- `skipped` — ai-watch unit is not installed/loadable, or container is not - running (data-mount only). Reserved for "ai-watch absent" — never used to - hide failures. -- `warn` — could not enumerate inputs (docker/systemctl failed, canonicalize - hit `ENOENT` / `EACCES`). Emits the OS error verbatim; we never silently - fall back to literal-string compare. -- `error` — both sides resolved and the canonical paths differ. - -## Dependencies - -| Dependency | Purpose | -|------------|---------| -| `sqlx` | Async SQLite driver | -| `axum` | HTTP server for MCP | -| `tokio` | Async runtime | -| `serde` | Serialization/Deserialization | -| `tracing` | Observability and logging | +`cortex compose` commands resolve the live Compose owner before mutation. They refuse ambiguous cwd fallback, stale Compose labels, listener conflicts, and destructive `down` without `--yes`. @@ -290,3 +283,34 @@ bd close # Complete work - NEVER say "ready to push when you are" - YOU must push - If push fails, resolve and retry until it succeeds + + +## Version Bumping + +**Every feature branch push MUST bump the version in ALL version-bearing files.** + +Bump type is determined by the commit message prefix: +- `feat!:` or `BREAKING CHANGE` → **major** (X+1.0.0) +- `feat` or `feat(...)` → **minor** (X.Y+1.0) +- Everything else (`fix`, `chore`, `refactor`, `test`, `docs`, etc.) → **patch** (X.Y.Z+1) + +**Canonical version-bearing files:** +- `Cargo.toml` — `version = "X.Y.Z"` in `[package]` +- `server.json` — MCP Registry `"version": "X.Y.Z"` plus package image tag +- `mcpb/manifest.json` — MCP Bundle `"version": "X.Y.Z"` +- `Cargo.lock` — updated when Cargo records the package version +- `CHANGELOG.md` — new entry under the bumped version + +Optional package metadata such as `package.json` or `pyproject.toml` must join +the same version set if introduced. Claude/Codex/Gemini plugin manifests are +intentionally unversioned; `scripts/check-plugin-manifest-versions.sh` rejects +top-level `version` keys in plugin manifests. + +All files MUST have the same version. Never bump only one file. +CHANGELOG.md must have an entry for every version bump. + +## Plugin setup hooks + +Plugin setup is owned by the binary. Keep `scripts/plugin-setup.sh` as a thin adapter that maps `CLAUDE_PLUGIN_OPTION_*` values to environment variables, prepares appdata, ensures `cortex` is on `PATH`, and then calls `cortex setup plugin-hook "$@"`. + +`cortex setup check` is read-only, `cortex setup repair` is idempotent, and `cortex setup plugin-hook --no-repair` is audit mode. Do not add Docker Compose, systemd, or service bootstrap logic back into the hook script. diff --git a/Cargo.lock b/Cargo.lock index d84686fa..7d27e957 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -418,7 +418,7 @@ checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" [[package]] name = "cortex" -version = "1.14.1" +version = "1.14.2" dependencies = [ "anyhow", "axum", diff --git a/Cargo.toml b/Cargo.toml index cb92190b..7ea7384d 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,8 +1,9 @@ [package] name = "cortex" -version = "1.14.1" +version = "1.14.2" edition = "2021" rust-version = "1.86" +license = "MIT" description = "Homelab intelligence platform — syslog/OTLP/Docker log aggregation, fleet awareness, and AI agent coordination over MCP, CLI, and HTTP" autobins = false @@ -17,6 +18,8 @@ tokio-util = { version = "0.7", features = ["rt"] } # HTTP / MCP transport axum = "0.8" +# rmcp 1.6.0 is the supported lower bound for cortex's current API surface. +# Cargo.lock may resolve a newer compatible 1.x release; see docs/RUST.md. rmcp = { version = "1.6.0", default-features = false, features = [ "server", "macros", diff --git a/README.md b/README.md index fd6f1ccf..fd9268d6 100644 --- a/README.md +++ b/README.md @@ -501,7 +501,7 @@ separate table: ```bash cortex shell index --path ~/.zsh_history --shell zsh cortex setup agent-command install -export CLAUDE_CODE_SHELL_PREFIX="$HOME/.local/bin/syslog-agent-command-wrapper" +export CLAUDE_CODE_SHELL_PREFIX="$HOME/.local/bin/cortex-agent-command-wrapper" cortex agent-command ingest-spool --path ~/.local/state/cortex/agent-command.jsonl ``` @@ -564,16 +564,17 @@ The installer puts the host `cortex` binary in `~/.local/bin` and then runs - `~/.cortex/compose/docker-compose.yml` — Docker Compose deployment assets - `~/.cortex/data/cortex.db` — SQLite database and WAL/SHM sidecars -Setup writes `COMPOSE_PROJECT_NAME=syslog-jmagar-lab` so direct -`docker compose` commands in `~/.cortex/compose` target the same canonical -container as `cortex compose`. +Setup writes the compose project name used by the shared host deployment. +Existing installations may still use the legacy `syslog-jmagar-lab` project +name for container-label compatibility; prefer `cortex compose ...` commands +because they resolve the live owner before mutating the stack. Useful installer controls: ```bash CORTEX_INSTALL_DRY_RUN=1 ./install.sh CORTEX_INSTALL_PREFIX=/opt/cortex ./install.sh -CORTEX_VERSION=0.25.4 ./install.sh +CORTEX_VERSION= ./install.sh CORTEX_INSTALL_SKIP_SETUP=1 ./install.sh ``` @@ -609,7 +610,7 @@ Install as a Claude Code plugin. The plugin handles deployment automatically — | `batch_size` | no | `100` | Number of parsed messages per SQLite batch | | `write_channel_capacity` | no | `10000` | Internal parsed-message queue capacity before listener backpressure | | `docker_ingest_enabled` | no | `false` | Pull container logs from remote `docker-socket-proxy` endpoints | -| `fleet_hosts` | no | — | SSH aliases of fleet hosts. Used for Docker ingest (when enabled, each becomes `http://:2375`) and the `syslog-deploy-dropins` skill | +| `fleet_hosts` | no | — | SSH aliases of fleet hosts. Used for Docker ingest (when enabled, each becomes `http://:2375`) and the `cortex-deploy-dropins` skill | **SessionStart hook automation** (in server mode): @@ -621,11 +622,11 @@ Install as a Claude Code plugin. The plugin handles deployment automatically — **Bundled skills**: -- `syslog-dr` — health check covering MCP, service status, syslog port, fleet drop-ins, and live log flow; tails service logs on failure -- `syslog-deploy-dropins` — SSH-based one-shot rsyslog drop-in deployment to every host in `fleet_hosts` -- `syslog-redeploy` — re-run plugin setup after config or plugin changes -- `syslog-logs` — Docker Compose service log tailing -- `syslog-version-check` — check whether the running Docker container matches the local Compose image; add `--pull` to pull first, otherwise checks only the local image cache +- `cortex-dr` — health check covering MCP, service status, syslog port, fleet drop-ins, and live log flow; tails service logs on failure +- `cortex-deploy-dropins` — SSH-based one-shot rsyslog drop-in deployment to every host in `fleet_hosts` +- `cortex-redeploy` — re-run plugin setup after config or plugin changes +- `cortex-logs` — Docker Compose service log tailing +- `cortex-version-check` — check whether the running Docker container matches the local Compose image; add `--pull` to pull first, otherwise checks only the local image cache The plugin deploys the server with Docker Compose through the same `cortex setup` path as the one-line installer. You can still build and run the binary locally @@ -838,10 +839,10 @@ The MCP query API (port 3100, default loopback) supports two auth modes: | Mode | Config | Effect | |------|--------|--------| -| Bearer token | `CORTEX_TOKEN=` | Static token grants `syslog:read` by default; set `CORTEX_STATIC_TOKEN_ADMIN=true` to also grant `syslog:admin` | +| Bearer token | `CORTEX_TOKEN=` | Static token grants `cortex:read` by default; set `CORTEX_STATIC_TOKEN_ADMIN=true` to also grant `cortex:admin` | | Google OAuth | `CORTEX_AUTH_MODE=oauth` | OAuth users authenticated via `CORTEX_AUTH_ADMIN_EMAIL` | -**Important**: Admin actions such as `ack_error`, `unack_error`, and `notifications_test` require `syslog:admin`. Static bearer tokens are read-only unless `CORTEX_STATIC_TOKEN_ADMIN=true` is explicitly set. +**Important**: Admin actions such as `ack_error`, `unack_error`, and `notifications_test` require `cortex:admin`. Static bearer tokens are read-only unless `CORTEX_STATIC_TOKEN_ADMIN=true` is explicitly set. The MCP port defaults to `127.0.0.1:3100` (loopback only). To expose it on a network interface, set `CORTEX_HOST=0.0.0.0` and configure a TLS-terminating reverse proxy in front of it. @@ -1084,7 +1085,8 @@ Before upgrading a populated database: 3. Start the new version and monitor logs for `Migration N: starting ...` and `Migration N: ... created`. 4. Keep the previous image or binary available until `/health` returns `ok` and `cortex stats` reports sane counts. -See [docs/runbooks/deploy.md](docs/runbooks/deploy.md) for the deploy checklist. +See [docs/RELEASE.md](docs/RELEASE.md) for the current release and deploy +gate checklist. --- @@ -1285,7 +1287,7 @@ Stdio mode does not use bearer auth because it is local child-process access. It { "mcpServers": { "cortex": { - "command": "/path/to/syslog", + "command": "/path/to/cortex", "args": ["mcp"], "env": { "CORTEX_DB_PATH": "/data/cortex.db", diff --git a/config/mcporter.json b/config/mcporter.json index a532dc2f..af0a334a 100644 --- a/config/mcporter.json +++ b/config/mcporter.json @@ -1,6 +1,6 @@ { "mcpServers": { - "syslog": { + "cortex": { "url": "http://localhost:3100/mcp", "transport": "http" } diff --git a/deny.toml b/deny.toml index 4db3ea25..e0b39b86 100644 --- a/deny.toml +++ b/deny.toml @@ -17,6 +17,8 @@ yanked = "deny" ignore = [ # RUSTSEC-2023-0071 — Marvin Attack (RSA timing side-channel) in the `rsa` crate. # Pulled in transitively: lab-auth → jsonwebtoken → rsa v0.9.10 / v0.10.0-rc.18. + # Owner: cortex maintainer. Review cadence: every release; next review no later + # than 2026-07-31 or when lab-auth/jsonwebtoken releases a hardened path. # No upstream fix exists yet (as of 2026-05). The RSA keys are used for JWT # signing/verification only (short-lived tokens, not long-term data encryption) # and the server is not on the PKCS#1 v1.5 decrypt path that is vulnerable. @@ -38,7 +40,6 @@ allow = [ "Unicode-3.0", "CC0-1.0", "Zlib", # foldhash (via hashbrown → rusqlite → lab-auth) - "MPL-2.0", # option-ext (via dirs-sys → dirs) "CDLA-Permissive-2.0", # webpki-roots / webpki-root-certs (via reqwest → rustls) ] # Confidence threshold for license text detection (0.0–1.0) @@ -47,10 +48,13 @@ confidence-threshold = 0.8 # ── Bans ────────────────────────────────────────────────────────────────────── [bans] -# Warn on multiple versions of the same crate — forces explicit resolution -multiple-versions = "warn" -# "warn" not "deny": git deps with a pinned rev have no semver version field, -# which cargo-deny treats as a wildcard — but the rev IS the pin. +# Duplicate versions are dominated by the transitive auth/MCP/Windows target +# stack and are reviewed during release dependency audits. Keep this check quiet +# so cargo-deny output highlights advisories, license drift, and source drift. +multiple-versions = "allow" +# Keep wildcard dependency requirements visible. Git deps are pinned by `rev`, +# and source policy below restricts remotes, but accidental broad crate specs +# should still show up during dependency audits. wildcards = "warn" # Allow path deps (e.g. the self-referencing dev dep) to omit a version. allow-wildcard-paths = true diff --git a/docs/CHECKLIST.md b/docs/CHECKLIST.md index ed032711..48e9bf44 100644 --- a/docs/CHECKLIST.md +++ b/docs/CHECKLIST.md @@ -1,10 +1,14 @@ -# Plugin Checklist -- cortex +# Release Audit Checklist -- cortex -Pre-release and quality checklist. Complete all items before tagging a release. +Supplemental pre-release audit checklist. `docs/RELEASE.md` is the source of +truth for hermetic and live release gates. ## Version and metadata -- [ ] All version-bearing files in sync: `Cargo.toml`, `.claude-plugin/plugin.json`, `.codex-plugin/plugin.json`, `gemini-extension.json`, `server.json` +- [ ] Version-bearing files in sync: `Cargo.toml`, `Cargo.lock`, + `server.json`, `mcpb/manifest.json`, and `CHANGELOG.md` +- [ ] Plugin manifests are unversioned: + `.claude-plugin/plugin.json` and `plugins/**/plugin.json` - [ ] `CHANGELOG.md` has an entry for the new version - [ ] README version badge is correct @@ -18,7 +22,7 @@ Pre-release and quality checklist. Complete all items before tagging a release. - [ ] `CLAUDE.md` is current and matches repo structure - [ ] `README.md` has up-to-date tool reference and environment variable table -- [ ] `skills/cortex/SKILL.md` has correct frontmatter and tool descriptions +- [ ] `plugins/cortex/skills/cortex/SKILL.md` has correct frontmatter and tool descriptions - [ ] Setup instructions work from a clean clone ## Security @@ -34,26 +38,27 @@ Pre-release and quality checklist. Complete all items before tagging a release. ## Build and test -- [ ] Docker image builds: `just docker-build` -- [ ] Docker healthcheck passes: `just health` -- [ ] CI pipeline passes: `just lint && just test` +- [ ] Docker image builds: `docker compose build` +- [ ] Docker healthcheck passes against the intended deployment +- [ ] CI pipeline passes the hermetic gates in `docs/RELEASE.md` - [ ] Live smoke test passes: `just test-live` -- [ ] `cargo clippy -- -D warnings` produces zero warnings +- [ ] `cargo clippy --all-targets -- -D warnings` produces zero warnings ## Deployment - [ ] `docker-compose.yml` uses correct ports (1514 UDP/TCP, 3100 TCP) -- [ ] `entrypoint.sh` is executable -- [ ] SWAG reverse proxy config tested (see `docs/syslog.subdomain.conf`) +- [ ] `cortex compose doctor` passes before lifecycle mutations +- [ ] Reverse proxy config tested when exposing the service externally ## Registry (if publishing) - [ ] `server.json` for MCP registry is valid JSON with correct version +- [ ] `mcpb/manifest.json` is valid JSON with matching package metadata - [ ] OCI image published to `ghcr.io/jmagar/cortex` - [ ] Crate published to crates.io (if applicable) - [ ] DNS verification for `tv.tootie/cortex` ## Marketplace (if applicable) -- [ ] Entry in `claude-homelab` marketplace manifest -- [ ] Plugin installs correctly: `/plugin marketplace add jmagar/claude-homelab` +- [ ] Entry in the active plugin marketplace manifest is current +- [ ] Plugin installs correctly from the current marketplace source diff --git a/docs/CLI.md b/docs/CLI.md index 8c5e0858..0a15047e 100644 --- a/docs/CLI.md +++ b/docs/CLI.md @@ -447,7 +447,7 @@ spool into SQLite. ```bash cortex setup agent-command install -export CLAUDE_CODE_SHELL_PREFIX="$HOME/.local/bin/syslog-agent-command-wrapper" +export CLAUDE_CODE_SHELL_PREFIX="$HOME/.local/bin/cortex-agent-command-wrapper" cortex agent-command ingest-spool --path ~/.local/state/cortex/agent-command.jsonl cortex agent-command wrap --spool ~/.local/state/cortex/agent-command.jsonl -- cargo test @@ -515,7 +515,7 @@ remain blocking errors. Installing the watch service disables the older ### `cortex setup debug-wrapper` Install, remove, or inspect the host-local debug wrapper at -`~/.local/bin/syslog`. +`~/.local/bin/cortex`. ```bash cortex setup debug-wrapper install @@ -524,11 +524,11 @@ cortex setup debug-wrapper remove ``` The wrapper is intentionally machine-local. It `cd`s into the configured repo -or worktree, builds `cargo build --bin syslog` into `.cache/cargo`, then execs +or worktree, builds `cargo build --bin cortex` into `.cache/cargo`, then execs the fresh debug binary. For non-server commands it defaults Docker ingest off and bearer auth mode on, so regular CLI checks do not accidentally start container-log ingestion or OAuth-only config paths. Override the source checkout -with `CORTEX_REPO=/path/to/cortex syslog ...`. +with `CORTEX_REPO=/path/to/cortex cortex ...`. ### `cortex setup debug-compose` @@ -544,10 +544,10 @@ cortex setup debug-compose remove The override is machine-local. It points the canonical Docker Compose project at the current repo/worktree and builds the `cortex:local-debug` image with the debug profile. This keeps `docker compose up -d --build` aligned with the same -code that the host debug wrapper builds. `cortex setup` also writes -`COMPOSE_PROJECT_NAME=syslog-jmagar-lab` to the setup `.env`, so direct -`docker compose` commands target the canonical project instead of a cwd-derived -project name. +code that the host debug wrapper builds. Existing setup environments may still +carry the legacy `COMPOSE_PROJECT_NAME=syslog-jmagar-lab` for container-label +compatibility; use `cortex compose ...` when possible because it resolves the +live owner before mutating the stack. ### `cortex setup doctor` @@ -626,7 +626,7 @@ bash scripts/smoke-ai-mcp.sh ``` The smoke scripts resolve `CORTEX_BIN` first, then `cortex` on `PATH`, then the -repo-local debug binary at `target/debug/syslog`. +repo-local debug binary at `target/debug/cortex`. With `syslog-ai-watch.service` installed, new transcript lines usually become searchable within a few seconds of the writer closing or flushing the file. @@ -993,7 +993,7 @@ models. The MCP-only `status` and `help` actions are runtime/protocol helpers, not direct database queries. Compose mutations (`up`, `down`, `restart`, `pull`, `logs`) are CLI-only and are not exposed over MCP. Admin MCP actions such as -`ack_error`, `unack_error`, and `notifications_test` require `syslog:admin` +`ack_error`, `unack_error`, and `notifications_test` require `cortex:admin` when auth is mounted. Use direct CLI mode for terminal queries and scripts on a host that can read the diff --git a/docs/CONFIG.md b/docs/CONFIG.md index 79f509a4..eddb02e6 100644 --- a/docs/CONFIG.md +++ b/docs/CONFIG.md @@ -41,8 +41,8 @@ cleanup_interval_secs = 60 host = "0.0.0.0" port = 3100 server_name = "cortex" -allowed_hosts = ["syslog.example.com", "syslog.example.com:443"] -allowed_origins = ["https://syslog.example.com"] +allowed_hosts = ["cortex.example.com", "cortex.example.com:443"] +allowed_origins = ["https://cortex.example.com"] [api] enabled = false @@ -61,9 +61,9 @@ allow_insecure_http = true Bind host fields (`CORTEX_RECEIVER_HOST` and `CORTEX_HOST`) must be hostnames or IP addresses without `:` because their ports are configured separately. `allowed_hosts` / `CORTEX_ALLOWED_HOSTS` are RMCP Host-header allow-list -entries and may include `host:port` values such as `syslog.example.com:443`. +entries and may include `host:port` values such as `cortex.example.com:443`. `allowed_origins` / `CORTEX_ALLOWED_ORIGINS` remain full browser origin URLs -such as `https://syslog.example.com`. +such as `https://cortex.example.com`. ## Environment variables @@ -209,7 +209,7 @@ For populated databases, treat heavy migrations as a planned upgrade step: 4. Wait for `curl -sf http://localhost:3100/health` to succeed. 5. Run `cortex stats --json` or `mcporter call ... action=stats` and confirm `total_logs`, storage metrics, and `write_blocked` match expectations. -If a migration must be abandoned, stop the new process before changing files, restore the WAL-safe backup, and restart the previous image or binary. See [runbooks/deploy.md](runbooks/deploy.md) for the full deploy checklist. +If a migration must be abandoned, stop the new process before changing files, restore the WAL-safe backup, and restart the previous image or binary. See [RELEASE.md](RELEASE.md) for the current deploy gate checklist. ## Validation rules @@ -232,12 +232,12 @@ When installed as a Claude Code plugin, users are prompted for: | Field | Sensitive | Description | | --- | --- | --- | -| `server_url` | no | Base server URL (e.g. `https://syslog.example.com`) | +| `server_url` | no | Base server URL (e.g. `https://cortex.example.com`) | | `api_token` | yes | Bearer token used by the plugin MCP client; enforced by the server unless `no_auth=true` | | `no_auth` | no | Explicit no-auth mode; non-loopback server binds also require `CORTEX_TRUSTED_GATEWAY_NO_AUTH=true` | | `is_server` | no | Whether this host owns the Docker Compose deployment | -These values are interpolated into `plugins/syslog/.mcp.json` via `${user_config.*}` syntax. See [plugin/CONFIG.md](plugin/CONFIG.md) for details. +These values are interpolated into `plugins/cortex/mcp.json` via `${user_config.*}` syntax. See [plugin/CONFIG.md](plugin/CONFIG.md) for details. ## .env.example conventions diff --git a/docs/GUARDRAILS.md b/docs/GUARDRAILS.md index 72cae584..947e6752 100644 --- a/docs/GUARDRAILS.md +++ b/docs/GUARDRAILS.md @@ -128,7 +128,7 @@ sudo iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to-port 1514 When exposing MCP over HTTPS via SWAG: - Add auth at the proxy layer or set `CORTEX_TOKEN` - Add public reverse-proxy hostnames to `CORTEX_ALLOWED_HOSTS` so RMCP Host validation accepts them -- See `docs/syslog.subdomain.conf` for a working nginx config +- Use `/config/nginx/proxy-confs/cortex.subdomain.conf` on the SWAG host, or an equivalent nginx vhost ## Input handling diff --git a/docs/INVENTORY.md b/docs/INVENTORY.md index ed948f63..13935a0c 100644 --- a/docs/INVENTORY.md +++ b/docs/INVENTORY.md @@ -64,7 +64,7 @@ that registry by `src/mcp/schemas.rs::tool_definitions()`. | `help` | Returns markdown documentation for all actions | no | Most MCP actions are read-only. `ack_error`, `unack_error`, and -`notifications_test` require `syslog:admin`; they mutate acknowledgement/audit +`notifications_test` require `cortex:admin`; they mutate acknowledgement/audit or notification state through service-owned actor and safety policy. ## Direct CLI commands @@ -169,6 +169,17 @@ and compact Docker inspect data including container status/health, image, published ports, networks, mounts, compose/route labels, and environment key names only. It does not store environment values. +SSH targets are validated before invoking OpenSSH, option-like hosts are +rejected, and the command builder inserts `--` before the host argument. +Inventory collectors and remote Docker event streams share strict host-key +defaults, a fleet-wide concurrency budget, and retry backoff. Deploy helpers +share host validation, the `--` delimiter, and the host-key argument policy, but +they do not use the inventory retry/concurrency context. The default is +`StrictHostKeyChecking=yes`; bootstrap TOFU is available only when explicitly +opted in with `CORTEX_INVENTORY_SSH_TRUST_ON_FIRST_USE=true`. Set +`CORTEX_INVENTORY_SSH_KNOWN_HOSTS` when automation should use a managed +known-hosts file. + Optional provider collectors are activated only when their URL/credential env vars are present. Supported media prefixes are `SONARR`, `RADARR`, `PROWLARR`, `SABNZBD`, `QBITTORRENT`, `PLEX`, `TAUTULLI`, and `OVERSEERR`. @@ -189,10 +200,10 @@ safe evidence samples, map-native `next_queries`, and graph `proof_queries`. | Surface | Present | Path | | --- | --- | --- | -| Skills | yes | `plugins/syslog/skills/` | +| Skills | yes | `plugins/cortex/skills/` | | Agents | no | -- | | Commands | no | -- | -| Hooks | yes | `plugins/syslog/hooks/` | +| Hooks | yes | `plugins/cortex/hooks/` | | Channels | no | -- | | Output styles | no | -- | | Schedules | no | -- | diff --git a/docs/OAUTH.md b/docs/OAUTH.md index 367f1a11..fc421f28 100644 --- a/docs/OAUTH.md +++ b/docs/OAUTH.md @@ -18,7 +18,7 @@ Client (browser/Claude) │ cortex HTTP :3100 │ Bearer static ──▶ constant-time compare │ │ │ │ RMCP tool dispatch │ - │ scope check (syslog:read) │ + │ scope check (cortex:read) │ │ → SyslogService / SQLite │ └────────────────────────────────────┘ @@ -39,7 +39,7 @@ Intentionally not mounted: 1. Client sends unauthenticated request to `/mcp` → receives `401 WWW-Authenticate: Bearer resource_metadata="…"`. 2. Client fetches `/.well-known/oauth-protected-resource` to discover the authorization server. 3. Client fetches `/.well-known/oauth-authorization-server` for the full metadata document. -4. Client constructs an `/authorize` URL (PKCE S256, `scope=syslog:read`), opens in browser. +4. Client constructs an `/authorize` URL (PKCE S256, `scope=cortex:read`), opens in browser. 5. User authenticates with Google; Google redirects to `/auth/google/callback`. 6. Server validates the Google email against `admin_email` plus any lab-auth `allowed_users` rows, issues an RS256 access token (1h TTL) and a refresh token (8h TTL). 7. Client uses `POST /token?grant_type=refresh_token` to obtain new access tokens without re-prompting. @@ -62,7 +62,7 @@ Intentionally not mounted: | Variable | Required | Description | |----------|----------|-------------| | `CORTEX_AUTH_MODE` | yes | Set to `oauth` to activate | -| `CORTEX_PUBLIC_URL` | yes | Base URL (e.g. `https://syslog.example.com`). Sets issuer + audience. | +| `CORTEX_PUBLIC_URL` | yes | Base URL (e.g. `https://cortex.example.com`). Sets issuer + audience. | | `CORTEX_GOOGLE_CLIENT_ID` | yes | From Google Console | | `CORTEX_GOOGLE_CLIENT_SECRET` | yes | From Google Console | | `CORTEX_AUTH_ADMIN_EMAIL` | yes | Bootstrap allowed Google account | @@ -76,7 +76,7 @@ These are **not** env vars — they go in `config.toml`: ```toml [mcp.auth] mode = "oauth" -public_url = "https://syslog.example.com" +public_url = "https://cortex.example.com" google_client_id = "..." # overridden by CORTEX_GOOGLE_CLIENT_ID google_client_secret = "..." # overridden by CORTEX_GOOGLE_CLIENT_SECRET @@ -108,6 +108,7 @@ disable_static_token_with_oauth = true # default: true - **`admin_email` is required**. It is the only config-backed OAuth email gate cortex passes into lab-auth today. lab-auth also honors rows in its `allowed_users` table. Startup rejects OAuth configs with a blank `admin_email`, and also rejects non-empty config-level `allowed_emails` until cortex can pass or enforce that list. - **`disable_static_token_with_oauth` defaults to `true` for `/mcp`**. OAuth-mode `/mcp` rejects `CORTEX_TOKEN` by default. Set `CORTEX_AUTH_DISABLE_STATIC_TOKEN_WITH_OAUTH=false` or `disable_static_token_with_oauth = false` in config.toml for break-glass bearer access. - **Non-loopback OAuth deployments still need `CORTEX_TOKEN` for OTLP `/v1/logs` unless OTLP exposure is loopback-only or service auth is explicitly disabled behind an upstream auth layer.** OTLP ingest does not accept OAuth JWTs today. +- **OAuth file-permission checks are Unix-oriented and fail closed on non-Unix platforms.** Run OAuth mode on Linux/Unix until cortex grows audited non-Unix ACL validation. - **Stdio mode always uses LoopbackDev**. `cargo run -- mcp` ignores the auth config entirely — no credentials are needed or enforced. - **Docker bind-mount ownership**. `auth.db` and `auth-jwt.pem` are written by the container UID. Host-side backup scripts may need `sudo` or a sidecar copy step. - **`/register` is never mounted**. cortex supports authorization-code OAuth routes but disables dynamic client registration. diff --git a/docs/README.md b/docs/README.md index 9262f1dd..9b6b4123 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,10 +1,10 @@ -# Syslog MCP Documentation +# Cortex Documentation Complete documentation for `cortex` -- a Rust syslog receiver and MCP server for homelab log intelligence. ## Directory index -### Root-level docs (this directory) +### Authoritative current docs | File | Purpose | | --- | --- | @@ -15,9 +15,13 @@ Complete documentation for `cortex` -- a Rust syslog receiver and MCP server for | `api.md` | REST API endpoint matrix (22 routes), versioning, perf, threat model, response caps, VACUUM caveats | | `architecture.md` | Caller → DB diagram (HTTP CLI default + direct-SQLite consumers) | | `rollout.md` | Manual v0.26 upgrade playbook for HTTP CLI cutover | -| `CHECKLIST.md` | Pre-release quality checklist -- version sync, security, CI, registry | +| `CHECKLIST.md` | Supplemental pre-release audit checklist -- current version policy and release gates point to `RELEASE.md` | | `GUARDRAILS.md` | Security guardrails -- credentials, Docker, auth, input handling | | `INVENTORY.md` | Component inventory -- tools, env vars, surfaces, dependencies | +| `OAUTH.md` | OAuth/JWT operator configuration and runtime model | +| `RUST.md` | Rust toolchain and rmcp dependency intent | +| `SECURITY.md` | Consolidated operator trust model | +| `RELEASE.md` | Release gates: hermetic CI versus live fleet checks | ### Subdirectories @@ -29,7 +33,12 @@ Complete documentation for `cortex` -- a Rust syslog receiver and MCP server for | `stack/` | Technology stack docs: prerequisites, architecture, Rust dependencies | | `upstream/` | Upstream service docs (cortex is self-contained -- no external API) | -### Preserved directories +### Preserved and archival directories + +Files in these directories are useful historical context, but they are not the +source of truth for current command names, plugin paths, auth scopes, release +version policy, or install examples. Prefer the authoritative docs above for +operator instructions. | Directory | Scope | | --- | --- | diff --git a/docs/RELEASE.md b/docs/RELEASE.md new file mode 100644 index 00000000..03a9139d --- /dev/null +++ b/docs/RELEASE.md @@ -0,0 +1,67 @@ +# Release Checklist + +Use this checklist before merging release-bound work. CI covers hermetic gates; +live fleet gates require a running cortex deployment and explicit operator +intent. + +## Hermetic Gates + +Run from the repo root: + +```bash +cargo fmt -- --check +cargo test +cargo clippy --all-targets -- -D warnings +cargo deny check +bash scripts/check-version-sync.sh +bash scripts/check-plugin-manifest-versions.sh +bash scripts/check-agent-memory-symlinks.sh +bash scripts/check-public-identity.sh +git diff --check +``` + +For release commits, also require: + +```bash +bash scripts/check-version-sync.sh --require-changelog +``` + +Version-bearing files are `Cargo.toml`, `server.json`, `mcpb/manifest.json`, +`Cargo.lock`, and `CHANGELOG.md`. Plugin manifests are intentionally +unversioned. + +## Live Gates + +Run these only against an intended test or production deployment: + +```bash +bash tests/test_live.sh +bash scripts/smoke-test.sh +bash scripts/smoke-test-http.sh +bash tests/mcporter/test-tools.sh +``` + +Live Docker ingest validation requires configured docker-socket-proxy endpoints +and `CORTEX_DOCKER_INGEST_ENABLED=true` with `CORTEX_DOCKER_HOSTS` set. + +Live SSH inventory validation requires configured SSH aliases or +`CORTEX_INVENTORY_SSH_HOSTS`, strict known-hosts coverage, and any intentional +TOFU bootstrap set explicitly with `CORTEX_INVENTORY_SSH_TRUST_ON_FIRST_USE=true`. + +Fleet drop-in deployment is intentionally outside hermetic CI. Validate first +with: + +```bash +cortex compose doctor +cortex inventory refresh --json +``` + +Then use the `cortex-deploy-dropins` plugin skill or the documented deploy +workflow only when the target `fleet_hosts` list is correct and reachable. + +## Commit Policy + +Every feature branch push bumps the version according to the repo policy in +`CLAUDE.md`. Patch bumps are appropriate for fixes, docs, CI, test, and policy +work. `CHANGELOG.md` must describe the operator-visible behavior, not just the +file list. diff --git a/docs/RUST.md b/docs/RUST.md index 6ef82253..39d082fe 100644 --- a/docs/RUST.md +++ b/docs/RUST.md @@ -45,3 +45,12 @@ explicit `./target` exclusion. All other settings (mold linker, profile tuning, Cranelift) are inherited from the global config. This repo has no xtask crate, so no `[alias]` section is needed. + +## rmcp version intent + +`Cargo.toml` declares `rmcp = "1.6.0"` as the supported lower bound for the +HTTP and stdio MCP API surface cortex uses. Cargo may resolve a newer compatible +`1.x` release in `Cargo.lock` (currently the lockfile resolves `rmcp 1.7.0`). +That is intentional semver behavior, not a mismatch. Do not pin the manifest to +the lockfile version unless cortex starts depending on an API that requires that +newer release. diff --git a/docs/SECURITY.md b/docs/SECURITY.md new file mode 100644 index 00000000..36e25855 --- /dev/null +++ b/docs/SECURITY.md @@ -0,0 +1,64 @@ +# Cortex Security Model + +This document collects operator-facing trust assumptions that are otherwise +spread across the code and setup docs. + +## Surfaces + +| Surface | Default | Trust boundary | +| --- | --- | --- | +| Syslog UDP/TCP `:1514` | unauthenticated | any reachable sender can submit frames; restrict by bind address, firewall, or `CORTEX_ALLOWED_SOURCE_CIDRS` | +| MCP HTTP `/mcp` | bearer auth when `CORTEX_TOKEN` is set | `cortex:read` for read actions, `cortex:admin` for write/admin actions | +| OAuth/JWT | disabled unless `CORTEX_AUTH_MODE=oauth` | Google identity plus the configured cortex allowlist; static token is disabled by default in OAuth mode | +| OTLP `/v1/logs` | loopback or bearer-token protected | OAuth JWTs do not authorize OTLP ingest today | +| Docker ingest | disabled unless configured | trust the docker-socket-proxy host and private network path; proxy must be read-only | +| SSH inventory/deploy | disabled unless hosts are configured | inventory and remote Docker events use validated host aliases, strict host keys, shared concurrency limits, and retry backoff; deploy uses the same host validation, `--` delimiter, and host-key argument policy | + +## Auth Scopes + +The current public scopes are `cortex:read` and `cortex:admin`. +`cortex:admin` satisfies `cortex:read`. Static bearer tokens receive +`cortex:read` by default; set `CORTEX_STATIC_TOKEN_ADMIN=true` only for +operators that need `ack_error`, `unack_error`, or `notifications_test`. + +## OAuth Platform Assumption + +OAuth/JWT key and database file permission checks are Linux/Unix oriented. +On non-Unix platforms cortex fails closed instead of silently accepting weaker +ACL validation. Treat OAuth mode as Linux-only unless non-Unix ACL validation is +implemented and tested. + +## SSH Host Keys + +Inventory and deploy SSH default to `StrictHostKeyChecking=yes`. Bootstrap TOFU +requires explicit opt-in with `CORTEX_INVENTORY_SSH_TRUST_ON_FIRST_USE=true`. +Use `CORTEX_INVENTORY_SSH_KNOWN_HOSTS` to point cortex at a managed known-hosts +file for fleet automation. + +## Identity Fields + +Syslog `hostname` is sender-claimed. For UniFi CEF messages, the stored +`hostname` comes from `UNIFIdeviceName` in the message body. `source_ip` is the +network-observed source identifier and is the better trust boundary for +correlation and inventory decisions. + +## Redaction Limits + +cortex redacts known credential-looking environment keys and sensitive setup +values before persisting inventory artifacts. Redaction is defensive, not a +formal data-loss-prevention guarantee. Treat log messages, transcript text, +paths, Docker metadata keys, and source-specific `metadata_json` as sensitive +operator data. + +## Dependency Exceptions + +`cargo deny check` ignores `RUSTSEC-2023-0071` for transitive RSA usage through +`lab-auth -> jsonwebtoken -> rsa`. The accepted path is JWT signing and +verification, not PKCS#1 v1.5 decryption. The owner is the cortex maintainer; +review the exception every release and remove it when `lab-auth` or +`jsonwebtoken` moves to a hardened dependency path. + +`cargo deny` also allows duplicate crate versions and wildcard git dependency +metadata because the current dependency graph includes transitive MCP/auth and +platform-target duplication plus a pinned `lab-auth` git revision. Source +allowlists, license policy, yanked crates, and advisory checks remain enforced. diff --git a/docs/SETUP.md b/docs/SETUP.md index ae6bd04d..9148731d 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -131,7 +131,7 @@ logger -n localhost -P 1514 --tcp "test from $(hostname)" curl -s -X POST http://localhost:3100/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ - -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"syslog","arguments":{"action":"tail","n":5}}}' | jq . + -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"tail","n":5}}}' | jq . ``` ## 8. Install as Claude Code plugin diff --git a/docs/contracts/mcp-actions-current.md b/docs/contracts/mcp-actions-current.md index 8695e5bd..7cbbe1c8 100644 --- a/docs/contracts/mcp-actions-current.md +++ b/docs/contracts/mcp-actions-current.md @@ -9,7 +9,7 @@ markdown file: - `src/mcp/actions.rs::ACTION_SPECS` registers action names, scopes, costs, and descriptions. - `src/mcp/actions.rs::action_names()` derives the schema enum. - `src/mcp/schemas.rs::tool_definitions()` builds `tools/list` and `cortex://schema/mcp-tool`. -- `src/mcp/tools.rs::tool_syslog()` dispatches handlers. +- `src/mcp/tools.rs::tool_cortex()` dispatches handlers. - `src/app/models.rs` defines typed request and response payloads. The MCP server exposes a single tool named `cortex`; every operation is selected @@ -21,56 +21,61 @@ Existing action names, required parameters, caps/defaults, and top-level response keys are stable. Renaming, removing, or tightening them is a breaking change. Adding optional parameters or optional response fields is non-breaking. -Most actions require `syslog:read` when auth is mounted. `ack_error`, -`unack_error`, and `notifications_test` require `syslog:admin`. `help` has no +Most actions require `cortex:read` when auth is mounted. `ack_error`, +`unack_error`, and `notifications_test` require `cortex:admin`. `help` has no action-level scope requirement, though the protected endpoint still requires transport auth when configured. ## Current Action Index -The live registry currently contains 40 actions: +The live registry currently contains 44 actions: | Action | Scope | Cost | Purpose | | --- | --- | --- | --- | -| `search` | `syslog:read` | cheap | Full-text search over syslog messages | -| `filter` | `syslog:read` | cheap | Filter logs by indexed fields without FTS5 | -| `tail` | `syslog:read` | cheap | Most recent log entries | -| `errors` | `syslog:read` | cheap | Error/warning summary | -| `hosts` | `syslog:read` | cheap | Known source hostnames | -| `correlate` | `syslog:read` | moderate | Time-window event correlation | -| `stats` | `syslog:read` | expensive | DB statistics and runtime observability | -| `status` | `syslog:read` | cheap | Lightweight health and runtime status | -| `apps` | `syslog:read` | cheap | Distinct application names with counts | -| `sessions` | `syslog:read` | cheap | AI transcript session inventory | -| `search_sessions` | `syslog:read` | cheap | FTS5 search over AI transcript sessions | -| `abuse` | `syslog:read` | moderate | Abuse-term hits with same-session context | -| `abuse_incidents` | `syslog:read` | moderate | Grouped abuse incident candidates | -| `abuse_investigate` | `syslog:read` | expensive | Evidence bundles for abuse incidents | -| `ai_correlate` | `syslog:read` | moderate | AI transcript anchors with nearby non-AI logs | -| `usage_blocks` | `syslog:read` | cheap | AI activity in 5-hour UTC blocks | -| `project_context` | `syslog:read` | moderate | AI project summary and recent entries | -| `list_ai_tools` | `syslog:read` | cheap | AI tools observed in transcripts | -| `list_ai_projects` | `syslog:read` | cheap | AI projects observed in transcripts | -| `source_ips` | `syslog:read` | cheap | Distinct source identifiers with counts | -| `timeline` | `syslog:read` | cheap | Bucketed log counts over time | -| `patterns` | `syslog:read` | expensive | Near-duplicate message template clusters | -| `context` | `syslog:read` | cheap | Logs surrounding a pivot id or timestamp | -| `get` | `syslog:read` | cheap | One log entry by id, including raw frame | -| `ingest_rate` | `syslog:read` | expensive | Recent ingest throughput and write-block state | -| `silent_hosts` | `syslog:read` | moderate | Hosts older than a staleness threshold | -| `clock_skew` | `syslog:read` | expensive | Per-host received_at minus timestamp distribution | -| `anomalies` | `syslog:read` | expensive | Recent vs baseline volume/error comparison | -| `compare` | `syslog:read` | expensive | Side-by-side comparison of two time ranges | -| `compose_status` | `syslog:read` | moderate | Redacted self Compose status projection | -| `compose_doctor` | `syslog:read` | expensive | Strict self Compose health diagnostics | -| `unaddressed_errors` | `syslog:read` | moderate | Unacknowledged repeating error signatures | -| `notifications_recent` | `syslog:read` | cheap | Recent notification firings | -| `similar_incidents` | `syslog:read` | moderate | FTS5 historical incident clusters | -| `ask_history` | `syslog:read` | moderate | AI transcript history with nearby log context | -| `incident_context` | `syslog:read` | moderate | Window bundle: log aggregates, errors, AI sessions | -| `ack_error` | `syslog:admin` | write | Acknowledge an error signature | -| `unack_error` | `syslog:admin` | write | Revoke an error acknowledgement | -| `notifications_test` | `syslog:admin` | write | Send a test Apprise notification | +| `search` | `cortex:read` | cheap | Full-text search over syslog messages | +| `filter` | `cortex:read` | cheap | Filter logs by indexed fields without FTS5 | +| `tail` | `cortex:read` | cheap | Most recent log entries | +| `errors` | `cortex:read` | cheap | Error/warning summary | +| `hosts` | `cortex:read` | cheap | Known source hostnames | +| `map` | `cortex:read` | moderate | Cached homelab inventory plus graph-backed topology answers | +| `host_state` | `cortex:read` | moderate | Latest bounded heartbeat state for one host | +| `fleet_state` | `cortex:read` | expensive | Fleet-wide heartbeat snapshot with pressure flags | +| `correlate` | `cortex:read` | moderate | Time-window event correlation | +| `correlate_state` | `cortex:read` | expensive | Correlate logs with heartbeat summaries around a reference time | +| `stats` | `cortex:read` | expensive | DB statistics and runtime observability | +| `status` | `cortex:read` | cheap | Lightweight health and runtime status | +| `apps` | `cortex:read` | cheap | Distinct application names with counts | +| `sessions` | `cortex:read` | cheap | AI transcript session inventory | +| `search_sessions` | `cortex:read` | cheap | FTS5 search over AI transcript sessions | +| `abuse` | `cortex:read` | moderate | Abuse-term hits with same-session context | +| `abuse_incidents` | `cortex:read` | moderate | Grouped abuse incident candidates | +| `abuse_investigate` | `cortex:read` | expensive | Evidence bundles for abuse incidents | +| `ai_correlate` | `cortex:read` | moderate | AI transcript anchors with nearby non-AI logs | +| `usage_blocks` | `cortex:read` | cheap | AI activity in 5-hour UTC blocks | +| `project_context` | `cortex:read` | moderate | AI project summary and recent entries | +| `list_ai_tools` | `cortex:read` | cheap | AI tools observed in transcripts | +| `list_ai_projects` | `cortex:read` | cheap | AI projects observed in transcripts | +| `source_ips` | `cortex:read` | cheap | Distinct source identifiers with counts | +| `timeline` | `cortex:read` | cheap | Bucketed log counts over time | +| `patterns` | `cortex:read` | expensive | Near-duplicate message template clusters | +| `context` | `cortex:read` | cheap | Logs surrounding a pivot id or timestamp | +| `get` | `cortex:read` | cheap | One log entry by id, including raw frame | +| `ingest_rate` | `cortex:read` | expensive | Recent ingest throughput and write-block state | +| `silent_hosts` | `cortex:read` | moderate | Hosts older than a staleness threshold | +| `clock_skew` | `cortex:read` | expensive | Per-host received_at minus timestamp distribution | +| `anomalies` | `cortex:read` | expensive | Recent vs baseline volume/error comparison | +| `compare` | `cortex:read` | expensive | Side-by-side comparison of two time ranges | +| `compose_status` | `cortex:read` | moderate | Redacted self Compose status projection | +| `compose_doctor` | `cortex:read` | expensive | Strict self Compose health diagnostics | +| `unaddressed_errors` | `cortex:read` | moderate | Unacknowledged repeating error signatures | +| `notifications_recent` | `cortex:read` | cheap | Recent notification firings | +| `similar_incidents` | `cortex:read` | moderate | FTS5 historical incident clusters | +| `ask_history` | `cortex:read` | moderate | AI transcript history with nearby log context | +| `incident_context` | `cortex:read` | moderate | Window bundle: log aggregates, errors, AI sessions | +| `graph` | `cortex:read` | moderate | Entity lookup and one-hop graph neighborhoods | +| `ack_error` | `cortex:admin` | write | Acknowledge an error signature | +| `unack_error` | `cortex:admin` | write | Revoke an error acknowledgement | +| `notifications_test` | `cortex:admin` | write | Send a test Apprise notification | | `help` | none | cheap | Markdown action reference | ## Schema Shape @@ -79,10 +84,10 @@ The generated MCP schema is an action-dispatched flat JSON schema. The `action` enum is derived from `ACTION_SPECS`; shared properties are declared at the top level and action handlers perform per-action validation. -The runtime schema also includes syslog-specific metadata: +The runtime schema also includes cortex-specific metadata: -- `x-syslog-action-metadata`: action names, costs, and descriptions. -- `x-syslog-agent-guidance`: cost ordering and suggested first-pass actions. +- `x-cortex-action-metadata`: action names, costs, and descriptions. +- `x-cortex-agent-guidance`: cost ordering and suggested first-pass actions. ## Response Envelope diff --git a/docs/mcp/CONNECT.md b/docs/mcp/CONNECT.md index e84b2ef5..09d9b9bd 100644 --- a/docs/mcp/CONNECT.md +++ b/docs/mcp/CONNECT.md @@ -82,7 +82,7 @@ run cleanup jobs, or require `CORTEX_TOKEN`. { "mcpServers": { "cortex": { - "command": "/path/to/syslog", + "command": "/path/to/cortex", "args": ["mcp"], "env": { "CORTEX_DB_PATH": "/data/cortex.db", @@ -107,7 +107,7 @@ The generated `dist/cortex--linux.mcpb` bundles the release `cortex` binary and launches it as: ```bash -server/syslog mcp +server/cortex mcp ``` The bundle is query-only. It reads `cortex.db` from the configured data @@ -171,7 +171,7 @@ curl -s -X POST http://localhost:3100/mcp \ curl -s -X POST http://localhost:3100/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ - -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"syslog","arguments":{"action":"stats"}}}' + -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"stats"}}}' ``` If connection fails, check: diff --git a/docs/mcp/DEPLOY.md b/docs/mcp/DEPLOY.md index 86401bf5..2c6e50f7 100644 --- a/docs/mcp/DEPLOY.md +++ b/docs/mcp/DEPLOY.md @@ -33,7 +33,7 @@ The installed binary is `cortex`. Use `cortex mcp` for local MCP clients that re curl -fsSL https://raw.githubusercontent.com/jmagar/cortex/main/install.sh | sh ``` -The installer installs the `cortex` binary to `~/.local/bin/syslog`, then runs +The installer installs the `cortex` binary to `~/.local/bin/cortex`, then runs `cortex setup`. Setup owns the shared Docker-only runtime layout: | Path | Purpose | @@ -170,7 +170,8 @@ Port 1514 is used instead of the standard syslog port 514 to avoid needing root ## SWAG reverse proxy -See `docs/syslog.subdomain.conf` for a working nginx config that exposes MCP over HTTPS at `https://cortex.tootie.tv/mcp`. +Use `/config/nginx/proxy-confs/cortex.subdomain.conf` on the SWAG host, or an +equivalent nginx vhost, to expose MCP over HTTPS at `https://cortex.tootie.tv/mcp`. The MCP endpoint uses RMCP Streamable HTTP in stateless JSON-response mode. Clients use `POST /mcp`; `GET` and `DELETE` on `/mcp` are not supported after diff --git a/docs/mcp/DEV.md b/docs/mcp/DEV.md index bf23dd0c..d2487e4a 100644 --- a/docs/mcp/DEV.md +++ b/docs/mcp/DEV.md @@ -50,7 +50,7 @@ cortex/ curl -s -X POST http://localhost:3100/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ - -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"syslog","arguments":{"action":"tail","n":10}}}' + -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"tail","n":10}}}' ``` 4. **Run checks**: ```bash @@ -64,11 +64,11 @@ cortex/ ## Adding a new MCP action 1. **Register the action** -- add an `ActionSpec` row to `src/mcp/actions.rs::ACTION_SPECS`. The schema enum and scope checks are derived from this table. -2. **Add adapter entry** -- add a match arm in `tool_syslog()`. +2. **Add adapter entry** -- add a match arm in `tool_cortex()`. 3. **Implement handler** -- write an async function that calls `SyslogService`. 4. **Add database query** -- implement the query function in `src/db.rs` with parameterized SQL. 5. **Add sidecar unit tests** -- place tests in the relevant `src/_tests.rs` file and keep the source module limited to the `#[cfg(test)] #[path = "..._tests.rs"] mod tests;` hook. -6. **Update syslog help** -- add the action to the help text in `tool_syslog_help()`. +6. **Update cortex help** -- add the action to the help text in `tool_cortex_help()`. 7. **Update public docs** -- refresh `docs/mcp/TOOLS.md`, `docs/mcp/SCHEMA.md`, `docs/mcp/TESTS.md`, and relevant skill docs. 8. **Update plugin manifests** -- keep the public tool name as `cortex`. @@ -95,9 +95,9 @@ RUST_LOG=cortex=debug,tower_http=info cargo run ### mcporter testing ```bash -mcporter list syslog --config config/mcporter.json -mcporter call --config config/mcporter.json syslog.cortex action=stats -mcporter call --config config/mcporter.json syslog.cortex action=tail n=10 +mcporter list cortex --config config/mcporter.json +mcporter call --config config/mcporter.json cortex.cortex action=stats +mcporter call --config config/mcporter.json cortex.cortex action=tail n=10 ``` ### MCP Inspector diff --git a/docs/mcp/ELICITATION.md b/docs/mcp/ELICITATION.md index c9927466..b2b290ce 100644 --- a/docs/mcp/ELICITATION.md +++ b/docs/mcp/ELICITATION.md @@ -8,13 +8,13 @@ Elicitation is an MCP protocol capability that allows servers to request informa cortex is a self-contained syslog receiver with no interactive first-run prompts. There are no upstream credentials to collect via MCP elicitation. All configuration is handled via environment variables, `config.toml`, and plugin `userConfig`. -Most MCP actions are read-only and require `syslog:read` when auth is mounted. A small set of state-changing/admin actions exists: +Most MCP actions are read-only and require `cortex:read` when auth is mounted. A small set of state-changing/admin actions exists: - `ack_error` - `unack_error` - `notifications_test` -Those actions require `syslog:admin`; they do not use elicitation confirmation gates. The action registry and scope mapping live in `src/mcp/actions.rs::ACTION_SPECS`. +Those actions require `cortex:admin`; they do not use elicitation confirmation gates. The action registry and scope mapping live in `src/mcp/actions.rs::ACTION_SPECS`. ## Configuration entry points diff --git a/docs/mcp/MCPORTER.md b/docs/mcp/MCPORTER.md index 960bf59d..affdd098 100644 --- a/docs/mcp/MCPORTER.md +++ b/docs/mcp/MCPORTER.md @@ -31,7 +31,7 @@ mcporter config is at `config/mcporter.json`: ```json { - "servers": { + "mcpServers": { "cortex": { "transport": "http", "url": "http://localhost:3100/mcp" @@ -44,16 +44,16 @@ mcporter config is at `config/mcporter.json`: ```bash # List available tools -mcporter list syslog --config config/mcporter.json +mcporter list cortex --config config/mcporter.json # Call actions through the single cortex tool -mcporter call --config config/mcporter.json syslog.cortex action=stats -mcporter call --config config/mcporter.json syslog.cortex action=tail n=10 -mcporter call --config config/mcporter.json syslog.cortex action=search query=error limit=5 -mcporter call --config config/mcporter.json syslog.cortex action=hosts -mcporter call --config config/mcporter.json syslog.cortex action=errors -mcporter call --config config/mcporter.json syslog.cortex action=status -mcporter call --config config/mcporter.json syslog.cortex action=help +mcporter call --config config/mcporter.json cortex.cortex action=stats +mcporter call --config config/mcporter.json cortex.cortex action=tail n=10 +mcporter call --config config/mcporter.json cortex.cortex action=search query=error limit=5 +mcporter call --config config/mcporter.json cortex.cortex action=hosts +mcporter call --config config/mcporter.json cortex.cortex action=errors +mcporter call --config config/mcporter.json cortex.cortex action=status +mcporter call --config config/mcporter.json cortex.cortex action=help ``` ## Test assertions @@ -77,8 +77,8 @@ The smoke test validates: ``` PASS: health endpoint returns ok - PASS: syslog search returns count field - FAIL: syslog tail count should be <= 10, got 50 + PASS: cortex search returns count field + FAIL: cortex tail count should be <= 10, got 50 --- 30 assertions: 29 PASS, 1 FAIL ``` diff --git a/docs/mcp/PATTERNS.md b/docs/mcp/PATTERNS.md index ba8a37a3..ca634048 100644 --- a/docs/mcp/PATTERNS.md +++ b/docs/mcp/PATTERNS.md @@ -10,12 +10,12 @@ cortex exposes one public MCP tool, `cortex`, and dispatches on the required ```rust async fn execute_tool(state: &AppState, name: &str, args: Value) -> anyhow::Result { match name { - "syslog" => tool_syslog(state, args).await, + "cortex" => tool_cortex(state, args).await, _ => Err(anyhow::anyhow!("Unknown tool: {name}")), } } -async fn tool_syslog(state: &AppState, args: Value) -> anyhow::Result { +async fn tool_cortex(state: &AppState, args: Value) -> anyhow::Result { match string_arg(&args, "action").as_deref() { Some("search") => tool_search_logs(state, args).await, Some("tail") => tool_tail_logs(state, args).await, @@ -24,7 +24,7 @@ async fn tool_syslog(state: &AppState, args: Value) -> anyhow::Result { Some("correlate") => tool_correlate_events(state, args).await, Some("stats") => tool_get_stats(state, args).await, Some("status") => tool_get_status(state, args).await, - Some("help") => tool_syslog_help().await, + Some("help") => tool_cortex_help().await, _ => Err(anyhow::anyhow!("action is required")), } } diff --git a/docs/mcp/PUBLISH.md b/docs/mcp/PUBLISH.md index a6a36cea..7020e7e7 100644 --- a/docs/mcp/PUBLISH.md +++ b/docs/mcp/PUBLISH.md @@ -19,13 +19,14 @@ All version-bearing files must match. Update together: | File | Field | | --- | --- | | `Cargo.toml` | `version = "X.Y.Z"` in `[package]` | -| `.claude-plugin/plugin.json` | `"version": "X.Y.Z"` | -| `.codex-plugin/plugin.json` | `"version": "X.Y.Z"` | -| `gemini-extension.json` | `"version": "X.Y.Z"` | | `server.json` | `"version": "X.Y.Z"` | | `mcpb/manifest.json` | `"version": "X.Y.Z"` | | `CHANGELOG.md` | New entry under `## X.Y.Z` | +Plugin manifests such as `.claude-plugin/plugin.json` are intentionally +unversioned. `scripts/check-plugin-manifest-versions.sh` is the guardrail that +prevents top-level plugin manifest `version` keys from coming back. + ## Publish workflow ```bash @@ -38,7 +39,7 @@ Steps executed: 2. Pull latest from origin 3. Read current version from `Cargo.toml` 4. Compute new version based on bump type -5. Update `Cargo.toml`, plugin manifests, and `gemini-extension.json` +5. Update `Cargo.toml`, `server.json`, `mcpb/manifest.json`, and `CHANGELOG.md` 6. Run `cargo check` to update `Cargo.lock` 7. Commit: `release: vX.Y.Z` 8. Tag: `vX.Y.Z` @@ -60,14 +61,13 @@ MCP Registry metadata at repo root: ```json { "name": "tv.tootie/cortex", - "title": "Syslog MCP", + "title": "Cortex", "description": "Syslog receiver and MCP server for homelab log intelligence.", - "version": "0.21.7", + "version": "X.Y.Z", "packages": [ { "registryType": "oci", - "identifier": "ghcr.io/jmagar/cortex:0.21.7", - "version": "0.21.7" + "identifier": "ghcr.io/jmagar/cortex:vX.Y.Z" } ] } diff --git a/docs/mcp/SCHEMA.md b/docs/mcp/SCHEMA.md index 72e12957..7cb6109b 100644 --- a/docs/mcp/SCHEMA.md +++ b/docs/mcp/SCHEMA.md @@ -9,7 +9,7 @@ Current source of truth: - `src/mcp/actions.rs::ACTION_SPECS` registers every action, its scope, cost, and description. - `src/mcp/actions.rs::action_names()` derives the schema action enum from `ACTION_SPECS`. - `src/mcp/schemas.rs::tool_definitions()` builds the MCP `tools/list` definition and the `cortex://schema/mcp-tool` resource from that action table. -- `src/mcp/tools.rs::tool_syslog()` dispatches the action handlers. +- `src/mcp/tools.rs::tool_cortex()` dispatches the action handlers. - `src/app/models.rs` defines request and response structs for typed action payloads. `docs/mcp/SCHEMA.md` is a human-maintained reference for that generated runtime @@ -24,50 +24,50 @@ selects one of these 44 actions: | Action | Scope | Cost | Purpose | | --- | --- | --- | --- | -| `search` | `syslog:read` | cheap | Full-text search over syslog messages | -| `filter` | `syslog:read` | cheap | Filter logs by indexed fields without FTS5 | -| `tail` | `syslog:read` | cheap | Most recent log entries | -| `errors` | `syslog:read` | cheap | Error/warning summary | -| `hosts` | `syslog:read` | cheap | Known source hostnames | -| `map` | `syslog:read` | moderate | Cached homelab inventory plus graph-backed topology answers | -| `host_state` | `syslog:read` | moderate | Latest bounded heartbeat state for one host | -| `fleet_state` | `syslog:read` | expensive | Fleet-wide heartbeat snapshot with pressure flags | -| `correlate` | `syslog:read` | moderate | Time-window event correlation | -| `correlate_state` | `syslog:read` | expensive | Correlate logs with heartbeat summaries around a reference time | -| `stats` | `syslog:read` | expensive | DB statistics and runtime observability | -| `status` | `syslog:read` | cheap | Lightweight health and runtime status | -| `apps` | `syslog:read` | cheap | Distinct application names with counts | -| `sessions` | `syslog:read` | cheap | AI transcript session inventory | -| `search_sessions` | `syslog:read` | cheap | FTS5 search over AI transcript sessions | -| `abuse` | `syslog:read` | moderate | Abuse-term hits with same-session context | -| `abuse_incidents` | `syslog:read` | moderate | Grouped abuse incident candidates | -| `abuse_investigate` | `syslog:read` | expensive | Evidence bundles for abuse incidents | -| `ai_correlate` | `syslog:read` | moderate | AI transcript anchors with nearby non-AI logs | -| `usage_blocks` | `syslog:read` | cheap | AI activity in 5-hour UTC blocks | -| `project_context` | `syslog:read` | moderate | AI project summary and recent entries | -| `list_ai_tools` | `syslog:read` | cheap | AI tools observed in transcripts | -| `list_ai_projects` | `syslog:read` | cheap | AI projects observed in transcripts | -| `source_ips` | `syslog:read` | cheap | Distinct source identifiers with counts | -| `timeline` | `syslog:read` | cheap | Bucketed log counts over time | -| `patterns` | `syslog:read` | expensive | Near-duplicate message template clusters | -| `context` | `syslog:read` | cheap | Logs surrounding a pivot id or timestamp | -| `get` | `syslog:read` | cheap | One log entry by id, including raw frame | -| `ingest_rate` | `syslog:read` | expensive | Recent ingest throughput and write-block state | -| `silent_hosts` | `syslog:read` | moderate | Hosts older than a staleness threshold | -| `clock_skew` | `syslog:read` | expensive | Per-host received_at minus timestamp distribution | -| `anomalies` | `syslog:read` | expensive | Recent vs baseline volume/error comparison | -| `compare` | `syslog:read` | expensive | Side-by-side comparison of two time ranges | -| `compose_status` | `syslog:read` | moderate | Redacted self Compose status projection | -| `compose_doctor` | `syslog:read` | expensive | Strict self Compose health diagnostics | -| `unaddressed_errors` | `syslog:read` | moderate | Unacknowledged repeating error signatures | -| `notifications_recent` | `syslog:read` | cheap | Recent notification firings | -| `similar_incidents` | `syslog:read` | moderate | FTS5 historical incident clusters | -| `ask_history` | `syslog:read` | moderate | AI transcript history with nearby log context | -| `incident_context` | `syslog:read` | moderate | Window bundle: log aggregates, errors, AI sessions | -| `graph` | `syslog:read` | moderate | Entity lookup and one-hop graph neighborhoods | -| `ack_error` | `syslog:admin` | write | Acknowledge an error signature | -| `unack_error` | `syslog:admin` | write | Revoke an error acknowledgement | -| `notifications_test` | `syslog:admin` | write | Send a test Apprise notification | +| `search` | `cortex:read` | cheap | Full-text search over syslog messages | +| `filter` | `cortex:read` | cheap | Filter logs by indexed fields without FTS5 | +| `tail` | `cortex:read` | cheap | Most recent log entries | +| `errors` | `cortex:read` | cheap | Error/warning summary | +| `hosts` | `cortex:read` | cheap | Known source hostnames | +| `map` | `cortex:read` | moderate | Cached homelab inventory plus graph-backed topology answers | +| `host_state` | `cortex:read` | moderate | Latest bounded heartbeat state for one host | +| `fleet_state` | `cortex:read` | expensive | Fleet-wide heartbeat snapshot with pressure flags | +| `correlate` | `cortex:read` | moderate | Time-window event correlation | +| `correlate_state` | `cortex:read` | expensive | Correlate logs with heartbeat summaries around a reference time | +| `stats` | `cortex:read` | expensive | DB statistics and runtime observability | +| `status` | `cortex:read` | cheap | Lightweight health and runtime status | +| `apps` | `cortex:read` | cheap | Distinct application names with counts | +| `sessions` | `cortex:read` | cheap | AI transcript session inventory | +| `search_sessions` | `cortex:read` | cheap | FTS5 search over AI transcript sessions | +| `abuse` | `cortex:read` | moderate | Abuse-term hits with same-session context | +| `abuse_incidents` | `cortex:read` | moderate | Grouped abuse incident candidates | +| `abuse_investigate` | `cortex:read` | expensive | Evidence bundles for abuse incidents | +| `ai_correlate` | `cortex:read` | moderate | AI transcript anchors with nearby non-AI logs | +| `usage_blocks` | `cortex:read` | cheap | AI activity in 5-hour UTC blocks | +| `project_context` | `cortex:read` | moderate | AI project summary and recent entries | +| `list_ai_tools` | `cortex:read` | cheap | AI tools observed in transcripts | +| `list_ai_projects` | `cortex:read` | cheap | AI projects observed in transcripts | +| `source_ips` | `cortex:read` | cheap | Distinct source identifiers with counts | +| `timeline` | `cortex:read` | cheap | Bucketed log counts over time | +| `patterns` | `cortex:read` | expensive | Near-duplicate message template clusters | +| `context` | `cortex:read` | cheap | Logs surrounding a pivot id or timestamp | +| `get` | `cortex:read` | cheap | One log entry by id, including raw frame | +| `ingest_rate` | `cortex:read` | expensive | Recent ingest throughput and write-block state | +| `silent_hosts` | `cortex:read` | moderate | Hosts older than a staleness threshold | +| `clock_skew` | `cortex:read` | expensive | Per-host received_at minus timestamp distribution | +| `anomalies` | `cortex:read` | expensive | Recent vs baseline volume/error comparison | +| `compare` | `cortex:read` | expensive | Side-by-side comparison of two time ranges | +| `compose_status` | `cortex:read` | moderate | Redacted self Compose status projection | +| `compose_doctor` | `cortex:read` | expensive | Strict self Compose health diagnostics | +| `unaddressed_errors` | `cortex:read` | moderate | Unacknowledged repeating error signatures | +| `notifications_recent` | `cortex:read` | cheap | Recent notification firings | +| `similar_incidents` | `cortex:read` | moderate | FTS5 historical incident clusters | +| `ask_history` | `cortex:read` | moderate | AI transcript history with nearby log context | +| `incident_context` | `cortex:read` | moderate | Window bundle: log aggregates, errors, AI sessions | +| `graph` | `cortex:read` | moderate | Entity lookup and one-hop graph neighborhoods | +| `ack_error` | `cortex:admin` | write | Acknowledge an error signature | +| `unack_error` | `cortex:admin` | write | Revoke an error acknowledgement | +| `notifications_test` | `cortex:admin` | write | Send a test Apprise notification | | `help` | none | cheap | Markdown action reference | ## Schema Pattern @@ -76,12 +76,12 @@ The runtime tool definition is a flat action-dispatched JSON schema: ```json { - "name": "syslog", + "name": "cortex", "description": "Query cortex logs with action-based subcommands...", - "x-syslog-action-metadata": [ + "x-cortex-action-metadata": [ { "name": "search", "cost": "cheap", "description": "..." } ], - "x-syslog-agent-guidance": { + "x-cortex-agent-guidance": { "cost_order": ["cheap", "moderate", "expensive", "write"], "first_pass": ["status", "errors", "tail", "search", "timeline", "context"], "escalate_only_when_scoped": [ @@ -134,7 +134,7 @@ handler and service layers. | `limit`, `offset` | Action-specific bounds; `offset` is for `apps` and `source_ips` pagination | | `host_limit`, `per_host_limit`, `section_limit`, `include_sections` | Node and inventory-section bounds for `map`; `per_host_limit` is accepted for v1 compatibility and ignored by map v2 | | `mode`, `host`, `domain`, `service`, `answer_limit`, `evidence_sample_limit`, `payload_budget` | Map snapshot mode and graph-backed map answer controls: `host_services`, `domain_routes`, and `service_dependencies` | -| `mode`, `entity_id`, `entity_type`, `key`, `alias_type`, `alias_key`, `depth`, `evidence_sample_limit`, `payload_budget` | Graph lookup and one-hop neighborhood controls for `graph` | +| `mode`, `entity_id`, `entity_type`, `key`, `alias_type`, `alias_key`, `depth`, `evidence_id`, `evidence_sample_limit`, `payload_budget` | Graph controls. Targeted modes require exactly one lookup strategy: `entity_id`, `entity_type` + `key`, or `alias_type` + `alias_key`. `evidence` requires `evidence_id`. | ## Correlation Arguments @@ -148,15 +148,15 @@ See [CORRELATION.md](CORRELATION.md) for the full behavior matrix. | `similar_incidents` | `query`, `hostname`, `app_name`, `severity_min`, `from`, `to`, `window_minutes`, `limit` | | `ask_history` | `query`, `hostname`, `app_name`, `from`, `to`, `limit` | | `incident_context` | `from`, `to`, `hostname`, `app_name`, `severity_min`, `limit`; `query` is accepted by the request shape but intentionally ignored in v1 | -| `graph` | `mode=entity|around`; exact lookup with `entity_type` + `key`, alias lookup with `alias_type` + `alias_key`, or one-hop `around` by `entity_id`; optional `limit`, `depth=1`, `evidence_sample_limit`, `payload_budget` | +| `graph` | `mode=entity|around|explain|evidence`; entity/around/explain require exactly one target lookup strategy (`entity_id`, `entity_type` + `key`, or `alias_type` + `alias_key`); `around` accepts `depth=1` only; `explain` accepts `depth=1..3`; `evidence` requires `evidence_id`; optional `limit`, `evidence_sample_limit`, `payload_budget` | ## Validation Input validation is action-specific: - `action` is required and must match `ACTION_SPECS`. -- Read actions require `syslog:read` when auth is mounted. -- Admin actions require `syslog:admin`. +- Read actions require `cortex:read` when auth is mounted. +- Admin actions require `cortex:admin`. - `help` has no scope gate, but auth policy still applies when the endpoint is protected. - Numeric parameters are capped by each action. - Timestamp parameters are parsed as RFC3339 and normalized where needed. diff --git a/docs/mcp/TESTS.md b/docs/mcp/TESTS.md index 63855863..9a2d0a52 100644 --- a/docs/mcp/TESTS.md +++ b/docs/mcp/TESTS.md @@ -50,7 +50,7 @@ rows, not just empty response envelopes. the HTTP MCP endpoint for `search_sessions`, `abuse`, `abuse_incidents`, `abuse_investigate`, `usage_blocks`, `project_context`, `list_ai_tools`, and `list_ai_projects`. The AI smoke scripts resolve `CORTEX_BIN` first, then `cortex` on `PATH`, then -the repo-local debug binary at `target/debug/syslog`, so repo-local builds do +the repo-local debug binary at `target/debug/cortex`, so repo-local builds do not require an installed shell binary. Action registry covered by live/script references: `search`, `filter`, `tail`, `errors`, @@ -66,38 +66,38 @@ Action registry covered by live/script references: `search`, `filter`, `tail`, ` ```bash # List available tools -mcporter list syslog --config config/mcporter.json +mcporter list cortex --config config/mcporter.json # Call actions through the single cortex tool -mcporter call --config config/mcporter.json syslog.cortex action=stats -mcporter call --config config/mcporter.json syslog.cortex action=status -mcporter call --config config/mcporter.json syslog.cortex action=tail n=10 -mcporter call --config config/mcporter.json syslog.cortex action=search query=error limit=5 -mcporter call --config config/mcporter.json syslog.cortex action=hosts -mcporter call --config config/mcporter.json syslog.cortex action=host_state host_id=host-id -mcporter call --config config/mcporter.json syslog.cortex action=sessions -mcporter call --config config/mcporter.json syslog.cortex action=abuse terms=ai-smoke-authentication limit=5 -mcporter call --config config/mcporter.json syslog.cortex action=abuse_incidents limit=3 -mcporter call --config config/mcporter.json syslog.cortex action=abuse_investigate limit=1 -mcporter call --config config/mcporter.json syslog.cortex action=correlate_state reference_time=2026-01-01T00:00:00Z window_minutes=10 -mcporter call --config config/mcporter.json syslog.cortex action=ai_correlate project=/tmp/cortex-ai-smoke limit=2 events_per_anchor=3 -mcporter call --config config/mcporter.json syslog.cortex action=apps -mcporter call --config config/mcporter.json syslog.cortex action=source_ips -mcporter call --config config/mcporter.json syslog.cortex action=timeline -mcporter call --config config/mcporter.json syslog.cortex action=patterns -mcporter call --config config/mcporter.json syslog.cortex action=context hostname=host timestamp=2026-01-01T00:00:00Z -mcporter call --config config/mcporter.json syslog.cortex action=get id=1 -mcporter call --config config/mcporter.json syslog.cortex action=ingest_rate -mcporter call --config config/mcporter.json syslog.cortex action=silent_hosts -mcporter call --config config/mcporter.json syslog.cortex action=clock_skew -mcporter call --config config/mcporter.json syslog.cortex action=anomalies -mcporter call --config config/mcporter.json syslog.cortex action=compare a_from=2026-01-01T00:00:00Z a_to=2026-01-01T01:00:00Z b_from=2026-01-01T01:00:00Z b_to=2026-01-01T02:00:00Z -mcporter call --config config/mcporter.json syslog.cortex action=compose_status -mcporter call --config config/mcporter.json syslog.cortex action=compose_doctor -mcporter call --config config/mcporter.json syslog.cortex action=graph mode=entity entity_type=host key=example-host -mcporter call --config config/mcporter.json syslog.cortex action=graph mode=around entity_type=host key=example-host depth=1 -mcporter call --config config/mcporter.json syslog.cortex action=graph mode=explain entity_type=host key=example-host depth=2 -mcporter call --config config/mcporter.json syslog.cortex action=graph mode=evidence evidence_id=12345 +mcporter call --config config/mcporter.json cortex.cortex action=stats +mcporter call --config config/mcporter.json cortex.cortex action=status +mcporter call --config config/mcporter.json cortex.cortex action=tail n=10 +mcporter call --config config/mcporter.json cortex.cortex action=search query=error limit=5 +mcporter call --config config/mcporter.json cortex.cortex action=hosts +mcporter call --config config/mcporter.json cortex.cortex action=host_state host_id=host-id +mcporter call --config config/mcporter.json cortex.cortex action=sessions +mcporter call --config config/mcporter.json cortex.cortex action=abuse terms=ai-smoke-authentication limit=5 +mcporter call --config config/mcporter.json cortex.cortex action=abuse_incidents limit=3 +mcporter call --config config/mcporter.json cortex.cortex action=abuse_investigate limit=1 +mcporter call --config config/mcporter.json cortex.cortex action=correlate_state reference_time=2026-01-01T00:00:00Z window_minutes=10 +mcporter call --config config/mcporter.json cortex.cortex action=ai_correlate project=/tmp/cortex-ai-smoke limit=2 events_per_anchor=3 +mcporter call --config config/mcporter.json cortex.cortex action=apps +mcporter call --config config/mcporter.json cortex.cortex action=source_ips +mcporter call --config config/mcporter.json cortex.cortex action=timeline +mcporter call --config config/mcporter.json cortex.cortex action=patterns +mcporter call --config config/mcporter.json cortex.cortex action=context hostname=host timestamp=2026-01-01T00:00:00Z +mcporter call --config config/mcporter.json cortex.cortex action=get id=1 +mcporter call --config config/mcporter.json cortex.cortex action=ingest_rate +mcporter call --config config/mcporter.json cortex.cortex action=silent_hosts +mcporter call --config config/mcporter.json cortex.cortex action=clock_skew +mcporter call --config config/mcporter.json cortex.cortex action=anomalies +mcporter call --config config/mcporter.json cortex.cortex action=compare a_from=2026-01-01T00:00:00Z a_to=2026-01-01T01:00:00Z b_from=2026-01-01T01:00:00Z b_to=2026-01-01T02:00:00Z +mcporter call --config config/mcporter.json cortex.cortex action=compose_status +mcporter call --config config/mcporter.json cortex.cortex action=compose_doctor +mcporter call --config config/mcporter.json cortex.cortex action=graph mode=entity entity_type=host key=example-host +mcporter call --config config/mcporter.json cortex.cortex action=graph mode=around entity_type=host key=example-host depth=1 +mcporter call --config config/mcporter.json cortex.cortex action=graph mode=explain entity_type=host key=example-host depth=2 +mcporter call --config config/mcporter.json cortex.cortex action=graph mode=evidence evidence_id=12345 ``` For graph proof UX smoke, use a real bounded evidence id from @@ -145,31 +145,31 @@ curl http://localhost:3100/health curl -s -X POST http://localhost:3100/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ - -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"syslog","arguments":{"action":"tail","n":10}}}' + -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"tail","n":10}}}' # Search curl -s -X POST http://localhost:3100/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ - -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"syslog","arguments":{"action":"search","query":"error","limit":5}}}' + -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"search","query":"error","limit":5}}}' # Stats curl -s -X POST http://localhost:3100/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ - -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"syslog","arguments":{"action":"stats"}}}' + -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"stats"}}}' # Status curl -s -X POST http://localhost:3100/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ - -d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"syslog","arguments":{"action":"status"}}}' + -d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"status"}}}' ``` ## Testing checklist -- [ ] **All actions return expected shape** -- syslog search, syslog tail, syslog errors, syslog hosts, syslog host_state, syslog sessions, syslog correlate, syslog stats, syslog status, syslog help -- [ ] **AI session analytics return expected shape and seeded rows** -- syslog search_sessions, syslog abuse, syslog ai_correlate, syslog usage_blocks, syslog project_context, syslog list_ai_tools, syslog list_ai_projects +- [ ] **All actions return expected shape** -- cortex search, cortex tail, cortex errors, cortex hosts, cortex host_state, cortex sessions, cortex correlate, cortex stats, cortex status, cortex help +- [ ] **AI session analytics return expected shape and seeded rows** -- cortex search_sessions, cortex abuse, cortex ai_correlate, cortex usage_blocks, cortex project_context, cortex list_ai_tools, cortex list_ai_projects - [ ] **Auth: valid token** -- 200 with correct Bearer token - [ ] **Auth: invalid token** -- 401 Unauthorized - [ ] **Auth: no token when required** -- 401 Unauthorized diff --git a/docs/mcp/TOOLS.md b/docs/mcp/TOOLS.md index 10843ff1..06f53739 100644 --- a/docs/mcp/TOOLS.md +++ b/docs/mcp/TOOLS.md @@ -53,7 +53,7 @@ cortex exposes one MCP tool named `cortex`. The required | `graph` | Resolve graph entities, neighborhoods, evidence-backed explanations, and evidence proof rows | | `help` | Markdown reference for all actions | -## syslog search +## cortex search Full-text search across all syslog messages. Uses SQLite FTS5 with porter stemming. @@ -61,7 +61,7 @@ Required argument: `action = "search"` Optional arguments: `query`, `hostname`, `source_ip`, `severity`, `app_name`, `facility`, `process_id`, `from`, `to`, `limit`. -## syslog filter +## cortex filter Structured filter-only log retrieval. This action rejects `query`; use `search` for FTS5 message-body search. @@ -69,7 +69,7 @@ Required argument: `action = "filter"` Optional arguments: `hostname`, `source_ip`, `source_kind`, `tool`, `project`, `session_id`, `container`, `docker_host`, `stream`, `event_action`, `severity`, `app_name`, `facility`, `exclude_facility`, `process_id`, `from`, `to`, `received_from`, `received_to`, `limit`. -## syslog tail +## cortex tail Get the N most recent log entries. Equivalent to `tail -f` across all hosts. @@ -77,7 +77,7 @@ Required argument: `action = "tail"` Optional arguments: `hostname`, `source_ip`, `app_name`, `severity_min`, `n`. -## syslog errors +## cortex errors Get a summary of errors and warnings across all hosts in a time window, grouped by hostname and severity. @@ -87,13 +87,13 @@ Optional arguments: `from`, `to`, `group_by`. `group_by` currently supports `app_name` for hostname + app + severity grouping. -## syslog hosts +## cortex hosts List all hosts that have sent syslog messages. Required argument: `action = "hosts"` -## syslog map +## cortex map Return a bounded homelab infrastructure snapshot from `~/.cortex/inventory` plus live Cortex host/heartbeat overlay, or answer graph-backed topology @@ -112,7 +112,7 @@ Use `mode = "host_services"` with `host`, `mode = "domain_routes"` with `service` to get a `graph_answer` envelope with answer status, topology rows, safe evidence, map follow-ups, and graph proof queries. -## syslog host_state +## cortex host_state Return latest bounded heartbeat state for one host. @@ -120,7 +120,7 @@ Required argument: `action = "host_state"` plus either `host_id` or uniquely res Optional arguments: `since`, `limit` (default 1, max 100). -## syslog correlate_state +## cortex correlate_state Correlate non-AI logs with per-host heartbeat window summaries around a reference time. Bounded by default; never performs a full-history scan. @@ -135,7 +135,7 @@ max 500). Response includes the resolved `window`, a `heartbeat_summary` plus matching `logs` per host, and a `truncated` flag. -## syslog sessions +## cortex sessions List AI transcript sessions grouped by project, tool, session, and host. @@ -143,7 +143,7 @@ Required argument: `action = "sessions"` Optional arguments: `project`, `tool`, `hostname`, `from`, `to`, `limit`. -## syslog search_sessions +## cortex search_sessions Search AI transcript rows with FTS5 and return grouped session results ranked by relevance. @@ -151,7 +151,7 @@ Required arguments: `action = "search_sessions"`, `query` Optional arguments: `project`, `tool`, `from`, `to`, `limit`. -## syslog abuse +## cortex abuse Detect abuse in AI transcript rows and return the hit plus surrounding rows from the same AI session. @@ -163,7 +163,7 @@ Optional arguments: `project`, `tool`, `from`, `to`, `limit`, `before`, `after`, `terms` replaces the built-in abuse detector list when provided. `before` and `after` default to 2 and are capped at 20. -## syslog abuse_incidents +## cortex abuse_incidents Groups AI transcript abuse hits into scored incident candidates by `(project, tool, session_id, hostname)` within a configurable time window. Returns incidents ordered by priority score with labels: `low` / `medium` / `high` / `critical`. @@ -171,7 +171,7 @@ Response includes `incidents`, `total_incidents`, `candidate_rows`, `candidate_c Optional arguments: `project`, `tool`, `from`, `to`, `limit` (default 20, max 100), `window_minutes` (default 10, max 120), `terms`. -## syslog abuse_investigate +## cortex abuse_investigate Expands the top abuse incidents into deterministic evidence bundles. Each bundle includes transcript context before and after the incident, the abuse anchor entries, and nearby non-AI syslog/Docker logs in the correlation window. @@ -190,7 +190,7 @@ Response includes `evidence` (array of bundles), `total_incidents`, `truncated`. Optional arguments: `project`, `tool`, `from`, `to`, `limit` (default 3, max 10), `window_minutes`, `correlation_window_minutes` (default 5, max 120), `terms`. -## syslog ai_correlate +## cortex ai_correlate Use AI transcript rows as timeline anchors and pull nearby non-AI syslog, Docker, OTLP, and host events from the same database. Related logs explicitly @@ -205,7 +205,7 @@ Optional arguments: `project`, `tool`, `session_id`, `ai_query`, `log_query`, `limit` caps AI anchors at 50. `events_per_anchor` caps related non-AI rows at 200 per anchor. `window_minutes` searches before and after each AI timestamp. -## syslog usage_blocks +## cortex usage_blocks Bucket AI activity into deterministic 5-hour UTC windows. @@ -213,7 +213,7 @@ Required argument: `action = "usage_blocks"` Optional arguments: `project`, `tool`, `from`, `to`. -## syslog project_context +## cortex project_context Summarize one AI project path with tools, sessions, hosts, counts, and recent representative entries. @@ -221,7 +221,7 @@ Required arguments: `action = "project_context"`, `project` Optional arguments: `tool`, `limit`. -## syslog list_ai_tools +## cortex list_ai_tools List distinct AI tools with counts and first/last seen timestamps. @@ -229,7 +229,7 @@ Required argument: `action = "list_ai_tools"` Optional arguments: `project`, `from`, `to`. -## syslog list_ai_projects +## cortex list_ai_projects List distinct AI projects with counts, tools used, and first/last seen timestamps. @@ -237,7 +237,7 @@ Required argument: `action = "list_ai_projects"` Optional arguments: `tool`, `from`, `to`. -## syslog correlate +## cortex correlate Search for related events across multiple hosts within a time window. @@ -245,19 +245,19 @@ Required arguments: `action = "correlate"`, `reference_time`. Optional arguments: `window_minutes`, `severity_min`, `hostname`, `source_ip`, `query`, `limit`. -## syslog stats +## cortex stats Get database statistics including storage health, runtime ingest counters, queue depth, writer failure/drop state, and OTLP receiver counters. Required argument: `action = "stats"` -## syslog status +## cortex status Get lightweight runtime status without the full DB statistics query. Required argument: `action = "status"` -## syslog compose_status +## cortex compose_status Get redacted read-only Docker Compose diagnostics for the canonical cortex deployment. MCP output omits host paths, mount sources, image ids, and raw command output. @@ -265,13 +265,13 @@ Required argument: `action = "compose_status"` Target override arguments such as `project_dir`, `compose_file`, `project_name`, `container`, and `container_name` are rejected. -## syslog compose_doctor +## cortex compose_doctor Run strict deployment-health checks for the canonical cortex Compose deployment. It returns the same redacted diagnostic shape as `compose_status` when healthy, and returns a tool error when Docker/Compose ownership or runtime checks are not ready for lifecycle work. Compose lifecycle mutations are CLI-only. Required argument: `action = "compose_doctor"` -## syslog help +## cortex help Return markdown documentation for all actions. diff --git a/docs/mcp/TRANSPORT.md b/docs/mcp/TRANSPORT.md index 688759f1..dfd408a0 100644 --- a/docs/mcp/TRANSPORT.md +++ b/docs/mcp/TRANSPORT.md @@ -109,8 +109,8 @@ curl -s -X POST http://localhost:3100/mcp \ RMCP validates the `Host` header to reduce DNS rebinding risk. Loopback hosts and the configured bind host are allowed by default. Add public names or proxy authorities with: ```bash -CORTEX_ALLOWED_HOSTS=syslog.example.com,syslog.example.com:443 -CORTEX_ALLOWED_ORIGINS=https://syslog.example.com +CORTEX_ALLOWED_HOSTS=cortex.example.com,cortex.example.com:443 +CORTEX_ALLOWED_ORIGINS=https://cortex.example.com ``` ## Direct stdio transport @@ -131,7 +131,7 @@ Ingestion still requires the daemon to be running somewhere. Stdio mode only que { "mcpServers": { "cortex": { - "command": "/path/to/syslog", + "command": "/path/to/cortex", "args": ["mcp"], "env": { "CORTEX_DB_PATH": "/data/cortex.db", diff --git a/docs/mcp/WEBMCP.md b/docs/mcp/WEBMCP.md index c17e2ee2..1239f7ef 100644 --- a/docs/mcp/WEBMCP.md +++ b/docs/mcp/WEBMCP.md @@ -34,7 +34,7 @@ MCP CLI clients (mcporter, curl, Claude Code) are not browser-based and ignore C Set a comma-separated origin allow-list: ```bash -CORTEX_ALLOWED_ORIGINS=https://syslog.example.com,https://logs.example.com +CORTEX_ALLOWED_ORIGINS=https://cortex.example.com,https://logs.example.com ``` Each value must be a full browser origin URL. Add matching reverse-proxy Host diff --git a/docs/plugin/AGENTS.md b/docs/plugin/AGENTS.md deleted file mode 100644 index b43acacf..00000000 --- a/docs/plugin/AGENTS.md +++ /dev/null @@ -1,14 +0,0 @@ -# Agent Definitions -- cortex - -cortex does not define any agents. The MCP tools are consumed directly by external agents (Claude Code, Codex, Gemini) without an intermediary agent layer. - -## Why no agents - -cortex is a data source (log receiver + query interface), not an orchestration layer. Agents that consume syslog data are defined elsewhere: -- `claude-homelab` homelab-core agents can call cortex tools for log analysis -- Custom agents in other repos can connect to cortex via HTTP transport - -## See also - -- [../mcp/TOOLS.md](../mcp/TOOLS.md) -- tools available for agent consumption -- [../mcp/CONNECT.md](../mcp/CONNECT.md) -- how agents connect to cortex diff --git a/docs/plugin/AGENTS.md b/docs/plugin/AGENTS.md new file mode 120000 index 00000000..681311eb --- /dev/null +++ b/docs/plugin/AGENTS.md @@ -0,0 +1 @@ +CLAUDE.md \ No newline at end of file diff --git a/docs/plugin/CLAUDE.md b/docs/plugin/CLAUDE.md index 0402000d..2027d0d8 100644 --- a/docs/plugin/CLAUDE.md +++ b/docs/plugin/CLAUDE.md @@ -13,12 +13,25 @@ Index for the `plugin/` documentation subdirectory. These docs cover every Claud | File | Purpose | | --- | --- | -| `AGENTS.md` | Agent definitions (none -- cortex has no agents) | | `CHANNELS.md` | Channel integration (none) | | `CONFIG.md` | Plugin settings: userConfig, settings.json | -| `HOOKS.md` | Lifecycle hooks: SessionStart → `scripts/plugin-setup.sh` → `cortex setup repair` | +| `HOOKS.md` | Lifecycle hooks: SessionStart/ConfigChange → `bin/cortex setup plugin-hook` | | `MARKETPLACES.md` | Marketplace publishing: Claude, Codex, Gemini, MCP Registry | | `OUTPUT-STYLES.md` | Output style definitions (none) | | `PLUGINS.md` | Plugin manifest reference: .claude-plugin, .codex-plugin, gemini-extension | | `SCHEDULES.md` | Scheduled tasks (none) | -| `SKILLS.md` | Skill definitions under `plugins/syslog/skills/`, including MCP usage, reports, diagnostics, deployment, logs, and version checks | +| `SKILLS.md` | Skill definitions under `plugins/cortex/skills/`, including MCP usage, reports, diagnostics, deployment, logs, and version checks | + +## Agent definitions + +cortex does not define plugin-local agents. The MCP tools are consumed directly +by external agents (Claude Code, Codex, Gemini) without an intermediary agent +layer. + +The consuming agents live outside this repo: + +- `claude-homelab` homelab-core agents can call cortex tools for log analysis. +- Custom agents in other repos can connect to cortex via HTTP transport. + +See [../mcp/TOOLS.md](../mcp/TOOLS.md) for the tool surface and +[../mcp/CONNECT.md](../mcp/CONNECT.md) for connection patterns. diff --git a/docs/plugin/CONFIG.md b/docs/plugin/CONFIG.md index 0384546f..442e3450 100644 --- a/docs/plugin/CONFIG.md +++ b/docs/plugin/CONFIG.md @@ -38,7 +38,7 @@ installer: ```text plugin userConfig - --> scripts/plugin-setup.sh exports CORTEX_* / CORTEX_* overrides + --> bin/cortex setup plugin-hook exports CORTEX_* overrides --> cortex setup repair (same engine as cortex deploy local) --> ~/.cortex/.env + ~/.cortex/compose/docker-compose.yml --> Docker Compose cortex container diff --git a/docs/plugin/HOOKS.md b/docs/plugin/HOOKS.md index fbe1b961..16d1bfe9 100644 --- a/docs/plugin/HOOKS.md +++ b/docs/plugin/HOOKS.md @@ -13,19 +13,21 @@ Lifecycle hooks that run automatically during Claude Code sessions. ``` plugins/ - hooks/ - hooks.json # Hook definitions + cortex/ + hooks/ + hooks.json # Hook definitions scripts/ - plugin-setup.sh # SessionStart hook: shared setup repair + plugin-setup.sh # Manual/legacy thin adapter ``` ## Hook definitions -Hooks are registered in `plugins/syslog/hooks/hooks.json` and executed by Claude Code at the appropriate lifecycle point. +Hooks are registered in `plugins/cortex/hooks/hooks.json` and executed by Claude Code at the appropriate lifecycle point. -### SessionStart — plugin-setup.sh +### SessionStart — cortex setup plugin-hook -Runs `${CLAUDE_PLUGIN_ROOT}/scripts/plugin-setup.sh` at the start of every Claude Code session. +Runs `${CLAUDE_PLUGIN_ROOT}/bin/cortex setup plugin-hook` at the start of +every Claude Code session. Responsibilities: - Server mode: exports current Claude Code `userConfig` values as @@ -46,7 +48,7 @@ Responsibilities: "hooks": [ { "type": "command", - "command": "${CLAUDE_PLUGIN_ROOT}/scripts/plugin-setup.sh" + "command": "${CLAUDE_PLUGIN_ROOT}/bin/cortex setup plugin-hook" } ] } @@ -57,7 +59,15 @@ Responsibilities: ## Manual execution -Run the setup script outside of Claude Code: +Run the binary-owned hook outside of Claude Code: + +```bash +CLAUDE_PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$PWD/plugins/cortex}" \ + "$CLAUDE_PLUGIN_ROOT/bin/cortex" setup plugin-hook +``` + +The legacy script remains a thin manual adapter for environments that still +need to map `CLAUDE_PLUGIN_OPTION_*` values before delegating to the binary: ```bash bash scripts/plugin-setup.sh diff --git a/docs/plugin/MARKETPLACES.md b/docs/plugin/MARKETPLACES.md index 84e5402c..056b36fd 100644 --- a/docs/plugin/MARKETPLACES.md +++ b/docs/plugin/MARKETPLACES.md @@ -1,7 +1,6 @@