feat: add managed file-tail ingest

.env.example

@@ -28,7 +28,7 @@
 # CORTEX_TRUSTED_GATEWAY_NO_AUTH=false
 
 # Grant the static bearer token cortex:admin in addition to cortex:read
-# (enables ack_error / unack_error / notifications_test).
+# (enables ack_error / unack_error / file_tails / notifications_test).
 # CORTEX_STATIC_TOKEN_ADMIN=false
 
 # Auth mode: "bearer" (default, static token only) or "oauth" (Google OAuth + JWT).
@@ -284,6 +284,26 @@
 # Env file the Compose service loads (optional).
 # CORTEX_ENV_FILE=.env
 
+# --- Managed file-tail ingest ---
+#
+# Sources are stored in <data-dir>/file-tails.json, where <data-dir> is the
+# parent directory of CORTEX_DB_PATH. Manage them at runtime:
+#
+#   cortex file-tail add --id swag-access \
+#     --path /file-tail-root/swag/log/nginx/access.log \
+#     --tag swag-access --hostname squirts --facility local4
+#   cortex file-tail list --json
+#
+# The default starts at EOF; add --from-start only for intentional backfills.
+# REST /api/file-tails needs the normal API bearer plus X-Cortex-Admin-Token.
+# CORTEX_API_ADMIN_TOKEN=change-me-admin-token
+# Host directory mounted read-only at /file-tail-root by Compose.
+# CORTEX_FILE_TAIL_LOG_VOLUME=/var/log
+# Dedicated container roots file-tail may read from. Keep the default narrow;
+# opt into broader read-only roots explicitly after mounting them.
+# CORTEX_FILE_TAIL_ALLOWED_ROOTS=/file-tail-root
+# CORTEX_FILE_TAIL_ALLOWED_ROOTS=/file-tail-root,/var/log,/logs
+
 # Shared cortex home (inventory cache, setup env) mounted at /cortex-home.
 # CORTEX_HOME_VOLUME=~/.cortex
 

CHANGELOG.md

@@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.20.1] - 2026-06-12
+
+### Changed
+
+- Require an explicit `hostname` when adding managed file-tail sources so rows are attributed to the intended host instead of falling back to the Cortex container identity.
+- Report file-tail durable-writer backpressure in the `status` action and mark overall status as `degraded` when tailers are blocked on writer ack.
+
+### Fixed
+
+- Prevent duplicate file-tail `add` calls from resetting checkpoints, avoid retrying stateful HTTP file-tail mutations after 503 responses, and make committed-but-reconcile-failed mutations explicit.
+- Harden file-tail resume and rotation handling: stale checkpoints now restart replacement files from the beginning, same-inode copytruncate/regrow is detected by prefix fingerprint, and rename-create rotation gets a short old-file drain window before switching.
+- Fix file-tail live smoke harness permissions so non-root containers can read the mounted smoke log, and tighten admin-scope predicates across smoke scripts.
+- Canonicalize configured file-tail allowed roots so symlinked operational roots validate against canonical file paths.
+
+## [1.20.0] - 2026-06-11
+
+### Added
+
+- Added managed file-tail ingest sources with CLI, REST API, and MCP control.
+- Added `file-tail` source kind for rows ingested from local log files.
+- Documented SWAG, fail2ban, Authelia, and AdGuard file-tail recipes for replacing rsyslog `imfile` drop-ins.
+
 ## [1.19.0] - 2026-06-11
 
 ### Added
@@ -2268,7 +2290,8 @@ start and verify with `cortex --http db status`.
 
 ---
 
-[Unreleased]: https://github.com/jmagar/cortex/compare/v1.14.0...HEAD
+[Unreleased]: https://github.com/jmagar/cortex/compare/v1.20.0...HEAD
+[1.20.0]: https://github.com/jmagar/cortex/compare/v1.19.0...v1.20.0
 [1.14.0]: https://github.com/jmagar/cortex/compare/v1.13.3...v1.14.0
 [1.13.3]: https://github.com/jmagar/cortex/compare/v1.13.2...v1.13.3
 [1.13.2]: https://github.com/jmagar/cortex/compare/v1.13.1...v1.13.2

CLAUDE.md

@@ -63,9 +63,9 @@ Tests: unit tests live in sidecar files beside their source modules (e.g. `src/d
 
 ## MCP Tools
 
-One MCP tool: **`cortex`** — dispatches by `action` argument. 45 actions, generated from `ACTION_SPECS` in `src/mcp/actions.rs` (the single authoritative registry — regenerate this table from there).
+One MCP tool: **`cortex`** — dispatches by `action` argument. 46 actions, generated from `ACTION_SPECS` in `src/mcp/actions.rs` (the single authoritative registry — regenerate this table from there).
 
-Scope taxonomy: every action requires `cortex:read` except the three **admin** actions `ack_error`, `unack_error`, and `notifications_test`, which require `cortex:admin` (static bearer tokens get read-only unless `CORTEX_STATIC_TOKEN_ADMIN=true`); `help` is info-only (no scope gate).
+Scope taxonomy: every action requires `cortex:read` except the four **admin** actions `ack_error`, `unack_error`, `file_tails`, and `notifications_test`, which require `cortex:admin` (static bearer tokens get read-only unless `CORTEX_STATIC_TOKEN_ADMIN=true`); `help` is info-only (no scope gate).
 
 | Action | Description |
 |--------|-------------|
@@ -110,6 +110,7 @@ Scope taxonomy: every action requires `cortex:read` except the three **admin** a
 | `ask_history` | Query AI transcript history |
 | `incident_context` | Full context for an incident |
 | `graph` | Resolve graph entities, neighborhoods, and evidence-backed explanations |
+| `file_tails` | **(admin)** Manage Cortex-owned file-tail ingest sources |
 | `ack_error` | **(admin)** Acknowledge an error signature |
 | `unack_error` | **(admin)** Revoke an error signature acknowledgement |
 | `notifications_test` | **(admin)** Send a test notification via Apprise |
@@ -164,6 +165,10 @@ CORTEX_GOOGLE_CLIENT_SECRET=...     # required when CORTEX_AUTH_MODE=oauth
 # Non-MCP REST API (always on; gated by its token)
 CORTEX_API_TOKEN=your-api-token         # REQUIRED at startup — /api/* is always mounted
 
+# Managed file-tail sources
+# Stored in the parent directory of CORTEX_DB_PATH as file-tails.json.
+# Manage with: cortex file-tail list|status|add|remove|enable|disable
+
 # Docker container log ingestion (disabled by default)
 CORTEX_DOCKER_INGEST_ENABLED=false      # set true to ingest from docker-socket-proxy hosts
 CORTEX_DOCKER_HOSTS=host-a,host-b      # comma-separated hostnames → http://<host>:2375
@@ -182,7 +187,7 @@ RUST_LOG=info
 | `docker-compose.yml` | Production deployment (ports 1514, 3100) |
 | `docs/SETUP.md` | Setup guide (clone, build, configure, deploy, verify); per-host forwarder configs (rsyslog, UniFi, ATT router, WSL) live in README "Syslog Forwarder Setup" |
 | `src/db/queries.rs` | All SQL queries and FTS5 search implementation |
-| `src/mcp/actions.rs` | `ACTION_SPECS` — authoritative registry of all 45 MCP actions and their scopes |
+| `src/mcp/actions.rs` | `ACTION_SPECS` — authoritative registry of all 46 MCP actions and their scopes |
 | `src/mcp/tools.rs` | Single `cortex` tool with action dispatch |
 | `config/mcporter.json` | mcporter config (HTTP transport to localhost:3100) |
 | `config/systemd/` | `cortex-backup.service` / `.timer` — daily WAL-safe backup units |