AI scouts that watch public pages, news coverage, social profiles, civic records, and transport activity on a schedule. They turn what they find into source-linked, deduplicated information units for journalists and AI assistants — with human verification and optional tamper-evident evidence snapshots. Self-host it, or sign in at scoutpost.ai.
Quick Start | Deployment | Docs | FAQ | scoutpost.ai
Built by Buried Signals • Supported by IMJ • tom@buriedsignals.com
Scoutpost is monitoring infrastructure for investigative and local journalism.
Journalists define recurring scouts; each run finds source-linked information
units, deduplicates repeated coverage, and sends new leads to an editorial
inbox. The same workflow is available to people in the web app and to agents
through MCP, REST, or the scout CLI.
- Page Scouts watch one public URL for meaningful changes.
- Beat Scouts follow a topic or geography across relevant coverage.
- Social Scouts track new and deleted posts from public profiles.
- Civic Scouts follow council pages, agendas, minutes, and PDFs.
- Fleet Scouts alert when watched vessels, aircraft, or satellites enter a defined area.
Scouts run on schedules and notify the journalist when their criteria are met. Information units are leads until a human verifies them; agents can organize, search, and draft from the material, but must preserve that editorial boundary.
Page Scouts can optionally capture tamper-evident evidence snapshots of each
change (MHTML + screenshot + RFC 3161 timestamp, with optional Internet Archive
submission). Archiving is toggled per scout (archive_enabled) and snapshots are
retrievable through the UI-less agent surfaces — the scout snapshots CLI, the
list_snapshots / get_snapshot_url MCP tools, and the /snapshots REST
endpoints. See docs/features/page-archive.md.
- Frontend: SvelteKit + TailwindCSS (static SPA)
- Backend: FastAPI (Python) — auth broker, feedback, admin, public
/api/v1 - Scout runtime: Supabase Edge Functions + pg_cron (post-2026-04-22 cutover)
- Database: Supabase Postgres with pgvector + HNSW for hybrid search
- Auth: MuckRock OAuth 2.0 (SaaS) / Supabase Auth (OSS / self-hosted)
- AI: Google Gemini models through OpenRouter's Google Vertex route + Firecrawl
- Hosting: Render (Docker) for the FastAPI service; Supabase for EFs + DB
- Node.js 22 LTS
- Python 3.13+ (3.11 works locally but CI runs 3.13)
- Deno 2.x (for the
cli/package)
# Frontend
cd frontend && nvm use && npm install
npm run dev # private repo default: local FastAPI auth broker + hosted account data on localhost
npm run dev:hosted-broker # diagnostic: same frontend, but use the deployed broker path
npm run dev:supabase-local-demo # disposable local Supabase auth + local-only onboarding demo
# Backend
cd backend && python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload
# CLI (scout)
cd cli && deno task run --helpCopy .env.example to .env and fill in the values. See
docs/architecture/api-surface.md for the current set of
load-bearing variables.
Scoutpost uses one external AI credential: OPENROUTER_API_KEY. Structured
extraction uses google/gemini-2.5-flash-lite, and text embeddings use
google/gemini-embedding-001 with dimensions: 768. Every request pins
google-vertex, requires ZDR, denies provider data collection, disables
fallbacks, and sends X-OpenRouter-Cache: false.
PDF parsing stays local-first: Poppler pdftotext -layout is the deterministic
primary parser. Only low-yield or scanned PDFs use Google's native PDF handling
through OpenRouter. That request forces the native PDF engine, so OpenRouter
does not invoke Mistral, Cloudflare, or another parser; when the constrained
route is unavailable the document returns needs_ocr.
The FastAPI service auto-deploys to Render on push to main.
Supabase Edge Functions deploy via supabase functions deploy <name>
(see supabase/functions/CLAUDE.md).
Always go via PR — never push to main. CI must show 4 green
checks (build-frontend, test-frontend, test-backend, lint)
before merge.
- Newsroom Docker install
- API surface (post-cutover)
- FastAPI endpoints (legacy + auth + v1)
- Supabase Edge Functions
- Developer guide
scoutCLI
├── frontend/ # SvelteKit SPA
├── backend/ # FastAPI service (auth, feedback, admin, /api/v1)
├── supabase/ # Edge Functions + migrations + pg_cron
├── cli/ # `scout` Deno CLI — talks to FastAPI or EFs
├── docs/ # Architecture + features + supabase docs
├── scripts/ # OSS strip + EF bundler + helpers
└── Dockerfile # Production build for the FastAPI service
Scoutpost stands on open work and specialist data services that do the heavy operational lifting. A sincere thank-you to every project and provider below. (Listing does not imply affiliation or endorsement.)
| Category | Projects we're grateful to |
|---|---|
| Scraping & browser automation | Crawl4AI (unclecode, Apache-2.0 — the primary scraper) · Playwright (browser automation under the scraper) · Poppler (pdftotext — civic-PDF extraction) |
| Application runtime | Deno (MIT — the scout CLI, MCP bridge, and every Edge Function) · FastAPI (Sebastián Ramírez, MIT — the backend API) |
| Search & analysis | pgvector (Andrew Kane — vector search behind semantic scout matching) · langdetect (language detection in dedup scoring) |
| Evidence archiving | Internet Archive / Wayback Machine (optional evidence-snapshot submission) |
| Fleet & transport data | adsb.lol (community ADS-B network — live aircraft positions) · VesselAPI (paid exact-MMSI vessel positions) · CelesTrak (T.S. Kelso — satellite orbital elements) · satellite.js (shashwatak, MIT — orbital math) |
These and the app's full credits also live at scoutpost.ai/acknowledgements.
Built something here we should credit, or want a listing changed or removed? Open an issue or PR — we'll fix it fast.