Skip to content

README.md

Latest commit

 

History

History
288 lines (218 loc) · 26.1 KB

README.md

File metadata and controls

288 lines (218 loc) · 26.1 KB
Raw

@loopover/miner

Foundation CLI for the local LoopOver miner runtime.

This package is the future home of the autonomous discover → analyze → plan → prepare → create → manage miner workflow. In this foundation phase it provides the package scaffold, a minimal CLI surface for --help and --version, and a non-blocking npm registry version nudge on startup.

Status

Current scope is intentionally small:

  • workspace package wiring
  • CLI entry point
  • --help and version commands
  • startup npm version nudge (override with --no-update-check or LOOPOVER_MINER_NO_UPDATE_CHECK=1)

Environment variables read by the miner are documented in docs/env-reference.md. Regenerate that file with npm run miner:env-reference from the repo root after adding or removing env reads.

Config precedence (.loopover-miner.yml vs operator env vs CLI flags) is documented in docs/config-precedence.md.

A committed micro-benchmark for the discovery-ranking and local-store read/write paths lives at BENCHMARKS.md — run it with npm run benchmark:miner from the repo root.

The cross-repo evaluation harness (#4788) lives at docs/cross-repo-evaluation.md — run it with npm run cross-repo-eval --workspace @loopover/miner after cloning the benchmark repos.

Real miner commands land in follow-up issues.

The package also includes the first metadata-only discovery primitive: fetchCandidateIssues lists open issue metadata across target repos, and searchCandidateIssues does the same from a GitHub issue-search query. Both paths hard-skip repos whose AI-USAGE.md or CONTRIBUTING.md explicitly bans AI-generated PRs. They perform GitHub GET requests only, never clone source, never upload source, and never write to GitHub.

The package also includes a metadata-only ranker: rankCandidateIssues composes deterministic engine signals (potential, feasibility, lane fit, freshness, dup risk) and returns fan-out candidates sorted by rankScore. It never clones source and never writes to GitHub.

Discovery is per-tenant, not github.com-specific (#4784): lib/forge-config.js (resolveForgeConfig) holds the forge base URL, API version, request headers, repo path, search endpoint/qualifiers, user-agent, and credential env var behind one resolver with gittensory's github.com values as the only defaults, so the fan-out targets another forge unchanged. loopover-miner discover surfaces --api-base-url <url> and --token-env <VAR> and forwards a tenant goal spec to the ranker, printing usedDefaultGoalSpec so a fall-back to the built-in rubric is explicit rather than silent. See docs/repo-agnostic-capability-audit.md for the #4780 audit this executes.

The package also includes repo stack auto-detection: detectRepoStack (lib/stack-detection.js) inspects an already-cloned target repo's manifest / lockfile / config files and returns a structured description — language, package manager, and the build / test / lint / format commands — for Node (npm/yarn/pnpm/bun), Python (pip/poetry/pipenv/uv), Rust, Go, Maven, and Gradle. It is pure (injectable existsSync / readFileSync), never throws, and per its acceptance criteria fails closed — a repo with no recognized manifest returns { detected: false, reason } and a command that can't be inferred without guessing stays null, rather than being assumed. The attempt path consumes it: buildCodingTaskSpec appends the real stack summary (and any confidently-inferred build/test/lint/format commands) to the coding-agent instructions so validation uses the target repo's own tooling rather than assuming LoopOver/gittensory CI (#4786). (#4785)

The package also includes an append-only governor decision ledger: initGovernorLedger / appendGovernorEvent persist structured allow/deny/throttle/kill-switch outcomes in local SQLite for contributor audit. Insert-only — no enforcement wiring yet. (#2328)

The package also includes a real, persisted governor pause/resume control surface: loopover-miner governor pause [--reason <text>] / governor resume / governor status toggle a paused/reason/pausedAt flag on governor-state.js's existing scalar-state row, and loop-cli.js's iteration loop checks it at the same two boundaries as the kill switch (before the first cycle and at the top of every subsequent one) — pausing mid-run stops the loop before its next cycle claims anything, and resuming (clear the flag, re-invoke loop) continues from the already-persisted queue/run state exactly where it left off. Distinct from the read-only kill switch (env/YAML inputs this package never writes) and the one-way run-halt breaker (no resume path at all) — this is the first genuinely operator/governor-writable stop/go control; the autonomous logic that decides when to pause is separate, later-wave scope. (#4851)

The package also includes a local soft-claim ledger: openClaimLedger / claimIssue / releaseClaim / listActiveClaims persist which issues this miner instance has claimed on this machine. The table is local bookkeeping only — duplicate winners are adjudicated elsewhere via @loopover/engine. (#2291)

The package also includes an append-only event ledger: initEventLedger / appendEvent / readEvents persist immutable miner-loop events in local SQLite for contributor audit. Insert-only — rows are never updated or deleted. (#2322)

The package also records local PR outcomes: recordPrOutcomeSnapshot / readPrOutcomes write and reduce the miner's OWN record of the outcomes of its OWN PRs (merged / closed, with an optional rejection-reason bucket) over the append-only event ledger above. This is DISTINCT from the gittensory server's recordPrOutcome (src/review/outcomes-wire.ts), which writes hosted-backend audit rows from the GitHub App's webhook stream — same concept name, different codebase layer, no shared code (a laptop-mode miner may have no webhook relay at all). (#4274)

The package also includes an append-only prediction ledger: initPredictionLedger / appendPrediction / readPredictions persist each predicted-gate verdict (conclusion / pack / readiness score + blocker/warning codes, plus the producing ENGINE_VERSION) in local SQLite, so a later self-improve pass can score predictions against realized outcomes. Insert-only. (#4263)

The package also includes the Phase 7 calibration runner: runHistoricalReplayCalibrationCycle (lib/calibration-run.js) scores a completed historical-replay run with the deterministic objective-anchor scorer, folds the composite into the engine's computePhase7CalibrationLoop combine alongside the existing pr_outcome signal, and persists the combined snapshot as a calibration_snapshot event — queryable with loopover-miner ledger list --type calibration_snapshot or readCalibrationSnapshots / latestCalibrationSnapshot. It measures and records only; acting on the metric (autonomy bumps, threshold tuning) stays maintainer-only. See docs/miner-selfimprove-calibration.md. (#4248)

loopover-miner manage status now also folds each tracked repo's current discover/plan/prepare run state (run-state.js) alongside its managed PR rows into a "run portfolio" view — collectRunPortfolio / renderRunPortfolioTable — so a repo actively being discovered or planned shows up even with zero PRs yet. Additive only: the existing rows JSON key and PR table are unchanged; runPortfolio is a new key printed after the existing table. A real GUI dashboard surface is out of scope here — apps/loopover-miner-ui/ is Phase 6 of the same roadmap tracker and hasn't been scaffolded yet. (#4279)

Local storage

Independent local SQLite stores back the commands above. Each keeps its own file, its own table(s), and (for most stores) its own env-var override — this is a DRY pass over their shared path-resolution/open boilerplate (local-store.js), not a merge into one database. (#4272)

See docs/env-reference.md for the full LOOPOVER_MINER_* / MINER_* env-var list.

Store File Primary table(s) Module Env var override
Laptop bootstrap laptop-state.sqlite3 laptop_meta laptop-init.js LOOPOVER_MINER_CONFIG_DIR (path only)
Run state run-state.sqlite3 miner_run_state run-state.js LOOPOVER_MINER_RUN_STATE_DB
Claim ledger claim-ledger.sqlite3 miner_claims claim-ledger.js LOOPOVER_MINER_CLAIM_LEDGER_DB
Portfolio queue portfolio-queue.sqlite3 miner_portfolio_queue portfolio-queue.js LOOPOVER_MINER_PORTFOLIO_QUEUE_DB
Event ledger event-ledger.sqlite3 miner_event_ledger event-ledger.js LOOPOVER_MINER_EVENT_LEDGER_DB
Plan store plan-store.sqlite3 miner_plans plan-store.js LOOPOVER_MINER_PLAN_STORE_DB
Governor ledger governor-ledger.sqlite3 governor_events governor-ledger.js LOOPOVER_MINER_GOVERNOR_LEDGER_DB
Governor state governor-state.sqlite3 governor_scalar_state, governor_reputation_history, governor_own_submissions governor-state.js LOOPOVER_MINER_GOVERNOR_STATE_DB
Attempt log attempt-log.sqlite3 attempt_log_events attempt-log.js LOOPOVER_MINER_ATTEMPT_LOG_DB
Prediction ledger prediction-ledger.sqlite3 predictions prediction-ledger.js LOOPOVER_MINER_PREDICTION_LEDGER_DB
Replay snapshot replay-snapshot.sqlite3 replay_snapshots replay-snapshot.js LOOPOVER_MINER_REPLAY_SNAPSHOT_DB
Deny-hook synthesis deny-hook-synthesis.sqlite3 deny_rule_proposals deny-hook-synthesis.js LOOPOVER_MINER_DENY_HOOK_SYNTHESIS_DB
Worktree allocator worktree-allocator.sqlite3 worktree_slots worktree-allocator.js LOOPOVER_MINER_WORKTREE_ALLOCATOR_DB
Orb export orb-export.sqlite3 orb_export_meta orb-export.js LOOPOVER_MINER_ORB_EXPORT_DB
Policy-doc cache policy-doc-cache.sqlite3 policy_doc_cache policy-doc-cache.js LOOPOVER_MINER_POLICY_DOC_CACHE_DB
Policy-verdict cache policy-verdict-cache.sqlite3 policy_verdict_cache policy-verdict-cache.js LOOPOVER_MINER_POLICY_VERDICT_CACHE_DB

The policy-doc and policy-verdict caches are the only stores above that hold no miner state of their own — both are pure optimization, and deleting either file only forces the next run to redo the work it would have skipped. The policy-doc cache caches the last-known ETag + body of each target repo's fetched policy docs (AI-USAGE.md/CONTRIBUTING.md) so a repeated discover revalidates them with a conditional GET (If-None-Match) instead of re-downloading static content, spending no extra rate-limit budget when GitHub answers 304 Not Modified (#4842). The policy-verdict cache goes one step further: once a repo's deciding doc's ETag is confirmed unchanged, it reuses the already-resolved AI-usage-policy verdict instead of re-resolving it from the (identical) doc text (#4843).

Every store resolves its file the same way: the store-specific env var above, else LOOPOVER_MINER_CONFIG_DIR, else XDG_CONFIG_HOME (falling back to ~/.config), joined with loopover-miner/<file>. Every store also opens its file with 0700/0600 permissions and a shared PRAGMA busy_timeout so two instances on the same file serialize writes instead of racing.

Opening a store through local-store.js also registers it with the CLI's crash-safety chokepoint (process-lifecycle.js): the entrypoint calls installCliSignalHandlers() once at startup, so a SIGINT/SIGTERM mid-run — or an uncaught exception / unhandled rejection — closes every still-open ledger cleanly and exits with a conventional code (130/143 for signals, non-zero for a crash) instead of dying mid-write. A store's normal close() unregisters itself first, so the happy path never double-closes and a long-running loop never accumulates stale handles. Cleanup only — no command business logic is affected. (#4826)

The "PR portfolio" manage status renders is currently a read-time join, not a dedicated table: collectManageStatus reads portfolio-queue.js rows (via the pr:{number} identifier convention) and joins them against event-ledger.js's free-form manage_pr_update JSON events at query time, on every read. Decision: keep this as a read-time join for now; revisit a dedicated indexed table only if/when PR-portfolio reads become frequent enough (e.g. a live-polling dashboard) that the per-read linear event-ledger scan becomes a measured bottleneck.

Install

See docs/miner-goal-spec.md for the .loopover-miner.yml field reference and .loopover-miner.yml.example at the repo root.

See docs/cross-repo-discovery-phase1.md for the Phase 1 cross-repo discovery scope (re-scoped from #1060, paper trail for #2299).

See docs/discovery-plane-operator-guide.md for the optional hosted discovery-index plane (opt-in default OFF; contrasts with Orb's opt-out-only export — #4309).

See DEPLOYMENT.md for laptop vs fleet deployment.

See docs/operations-runbook.md for SQLite concurrency guarantees, corruption recovery, multi-process collision response, and post-upgrade ledger migration (#4875).

See docs/sizing.md for measured CPU/RAM/disk numbers across laptop mode and fleet mode at different worker counts.

Laptop-mode quickstart

Zero-infra local install — no Docker, Redis, or Postgres required:

npm install -g @loopover/miner
loopover-miner init
loopover-miner doctor
loopover-miner status

init creates ~/.config/loopover-miner/ (or LOOPOVER_MINER_CONFIG_DIR / XDG_CONFIG_HOME overrides) and a local laptop-state.sqlite3 bootstrap file. Re-running init is idempotent. Pass --verify-token to make one authenticated GitHub API call up front and fail fast if GITHUB_TOKEN is invalid or missing repository access scopes. doctor reports Node, the state directory, SQLite readiness, and whether Docker is installed (informational only). Every local store already applies its own pending schema migrations automatically the moment some other command first opens it, but migrate lets an operator proactively bring every EXISTING store file up to date in one pass (e.g. right after upgrading) instead of relying on whichever command happens to touch a given store first; a store file that hasn't been created yet is reported as skipped, not created.

First-time operators can instead run loopover-miner init --interactive (#5176): a guided prompt for GITHUB_TOKEN (input hidden, never echoed or written to any log) and an optional coding-agent provider — plus that provider's model/timeout companion vars, each individually skippable with Enter — writes a starter .env to the state dir, then automatically reruns doctor against the collected values so setup problems surface immediately. --interactive makes no network calls of its own beyond what doctor already makes (none); non-interactive init invocations are unaffected.

The primary way to authenticate AMS's git operations (#6116) is loopover-mcp login (the GitHub App device flow, npm install -g @loopover/mcp@latest then loopover-mcp login) — the same one command that already authenticates ORB's MCP session. AMS resolves a live token from that session automatically; no GITHUB_TOKEN setup is required for the common case. A manually-created GITHUB_TOKEN PAT remains supported as an explicit fallback (it always wins over a session token when set) for self-host operators who already have one configured, or who need a token with different scopes than the App grants. doctor's github-token check reports either source as satisfying the requirement.

From a local checkout:

npm install
npm --workspace @loopover/miner run build
npm link --workspace @loopover/miner

Coding-agent driver configuration

The production driver is selected by MINER_CODING_AGENT_PROVIDER. The value is a comma-separated preference list: the first configured name wins, unknown names are skipped, and an empty/unset list leaves production driver construction fail-closed. See docs/coding-agent-driver.md for the interface-level contract and provider behavior.

Env var Accepted values Default / behavior
MINER_CODING_AGENT_PROVIDER noop, claude-cli, codex-cli, agent-sdk Unset / empty means no provider is configured; the miner fails closed until a valid provider name is supplied.
MINER_CODING_AGENT_CLAUDE_MODEL Any Claude model string accepted by the local claude CLI Unset means claude-cli uses the CLI's own default model. Ignored by noop, codex-cli, and agent-sdk.
MINER_CODING_AGENT_CODEX_MODEL Any Codex model string accepted by the local codex CLI Unset means codex-cli uses the CLI's own default model. Ignored by noop, claude-cli, and agent-sdk.
MINER_CODING_AGENT_TIMEOUT_MS Positive integer milliseconds Unset or invalid falls back to the CLI driver's default wall-clock timeout of 120000 ms. Ignored by noop and agent-sdk.

Recognizing a stale or missing coding-agent credential

When an attempt fails on a claude-cli / codex-cli provider, the CLI-subprocess driver folds the CLI's own output into a machine-readable error string on the attempt result. The credential/auth failure modes below map that exact string to a symptom and a concrete remediation — mirroring ORB's hosted-side Recognizing a stale or missing credential table. Every string is emitted by cli-subprocess-driver.ts; nothing here is speculative.

Error string / pattern Symptom Remediation
claude_code_error_<status> The claude CLI ran but its --output-format json envelope reported is_error: true (e.g. claude_code_error_invalid_api_key) — the OAuth token is missing, rejected, or expired. loopover-miner doctor reports the same condition up front as not authenticated: set CLAUDE_CODE_OAUTH_TOKEN. Regenerate a long-lived token with claude setup-token and set CLAUDE_CODE_OAUTH_TOKEN, then retry.
codex_no_auth codex exec exited non-zero with no structured error in its JSONL stdout and only the Reading prompt from stdin... banner on stderr — its auth.json credential is missing or expired. The driver appends its own remediation hint (... auth.json missing or expired -- run codex auth to authenticate). Run codex auth to re-authenticate; the next attempt reads the refreshed auth.json with no further restart.
<command>_exit_<code> Generic non-zero-exit fallback when neither structured parser matched (e.g. codex_exit_1: ..., claude_exit_1: ...). The driver appends the redacted stderr slice — an auth failure that neither parser recognized still surfaces here. Read the appended detail; if it points at authentication, re-run claude setup-token or codex auth for the failing provider, otherwise address the reported error directly.

Commands

loopover-miner --help
loopover-miner help
loopover-miner --version
loopover-miner version
loopover-miner init [--json] [--verify-token]
loopover-miner status [--json]
loopover-miner doctor [--json]
loopover-miner migrate [--json]
loopover-miner manage status [--json]
loopover-miner manage poll <owner/repo> <pr#> [--branch <name>] [--json]

MCP server

The package ships a second bin entry, loopover-miner-mcp, a minimal Model Context Protocol stdio server that any MCP-compatible client can connect to:

loopover-miner-mcp

It exposes these read-only tools:

  • loopover_miner_ping (#5153) — a health check returning a static { "status": "ok", "tool": "loopover_miner_ping" } object. Reads no AMS state, takes no arguments.

  • loopover_miner_get_portfolio_dashboard (#5155) — the per-repo portfolio-queue backlog dashboard: status counts (queued / in_progress / done), totals, and the oldest-queued age. Wraps collectPortfolioDashboard() (no new logic) — the same data loopover-miner queue dashboard --json prints locally. Read-only, takes no arguments.

  • loopover_miner_get_manage_status (#5822) — read-only manage-phase status: the per-managed-PR rows (branch, CI state, gate verdict, outcome, last-polled-at, queue status/priority) plus the run-level portfolio view (one row per tracked repo: run state, updated-at, PR count). Joins the portfolio queue, the event ledger, and run-state by reusing collectManageStatus() / collectRunPortfolio() (no new join logic) — the same { rows, runPortfolio } shape loopover-miner manage status --json prints. Read-only: never calls GitHub, never mutates local stores; takes no arguments.

  • loopover_miner_list_claims (#5156) — lists the local claim ledger (repo, issue number, status, claimed-at, note) via listClaims(). Optional repoFullName / status filters pass through to the query. Read-only — exposes no claim/release mutation.

  • loopover_miner_get_audit_feed (#5158) — read-only, metadata-only event-ledger audit feed (eventType, repoFullName, outcome, actor, detail, createdAt). Wraps collectEventLedgerAuditFeed() with the same filters as loopover-miner ledger list (--repo, --since, --type). Never returns payload_json or other raw ledger columns.

  • loopover_miner_get_run_state (#5160) — read-only per-repo run-state (idle / discovering / planning / preparing) via getRunState / listRunStates. Pass repoFullName for one repo (a null state means none recorded yet), or omit it to list all. The read-only analog of ORB's loopover_get_automation_state; adds no state-set mutation.

  • loopover_miner_list_plans / loopover_miner_get_plan (#5161) — read-only access to the persisted plan store (planId, plan DAG, status, updatedAt) via listPlans / loadPlan; list_plans takes an optional status filter, get_plan takes a planId and returns an explicit { planId, found: false } for an unknown id. These read the store-backed AMS plan store — distinct from ORB's stateless loopover_plan_status tool.

  • loopover_miner_get_governor_decisions (#5159) — read-only projection of the governor decision log (id, ts, eventType, repoFullName, actionClass, decision, reason), optionally filtered by repoFullName. The projection excludes the sensitive payload_json column by construction — governor-ledger.js reads it with an explicit named-column SELECT, never SELECT *.

  • loopover_miner_status (#5154) — read-only status + doctor diagnostics, returning { status, doctor }: status = package/engine versions (and skew), node version, state-dir + config-file paths, and the resolved coding-agent driver (provider name, the model env-var NAME never its value, CLI-present boolean); doctor = the checks loopover-miner doctor runs (Docker/CLI presence, config validity, …) as { name, ok, detail }. Reuses collectStatus / runDoctorChecks so it can't drift from the CLI, and returns only names / booleans / paths — never any env-var value, token, or credential.

  • loopover_miner_get_calibration_report (#5821) — read-only miner-local prediction-accuracy report: per-project merge/close precision, joining this miner's own recorded gate predictions (prediction ledger) with the realized PR outcomes it later observed (pr_outcome events). Wraps calibration-cli.js's existing toPredictionRecords / toOutcomeRecords mappers and calibration.js's buildCalibrationReport composer — no new join/scoring logic. Strictly local and offline; distinct from ORB's hosted, maintainer-authenticated loopover_get_outcome_calibration tool, which reads a different (D1) data source.

This completes the read-only AMS MCP tool surface (status, portfolio, claims, event-ledger, governor-ledger, run-state, plan-store, calibration).

Client config

loopover-mcp (ORB's hosted contributor-workflow tools) and loopover-miner-mcp (AMS's own local state-visibility tools above) can run as two separate stdio servers in the same MCP client session — useful for a dual-role operator running both ORB and AMS on the same box. Generate ORB's half with loopover-mcp init-client --print claude (see the @loopover/mcp README); loopover-miner-mcp takes no flags, so its entry is just the bin name. Combined, a Claude Desktop / Claude Code style config looks like:

{
  "mcpServers": {
    "loopover": {
      "command": "loopover-mcp",
      "args": ["--stdio"]
    },
    "loopover-miner": {
      "command": "loopover-miner-mcp",
      "args": []
    }
  }
}

loopover exposes ORB's hosted contributor-workflow tools (issue ranking, PR packet prep, decision packs). loopover-miner exposes AMS's own local state-visibility tools listed above (portfolio dashboard, claims, audit feed, run state, plans, calibration) — a fully separate, 100% local tool surface with no shared code or network calls between the two. Both follow the same loopover_* tool-naming convention (loopover_... vs. loopover_miner_...), but back onto different stores: ORB's tools read the hosted loopover backend, AMS's tools read this machine's own local SQLite files (see Local storage) — a handful of AMS tools even name the ORB tool they mirror (e.g. loopover_miner_get_run_state is the read-only analog of loopover_get_automation_state) so the relationship is explicit at the point of use, not just here.

Version check

On every invocation the CLI starts an async npm registry lookup (5s timeout). When the installed package is behind @loopover/miner@latest, it prints a one-line upgrade command to stderr without blocking or failing the requested command. Set LOOPOVER_NPM_REGISTRY_URL to point at a mirror, same as @loopover/mcp.