mastra-oncall-agent

An on-call incident agent built on Mastra, plus the harness that answers the question most "build an agent" posts skip: what actually happens when the process dies mid-run?

The agent takes an alert, triages it, gathers evidence, proposes a fix, waits for a human to approve, performs the action, and writes a handover note. The interesting part is src/experiments/, which kills the process at the worst possible moment and measures the damage.

Companion to We Built an On-Call Agent in Mastra.

The headline result

A SIGKILL lands the instant the side effect commits, while the step is still running. Then a fresh process recovers the run.

	side effects after crash and recovery
no idempotency guard	2, the rollback ran twice
idempotency key on the action	1, exactly once

Durable execution gives you at-least-once, not exactly-once. A step that dies in flight is re-executed from the top on recovery, so any side effect inside it happens again. This is not specific to Mastra; it is how Temporal, DBOS and Restate behave too. Idempotency remains your job.

Reproduce both rows:

npm run crash-test               # no guard
IDEMPOTENT=1 npm run crash-test  # with the guard

Setup

npm install
export OPENAI_API_KEY=...      # or DO_INFERENCE_KEY for the DigitalOcean gateway
npm run incident               # one incident, end to end

The model is configured in one place, src/mastra.ts:

const defaults = openaiKey
  ? { id: 'openai/gpt-5', url: 'https://api.openai.com/v1' }
  : { id: 'openai/deepseek-v4-pro', url: 'https://inference.do-ai.run/v1' };

Set OPENAI_API_KEY and it talks to OpenAI directly. Set DO_INFERENCE_KEY instead and it goes through DigitalOcean's gateway. Override MODEL_ID and MODEL_URL for anything else OpenAI-compatible. Nothing else in the codebase changes.

What is in here

Path	What it is
src/workflow.ts	The six-step incident workflow: triage, gather, propose, approve, act, writeup
src/mastra.ts	Agents, model config, LibSQL storage
src/ledger.ts	The side-effect ledger. record vs recordOnce is the whole experiment
src/experiments/crash-resume.ts	Starts a run, SIGKILLs it mid-action, recovers it, counts the damage
src/experiments/eval-triage.ts	Does an eval catch a plausible prompt regression?
src/experiments/baseline-loop.ts	The same job with no framework, for comparison

Scripts

npm run incident      # one incident end to end, auto-approved
npm run crash-test    # the crash and recovery experiment
npm run eval          # prompt regression check
npm run baseline      # the same task without a framework
npm run typecheck

Notes on reproducibility

The world the agent investigates (src/incident-data.ts) is a fixture, so the only non-determinism is the model. Alert payloads, deploy history and error rates are fixed, which means a run differs only in wording, not in facts.

The crash experiment does not guess at timing. It watches the ledger and sends SIGKILL the moment the side effect lands, so the process always dies inside the dangerous window rather than near it.

Licence

MIT.