Skip to content

Commit bc2a90b

Browse files
committed
feature: add heartbeat loop for periodic health checks
New heartbeat.ts extension β€” a periodic timer that reads HEARTBEAT.md and injects it as a follow-up prompt (default: every 10 min). - Configurable via env vars (HEARTBEAT_INTERVAL_MS, HEARTBEAT_FILE, HEARTBEAT_ENABLED) - Error backoff: exponential delay on consecutive failures (2x per error, max 1 hour) to prevent token burn - heartbeat tool: status/pause/resume/trigger/config actions - Default checklist checks agent sessions, Slack bridge, email monitor, stale worktrees, and stuck todos - If HEARTBEAT.md is empty or missing, no heartbeat fires (zero cost) - deploy.sh deploys HEARTBEAT.md (always overwrites β€” admin-managed) Inspired by OpenClaw's HEARTBEAT.md pattern.
1 parent c56c09f commit bc2a90b

7 files changed

Lines changed: 367 additions & 0 deletions

File tree

β€ŽAGENTS.mdβ€Ž

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,13 @@ pi/
2424
extensions/ source of truth for pi agent extensions
2525
tool-guard.ts πŸ”’ tool call interception (blocks dangerous patterns)
2626
tool-guard.test.mjs πŸ”’ 86 tests for tool-guard
27+
heartbeat.ts periodic health check loop
2728
auto-name.ts session naming
2829
control.ts inter-session communication
2930
...
3031
skills/ source of truth for agent skill templates
3132
control-agent/ orchestration agent
33+
HEARTBEAT.md health check checklist (deployed to ~/.pi/agent/)
3234
dev-agent/ coding agent
3335
sentry-agent/ monitoring/triage agent
3436
settings.json pi agent settings
@@ -67,6 +69,7 @@ Agent runtime layout:
6769
β”œβ”€β”€ .pi/agent/
6870
β”‚ β”œβ”€β”€ extensions/ deployed extensions
6971
β”‚ β”œβ”€β”€ skills/ agent-owned (can modify freely)
72+
β”‚ β”œβ”€β”€ HEARTBEAT.md periodic health check checklist (admin-managed)
7073
β”‚ β”œβ”€β”€ baudbot-version.json deploy version (git SHA, timestamp)
7174
β”‚ └── baudbot-manifest.json SHA256 hashes of all deployed files
7275
β”œβ”€β”€ workspace/ project repos + git worktrees

β€ŽCONFIGURATION.mdβ€Ž

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,14 @@ Set during `setup.sh` via env vars (or edit `~/.gitconfig` after):
8585
| `GIT_USER_NAME` | Git commit author name | `baudbot-agent` |
8686
| `GIT_USER_EMAIL` | Git commit author email | `baudbot-agent@users.noreply.github.com` |
8787

88+
### Heartbeat
89+
90+
| Variable | Description | Default |
91+
|----------|-------------|---------|
92+
| `HEARTBEAT_INTERVAL_MS` | Interval between heartbeat checks (milliseconds) | `600000` (10 min) |
93+
| `HEARTBEAT_FILE` | Path to heartbeat checklist file | `~/.pi/agent/HEARTBEAT.md` |
94+
| `HEARTBEAT_ENABLED` | Set to `0` or `false` to disable heartbeats | enabled |
95+
8896
### Bridge
8997

9098
| Variable | Description | Default |

β€ŽREADME.mdβ€Ž

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,17 @@ Slack β†’ bridge (access control + content wrapping) β†’ pi agent β†’ tools (too
9999

100100
Every layer assumes the previous one failed. The bridge wraps content and rate-limits, but tool-guard blocks dangerous commands even if wrapping is bypassed. Safe-bash blocks patterns even if tool-guard is evaded. The firewall blocks non-standard ports even if all software layers fail.
101101

102+
### Heartbeat
103+
104+
The control agent runs a periodic heartbeat loop (default: every 10 minutes) that checks system health:
105+
106+
- Are all agent sessions alive?
107+
- Is the Slack bridge responsive?
108+
- Is the email monitor running?
109+
- Are there stale worktrees or stuck todos?
110+
111+
The checklist lives in `HEARTBEAT.md` β€” edit it to add custom checks. The heartbeat extension (`heartbeat.ts`) handles scheduling, error backoff, and the `heartbeat` tool for runtime control. If the checklist is empty, no heartbeat fires (saves tokens).
112+
102113
## Architecture
103114

104115
```
@@ -115,6 +126,7 @@ baudbot_agent (unprivileged uid)
115126
β”œβ”€β”€ ~/.pi/agent/
116127
β”‚ β”œβ”€β”€ extensions/ deployed extensions (read-only)
117128
β”‚ β”œβ”€β”€ skills/ agent-owned (can modify)
129+
β”‚ β”œβ”€β”€ HEARTBEAT.md periodic health check checklist
118130
β”‚ └── baudbot-manifest.json SHA256 integrity hashes
119131
β”œβ”€β”€ ~/workspace/ project repos + worktrees
120132
└── ~/.config/.env secrets (600 perms)

β€Žbin/deploy.shβ€Ž

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,24 @@ else
140140
log "would copy: skills/"
141141
fi
142142

143+
# ── Heartbeat ────────────────────────────────────────────────────────────────
144+
145+
echo "Deploying heartbeat checklist..."
146+
147+
HEARTBEAT_SRC="$STAGE_DIR/skills/control-agent/HEARTBEAT.md"
148+
HEARTBEAT_DEST="$BAUDBOT_HOME/.pi/agent/HEARTBEAT.md"
149+
150+
if [ "$DRY_RUN" -eq 0 ]; then
151+
# HEARTBEAT.md β€” always overwrite (admin-managed checklist)
152+
if [ -f "$HEARTBEAT_SRC" ]; then
153+
as_agent cp "$HEARTBEAT_SRC" "$HEARTBEAT_DEST"
154+
as_agent chmod 644 "$HEARTBEAT_DEST"
155+
log "βœ“ HEARTBEAT.md"
156+
fi
157+
else
158+
log "would copy: HEARTBEAT.md"
159+
fi
160+
143161
# ── Slack Bridge ─────────────────────────────────────────────────────────────
144162

145163
echo "Deploying slack-bridge..."

β€Žpi/extensions/heartbeat.tsβ€Ž

Lines changed: 298 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,298 @@
1+
/**
2+
* Heartbeat Extension
3+
*
4+
* Periodically injects a heartbeat prompt into the agent's conversation so it
5+
* can perform health checks, clean up stale resources, and act proactively
6+
* without waiting for external events.
7+
*
8+
* The heartbeat reads a configurable checklist file (HEARTBEAT.md) and sends
9+
* it as a follow-up message. If the file is empty or missing, no heartbeat
10+
* fires (saves tokens).
11+
*
12+
* Configuration (env vars):
13+
* HEARTBEAT_INTERVAL_MS β€” interval between heartbeats (default: 600000 = 10 min)
14+
* HEARTBEAT_FILE β€” path to checklist file (default: ~/.pi/agent/HEARTBEAT.md)
15+
* HEARTBEAT_ENABLED β€” set to "0" or "false" to disable (default: enabled)
16+
*
17+
* Inspired by OpenClaw's HEARTBEAT.md pattern β€” a user-configurable Markdown
18+
* checklist that the agent evaluates on each tick.
19+
*/
20+
21+
import type { ExtensionAPI, ExtensionContext } from "@mariozechner/pi-coding-agent";
22+
import { Type } from "@sinclair/typebox";
23+
import { StringEnum } from "@mariozechner/pi-ai";
24+
import { existsSync, readFileSync } from "node:fs";
25+
import { homedir } from "node:os";
26+
import { join } from "node:path";
27+
28+
const DEFAULT_INTERVAL_MS = 10 * 60 * 1000; // 10 minutes
29+
const DEFAULT_HEARTBEAT_FILE = join(homedir(), ".pi", "agent", "HEARTBEAT.md");
30+
31+
// Minimum interval to prevent accidental token burn (2 minutes)
32+
const MIN_INTERVAL_MS = 2 * 60 * 1000;
33+
34+
// Maximum consecutive errors before backing off
35+
const MAX_CONSECUTIVE_ERRORS = 5;
36+
const BACKOFF_MULTIPLIER = 2;
37+
const MAX_BACKOFF_MS = 60 * 60 * 1000; // 1 hour
38+
39+
type HeartbeatState = {
40+
enabled: boolean;
41+
intervalMs: number;
42+
heartbeatFile: string;
43+
lastRunAt: number | null;
44+
consecutiveErrors: number;
45+
totalRuns: number;
46+
};
47+
48+
const HEARTBEAT_STATE_ENTRY = "heartbeat-state";
49+
50+
function isDisabledByEnv(): boolean {
51+
const val = process.env.HEARTBEAT_ENABLED?.trim().toLowerCase();
52+
return val === "0" || val === "false" || val === "no";
53+
}
54+
55+
function resolveConfig(): { intervalMs: number; heartbeatFile: string; enabled: boolean } {
56+
const envInterval = parseInt(process.env.HEARTBEAT_INTERVAL_MS || "", 10);
57+
const intervalMs = Math.max(
58+
MIN_INTERVAL_MS,
59+
Number.isFinite(envInterval) ? envInterval : DEFAULT_INTERVAL_MS
60+
);
61+
const heartbeatFile = process.env.HEARTBEAT_FILE?.trim() || DEFAULT_HEARTBEAT_FILE;
62+
const enabled = !isDisabledByEnv();
63+
return { intervalMs, heartbeatFile, enabled };
64+
}
65+
66+
function readHeartbeatFile(filepath: string): string | null {
67+
try {
68+
if (!existsSync(filepath)) return null;
69+
const content = readFileSync(filepath, "utf-8").trim();
70+
// Skip if empty or only comments/whitespace
71+
const meaningful = content
72+
.split("\n")
73+
.filter((line) => {
74+
const trimmed = line.trim();
75+
return trimmed.length > 0 && !trimmed.startsWith("#");
76+
})
77+
.join("\n")
78+
.trim();
79+
return meaningful.length > 0 ? content : null;
80+
} catch {
81+
return null;
82+
}
83+
}
84+
85+
function computeBackoffMs(consecutiveErrors: number, baseInterval: number): number {
86+
if (consecutiveErrors <= 0) return baseInterval;
87+
const backoff = baseInterval * Math.pow(BACKOFF_MULTIPLIER, consecutiveErrors);
88+
return Math.min(backoff, MAX_BACKOFF_MS);
89+
}
90+
91+
export default function heartbeatExtension(pi: ExtensionAPI): void {
92+
let timer: ReturnType<typeof setTimeout> | null = null;
93+
let state: HeartbeatState = {
94+
enabled: true,
95+
intervalMs: DEFAULT_INTERVAL_MS,
96+
heartbeatFile: DEFAULT_HEARTBEAT_FILE,
97+
lastRunAt: null,
98+
consecutiveErrors: 0,
99+
totalRuns: 0,
100+
};
101+
102+
function saveState() {
103+
pi.appendEntry(HEARTBEAT_STATE_ENTRY, {
104+
lastRunAt: state.lastRunAt,
105+
consecutiveErrors: state.consecutiveErrors,
106+
totalRuns: state.totalRuns,
107+
});
108+
}
109+
110+
function armTimer() {
111+
if (timer) clearTimeout(timer);
112+
timer = null;
113+
114+
if (!state.enabled) return;
115+
116+
const delay = computeBackoffMs(state.consecutiveErrors, state.intervalMs);
117+
timer = setTimeout(() => {
118+
fireHeartbeat();
119+
}, delay);
120+
}
121+
122+
function fireHeartbeat() {
123+
const content = readHeartbeatFile(state.heartbeatFile);
124+
if (!content) {
125+
// No checklist β€” skip silently, re-arm for next interval
126+
armTimer();
127+
return;
128+
}
129+
130+
const now = Date.now();
131+
state.lastRunAt = now;
132+
state.totalRuns += 1;
133+
134+
const prompt = [
135+
`πŸ«€ **Heartbeat** (run #${state.totalRuns}, ${new Date(now).toISOString()})`,
136+
``,
137+
`Review the following checklist and take action on any items that need attention.`,
138+
`If everything is healthy, respond briefly with what you checked. Do NOT take action unless something is wrong.`,
139+
``,
140+
`---`,
141+
content,
142+
`---`,
143+
``,
144+
`If you find issues, fix them. If everything looks good, say so briefly and move on.`,
145+
].join("\n");
146+
147+
pi.sendMessage(
148+
{
149+
customType: "heartbeat",
150+
content: prompt,
151+
display: true,
152+
},
153+
{
154+
deliverAs: "followUp",
155+
triggerTurn: true,
156+
}
157+
);
158+
159+
saveState();
160+
// Re-arm after firing (the agent_end handler will also re-arm on error)
161+
armTimer();
162+
}
163+
164+
function stopTimer() {
165+
if (timer) {
166+
clearTimeout(timer);
167+
timer = null;
168+
}
169+
}
170+
171+
// ── Tool: heartbeat control ───────────────────────────────────────────────
172+
173+
pi.registerTool({
174+
name: "heartbeat",
175+
label: "Heartbeat",
176+
description:
177+
"Manage the periodic heartbeat loop. " +
178+
"Actions: status (check state), pause (stop heartbeats), resume (restart), " +
179+
"trigger (fire one now), config (show configuration).",
180+
parameters: Type.Object({
181+
action: StringEnum(["status", "pause", "resume", "trigger", "config"] as const),
182+
}),
183+
async execute(_toolCallId, params, _signal, _onUpdate, _ctx) {
184+
switch (params.action) {
185+
case "status": {
186+
const nextIn = timer
187+
? `~${Math.round(computeBackoffMs(state.consecutiveErrors, state.intervalMs) / 1000)}s`
188+
: "paused";
189+
return {
190+
content: [
191+
{
192+
type: "text" as const,
193+
text: [
194+
`Heartbeat Status:`,
195+
` Enabled: ${state.enabled ? "βœ…" : "⏹"}`,
196+
` Interval: ${state.intervalMs / 1000}s`,
197+
` Next fire: ${nextIn}`,
198+
` Total runs: ${state.totalRuns}`,
199+
` Consecutive errors: ${state.consecutiveErrors}`,
200+
` Last run: ${state.lastRunAt ? new Date(state.lastRunAt).toISOString() : "never"}`,
201+
` Checklist: ${state.heartbeatFile}`,
202+
].join("\n"),
203+
},
204+
],
205+
};
206+
}
207+
208+
case "pause": {
209+
state.enabled = false;
210+
stopTimer();
211+
saveState();
212+
return {
213+
content: [{ type: "text" as const, text: "⏹ Heartbeat paused." }],
214+
};
215+
}
216+
217+
case "resume": {
218+
state.enabled = true;
219+
state.consecutiveErrors = 0;
220+
armTimer();
221+
saveState();
222+
return {
223+
content: [
224+
{
225+
type: "text" as const,
226+
text: `βœ… Heartbeat resumed (every ${state.intervalMs / 1000}s).`,
227+
},
228+
],
229+
};
230+
}
231+
232+
case "trigger": {
233+
fireHeartbeat();
234+
return {
235+
content: [{ type: "text" as const, text: "πŸ«€ Heartbeat triggered." }],
236+
};
237+
}
238+
239+
case "config": {
240+
const content = readHeartbeatFile(state.heartbeatFile);
241+
return {
242+
content: [
243+
{
244+
type: "text" as const,
245+
text: [
246+
`Heartbeat Configuration:`,
247+
` File: ${state.heartbeatFile}`,
248+
` File exists: ${existsSync(state.heartbeatFile) ? "yes" : "no"}`,
249+
` Has content: ${content ? "yes" : "no (empty or comments only)"}`,
250+
` Interval: ${state.intervalMs / 1000}s (env: HEARTBEAT_INTERVAL_MS)`,
251+
` Min interval: ${MIN_INTERVAL_MS / 1000}s`,
252+
` Backoff multiplier: ${BACKOFF_MULTIPLIER}x per error`,
253+
` Max backoff: ${MAX_BACKOFF_MS / 1000}s`,
254+
``,
255+
content ? `Current checklist:\n${content}` : `(no checklist loaded)`,
256+
].join("\n"),
257+
},
258+
],
259+
};
260+
}
261+
262+
default:
263+
return {
264+
content: [{ type: "text" as const, text: `Unknown action: ${params.action}` }],
265+
};
266+
}
267+
},
268+
});
269+
270+
// ── Lifecycle ─────────────────────────────────────────────────────────────
271+
272+
pi.on("session_start", async (_event, ctx) => {
273+
// Restore persisted state
274+
for (const entry of ctx.sessionManager.getEntries()) {
275+
const e = entry as { type: string; customType?: string; data?: any };
276+
if (e.type === "custom" && e.customType === HEARTBEAT_STATE_ENTRY && e.data) {
277+
if (typeof e.data.consecutiveErrors === "number")
278+
state.consecutiveErrors = e.data.consecutiveErrors;
279+
if (typeof e.data.totalRuns === "number") state.totalRuns = e.data.totalRuns;
280+
if (typeof e.data.lastRunAt === "number") state.lastRunAt = e.data.lastRunAt;
281+
}
282+
}
283+
284+
// Apply env config
285+
const config = resolveConfig();
286+
state.intervalMs = config.intervalMs;
287+
state.heartbeatFile = config.heartbeatFile;
288+
state.enabled = config.enabled;
289+
290+
if (state.enabled) {
291+
armTimer();
292+
}
293+
});
294+
295+
pi.on("session_shutdown", async () => {
296+
stopTimer();
297+
});
298+
}
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Heartbeat Checklist
2+
3+
Check each item and take action only if something is wrong.
4+
5+
- Check all agent sessions are alive (`list_sessions` β€” confirm `sentry-agent` exists, check for orphaned `dev-agent-*` sessions with no matching active todo)
6+
- Verify Slack bridge is responsive (`curl -s -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1:7890/send -H 'Content-Type: application/json' -d '{}'` β†’ should return 400)
7+
- Check email monitor is running (`email_monitor status` β€” should show active)
8+
- Check for stale worktrees in `~/workspace/worktrees/` that don't correspond to active in-progress todos β€” clean them up with `git worktree remove`
9+
- Check for stuck todos (status `in-progress` for more than 2 hours with no corresponding dev-agent session) β€” escalate to user via Slack

0 commit comments

Comments
Β (0)