Skip to content

Commit b189eaf

Browse files
JSONboredJSONbored
authored
feat(miner): add backup/restore tooling for local SQLite state (#5623)
Adds scripts/backup-miner.sh and restore-miner.sh: every *.sqlite3 store under GITTENSORY_MINER_CONFIG_DIR is discovered by glob (not a hardcoded filename list) and backed up via SQLite's own online .backup command + a PRAGMA integrity_check, mirroring the safe-backup pattern operations-runbook.md's corruption scenario already documents. Restore validates every file in the chosen backup before copying anything into place, requires --yes (overwrites live state), and clears stale -wal/-shm sidecars after restoring. Documents both in a new "Backup and restore" section in operations-runbook.md, and points the corruption scenario's Tier B at the new restore script instead of a vague manual-copy reference.
1 parent b51220e commit b189eaf

4 files changed

Lines changed: 563 additions & 1 deletion

File tree

‎packages/gittensory-miner/docs/operations-runbook.md‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,42 @@ gittensory-miner status --json
100100

101101
4. **Claims are local bookkeeping only.** Two miners on different machines claiming the same GitHub issue is a **fleet coordination** problem (duplicate-cluster adjudication in the engine), not something SQLite resolves — split state dirs and use operational claim hygiene.
102102

103+
## Backup and restore
104+
105+
Proactive tooling (#4872), not just the reactive "ledger corrupted" scenario below — run `backup-miner.sh` on a
106+
schedule (cron, systemd timer, etc.) so a good restore point always exists before anything goes wrong.
107+
108+
- **[`scripts/backup-miner.sh`](../../../scripts/backup-miner.sh)** — backs up every `*.sqlite3` file currently
109+
present under `GITTENSORY_MINER_CONFIG_DIR` into a new timestamped directory, using SQLite's own online
110+
`.backup` command (safe even while the miner is running — see the corruption scenario's warning below about
111+
why a plain `cp` is not) plus a `PRAGMA integrity_check` on each resulting file before it's kept. Stores
112+
discovered by glob, not a hardcoded list, so a newly added store is backed up automatically without this doc
113+
or the script needing an update.
114+
115+
```sh
116+
sh scripts/backup-miner.sh
117+
# Env overrides: GITTENSORY_MINER_CONFIG_DIR (source), GITTENSORY_MINER_BACKUP_DIR (default
118+
# $GITTENSORY_MINER_CONFIG_DIR/backups), GITTENSORY_MINER_BACKUP_RETAIN (default 7 — oldest backups beyond
119+
# this count are pruned after a fully successful run; a run with any failed store skips pruning so no older,
120+
# good backup is ever lost to make room for a bad one).
121+
```
122+
123+
- **[`scripts/restore-miner.sh`](../../../scripts/restore-miner.sh)** — the read side. **Stop the miner first**
124+
(this script does not detect a running process). Validates every store file in the chosen backup with
125+
`PRAGMA integrity_check` **before** copying anything into place — a half-good backup can never produce a
126+
half-restored state directory. Requires an explicit `--yes` flag (it overwrites live state) and defaults to
127+
the newest backup when no directory is given:
128+
129+
```sh
130+
sh scripts/restore-miner.sh --yes # newest backup
131+
sh scripts/restore-miner.sh --yes /path/to/backups/<ts> # a specific one
132+
gittensory-miner doctor --json # verify afterward
133+
```
134+
135+
Also removes any leftover `-wal`/`-shm` sidecar files from the live directory after restoring each store —
136+
those hold in-flight writes from *before* the restore, and leaving them in place would let SQLite silently
137+
replay stale pre-restore writes back on top of the freshly restored file on next open.
138+
103139
## Scenario: ledger corrupted
104140

105141
**Symptoms**
@@ -136,7 +172,7 @@ gittensory-miner status --json
136172
| Tier | When | Action |
137173
|------|------|--------|
138174
| **A — single store reset** | One ledger is corrupt; others healthy; you accept losing that store's history | Remove only the bad `*.sqlite3` (and any `-wal`/`-shm` siblings). Next command recreates an empty store. |
139-
| **B — restore from backup** | You have a recent quiesced backup | Stop miner → restore the known-good file → restart. |
175+
| **B — restore from backup** | You have a recent backup from `backup-miner.sh` (see **Backup and restore** above) | Stop miner → `sh scripts/restore-miner.sh --yes` → restart. |
140176
| **C — full re-init** | Multiple files suspect or state is disposable | Archive dir → `gittensory-miner init` → reconfigure env/goals. Rebuild claims/plans from GitHub metadata as needed. |
141177

142178
4. **Never copy a live SQLite file** from a running miner as backup — stop first, or use SQLite's `.backup` command:

‎scripts/backup-miner.sh‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
#!/bin/sh
2+
# gittensory-miner local-state backup (#4872): every store is an independent SQLite file directly under
3+
# GITTENSORY_MINER_CONFIG_DIR (packages/gittensory-miner/docs/operations-runbook.md's "Local state at a
4+
# glance") -- there is no Postgres/Qdrant involved, so this is deliberately a simpler sibling to
5+
# scripts/backup.sh, not a reuse of it (that script's manifest/multi-target logic has nothing to compose with
6+
# here). Backs up EVERY *.sqlite3 file currently present, discovered by glob rather than a hardcoded filename
7+
# list -- the miner package grows new stores over time (17 as of this writing), and a hardcoded list would
8+
# silently go stale the next time one is added, exactly the kind of drift this repo's generated-reference
9+
# checks exist to prevent elsewhere.
10+
#
11+
# Uses SQLite's own online-backup command (".backup"), the same safe-even-while-live mechanism
12+
# operations-runbook.md's "ledger corrupted" scenario already documents -- NOT a plain `cp`, which can capture
13+
# a torn snapshot mid-write. Each backed-up file is then integrity-checked before being kept.
14+
#
15+
# Usage:
16+
# sh scripts/backup-miner.sh
17+
# GITTENSORY_MINER_CONFIG_DIR=/data/miner GITTENSORY_MINER_BACKUP_RETAIN=14 sh scripts/backup-miner.sh
18+
set -eu
19+
20+
STATE_DIR="${GITTENSORY_MINER_CONFIG_DIR:-$HOME/.config/gittensory-miner}"
21+
OUT_DIR="${GITTENSORY_MINER_BACKUP_DIR:-$STATE_DIR/backups}"
22+
RETAIN="${GITTENSORY_MINER_BACKUP_RETAIN:-7}"
23+
24+
if ! command -v sqlite3 >/dev/null 2>&1; then
25+
echo "[backup-miner] sqlite3 not found; cannot take a safe online backup" >&2
26+
exit 1
27+
fi
28+
29+
if [ ! -d "$STATE_DIR" ]; then
30+
echo "[backup-miner] state dir not found: $STATE_DIR (nothing to back up)" >&2
31+
exit 1
32+
fi
33+
34+
TS=$(date -u +%Y%m%dT%H%M%SZ)
35+
DEST="$OUT_DIR/$TS"
36+
mkdir -p "$DEST"
37+
38+
TOTAL=0
39+
FAILED=0
40+
BACKED_UP=0
41+
for db in "$STATE_DIR"/*.sqlite3; do
42+
[ -e "$db" ] || continue # glob matched nothing
43+
TOTAL=$((TOTAL + 1))
44+
name=$(basename "$db")
45+
dest_db="$DEST/$name"
46+
if ! sqlite3 "$db" ".backup '$dest_db'" 2>/dev/null; then
47+
echo "[backup-miner] ERROR: online backup failed for $name" >&2
48+
FAILED=1
49+
continue
50+
fi
51+
result=$(sqlite3 "$dest_db" 'PRAGMA integrity_check;' 2>/dev/null | head -1 || true)
52+
if [ "$result" != "ok" ]; then
53+
echo "[backup-miner] ERROR: integrity check failed for $name (${result:-no output})" >&2
54+
rm -f "$dest_db"
55+
FAILED=1
56+
continue
57+
fi
58+
chmod 600 "$dest_db"
59+
BACKED_UP=$((BACKED_UP + 1))
60+
echo "[backup-miner] backed up $name"
61+
done
62+
63+
# Two distinct empty-outcome cases, deliberately handled differently: the glob matching NOTHING means there
64+
# was never anything to back up (remove the now-useless empty timestamped dir); every MATCHED file failing is
65+
# a real backup failure (fall through to the FAILED branch below, which -- like backup.sh's own equivalent --
66+
# keeps the directory and its error output for debugging, and skips retention pruning).
67+
if [ "$TOTAL" -eq 0 ]; then
68+
echo "[backup-miner] no *.sqlite3 files found under $STATE_DIR" >&2
69+
rmdir "$DEST" 2>/dev/null || true
70+
exit 1
71+
fi
72+
chmod 700 "$DEST"
73+
74+
if [ "$FAILED" = 1 ]; then
75+
echo "[backup-miner] FAILED ($TS): one or more stores did not back up cleanly; see errors above" >&2
76+
echo "[backup-miner] skipping retention prune so no older, fully-good backup is lost" >&2
77+
exit 1
78+
fi
79+
80+
# Retention: keep the newest $RETAIN timestamped backup directories, prune the rest. Mirrors
81+
# scripts/backup.sh's own `ls -1t | tail -n +N+1 | while read` idiom.
82+
# shellcheck disable=SC2012
83+
ls -1t "$OUT_DIR" 2>/dev/null | tail -n +"$((RETAIN + 1))" | while IFS= read -r old; do
84+
echo "[backup-miner] pruning old backup: $old"
85+
rm -rf "${OUT_DIR:?}/$old"
86+
done
87+
88+
echo "[backup-miner] complete ($TS); backed up $BACKED_UP store(s) to $DEST; retaining newest $RETAIN"

‎scripts/restore-miner.sh‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
#!/bin/sh
2+
# gittensory-miner local-state restore (#4872): the read side of scripts/backup-miner.sh. STOP the miner (and
3+
# any loop/systemd/docker service) before running this -- it overwrites the live state directory and does not
4+
# detect a running process itself, the same "stop first" precondition operations-runbook.md's "ledger
5+
# corrupted" scenario already documents for manual recovery.
6+
#
7+
# Validates EVERY store file in the chosen backup (via PRAGMA integrity_check) BEFORE copying anything into
8+
# place: a half-good backup must never produce a half-restored, mismatched-vintage state directory. Removes
9+
# any leftover -wal/-shm sidecar files from the LIVE directory after restoring each store -- those hold
10+
# in-flight, not-yet-checkpointed writes from BEFORE the restore, and leaving them in place would let SQLite
11+
# silently replay stale pre-restore writes back on top of the freshly restored file on next open.
12+
#
13+
# Usage:
14+
# sh scripts/restore-miner.sh --yes # restores the newest backup
15+
# sh scripts/restore-miner.sh --yes /path/to/backups/<ts> # restores a specific backup
16+
set -eu
17+
18+
STATE_DIR="${GITTENSORY_MINER_CONFIG_DIR:-$HOME/.config/gittensory-miner}"
19+
BACKUP_DIR="${GITTENSORY_MINER_BACKUP_DIR:-$STATE_DIR/backups}"
20+
21+
usage() {
22+
cat <<USAGE >&2
23+
Usage: $0 --yes [BACKUP_DIR]
24+
25+
Restores gittensory-miner local state from a backup produced by backup-miner.sh.
26+
27+
BACKUP_DIR A specific timestamped backup directory. Defaults to the newest one
28+
under \$GITTENSORY_MINER_BACKUP_DIR ($BACKUP_DIR).
29+
--yes Required. This OVERWRITES the live state directory ($STATE_DIR).
30+
31+
STOP the miner (and any loop/systemd/docker service using this state dir) first --
32+
this script does not check whether one is still running.
33+
USAGE
34+
}
35+
36+
CONFIRMED=0
37+
SOURCE=""
38+
for arg in "$@"; do
39+
case "$arg" in
40+
--yes) CONFIRMED=1 ;;
41+
-h | --help)
42+
usage
43+
exit 0
44+
;;
45+
*) SOURCE=$arg ;;
46+
esac
47+
done
48+
49+
if [ "$CONFIRMED" != 1 ]; then
50+
usage
51+
echo "[restore-miner] refusing to restore without --yes (this overwrites live state)" >&2
52+
exit 1
53+
fi
54+
55+
if ! command -v sqlite3 >/dev/null 2>&1; then
56+
echo "[restore-miner] sqlite3 not found; cannot verify backup integrity before restoring" >&2
57+
exit 1
58+
fi
59+
60+
if [ -z "$SOURCE" ]; then
61+
SOURCE=$(ls -1dt "$BACKUP_DIR"/*/ 2>/dev/null | head -1 || true)
62+
if [ -z "$SOURCE" ]; then
63+
echo "[restore-miner] no backups found under $BACKUP_DIR" >&2
64+
exit 1
65+
fi
66+
fi
67+
SOURCE=${SOURCE%/}
68+
69+
if [ ! -d "$SOURCE" ]; then
70+
echo "[restore-miner] backup directory not found: $SOURCE" >&2
71+
exit 1
72+
fi
73+
74+
FOUND=0
75+
for db in "$SOURCE"/*.sqlite3; do
76+
[ -e "$db" ] || continue
77+
FOUND=1
78+
result=$(sqlite3 "$db" 'PRAGMA integrity_check;' 2>/dev/null | head -1 || true)
79+
if [ "$result" != "ok" ]; then
80+
echo "[restore-miner] ERROR: $(basename "$db") failed integrity check (${result:-no output}); aborting, nothing was restored" >&2
81+
exit 1
82+
fi
83+
done
84+
if [ "$FOUND" -eq 0 ]; then
85+
echo "[restore-miner] no *.sqlite3 files found in $SOURCE" >&2
86+
exit 1
87+
fi
88+
89+
echo "[restore-miner] restoring from $SOURCE into $STATE_DIR"
90+
mkdir -p "$STATE_DIR"
91+
chmod 700 "$STATE_DIR"
92+
for db in "$SOURCE"/*.sqlite3; do
93+
[ -e "$db" ] || continue
94+
name=$(basename "$db")
95+
cp "$db" "$STATE_DIR/$name"
96+
chmod 600 "$STATE_DIR/$name"
97+
rm -f "$STATE_DIR/$name-wal" "$STATE_DIR/$name-shm"
98+
echo "[restore-miner] restored $name"
99+
done
100+
101+
echo "[restore-miner] complete. Run 'gittensory-miner doctor --json' to verify."

0 commit comments

Comments
 (0)