Skip to content

Latest commit

 

History

History
124 lines (94 loc) · 5.56 KB

README.md

File metadata and controls

124 lines (94 loc) · 5.56 KB

Vultr Benchmark Orchestration

⚠️ Fair warning: This was vibe-coded from a plane using OpenClaw. It works, but don't expect enterprise-grade polish. PRs welcome, flames not so much. 🦀✈️

Instances are created with a model list baked into their userdata, run benchmarks autonomously, and self-destruct when done. Your laptop only needs to be running long enough to fire the vultr instance create calls (~10 seconds).

How It Works

Your laptop                    Vultr API              Vultr Instance
    |                              |                        |
    |-- instance create x N ------>|                        |
    |   (userdata = model list)    |                        |
    |<-- instance IDs -------------|                        |
    |  [laptop can close now]      |                        |
                                   |-- boots from snapshot ->|
                                                            |-- reads models from
                                                            |   metadata API
                                                            |-- uv run benchmark.py --register
                                                            |-- [Slack: started + claim URL]
                                                            |-- uv run benchmark.py --model ...
                                                            |-- uv run benchmark.py --model ...
                                                            |-- [Slack: done + result URLs]
                                                            |-- vultr instance delete $SELF

Each instance also schedules a safety-net self-destruct via at now + 5 hours at startup, so orphaned instances are cleaned up even if the runner crashes.

Running Benchmarks

uv run orchestrate_vultr.py --count 10

# Or override with an explicit list
uv run orchestrate_vultr.py --count 10 \
  --models \
  anthropic/claude-opus-4.5 \
  openai/gpt-4o \
  google/gemini-2.5-flash \
  ...

Models are distributed round-robin across instances (e.g. 30 models across 10 instances = 3 models per instance). The script exits as soon as all instances are created.

Options:

Option Default Description
--models (optional) Model IDs to benchmark (space-separated)
--models-file default-models.yml YAML file used when --models is not provided
--count 1 Number of instances; models distributed across them
--region atl Vultr region
--plan vc2-1c-2gb Vultr instance plan
--snapshot (see VultrConfig in orchestrate_vultr.py) Vultr snapshot ID — update after re-bootstrapping
--ssh-keys a4b8f6d9-... Vultr SSH key ID

Monitoring:

# Watch instances disappear as they finish
watch vultr instance list

# Tail logs on a running instance
ssh root@<ip> tail -f /var/log/bench-runner.log

# View systemd service output
ssh root@<ip> journalctl -u bench-runner -f

Bootstrapping a New Vultr Snapshot

See docs/bootstrapping-snapshot.md

Files

File Purpose
orchestrate_vultr.py Fire-and-forget launcher — creates instances and exits
bench_runner.sh Runs on each instance; reads models from metadata, benchmarks, self-destructs
bench-runner.service systemd unit that starts bench_runner.sh on first boot
bootstrap_instance.sh One-shot setup script for building a new snapshot image
setup_snapshot.sh Lighter alternative to bootstrap — installs just the runner files onto an existing instance
create_instance.sh Convenience shell script with the full model list pre-filled
utilities/delete_instances.sh Emergency cleanup: delete instances by ID
utilities/reaper.sh Automated cleanup: delete stale instances older than TTL

Automated Cleanup (Reaper)

While instances self-destruct on completion, sometimes things go wrong. The reaper script provides an external safety net by deleting any bench-* instances older than a configurable TTL (default: 6 hours).

# Preview what would be deleted
./utilities/reaper.sh --dry-run

# Delete stale instances (default: older than 6 hours)
./utilities/reaper.sh

# Custom TTL (2 hours = 7200 seconds)
./utilities/reaper.sh --ttl 7200

Recommended cron setup (run every hour):

0 * * * * /path/to/pinchbench-scripts/utilities/reaper.sh >> /var/log/reaper.log 2>&1

This catches any instances that:

  • Failed to self-destruct due to crashes
  • Got stuck in a hung state
  • Had their at safety-net fail for any reason