dotbot: site area roles, a default site, site packs and the calibration span

[geonnave]

Problem

Every default that needs "the place where experiments happen" looked for an area literally named arena: swarm calibrate-lh2 collect defaulted to arena:corners, the console's calibration setup opened on arena, the console session API defaulted to arena:corners, and the simulator placed world-file robots in arena while a --robots N fleet went to field. A site that did not use that exact name got silent fallbacks, and a site could not say which of its areas is for parking, or that a small bench patch should stay out of the way.

Around that, three gaps made a site hard to start with and hard to share. A fresh install had no site at all, so nothing worked until someone measured a room and typed a table, and the root config_sample.toml predated --conn. A site lived only as an inline table in one person's dotbot.toml, with its calibrations in a separate home directory, so handing a site to a colleague meant copying pieces by hand; nothing stopped the controller from loading a calibration made for another site. And a loaded calibration was invisible on the map: nothing showed which part of the site it actually covers, how old it is, or that robots hold homographies it never solved.

Approach

This PR lands in four phases plus the follow-ups from a demo run; each is reviewable on its own (commit ranges below).

Phase 2: area roles, one field, calibration defaults

Site areas get an optional role: field (where experiments happen and what gets calibrated), staging (where robots park and charge), or corner (a small patch that may overlap other areas, hidden in the console until turned on). An area named after a role has it; role = "..." gives one to any other name, and an explicit role beats the name. A site has at most one field.

One resolver, Site.field, answers "which area is the field" for every reader: the area with role field, else the first area that is neither staging nor corner, else the first area, else the whole extent. The simulator (both placement paths), calibrate-lh2 collect, the console session API, the camera collect --area default and the console's calibration setup all go through it, so "arena" no longer appears in any default.

Calibration keeps the four-corner procedure but chooses the corners differently: collect with no flag uses the field's corners; --over <area> uses another area's corners (e.g. --over dev-corner for bench work); --square <mm> uses a centred square in the field and says the rest is extrapolated; --points ... is unchanged. The three are mutually exclusive. The saved calibration records how its points were chosen as a points_from table, { kind = "field" | "over" | "square" | "points", area?, side_mm? }, which the site payload carries as the same object and the console renders from its fields; it stays out of the calibration id. A file with the pre-release string form of points_from is refused rather than read. The whole field is the default because a four-point homography is tight inside its points' span and its error grows outside it; a smaller span only saves taping.

GET /controller/site lists areas in declared order with their roles and names the field. The console starts corners hidden and stores per-area visibility as {name: shown} over the role defaults. The simulator example becomes a generic site, virtual-lab, with a field and a staging area, and its config is a plain dotbot.toml, so dotbot run simulator --robots 200 works from its folder with no -c and no --site.

Phase 3: a default site from config init, examples that read the site

dotbot config init now writes site = "default" and a [sites.default] table: a 2 x 2 m field centred in a 5 x 5 m extent, with a 0.6 m staging strip along the field's bottom edge. --field sizes it (1500, 1.5m, 1500mm, 2000x3000, 1.5x2m; a bare number is mm, decimals only on m, 100 mm to 100 m, with a "did you mean 1.5m?" hint below that), everything else derives from the field, and above 5 m on a side it warns that one LH2 station rarely covers that well. --site names the site; --global writes the same template into ~/.dotbot/config.toml; --force keeps its whole-file meaning. The site is plain TOML in the user's own file rather than a hidden built-in, so "open dotbot.toml" is the whole step from the default to a measured site.

config_sample.toml is replaced by dotbot.example.toml, equal to what config init writes, with a test that keeps the two identical. It is not named dotbot.toml because that would be discovered from the repo root and shadow ~/.dotbot/config.toml.

Site.staging and Site.field_or_fallback serve the examples: motions places its shapes in an area (--area, default the field), the naming game walks inside the field, and the charging example queues on staging's border with the field, charges at staging's far edge, and parks along the field's opposite edge.

Phase 4: site packs, dotbot site add/export, refusing another site's calibration

A site pack is a folder <name>/site.toml (the keys of [sites.<name>]) with optional <name>/calibrations/. A new top-level site_dirs (default ["sites", "~/.dotbot/sites"], relative entries resolved from the config file's folder) lists where packs are found. An inline [sites.<name>] wins over a pack of the same name, with a one-line notice naming the pack it shadows; config show prints each site's source.

Calibration lookup keeps explicit selection (path, tag or id prefix, never "latest") but searches ordered folders: the pack's calibrations/, then ~/.dotbot/calibrations/<site>/, where collect keeps writing. The first folder with a match wins. At load, the controller refuses an LH2 or camera calibration whose recorded site differs from the active one, or whose anchor differs when both record one; a refused load is a CLI error, not a traceback. Push is already gated on the robots' reported site, and reframe reads another site's file by design, so neither gains the check.

dotbot site add <folder|zip|git URL|-> copies a pack into ~/.dotbot/sites/<name>/. - reads a zip from stdin, so curl -L https://.../lab.zip | dotbot site add - works; the site's name is then the zip's top folder, as site export writes it, and a TTY stdin or a bare https://...zip argument gets an error saying to pipe it. The copy is staged in a hidden sibling folder and renamed into place, so a --force that fails part-way leaves the old pack as it was. It refuses to overwrite without --force, and refuses a name that is not letters, digits, - and _, a pack holding links, and git's command-running ext:: transport); dotbot site export <name> [--out file.zip] [--with-calibrations] writes one, turning an inline site into a site.toml. The same plain files work with git clone, unzip or cp.

Phase 5: the calibrated span on the map, calibration warnings

GET /controller/site gains calibration: id, tag, created_at, and each placement's points_mm and points_from. The console outlines each placement's convex hull and hatches the rest of the site's extent, so an extrapolated position is visible rather than a footnote; overlapping placements leave one clear region, and a site with no extent gets the outline only. The tooltip names the tag and id, the age in days and how the points were chosen. It has its own Layers section, shown only when a calibration is loaded.

The controller warns at load when the LH2 calibration is older than [run.controller] lh2_calibration_max_age_days (default 30, 0 disables it, also settable through its DOTBOT_RUN_CONTROLLER_ env var), and when robots hold homographies for station indices the loaded calibration did not solve, as one line naming the stations, logged again only when that set changes. A moved station keeps its index, so age is the only guard for that case.

Follow-ups from the demo run

• run simulator --area <spec> places the simulated fleet (a --robots N fleet, and any world-file robot without a position) in a named area, a +-joined composite such as field+staging, or x,y,w,h; the default is still the field, and [run.controller] simulator_area sets it from the config. The name matches the other --area flags (camera collect, motions): an area spec, defaulting to the field.
• The fleet grid takes the area's shape. The column count follows the area's aspect ratio, capped by what fits across its width, so a 2 x 4 m area holds 200 robots at 200 mm (it held 100 with the old near-square grid) and a square field still gets a square block. Capacity is floor(w / pitch) * floor(h / pitch), keeping the half-pitch margin all round.
• Every generated robot faces up. Positions are reported at the photodiode, a lever arm ahead of the axle, so the old two halves facing apart showed about 300 mm between them against 200 mm everywhere else.
• The unsolved-station warning is one line, instead of one per robot and station (300 lines for 50 simulated robots). Simulated robots also stop claiming all eight stations: without a calibrated in their world file, they hold exactly the stations of the controller's loaded calibration, and all eight only when none is loaded.
• The virtual-lab example has one staging area. Its separate charging strip, which also had role = "staging", read as a second staging area in the console; the charging example uses the site's staging, which is unchanged.
• Roles show in the console: each Layers row carries its role as a small tag, areas are coloured by role (every field alike, every staging area alike) with role-less areas on the remaining colours, and a shown corner is drawn with a heavier, finer-dashed line. config show names the active site and prints each area's role, saying when it is implied by the name.
• Wording: "1 calibration file", "Using config file at" / "No config file found" capitalised alike, no config notice for site add (it only writes a pack), config show no longer prints "No config file found" beneath a site pack it listed, and the --background-map help typo.

Breaking changes

The project is in beta and these are clean breaks, with no compatibility shims:

• Areas are ordered as declared, in the site API and everywhere the "first area" fallback applies.
• The "arena" defaults are gone. collect, the session API, the console calibration setup and simulator placement use the field. A site whose experiment area is called arena should rename it to field or give it role = "field".
• The simulator example's site is renamed to virtual-lab, and its config file to dotbot/examples/simulator_fleet/dotbot.toml.
• The console's area-visibility storage key changed from dotbot.console.hiddenAreas to dotbot.console.areaVisibility, with no migration: areas a browser had hidden show again once, and corner areas start hidden.
• Two fields in one site is a config error, naming both.
• motions drops --arena-size for --area, defaulting to the controller's field.
• The charging example needs a staging area and refuses a site without one.
• The controller refuses a calibration from another site, by name, and by anchor when both record one, including a file passed by path.
• config_sample.toml is removed in favour of dotbot.example.toml.
• A --robots N fleet is laid out differently: its grid follows the area's shape, every robot faces up, and more robots fit a non-square area. A file written with --write-init-state before this change still runs as written.
• A simulated robot without calibrated in its world file reports the loaded calibration's stations rather than all eight.
• Console area colours follow roles, so an area may change colour once.
• The virtual-lab example drops its charging area.

What a reviewer should check

• The field fallback order in Site.field (dotbot/site.py): role, then first non-staging/non-corner, then first area, then the extent. It decides behaviour on every existing config without roles.
• An explicit role beating the one the name implies (area_role in dotbot/area.py).
• --square reporting extrapolation, and the error when two of --points / --over / --square are given (dotbot/calibration/points.py, dotbot/cli/swarm_lh2.py).
• --field parsing and the derived layout (dotbot/cli/config_cmd.py), and that dotbot.example.toml stays byte-equal to config init's output.
• Pack precedence and lookup (dotbot/site_packs.py, resolve_calibration_spec in dotbot/calibration/lighthouse2.py): inline beats pack, first folder wins, and anchors compared only when both sides record one (the default site and older configs record none).
• The "unsolved station" interpretation (dotbot/controller.py). Robots do not report which stations they see, only the calibrated bitmask of the homographies they hold, so the warning fires when robots hold a homography for an index the loaded calibration does not solve; its robot count is the count when the set of stations last changed. It cannot catch a station that is visible but never calibrated. If that reading of the requirement is wrong, this is the place to say so.
• The hatch geometry (dotbot/console-web/src/calibrationSpan.ts): site extent masked by the placement hulls.
• The grid shape (_grid_shape in dotbot/dotbot_simulator.py): columns from the area's aspect ratio, raised so the rows fit and capped at what fits across. Rows are not always filled to the full width: a 2 x 4 m area takes 150 robots as 17 rows of 9, and 200 as 20 rows of 10.

Validation

• hatch test: 1285 passed, plus 5 failures specific to the local environment (they pass with a scratch HOME).
• CI green on all checks at b7eb271 (Linux, macOS, Windows, console, documentation, readthedocs), including the Windows run, whose home-directory and path handling the site pack tests now account for.
• Scenario tests: 14/14.
• Console: lint, vitest (792 passed) and npm run build (vitest does not typecheck, so the build is the type check).
• pre-commit run --all-files clean; docs build with -W --keep-going -n clean after the docs sweep, and readthedocs green.
• After the docs sweep: pytest 1291 passed, plus one local failure on port 8001 held by a server already running on this machine; console build clean. The charging example ran against config init's default site and queued its robots along the field and staging border.
• Smoke runs:
  • dotbot run simulator --robots 50 from inside dotbot/examples/simulator_fleet/, without -c: GET /controller/site names virtual-lab with its areas.
  • config init in an empty folder, then dotbot run simulator --robots 20: site default, every robot inside the field, and motions -m square centred on (2500, 2500). With a C405-shaped layout, the charging example queued robots along y = 2000, charged at y = 3893 and parked at (300, 307).
  • With HOME in a scratch folder: site export c405-arena --out c405.zip --with-calibrations, site add c405.zip, then config show from another folder listed c405-arena from ~/.dotbot/sites/, and run controller --site c405-arena --lh2-calibration <id> loaded the calibration from the pack.
  • dotbot -c <C405 config> run simulator --robots 150 --area field+staging: 150 robots in 17 rows of 9, photodiodes 200 mm apart in both directions, all inside the 2 x 4 m area; --robots 200 fills it as 20 rows of 10; without --area the same command refuses with "at most 100 do". With a world file whose robots claim all eight stations and a one-station calibration, the controller logged one warning line.
  • The simulator with a fixture calibration and a 7-day limit logged the age warning (19 days); /controller/site carried the placement, and a headless screenshot of /console/ showed the outline over the field with staging hatched.

Nothing here reaches the robots: the firmware carries a site name and a validity rectangle, not areas, so no hardware validation applies.

Commits by phase

Review in this order; each range builds on the one before.

Phase	Commits	Range
2: roles, field resolver, calibration defaults, virtual-lab	6	0c46279...e109de3
3: config init default site, dotbot.example.toml, examples read the site	4	e109de3...70c1a86
4: site packs, calibration lookup and site check, dotbot site	3	70c1a86...e9c4555
5: calibration warnings, span on the map	3	e9c4555...ece9501
CI fix: Windows paths in Phase 4's tests	2	ece9501...eb41d73
Review fixes: site add hardening, test isolation, age parsing, cleanups	8	eb41d73...a5b6f4c
Review changes: structured points_from, safe site add --force, site add -	3	a5b6f4c...520ec95
Demo follow-ups: run simulator --area, area-shaped grid, one staging area, warning summary, roles in the console	5	520ec95...d790664
Docs sweep: README, doc/, API pages, example READMEs	3	d790664...b7eb271

Size

The PR grew from one phase to four at review time, by decision, so it is larger than a single-phase PR would be. Tests are close to half of it.

	Files	+	-
Python source	28	+1646	-323
Python tests	17	+1500	-58
Frontend source (TS/React)	9	+324	-68
Frontend tests	15	+331	-39
Docs: README, doc/ and example READMEs	21	+498	-173
Config and example configs	5	+37	-62
Total	95	+4336	-723

Reviewed; the docs sweep is in, and this is ready to merge.

Docs

The README quickstart now starts from dotbot config init, then the simulator. doc/index.md explains sites and the three area roles; doc/cli/config.md covers the template, --field, --site, --global and --force; a new doc/cli/site.md covers site add (folder, zip, git URL, - from stdin) and site export; the configuration reference gains a Sites section (the site table, roles, the one-field rule and the fallback order, site packs and site_dirs, where calibrations are found and the refusal of another site's, points_from) plus site, site_dirs and lh2_calibration_max_age_days in its key tables. The LH2 calibration guide is rewritten around the field default, --over / --square / --points, the calibrate app's button, the span overlay and the refusal at load, with a note to use base-station channels 1..N with no gaps and to avoid channel 14 for now (a known issue); it drops the retired -d square sizing and every "arena". dotbot.site_packs joins the API pages, the dotbot/config.py module example shows site_dirs, and the example READMEs run dotbot run simulator (the charging example saying it needs a staging area).

Design notes

The design lives in an internal plan (rendered: https://claude.ai/artifact/Y4j8T4oW75CDU4DNre3g4s, source: https://github.com/DotBots/dotbot-workspace/blob/main/plans/site-areas/plan.html). Both links are internal; the rationale that matters for review is paraphrased above.

[codecov]

Codecov Report

❌ Patch coverage is 96.06987% with 63 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.11%. Comparing base (eb7c470) to head (b7eb271).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dotbot/examples/motions/motions.py	50.00%	17 Missing ⚠️
dotbot/console-web/src/RightPane.tsx	66.66%	10 Missing ⚠️
dotbot/cli/site_cmd.py	94.73%	9 Missing ⚠️
...tbot/examples/charging_station/charging_station.py	86.04%	6 Missing ⚠️
dotbot/cli/config_cmd.py	95.09%	5 Missing ⚠️
dotbot/controller_app.py	84.37%	5 Missing ⚠️
dotbot/cli/_site.py	84.61%	2 Missing ⚠️
dotbot/site_packs.py	97.29%	2 Missing ⚠️
dotbot/tests/test_site_packs.py	99.19%	2 Missing ⚠️
dotbot/console-web/src/App.tsx	91.66%	1 Missing ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #307      +/-   ##
==========================================
+ Coverage   84.43%   85.11%   +0.67%     
==========================================
  Files         209      214       +5     
  Lines       29264    30673    +1409     
  Branches     2053     2100      +47     
==========================================
+ Hits        24710    26108    +1398     
- Misses       4550     4561      +11     
  Partials        4        4              

Flag	Coverage Δ
console	77.72% <94.52%> (+0.21%)	⬆️
python	89.31% <96.29%> (+0.73%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dotbot/adapter.py	86.78% <100.00%> (+0.23%)	⬆️
dotbot/area.py	92.53% <100.00%> (+1.01%)	⬆️
dotbot/camera/registration.py	98.00% <100.00%> (+0.04%)	⬆️
dotbot/cli/camera_calibrate.py	75.00% <100.00%> (+1.01%)	⬆️
dotbot/cli/main.py	100.00% <100.00%> (ø)
dotbot/cli/swarm_lh2.py	77.66% <100.00%> (+1.64%)	⬆️
dotbot/config.py	98.37% <100.00%> (+0.10%)	⬆️
dotbot/console-web/src/MapView.tsx	97.54% <100.00%> (+0.11%)	⬆️
dotbot/console-web/src/Minimap.tsx	100.00% <100.00%> (ø)
dotbot/console-web/src/areaColor.ts	100.00% <100.00%> (ø)
... and 40 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.