fix: harden Labtasker evaluation and simplify setup

Author: luocfprime

.github/workflows/ci.yml

@@ -21,7 +21,7 @@ jobs:
         run: pip install ruff
 
       - name: Lint
-        run: ruff check openwam/ scripts/ tests/ docker/
+        run: ruff check openwam/ benchmarks/ scripts/ tests/ docker/
 
   tests:
     runs-on: ubuntu-latest
@@ -38,8 +38,7 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           python -m pip install --index-url https://download.pytorch.org/whl/cpu torch==2.7.1 torchvision==0.22.1
-          python -m pip install -e .
-          python -m pip install pytest labtasker==2.5.0
+          python -m pip install -e ".[dev]"
 
       - name: Run standard checks
         run: make check

Makefile

@@ -26,14 +26,14 @@ tri-system-smoke:
 	$(PYTHON) -m pytest -q $(TRI_SYSTEM_SMOKE_ARGS)
 
 lint:
-	$(PYTHON) -m ruff check openwam/ scripts/ tests/ docker/
+	$(PYTHON) -m ruff check openwam/ benchmarks/ scripts/ tests/ docker/
 
 format:
-	$(PYTHON) -m ruff format openwam/ scripts/ tests/ docker/
-	$(PYTHON) -m ruff check --fix openwam/ scripts/ tests/ docker/
+	$(PYTHON) -m ruff format openwam/ benchmarks/ scripts/ tests/ docker/
+	$(PYTHON) -m ruff check --fix openwam/ benchmarks/ scripts/ tests/ docker/
 
 compile:
-	$(PYTHON) -m compileall openwam scripts tests docker
+	$(PYTHON) -m compileall openwam benchmarks scripts tests docker
 
 check: compile test
 

benchmarks/libero/LABTASKER.md

@@ -3,8 +3,8 @@
 This lightweight integration gives you more built-in features with less custom
 orchestration. Labtasker handles task scheduling, retries and progress tracking,
 keeping the evaluation adapter small and the workflow simple: submit Tasks,
-start Workers and inspect results. On a single machine, its local server starts
-automatically.
+start Workers and inspect results. On a single machine with a local-filesystem
+checkout, you can explicitly authorize its managed local server on first use.
 
 Compared with the traditional launcher, you get:
 
@@ -43,11 +43,70 @@ export LIBERO_PATH=/path/to/LIBERO
 python -m pip install -r benchmarks/libero/requirements-labtasker.txt
 ```
 
-On one machine, no Labtasker server setup is needed.
+**On one machine with the repository on a local disk, you can continue to
+the first-run commands below.** Their `--auto-start-local-server` flag starts
+Labtasker for you; later commands reconnect automatically. No separate server
+command, port or database configuration is needed. The project GPU Docker image
+already includes the full Labtasker package needed for this workflow.
+
+For multiple machines, or a checkout on NFS, WekaFS or Lustre, use the setup
+below first. Use it also if automatic startup reports an unknown filesystem.
+
+<details>
+<summary>Setup for multiple machines or a shared-filesystem checkout</summary>
+
+If your team already runs a Labtasker server, ask for its URL and token and
+skip to the client settings below. Otherwise, run this once on one machine:
+
+```bash
+# Supply your own secret token.
+export LABTASKER_SERVER_TOKEN=...  # do not commit the token
+labtasker-server serve --connection http --host 0.0.0.0
+```
+
+By default, the server saves its records in `.labtasker/` under the directory
+where you start it. Its automatically created `server.db` stores submitted
+task inputs, task status and reported results. Keep this directory to retain
+your evaluation history.
+
+To choose another location, add just `--labtasker-root /path/to/openwam-eval`;
+`server.db` follows that directory automatically. `--database` only overrides
+the database file's location separately and is unnecessary for this setup.
+Local disks and NFS/shared storage are supported: the server detects the
+filesystem automatically. On shared storage, run only one server for that
+directory/database, even when several machines can access it. The local-disk
+restriction applies to `--auto-start-local-server`, not this manual command.
+
+Workers keep their execution logs and run records under their own
+`.labtasker/runs/`; setting the server's `--labtasker-root` does not move remote
+Worker logs. Benchmark logs, videos and detailed evaluation results still go
+to the output directories described below.
+
+The server listens on port **8000** by default. Keep this command running in
+its terminal or under your process supervisor.
+
+In each shell that submits tasks, runs Workers or reads results, set:
+
+```bash
+export LABTASKER_URL=http://SERVER_HOST:8000
+export LABTASKER_TOKEN=...  # same token as the server
+```
+
+Replace `SERVER_HOST` with the server machine's reachable hostname or IP.
+Then follow the evaluation steps below, omitting `--auto-start-local-server`.
+All Workers must be able to access the submitted checkpoint and manifest paths.
+
+</details>
+
+For background, see the optional
+[Labtasker guide and demo video](https://luocfprime.github.io/labtasker/latest/).
 
 > [!TIP]
-> Labtasker uses a server–client architecture and a shared task queue to manage
-> concurrent Workers. See the [conceptual guide and demo video](https://luocfprime.github.io/labtasker/latest/).
+> To let your coding agent help with Labtasker setup, task submission, progress
+> monitoring, troubleshooting and result summaries, install the **Labtasker skill**.
+> See the [Agent Skill guide](https://github.com/luocfprime/labtasker/blob/main/docs/guides/agent-skill.md)
+> for installation instructions. Describe the evaluation you want to run; the
+> agent can help translate it into the required Labtasker commands and settings.
 
 <a id="first-run"></a>
 
@@ -60,7 +119,8 @@ On one machine, no Labtasker server setup is needed.
 ### 1. Submit manifest-building Tasks
 
 ```bash
-python benchmarks/libero/labtasker_submit.py --operation build_manifest
+python benchmarks/libero/labtasker_submit.py --auto-start-local-server \
+  --operation build_manifest
 ```
 
 Save the submission ID printed by this command.
@@ -98,7 +158,7 @@ Replace `SUBMISSION_ID` with the printed ID. Continue when the count reaches zer
 Once the cache is ready:
 
 ```bash
-python benchmarks/libero/labtasker_submit.py \
+python benchmarks/libero/labtasker_submit.py --auto-start-local-server \
   -- --ckpt-dir /path/to/checkpoint
 ```
 

benchmarks/libero/labtasker_runtime.py

@@ -32,11 +32,13 @@
 from benchmarks.utils.eval_sharding import split_range  # noqa: E402
 from benchmarks.utils.labtasker_utils import (  # noqa: E402
     count_task_statuses,
+    create_client,
     list_submission_tasks,
     print_submission_id,
     print_summary_table,
     project_context,
     submit_tasks,
+    validate_submission_coverage,
 )
 from benchmarks.utils.labtasker_utils import new_submission_id as new_submission_id  # noqa: E402, F401
 from benchmarks.utils.owned_processes import ProcessRegistry  # noqa: E402, F401
@@ -632,6 +634,11 @@ def _build_parser(mode: str = "submit") -> argparse.ArgumentParser:
     worker = mode == "worker"
     parser.add_argument("--route", help="Task route (default: openwam-libero-<operation>)")
     parser.add_argument("--queue")
+    parser.add_argument(
+        "--auto-start-local-server",
+        action="store_true",
+        help="explicitly authorize startup of the project-local Labtasker Server",
+    )
     parser.add_argument("--operation", choices=("build_manifest", "run_eval"), default="run_eval")
     if not worker:
         parser.add_argument("--policy-config", type=Path, default=DEFAULT_POLICY_CONFIG)
@@ -757,6 +764,7 @@ def submit(
     max_attempts: int = 3,
     priority: int = 0,
     note: str | None = None,
+    auto_start_local_server: bool = False,
 ) -> list[str]:
     """Retry the original definition; never rewrite or requeue an existing Task."""
     inputs = list(inputs)
@@ -785,7 +793,7 @@ def submit(
                 f"libero({item['suite']}):task{item['task_id']:02d}_{grouped_indices[key]:0{width}d}_{total:0{width}d}"
             )
         names.append(display + (f"_{note}" if note else ""))
-    with project_context(), labtasker.Client(queue=queue) as client:
+    with project_context(), create_client(queue=queue, auto_start_local_server=auto_start_local_server) as client:
         task_ids = submit_tasks(
             client,
             inputs,
@@ -846,11 +854,18 @@ def validated_task_result(task: labtasker.Task) -> dict[str, Any] | None:
     return result
 
 
-def summarize(submission_id: str, output: str | Path, *, queue: str | None = None) -> int:
-    with project_context(), labtasker.Client(queue=queue) as client:
+def summarize(
+    submission_id: str,
+    output: str | Path,
+    *,
+    queue: str | None = None,
+    auto_start_local_server: bool = False,
+) -> int:
+    with project_context(), create_client(queue=queue, auto_start_local_server=auto_start_local_server) as client:
         tasks = list_libero_submission_tasks(client, submission_id, "run_eval")
     if not tasks:
         raise ValueError(f"LIBERO eval submission not found: {submission_id}")
+    validate_submission_coverage(tasks)
     _summarize(Path(output), tasks)
     return 0
 

benchmarks/libero/labtasker_submit.py

@@ -27,6 +27,7 @@ def main(argv=None):
         max_attempts=args.max_attempts,
         priority=args.priority,
         note=args.note,
+        auto_start_local_server=args.auto_start_local_server,
     )
     return 0
 

benchmarks/libero/labtasker_summarize.py

@@ -5,16 +5,22 @@
 import sys
 from pathlib import Path
 
-from labtasker_runtime import summarize
+import labtasker_runtime as runtime
 
 
 def main(argv=None):
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("submission_id")
     parser.add_argument("--output-dir", type=Path, required=True)
     parser.add_argument("--queue")
+    parser.add_argument("--auto-start-local-server", action="store_true")
     args = parser.parse_args(argv)
-    return summarize(args.submission_id, args.output_dir.expanduser().absolute(), queue=args.queue)
+    return runtime.summarize(
+        args.submission_id,
+        args.output_dir.expanduser().absolute(),
+        queue=args.queue,
+        auto_start_local_server=args.auto_start_local_server,
+    )
 
 
 if __name__ == "__main__":

benchmarks/libero/labtasker_worker.py

@@ -193,6 +193,9 @@ def main(argv: list[str] | None = None) -> int:
         args.gpu, args.render_gpu if args.render_gpu is not None else args.gpu, args.port
     )
     try:
+        if args.auto_start_local_server:
+            with project_context(), runtime.create_client(queue=args.queue, auto_start_local_server=True) as client:
+                client.count_tasks()
         print(f"[worker] route={args.route} operation={args.operation} gpu={worker_resources.gpu}", flush=True)
         with project_context():
             run_worker(args, worker_resources, server)

benchmarks/libero/requirements-labtasker.txt

@@ -1,3 +1,3 @@
 # Install in SERVER_PYTHON; the isolated LIBERO environment is unchanged.
-labtasker==2.5.0
+labtasker==2.6.0
 PyYAML>=6.0

benchmarks/robotwin/LABTASKER.md

@@ -3,8 +3,8 @@
 This lightweight integration gives you more built-in features with less custom
 orchestration. Labtasker handles task scheduling, retries and progress tracking,
 keeping the evaluation adapter small and the workflow simple: submit Tasks,
-start Workers and inspect results. On a single machine, its local server starts
-automatically.
+start Workers and inspect results. On a single machine with a local-filesystem
+checkout, you can explicitly authorize its managed local server on first use.
 
 Compared with the traditional launcher, you get:
 
@@ -43,11 +43,70 @@ export ROBOTWIN_PATH=/path/to/RoboTwin
 python -m pip install -r benchmarks/robotwin/requirements-labtasker.txt
 ```
 
-On one machine, no Labtasker server setup is needed.
+**On one machine with the repository on a local disk, you can continue to
+the first-run commands below.** Their `--auto-start-local-server` flag starts
+Labtasker for you; later commands reconnect automatically. No separate server
+command, port or database configuration is needed. The project GPU Docker image
+already includes the full Labtasker package needed for this workflow.
+
+For multiple machines, or a checkout on NFS, WekaFS or Lustre, use the setup
+below first. Use it also if automatic startup reports an unknown filesystem.
+
+<details>
+<summary>Setup for multiple machines or a shared-filesystem checkout</summary>
+
+If your team already runs a Labtasker server, ask for its URL and token and
+skip to the client settings below. Otherwise, run this once on one machine:
+
+```bash
+# Supply your own secret token.
+export LABTASKER_SERVER_TOKEN=...  # do not commit the token
+labtasker-server serve --connection http --host 0.0.0.0
+```
+
+By default, the server saves its records in `.labtasker/` under the directory
+where you start it. Its automatically created `server.db` stores submitted
+task inputs, task status and reported results. Keep this directory to retain
+your evaluation history.
+
+To choose another location, add just `--labtasker-root /path/to/openwam-eval`;
+`server.db` follows that directory automatically. `--database` only overrides
+the database file's location separately and is unnecessary for this setup.
+Local disks and NFS/shared storage are supported: the server detects the
+filesystem automatically. On shared storage, run only one server for that
+directory/database, even when several machines can access it. The local-disk
+restriction applies to `--auto-start-local-server`, not this manual command.
+
+Workers keep their execution logs and run records under their own
+`.labtasker/runs/`; setting the server's `--labtasker-root` does not move remote
+Worker logs. Benchmark logs, videos and detailed evaluation results still go
+to the output directories described below.
+
+The server listens on port **8000** by default. Keep this command running in
+its terminal or under your process supervisor.
+
+In each shell that submits tasks, runs Workers or reads results, set:
+
+```bash
+export LABTASKER_URL=http://SERVER_HOST:8000
+export LABTASKER_TOKEN=...  # same token as the server
+```
+
+Replace `SERVER_HOST` with the server machine's reachable hostname or IP.
+Then follow the evaluation steps below, omitting `--auto-start-local-server`.
+All Workers must be able to access the submitted checkpoint and manifest paths.
+
+</details>
+
+For background, see the optional
+[Labtasker guide and demo video](https://luocfprime.github.io/labtasker/latest/).
 
 > [!TIP]
-> Labtasker uses a server–client architecture and a shared task queue to manage
-> concurrent Workers. See the [conceptual guide and demo video](https://luocfprime.github.io/labtasker/latest/).
+> To let your coding agent help with Labtasker setup, task submission, progress
+> monitoring, troubleshooting and result summaries, install the **Labtasker skill**.
+> See the [Agent Skill guide](https://github.com/luocfprime/labtasker/blob/main/docs/guides/agent-skill.md)
+> for installation instructions. Describe the evaluation you want to run; the
+> agent can help translate it into the required Labtasker commands and settings.
 
 <a id="first-run"></a>
 
@@ -60,7 +119,8 @@ On one machine, no Labtasker server setup is needed.
 ### 1. Submit manifest-building Tasks
 
 ```bash
-python benchmarks/robotwin/labtasker_submit.py --operation build_manifest --mode demo_clean
+python benchmarks/robotwin/labtasker_submit.py --auto-start-local-server \
+  --operation build_manifest --mode demo_clean
 ```
 
 Save the submission ID printed by this command.
@@ -98,7 +158,7 @@ Replace `SUBMISSION_ID` with the printed ID. Continue when the count reaches zer
 Once the cache is ready:
 
 ```bash
-python benchmarks/robotwin/labtasker_submit.py --mode demo_clean \
+python benchmarks/robotwin/labtasker_submit.py --auto-start-local-server --mode demo_clean \
   -- --ckpt-dir /path/to/checkpoint
 ```