diff --git a/docs/source-en/rst_source/real_world_robots/dual_franka.rst b/docs/source-en/rst_source/real_world_robots/dual_franka.rst index f78faf229..8976d41c2 100644 --- a/docs/source-en/rst_source/real_world_robots/dual_franka.rst +++ b/docs/source-en/rst_source/real_world_robots/dual_franka.rst @@ -8,27 +8,54 @@ Dual-Arm Franka Dual-Franka system for real-world experiments. -RPent can control a two-node dual-Franka setup through an RLinf +RPent controls a two-node dual-Franka setup through an RLinf ``RealWorldEnv`` worker. Watch the :doc:`task demos <../real_world_demos/franka>`. Install ------- -Install libfranka 0.19.0 first by following the `official quick-install guide -`_. - -Clone RPent and install its Python dependencies. If you already have the -checkout, enter it and run ``uv sync``: +Install RPent on both Linux control nodes with matching code, Python, Ray and +RLinf versions. Nodes running VLA or SAM3 locally need an NVIDIA GPU. +Use the same installation workflow as :doc:`../get_started/quickstart`: .. code-block:: bash git clone https://github.com/RLinf/RPent.git cd RPent - uv sync --extra franka --extra sam3 + uv venv + source .venv/bin/activate + uv pip install -e ".[franka,sam3]" + +If Python package downloads are slow, replace the installation command with the +`Tsinghua PyPI mirror `_: + +.. code-block:: bash + + uv pip install -e ".[franka,sam3]" \ + --index-url https://pypi.tuna.tsinghua.edu.cn/simple -This installs RLinf ``release/v0.4``, ``rpent-openpi``, SAM3, and Franka -camera, gripper, and teleoperation dependencies into ``.venv``. The included -``franky-control`` wheel bundles libfranka 0.19.0. +This option changes only the Python package index; it does not proxy Git +repositories, direct download URLs or model weights. + +Check libfranka compatibility with the robot firmware before connecting. +The calibration state reader also needs libfranka development files; see the +calibration steps below. + +Run subsequent commands from the repository root in this environment. In a new +terminal, activate ``.venv`` and set the required model variables again. + +Download SAM3 weights on the environment node if local segmentation is needed: + +.. code-block:: bash + + uv pip install modelscope + modelscope download --model facebook/sam3 sam3.pt \ + --local_dir ./checkpoints/sam3 + export SAM3_CHECKPOINT_PATH="$PWD/checkpoints/sam3/sam3.pt" + +Without this variable or ``--sam3-endpoint``, segmentation is unavailable; +installing the extra alone does not download weights. VLA checkpoints are +configured in :ref:`dual-franka-vla`. Calibration ----------- @@ -40,35 +67,25 @@ Wrist cameras are observation-only in the default configuration. To use a wrist RGBD camera for projection, configure its calibration and projection view as described below. -The tools support D435 and have been validated with a D435 setup. Other models -and stream configurations are unverified and may require more than parameter -changes. Zero distortion coefficients are accepted; nonzero coefficients are -supported only for ``distortion.brown_conrady``. Other nonzero models are -rejected with their model and coefficients, without automatic conversion or -ignoring distortion. +The collector uses RealSense RGB images and accepts zero distortion coefficients +or nonzero ``distortion.brown_conrady`` coefficients. Other nonzero distortion +models are rejected. **1. Prepare the environment** -On the camera node, activate the existing RLinf Franka environment and check -the required interfaces from the RPent repository root: +Use the installed environment from the RPent repository root on the camera node. +To install calibration dependencies separately, create a dedicated environment: .. code-block:: bash - source /absolute/path/to/franka-env/bin/activate - cd /absolute/path/to/RPent - PYTHONPATH=calibration_tools python -c "import numpy, scipy, yaml, pyrealsense2; from common import check_opencv; check_opencv()" - -This checks ChArUco, PnP, ``calibrateHandEye``, and the required method constants. -If dependencies are missing, create a separate calibration environment: - -.. code-block:: bash - - python3 -m venv .venv-calibration + uv venv .venv-calibration source .venv-calibration/bin/activate - pip install -r calibration_tools/requirements.txt pyrealsense2 + uv pip install -r calibration_tools/requirements.txt pyrealsense2 -On the right-arm control node, compile the reader against the libfranka -development library matching the robot firmware: +On the right-arm control node, first follow the `official libfranka installation +guide `_ +to install a firmware-compatible development library, including headers and CMake +configuration. Then compile the state reader: .. code-block:: bash @@ -76,13 +93,14 @@ development library matching the robot firmware: -DCMAKE_PREFIX_PATH=/absolute/path/to/libfranka/install cmake --build calibration_tools/build --parallel -The executable ``calibration_tools/build/read_franka_state`` uses ``readOnce()`` -to output end-effector poses, joint velocities, and robot status as JSON. It -sends no motion commands. Installing Python control packages does not guarantee -the headers and CMake configuration needed to build this reader. -Replace the camera serials, SSH alias, absolute reader path, and robot IP below. -Omit ``--ssh-host`` for a local reader; add ``--library-dir /path/to/lib`` when -its shared libraries need an explicit search path. +The build produces ``calibration_tools/build/read_franka_state``. Calibration +uses it to read robot state without moving the arm. + +* Local reader: omit ``--ssh-host`` when the reader and collector run on the same node. +* Remote reader: use ``--ssh-host`` to select the reader's node and configure + passwordless SSH login first. + +Replace the camera serials, robot IP and reader path below with your actual values. **2. Start the camera and collect samples** @@ -98,6 +116,7 @@ stationary. Defaults are 6×8 squares, 25 mm square length, 18 mm marker length, DICT_4X4_100, and a non-legacy layout. Configure other boards with ``--squares-x``, ``--squares-y``, ``--square-m``, ``--marker-m``, and ``--dictionary``; lengths are in meters. +Measure the printed square size and ensure it matches the configuration. In another terminal on the same camera node, activate the chosen environment: @@ -110,28 +129,26 @@ In another terminal on the same camera node, activate the chosen environment: --reader /absolute/path/to/read_franka_state --robot-ip ROBOT_IP \ --output calibration_tools/sessions/base-to-right -Open ``http://127.0.0.1:8767``. Move the right arm manually, release the guidance -button, wait until stationary, and click ``Capture pose``. Collect 20–30 distinct -poses with rotations about multiple axes; fitting requires at least ten. -The tool only reads state. Without ``--output``, sessions are saved under the -script directory's ``calibration_tools/sessions/``, regardless of the working -directory. - -Use the saved-sample table to inspect each sample's corner count and -reprojection RMS in pixels. Click ``Preview`` to open its annotated image, -then ``Exclude`` to remove an unsuitable sample from the next solve. Its -files remain in the session's ``excluded/`` directory. ``Active samples`` -counts only samples used by the solver; IDs remain unchanged, so gaps are -normal. Adding or excluding samples does not recalculate existing calibration -results: run the solver again before exporting. - -Stop the collector with ``Ctrl+C``. To start again after an interruption or -recollect the entire session, rerun the command with a new ``--output`` directory. -Existing session files remain on disk but are not loaded into the new collection. - -For D455, repeat with ``D455_SERIAL``, ``/raw/d455``, and -``calibration_tools/sessions/d455-to-right``. Check that its stream configuration -uses a supported distortion model as described above. +Open ``http://127.0.0.1:8767`` and collect samples: + +1. Confirm the arm is Idle and free of errors. Move the right arm manually, + release the guidance button, wait until stationary, then click ``Capture pose``. +2. Collect 20–30 different positions and orientations, with rotations about + multiple axes. Fitting requires at least 10 samples. +3. Use ``Preview`` to inspect samples and ``Exclude`` to remove unsuitable ones. + Run the solver again after changing the samples. +4. Press ``Ctrl+C`` when finished. To recollect, use a new ``--output`` directory. + +To view the page on your own computer, forward the port first, then open the +address above: + +.. code-block:: bash + + ssh -L 8767:127.0.0.1:8767 CAMERA_HOST + +For D455, reuse the collection command with serial ``D455_SERIAL``, URL +``http://127.0.0.1:8765/raw/d455``, and output directory +``calibration_tools/sessions/d455-to-right``. **3. Fit and export** @@ -162,8 +179,7 @@ List the independently validated YAMLs in your robot configuration: base_camera: /absolute/path/to/base_to_right.yaml d455_camera: /absolute/path/to/d455_to_right.yaml -Load this configuration with ``--robot-config`` below. Stop the calibration -camera service before starting RPent. Preserve and verify +Load this configuration with ``--robot-config`` below. Preserve and verify ``perception.base_frames`` and localization bounds; these tools do not calibrate the relationship between the two arm bases. @@ -180,8 +196,14 @@ points. Snapshot TCP poses must represent O_T_EE and declare their base frame. The default Lumos observation cameras are not automatically enabled for RGBD projection; selecting a wrist anchor requires this configuration. -Development Configuration -------------------------- +After calibration, stop all camera services, collectors, and robot-state readers +started for calibration before starting RPent. + +Robot Configuration +------------------- + +If calibration used a separate Python environment, reactivate RPent's +``.venv`` before running the following commands. Review and edit the checked-in development defaults before enabling motion: @@ -196,34 +218,166 @@ RPent translates this robot-focused schema into the internal two-node RLinf cluster and environment objects. To use a different file, pass ``--robot-config /path/to/robot_config.yaml``. -During ``rotate_delta``, the position target stays at the starting TCP position. -On real hardware, compliance/impedance control can allow the actual pose to -deviate from the target due to load, contact, or limited tracking accuracy. +The example configuration uses the cameras as follows: + +* ``base``, ``left_wrist`` and ``right_wrist`` provide RGB images for scene observation. +* ``d455`` provides RGB and aligned depth images to convert selected pixels into + 3D coordinates. Back-projection requires valid calibration for this camera. + +``perception.projection_views`` selects the views used for back-projection; it +does not enable depth capture. Use ``d455`` with the example configuration. + +``rotate_delta`` changes the end-effector orientation while trying to maintain +its starting position. ``rotation_hold.max_drift_m`` sets the allowed position +drift during rotation, defaulting to ``0.10`` m (10 cm). Exceeding this limit +aborts the call with an error. + +Task Selection +-------------- + +The main example uses task ``2``. On first connection, use task ``0`` to check +small translations, rotations and gripper actions. This is a planner-guided +check, not a fixed automated sequence, and does not require VLA. + +.. list-table:: + :header-rows: 1 + + * - ID + - Task + - VLA required + * - 0 + - Planner-guided primitive checks + - No + * - 1 + - Dirty/clean sorting of bowls and plates, with cup and utensils + - Yes + * - 2 + - Dirty/clean sorting of bowls, plates, and two cups; no utensils + - Yes + +.. note:: + + Task IDs select tasks in launch commands; sorting tasks use their names for + recipe and memory identifiers. Keep IDs and names associated with the same task when + editing definitions. Do not reorder IDs or reuse names for different tasks. -``rotation_hold.max_drift_m`` defaults to ``0.10`` m (10 cm) and bounds the -Euclidean distance from the TCP position at the start of this rotation. It is -separate from the position tolerance, ``move_tolerance_m`` (0.006 m). At or below -the drift limit, after rotation ends, ``move_delta`` returns the TCP to its -starting position only if the position error exceeds that tolerance. The call -reports ``ok=True`` only if compensation succeeds (when needed) and the final -position and angle both meet their tolerances; otherwise it reports ``False``. -Above the limit, a check during the rotation loop or at its end raises an RPC -error, ending the call without translation compensation. +In task ``1``, an egg tart or foil tray inside a bowl or on a plate marks it as +dirty. Dirty bowls and plates go into the metal basket; clean bowls and plates, +and the cup, go into the cardboard box. The chopsticks and spoon go inside the cup. -The robot's own protections can still trigger independently. +In task ``2``, chicken nuggets inside a bowl, or an egg-tart foil tray in a cup +or on a plate, mark it as dirty. Dirty objects go into the metal basket; clean +objects go into the cardboard box. There are no utensils. + +Both sorting tasks provide ``vla_right_grasp`` / ``vla_handoff`` / ``vla_left_place`` skills. + +.. _dual-franka-vla: + +.. _vla-grasp-demo: + +VLA Configuration +----------------- + +``PI05_CHECKPOINT_PATH`` points to the trained Pi0.5 checkpoint, while +``DUAL_FRANKA_REPO_ID`` is the dataset ID used to locate matching normalization +statistics. + +The environment variables correspond to ``--vla-model-path`` and +``--vla-repo-id``. The repo ID is the relative directory containing the matching +normalization statistics, not the model download repository name. The checkpoint +must contain: + +.. code-block:: text + + actor/model_state_dict/full_weights.pt + /norm_stats.json + +**Pretrained checkpoint** + +Download the `dual-Franka clean-desk checkpoint +`_ +from Hugging Face. It supplies the VLA skills; the RPent planner makes the +dirty/clean sorting decisions. + +.. code-block:: bash + + uv pip install huggingface_hub + hf download RLinf/RLinf-Pi05-RealWorld-DualFranka-Clean_Desk \ + --revision 4f43adc723dfe98fae806a131ad61999e1f95e4f \ + --local-dir ./models/pi05-dualfranka-clean-desk + + export PI05_CHECKPOINT_PATH="$PWD/models/pi05-dualfranka-clean-desk" + export DUAL_FRANKA_REPO_ID=id_0 + +This checkpoint includes ``id_0/norm_stats.json``. Keep it with the downloaded +weights and use ``id_0`` as shown above. + +.. warning:: + + This checkpoint is trained only on our in-house test environment (robot + poses, cameras, workspace layout, and objects), so it is expected to + generalize poorly to a different setup. To deploy on your own rig, collect + demonstrations and fine-tune your own checkpoint with RLinf by following the + `RLinf dual-Franka guide + `_ + (collect GELLO demos, convert to tcp_rot6d, run SFT, then deploy). + +.. _dual-franka-codex-isolation: + +Robot Codex Profile Isolation +----------------------------- + +In a dedicated terminal, activate ``.venv`` and set separate Codex configuration +and session directories from the repository root. Repeat this in each new +robot-control terminal: + +.. code-block:: bash + + export CODEX_HOME="$PWD/.codex-rpent-live" + export CODEX_SQLITE_HOME="$CODEX_HOME" + export CODEX_SERVICE_TIER=fast + mkdir -p "$CODEX_HOME" + unset CODEX_API_KEY CODEX_BASE_URL OPENAI_API_KEY OPENAI_BASE_URL CODEX_ACCESS_TOKEN + export CODEX_BIN="$(python -c 'from codex_cli_bin import bundled_codex_path; print(bundled_codex_path())')" + +On first use, enable file credential storage in ``$CODEX_HOME/config.toml``, +then log in with the CLI bundled with the SDK: + +.. code-block:: toml + + cli_auth_credentials_store = "file" + +.. code-block:: bash + + "$CODEX_BIN" login --device-auth + "$CODEX_BIN" login status + +If ``sqlite_home`` is already configured, point it at the separate directory too. +For a custom API, explicitly set this deployment's ``CODEX_API_KEY`` and +``CODEX_BASE_URL``. See `Codex authentication +`_ for other login methods. + +RPent does not enable this isolation automatically. It neither isolates shared +workspace files nor migrates old sessions. Task memory is selected separately +with ``--memory-dir``. Never commit the private directory or credentials. Start the Two-node Ray Cluster ------------------------------ -The two nodes have fixed, different roles (defined in -``robots/dual_franka/runtime_config.py``): +Activate the installed ``.venv`` on each node before starting Ray. The default +placement in ``robots/dual_franka/runtime_config.py`` is: -* Node ``0`` is the Ray head: it runs the dual-Franka environment - worker (all cameras, perception, and arm/gripper state), and the **left** - arm's real-time controller. For the VLA task, the - local VLA server also runs here. -* Node ``1`` is a Ray worker: it runs only the **right** arm's real-time - controller, with no cameras and no RPent process. +* Node ``0`` is the Ray head: it runs RPent, the environment worker, the + **left** arm/gripper controller, the base camera and the left wrist camera. + RPent opens the additional D455 locally here. Local VLA and SAM3 services + also run here when configured. +* Node ``1`` is a Ray worker: it runs the **right** arm/gripper controller and + its right wrist camera. It needs the installed environment and Ray, but no + separate RPent runner. + +RLinf creates camera handles on the configured nodes; it does not search other +nodes for a relocated USB camera. Connect each wrist camera to its arm's control +node and edit the configured type/serial for the actual device. .. warning:: @@ -245,104 +399,132 @@ Node ``1``: ray stop --force ray start --address=HEAD_IP:6379 --node-ip-address=WORKER_IP -Run a Smoke Test ----------------- - -Configure the planner and model service using :doc:`../guides/configure_planner` first. +Manual Skill Testing +-------------------- -Task ``0`` tests small translations, rotations, and gripper actions, one arm at a time: +List the available skills, then inspect the parameters and types accepted by a +skill. From the repository root: .. code-block:: bash - uv run --extra franka rpent --robot dual_franka --task-id 0 \ - --planner claude_code --model claude-opus-4-8 \ - --robot-config robots/dual_franka/config/example.yaml + python robots/dual_franka/dual_franka_manual_call.py --list-primitives + python robots/dual_franka/dual_franka_manual_call.py --schema vla_right_grasp -RPent starts ``robots/dual_franka/env_server.py`` with the current interpreter, -loads the RPent robot config, generates the internal RLinf adapter config, -connects to Ray, waits for ``healthz``, and records the initial state as step -``0``. Task ``0`` does not load the VLA. +Without service addresses, the script starts and connects to the required +services automatically; no port selection is needed. It stops only the services +it started when it exits. Supply ``--env-endpoint``, ``--vla-endpoint`` or +``--sam3-endpoint`` to reuse an existing service. Attaching to an existing Env +does not reset it. With a newly started Env, a manual action first requests +scene confirmation and performs one reset. An explicit manual ``reset`` also +waits for confirmation. Listing tools and printing schemas never starts services. -VLA Grasp Demo --------------- - -RPent provides a demo that uses a VLA to grasp objects. Task ``1`` exposes -``vla_right_grasp`` / ``vla_handoff`` / ``vla_left_place`` and can start the dual-Franka VLA server locally. -``PI05_CHECKPOINT_PATH`` points to the trained Pi-05 checkpoint, while -``DUAL_FRANKA_REPO_ID`` is the dataset ID used to locate matching normalization -statistics: +For example, check the schema before running a small left-arm translation: .. code-block:: bash - export PI05_CHECKPOINT_PATH=/path/to/checkpoints/global_step_N - export DUAL_FRANKA_REPO_ID=org/dual-franka-tcp-rot6d - - uv run --extra franka rpent --robot dual_franka --task-id 1 \ - --cuda-device 0 \ - --planner claude_code --model claude-opus-4-8 \ + python robots/dual_franka/dual_franka_manual_call.py --schema move_delta + python robots/dual_franka/dual_franka_manual_call.py \ + --primitive move_delta --params '{"arm":"left","delta_xyz":[0.01,0,0]}' \ --robot-config robots/dual_franka/config/example.yaml -The checkpoint must contain: +Use ``--primitive NAME --params JSON`` to call a tool; manual calls do not select +a task ID. The example config already sets ``vla.instruction`` for the clean-desk +policy. Change it for a different checkpoint, or override it with +``--instruction``. This setting applies only to manual calls; task runners still +use their registered task instructions. ``prompt`` inside ``--params`` records +the skill intent and does not replace the policy instruction. -.. code-block:: text +With the VLA checkpoint and normalization variables set, run one grasp skill: - actor/model_state_dict/full_weights.pt - /norm_stats.json +.. code-block:: bash -**Pretrained checkpoint** + python robots/dual_franka/dual_franka_manual_call.py \ + --primitive vla_right_grasp \ + --params '{"prompt":"grasp the bowl","max_chunks":1}' \ + --robot-config robots/dual_franka/config/example.yaml + +VLA Diagnostic Console +~~~~~~~~~~~~~~~~~~~~~~ -A ready-made task ``1`` checkpoint is published on ModelScope: -`Brunchlife/pi05-dualfranka-tcp-rot6d-clean-desk-532-delect-76000 -`_. -Download it, point ``PI05_CHECKPOINT_PATH`` at the downloaded directory, and set -``DUAL_FRANKA_REPO_ID`` to the subdirectory that holds ``norm_stats.json``: +The console runs VLA independently of the agent and registered tasks. The policy +instruction comes from ``vla.instruction`` in the robot config, unless overridden +with ``--instruction``. Run it from a source checkout; it starts Env and VLA +services as needed, without SAM3. .. code-block:: bash - modelscope download \ - --model Brunchlife/pi05-dualfranka-tcp-rot6d-clean-desk-532-delect-76000 \ - --local_dir /path/to/pi05-dualfranka-clean-desk + python -m tests.e2e_tests.dual_franka.dual_franka_vla \ + --robot-config robots/dual_franka/config/example.yaml - export PI05_CHECKPOINT_PATH=/path/to/pi05-dualfranka-clean-desk +Once the console starts, enter: -.. warning:: +* ``run N``: predict and execute N action chunks (sequences produced by the model + in one prediction). Omit N for continuous rollout without skill-boundary stops. +* ``reset``: reset both arms in the same environment after the operator replies + ``done``. Replying ``abort`` cancels the reset. +* ``quit``: exit the console. - This checkpoint is trained only on our in-house test environment (robot - poses, cameras, workspace layout, and objects), so it is expected to - generalize poorly to a different setup. To deploy on your own rig, collect - demonstrations and fine-tune your own checkpoint with RLinf by following the - `RLinf dual-Franka guide - `_ - (collect GELLO demos, convert to tcp_rot6d, run SFT, then deploy). +Startup does not reset the arms. During rollout, press ``Ctrl+C`` or create the +``STOP`` file in the log directory to stop. An in-flight RPC may still finish; +this is not a hardware emergency stop. Inputs and predictions are saved as +JSON/NPZ. If execution state is uncertain, check the physical scene and restart. -When ``--vla-endpoint`` is absent, RPent starts -``rpent/robots/components/pi05_vla_server.py`` and loads -``pi05_dualfranka_tcp_rot6d`` once. +The action chunk length is read from the VLA server, or can be set explicitly +with ``--expected-action-steps``. -To run the VLA service separately: +Run a Task +---------- + +Complete calibration, robot configuration and Ray startup, then configure the +planner using :doc:`../guides/configure_planner`. Before using Codex, complete +:ref:`dual-franka-codex-isolation`. + +With the model variables set, the example below runs task ``2``. Select another +registered task with ``--task-id``. RPent starts the required +environment and model services automatically; do not start VLA in another terminal: .. code-block:: bash - uv run --extra franka python -m rpent.robots.components.pi05_vla_server \ - --embodiment dual_franka \ - --model-path /path/to/checkpoints/global_step_N \ - --repo-id org/dual-franka-tcp-rot6d \ - --cuda-device 0 --transport http --host 0.0.0.0 --port 6000 + rpent --robot dual_franka --task-id 2 \ + --planner codex --model gpt-6-astra --reasoning-effort xhigh \ + --robot-config robots/dual_franka/config/example.yaml \ + --memory-profile local --memory-dir ./memory/dual_franka -Then pass ``--vla-endpoint http://VLA_HOST:6000`` to ``rpent``. An external -endpoint always takes precedence over local auto-start. +Connection initializes controllers and grippers without resetting robot posture. +The agent first calls ``request_scene_reset``: secure held objects, restore the +tabletop and check both reset paths, then answer ``done`` to allow reset and a +fresh observation. Motion remains blocked if confirmation or reset fails. Each +new run/session needs its own confirmation. -External Environment Server ---------------------------- +The example uses Codex, ``gpt-6-astra`` and ``xhigh`` reasoning. Claude Code is +also available: replace the planner/model options with +``--planner claude_code --model claude-opus-4-8``. See +:doc:`../guides/configure_planner` for model and reasoning settings. -To attach RPent to an already-running dual-Franka environment service: +Append options to this same command as needed: -.. code-block:: bash +* ``--explore`` explores and accumulates task memory, as described below. +* ``--explore --interactive`` accepts terminal feedback during exploration; + ordinary tasks do not support ``--interactive`` on its own. +* ``--dashboard`` accepts tasks, messages and confirmations in the web input. + Add ``--explore`` for exploration. Do not combine it with ``--interactive``. + See :doc:`../guides/dashboard` for interface controls. - uv run --extra franka rpent --robot dual_franka --task-id 0 \ - --env-endpoint http://ROBOT_HOST:PORT \ - --planner claude_code --model claude-opus-4-8 \ - --robot-config robots/dual_franka/config/example.yaml +View Results +------------ + +Use a new ``--output-dir`` for each run. Inspect: + +* ``transcript_*.json`` for planner messages, tool calls and the finish result. +* ``run.log`` and component ``*_server.log`` files for runtime/startup errors. +* ``states.json`` and its referenced image, depth and robot-state artifacts for + the evidence after each step. +* ``operator_events.json`` for reset confirmations and human verdicts. + +Exploration keeps per-session state and operator records under +``sessions/session_/``. Physical task success comes from the operator's +verdict, not a successful RPC or the planner's own ``finish`` text. See +:ref:`run-output-files` for the common output layout. Tools and Artifacts ------------------- @@ -353,96 +535,20 @@ analytic motion selects exactly one arm, ``left`` or ``right``. Mutating tools capture per-arm state and synchronized left-wrist, base, and right-wrist images in RPent's central ``EnvState``. -Safety ------- - -Keep operators at both emergency stops. Validate task ``0`` with very small -single-arm motions before attempting a grasp. Stop when camera/state results -disagree, when the requested motion is not reached, or when any calibration is -uncertain. - -Manual Skill Testing --------------------- - -For ordinary terminal evaluation, omit ``--interactive`` and ``--dashboard`` -and answer human-confirmation prompts in a TTY. ``--explore`` also accepts -prompted answers without ``--interactive``. To guide exploration through -terminal conversation, add ``--interactive`` (``-i``); see -:doc:`../guides/dashboard` for Codex/Claude live input and API follow-up behavior. - -With ``--dashboard``, submit tasks, messages and confirmation replies in the -Dashboard message input; no TTY or ``--interactive`` is needed. The two flags -cannot be combined. Dashboard supports ``api``, ``claude_code`` and ``codex`` -(current physical experiments have only validated Codex). -For ordinary tasks, ``/success``, ``/failure`` and ``/continue`` answer a pending -``request_operator_verdict``; the agent then continues to ``finish``. -Ordinary tasks reset on environment connection; exploration waits for scene -confirmation before resetting. - -The deployment scripts live in ``robots/dual_franka/``. From the repository root: - -.. code-block:: bash - - robots/dual_franka/run_manual_skill.sh --list-primitives - robots/dual_franka/run_manual_skill.sh --schema vla_right_grasp - -Use ``--primitive NAME --params JSON`` to call a tool. ``--task-id`` selects -the task's configured ``vla_instruction`` for named VLA skills; the planner's -segment prompt is recorded separately. Existing clean-desk tasks retain their -checkpoint's original training instruction. ``--robot-config`` selects the -machine configuration; its ``perception.calibration`` mapping points at the -easy_handeye hand-eye YAMLs. Local SAM3 requires the ``sam3`` extra; a remote -SAM3 service can be attached with ``--sam3-endpoint``. - -Robot Codex Profile Isolation ----------------------------------------- - -The deployment wrappers select ``RPENT_CODEX_HOME`` (default: -``.codex-rpent-live`` inside the checkout), not the coding shell's -``CODEX_HOME``. Memory defaults to its ``memory`` subdirectory and the Codex -state database uses the dedicated directory too. Create a private ``config.toml`` -there if needed; do not overwrite existing private settings. - -For API deployments, explicitly set ``RPENT_CODEX_API_KEY`` and optionally -``RPENT_CODEX_BASE_URL``. The wrappers clear inherited ``CODEX_API_KEY``, -``CODEX_BASE_URL``, ``OPENAI_API_KEY`` and ``OPENAI_BASE_URL``. Otherwise, -authenticate separately in the dedicated profile. File-based credential storage -can be configured; check private configurations for shared OS keychain use. -Never commit credentials, private configuration or session records. - -``RPENT_CODEX_MODEL``, ``RPENT_REASONING_EFFORT`` and -``RPENT_CODEX_SERVICE_TIER`` default to ``gpt-5.5``, ``medium`` and ``fast``. -These isolation rules apply to the deployment wrappers, not the generic RPent -CLI. Prefer invoking a wrapper: sourcing its environment script directly changes -the current shell's environment. - -Directory isolation is not a security sandbox or workspace-file isolation. -The planner explicitly uses no interactive approvals and full filesystem access. -Editing private configuration does not override the planner's -explicit permissions. The connectivity probe remains read-only. - Attended Exploration -------------------- -``dual_franka --explore`` waits for scene confirmation before resetting the -robot and records human verdicts with camera and robot state. The following -example starts task 0 with terminal confirmation and the planner and VLA settings -described above. Add ``--dashboard`` to use the Dashboard message input instead: - -.. code-block:: bash - - rpent --robot dual_franka --task-id 0 --explore \ - --robot-config /path/to/robot.yaml \ - --memory-dir /path/to/memory/dual_franka \ - --explore-attempts-per-session 3 --explore-sessions 2 \ - --output-dir /path/to/new-run +Append ``--explore`` to a task command to enable exploration. Set attempts per planner session +with ``--explore-attempts-per-session`` and session count with +``--explore-sessions``. Add ``--interactive`` for terminal input or +``--dashboard`` for web interaction. Each planner session begins with ``request_scene_reset``. Secure held objects, restore the scene and reply ``done``. A successful reset and camera/state capture start an attempt; a failed reset keeps motion blocked. ``--explore-attempts-per-session`` limits attempts within each session. External ``--env-endpoint`` servers must advertise ``explicit_reset_only=True`` -to support this confirmation-before-reset workflow. +for both ordinary and exploration runs. ``request_operator_verdict`` asks for ``success``, ``failure``, ``continue`` or ``abort``, with optional notes. Its success/failure reply judges the current @@ -487,44 +593,14 @@ Enter each command on its own. Bare words such as ``success`` and ``failure`` are ordinary agent messages. Already-issued robot actions or RPCs may need to return before the task can end; these commands are not an emergency stop. -VLA Diagnostic Console -~~~~~~~~~~~~~~~~~~~~~~ - -Use the standalone console for prediction recording and explicit execution. -The existing manual primitive entry remains ``robots/dual_franka/run_manual_skill.sh``. -Diagnostics do not register task IDs 103/104 or dispatch through the shared runner. -``--task-id`` selects an existing VLA task profile (default 1); the policy instruction -comes from its ``vla_instruction`` unless overridden with ``--instruction``. -Run the following command from a source checkout. The diagnostic initializes only -the environment and VLA components; configured SAM3 services are not started or contacted. - -.. code-block:: bash - - python -m tests.e2e_tests.dual_franka.dual_franka_vla --task-id 1 \ - --robot-config /path/to/robot.yaml \ - --vla-model-path /path/to/checkpoint --vla-repo-id org/dataset - -Commands: ``prompt ``, ``infer`` (no execution), ``step`` -(fresh prediction and execution), ``run N`` (1–20 chunks), ``reset``, ``quit``. -Initialization may reset the robot. Inputs and predictions are saved before -execution as JSON/NPZ. Invalid observations/actions are rejected; uncertain -execution blocks further motion until restart. RPC success is not task success. -Action validation expects 20 steps per prediction chunk. For a checkpoint with a -different chunk length, set ``--expected-action-steps`` explicitly to match it. -External model servers use the standard VLA prediction and health-check RPCs. - -Flash replay +Flash Replay ------------ Dual Franka uses the shared :ref:`Franka Flash workflow `. -Generate with ``--robot dual_franka --task dual_franka_t0`` and replay with +The example below uses task ``0``; for another task, change the ID in both the +generation and replay arguments. Generate with ``--robot dual_franka --task dual_franka_t0`` and replay with ``--robot dual_franka --planner flash --task-id 0``. Use a dual-arm plan and robot configuration. Translation annotations name ``base``, ``d455``, ``left_wrist`` or ``right_wrist`` with depth and valid calibration. Primitive moves specify ``arm: left`` or ``arm: right``; left-arm workspace checks convert the shared right-base target into the left-base frame. - -Stop the Run ------------- - -Press Ctrl+C in the terminal to request RPent shutdown. Already-issued robot actions or an active RPC must return before cleanup completes; use the hardware emergency stop for an emergency. Check both arms and grippers before shutting down services through the controller’s shutdown procedure. diff --git a/docs/source-zh/rst_source/real_world_robots/dual_franka.rst b/docs/source-zh/rst_source/real_world_robots/dual_franka.rst index 5119ccc3b..321a97393 100644 --- a/docs/source-zh/rst_source/real_world_robots/dual_franka.rst +++ b/docs/source-zh/rst_source/real_world_robots/dual_franka.rst @@ -8,58 +8,63 @@ 用于真机实验的双臂 Franka 系统。 -RPent 可以通过 RLinf ``RealWorldEnv`` worker 控制双节点双臂 Franka 系统。 任务视频见 :doc:`双臂 Franka 演示 <../real_world_demos/franka>`。 +RPent 通过 RLinf ``RealWorldEnv`` worker 控制双节点双臂 Franka 系统。任务视频见 :doc:`双臂 Franka 演示 <../real_world_demos/franka>`。 安装 ---- -请先按照 `libfranka 官方快速安装指南 `_ 安装 libfranka 0.19.0。 - -克隆 RPent 并安装 Python 依赖。若已有仓库，进入仓库后执行 ``uv sync``： +在两台 Linux 控制节点分别安装 RPent，保持代码、Python、Ray 和 RLinf 版本一致。本地运行 VLA 或 SAM3 的节点需要 NVIDIA GPU。以下沿用 :doc:`../get_started/quickstart` 的安装方式： .. code-block:: bash git clone https://github.com/RLinf/RPent.git cd RPent - uv sync --extra franka --extra sam3 + uv venv + source .venv/bin/activate + uv pip install -e ".[franka,sam3]" + +下载 Python 依赖较慢时，可使用 `清华 PyPI 镜像 `_ 替换上面的安装命令： -该命令将 RLinf ``release/v0.4``、``rpent-openpi``、SAM3 以及 Franka 的相机、夹爪和遥操作依赖安装到 ``.venv``。其中的 ``franky-control`` wheel 已包含 libfranka 0.19.0。 +.. code-block:: bash -标定（Calibration） ----------------------------------------- + uv pip install -e ".[franka,sam3]" \ + --index-url https://pypi.tuna.tsinghua.edu.cn/simple -``base_camera`` 和 ``d455_camera`` 都需要标定到 **右臂基座**。可使用 -``calibration_tools/``，也可继续使用 ROS -`easy_handeye `_，最终加载格式均为 YAML。 -腕部相机在默认配置中仅用于观测；若使用腕部 RGBD 相机进行投影，需要按下文 -配置手眼标定和投影视图。 +该选项只替换 Python 包索引，不代理 Git 仓库、直接下载链接或模型权重。 -本工具支持 D435，已在 D435 配置下验证；其他型号和流配置尚未验证，不能保证 -仅修改参数即可使用。零畸变系数可直接使用；非零系数仅支持 -``distortion.brown_conrady``。不支持的非零畸变模型会拒绝采样，并显示模型和 -系数，不会自动转换或忽略畸变。 +连接机械臂前，请核对 libfranka 与机器人固件的兼容性。标定状态读取程序还需要 libfranka 开发库，见下文的标定步骤。 -**1. 准备环境** +后续命令在仓库根目录和该环境中执行。新开终端时，重新激活 ``.venv`` 并设置所需的模型环境变量。 -在相机节点激活已有的 RLinf Franka 运行环境，然后进入 RPent 仓库根目录， -检查所需接口： +需要本地分割时，在环境节点下载 SAM3 权重： .. code-block:: bash - source /absolute/path/to/franka-env/bin/activate - cd /absolute/path/to/RPent - PYTHONPATH=calibration_tools python -c "import numpy, scipy, yaml, pyrealsense2; from common import check_opencv; check_opencv()" + uv pip install modelscope + modelscope download --model facebook/sam3 sam3.pt \ + --local_dir ./checkpoints/sam3 + export SAM3_CHECKPOINT_PATH="$PWD/checkpoints/sam3/sam3.pt" + +未设置该变量或 ``--sam3-endpoint`` 时，分割功能不可用；仅安装 extra 不会下载权重。VLA checkpoint 配置见 :ref:`dual-franka-vla`。 + +标定 +---- + +``base_camera`` 和 ``d455_camera`` 都需要标定到 **右臂基座**。可使用 ``calibration_tools/`` 或 ROS `easy_handeye `_，最终加载格式均为 YAML。腕部相机默认仅用于观测；若使用腕部 RGBD 相机进行投影，需要配置手眼标定和投影视图。 -检查包含 ChArUco、PnP、``calibrateHandEye`` 及所需方法常量。 -若现有环境不满足依赖，可另建标定环境，避免改动策略运行环境： +采集器使用 RealSense RGB 图像，支持零畸变系数或非零的 ``distortion.brown_conrady`` 系数；其他非零畸变模型会拒绝采样。 + +**1. 准备环境** + +在相机节点的 RPent 仓库根目录使用已安装的环境。若需独立安装标定依赖，可创建单独环境： .. code-block:: bash - python3 -m venv .venv-calibration + uv venv .venv-calibration source .venv-calibration/bin/activate - pip install -r calibration_tools/requirements.txt pyrealsense2 + uv pip install -r calibration_tools/requirements.txt pyrealsense2 -在右臂控制节点使用匹配机器人固件的 libfranka 开发库编译状态读取程序： +在右臂控制节点，先按 `libfranka 官方安装指南 `_ 安装与机器人固件兼容的开发库（含头文件和 CMake 配置），再编译状态读取程序： .. code-block:: bash @@ -67,11 +72,12 @@ RPent 可以通过 RLinf ``RealWorldEnv`` worker 控制双节点双臂 Franka -DCMAKE_PREFIX_PATH=/absolute/path/to/libfranka/install cmake --build calibration_tools/build --parallel -产物为 ``calibration_tools/build/read_franka_state``，通过 ``readOnce()`` 读取末端 -位姿、关节速度和机器人状态并输出 JSON，不发送运动指令。Python 控制包的安装 -不保证包含此编译步骤所需的头文件和 CMake 配置。 -下文替换相机序列号、SSH 别名、读取程序绝对路径和机器人 IP；读取程序在本机时 -省略 ``--ssh-host``，需要指定共享库目录时追加 ``--library-dir /path/to/lib``。 +编译后得到 ``calibration_tools/build/read_franka_state``。标定时用它读取机械臂状态，不会移动机械臂。 + +* 本机读取：读取程序与采集器在同一节点，不传 ``--ssh-host``。 +* 远程读取：用 ``--ssh-host`` 指定读取程序所在节点，并提前配置 SSH 免密登录。 + +下文命令中的相机序列号、机器人 IP 和程序路径均需替换为实际值。 **2. 启动相机并采集** @@ -86,6 +92,7 @@ RPent 可以通过 RLinf ``RealWorldEnv`` worker 控制双节点双臂 Franka 标记边长 18 mm、DICT_4X4_100、非 legacy 布局。其他板可通过 ``--squares-x``、``--squares-y``、``--square-m``、``--marker-m`` 和 ``--dictionary`` 配置，长度单位为米。 +请实测打印后的格边长，确保与配置一致。 在同一相机节点另开终端，激活上一步选用的环境后运行： @@ -98,16 +105,20 @@ RPent 可以通过 RLinf ``RealWorldEnv`` worker 控制双节点双臂 Franka --reader /absolute/path/to/read_franka_state --robot-ip ROBOT_IP \ --output calibration_tools/sessions/base-to-right -打开 ``http://127.0.0.1:8767``，人工调整右臂，释放引导按钮并停稳后点击 -``Capture pose``。建议采集 20–30 个不同姿态，覆盖多个旋转轴；求解至少需要十组。 -工具只读取状态，不会移动机械臂。不传 ``--output`` 时，默认写入脚本所在目录下的 -``calibration_tools/sessions/``，与启动时的工作目录无关。 +打开 ``http://127.0.0.1:8767``，按以下步骤采样： + +1. 确认机械臂处于 Idle 且无报错。人工调整右臂后，释放引导按钮，停稳后点击 ``Capture pose``。 +2. 采集 20–30 个不同位置和角度，覆盖多个旋转轴；求解至少需要 10 组。 +3. 用 ``Preview`` 查看样本，``Exclude`` 排除不合适的样本。样本调整后需重新求解。 +4. 采完按 ``Ctrl+C`` 停止采集器。重新采集时，为 ``--output`` 指定新目录。 -在已保存样本表格中查看角点数量和重投影 RMS（像素）。点击 ``Preview`` 打开带标注的样本图像，再通过 ``Exclude`` 将不合适的样本排除出下一次求解；文件仍保留在会话的 ``excluded/`` 目录中。``Active samples`` 只统计参与求解的有效样本；样本编号保持不变，出现空缺是正常的。增加或排除样本不会自动重算已有标定结果，应重新运行求解器后再导出。 +若从自己的电脑访问网页，先转发端口，再打开上面的地址： + +.. code-block:: bash -按 ``Ctrl+C`` 停止采集器。中断后重新开始或整批重采时，重新运行上面的命令，并为 ``--output`` 指定新会话目录。旧会话文件仍保留在磁盘上，但不会加载到新的采集会话中。 + ssh -L 8767:127.0.0.1:8767 CAMERA_HOST -标定 D455 时重复上述命令，将序列号换成 ``D455_SERIAL``、URL 换成 ``/raw/d455``、输出目录换成 ``calibration_tools/sessions/d455-to-right``。D455 流配置需先确认符合上面的畸变模型支持范围。 +标定 D455 时复用采集命令，将序列号改为 ``D455_SERIAL``、URL 改为 ``http://127.0.0.1:8765/raw/d455``、输出目录改为 ``calibration_tools/sessions/d455-to-right``。 **3. 求解并导出** @@ -136,7 +147,7 @@ D455 使用同样步骤，将会话目录和输出文件改为 ``d455-to-right`` base_camera: /absolute/path/to/base_to_right.yaml d455_camera: /absolute/path/to/d455_to_right.yaml -通过下文的 ``--robot-config`` 加载配置。启动 RPent 前停止标定相机服务。 +通过下文的 ``--robot-config`` 加载配置。 保留并核对 ``perception.base_frames`` 和定位边界；本工具不会自动标定左右基座关系。 腕部投影为可选功能。在示例配置中取消 ``perception.calibration`` 和 @@ -150,8 +161,12 @@ RPent 将其与所选快照中的末端位姿组合；若位姿使用 ``left_bas 并声明基座坐标系。默认的 Lumos 观测相机不会自动启用 RGBD 投影；选择腕部锚点前 必须完成这些配置。 -开发配置 --------- +标定结束后，停止本次标定启动的全部相机服务、采集器和机器人状态读取程序，再启动 RPent。 + +机器人配置 +---------- + +若标定使用了独立 Python 环境，执行后续命令前请重新激活 RPent 的 ``.venv``。 启用机械臂运动前，请检查并修改仓库中的开发默认值： @@ -159,19 +174,120 @@ RPent 将其与所选快照中的末端位姿组合；若位姿使用 ``left_bas RPent 会将该机器人配置转换成内部双节点 RLinf cluster 和环境对象。如需使用其他文件，请传入 ``--robot-config /path/to/robot_config.yaml``。 -``rotate_delta`` 旋转时，位置目标保持在本次旋转开始时的 TCP 位置。真实硬件采用柔顺/阻抗控制，实际位姿可能因负载、接触或有限的跟踪精度而偏离目标。 +示例配置中，各相机的用途如下： + +* ``base``、``left_wrist``、``right_wrist``：提供 RGB 图像，用于观察场景。 +* ``d455``：提供 RGB 和对齐深度图像，可将图像中的选点转换为三维坐标。反投影需要该相机的有效标定。 + +``perception.projection_views`` 用于选择反投影视角，不控制相机是否采集深度；示例配置中应使用 ``d455``。 + +``rotate_delta`` 调整末端朝向，并尽量保持旋转开始时的位置。``rotation_hold.max_drift_m`` 设置旋转时允许的位置偏移上限，默认 ``0.10`` m（10 cm）；超出上限时中止本次调用并报错。 + +任务选择 +-------- + +主要示例使用任务 ``2``。首次连接时可先用任务 ``0`` 检查小幅平移、旋转和夹爪动作；该任务由规划器驱动，不是固定自动测试序列，也不需要 VLA。 + +.. list-table:: + :header-rows: 1 + + * - ID + - 内容 + - 是否需要 VLA + * - 0 + - 规划器驱动的单技能检查 + - 否 + * - 1 + - 碗盘脏污分类，包含杯子和筷勺 + - 是 + * - 2 + - 碗盘及两个杯子脏污分类，无筷勺 + - 是 + +.. note:: + + 任务编号用于启动命令，分类任务使用任务名称生成 recipe 和记忆标识。修改任务时，请保持编号、名称与任务内容的对应关系，不要随意重排编号或将已有名称用于不同任务。 + +任务 ``1`` 中，碗盘内的蛋挞或蛋挞锡盘标记脏污：脏碗盘放入金属筐，干净碗盘和杯子放入纸盒；筷勺插入杯中。 + +任务 ``2`` 中，碗内的鸡块、盘子或杯子内的蛋挞锡盘标记脏污：脏物放入金属筐，干净物放入纸盒，不含筷勺。 + +两个分类任务均提供 ``vla_right_grasp`` / ``vla_handoff`` / ``vla_left_place`` 技能。 + +.. _dual-franka-vla: + +VLA 配置 +-------- + +``PI05_CHECKPOINT_PATH`` 指向训练好的 Pi0.5 checkpoint，``DUAL_FRANKA_REPO_ID`` 是用于查找对应归一化统计的数据集 ID。 + +上述环境变量对应 ``--vla-model-path`` 和 ``--vla-repo-id``。repo ID 是 checkpoint 内对应归一化统计的相对目录，不是模型下载仓库名。checkpoint 必须包含： + +.. code-block:: text + + actor/model_state_dict/full_weights.pt + /norm_stats.json + +**预训练 checkpoint** + +从 Hugging Face 下载 `双臂 Franka clean-desk checkpoint `_。它用于执行 VLA 技能，物体的脏污分类由 RPent 规划器判断。 + +.. code-block:: bash + + uv pip install huggingface_hub + hf download RLinf/RLinf-Pi05-RealWorld-DualFranka-Clean_Desk \ + --revision 4f43adc723dfe98fae806a131ad61999e1f95e4f \ + --local-dir ./models/pi05-dualfranka-clean-desk + + export PI05_CHECKPOINT_PATH="$PWD/models/pi05-dualfranka-clean-desk" + export DUAL_FRANKA_REPO_ID=id_0 + +该 checkpoint 已包含 ``id_0/norm_stats.json``，请与权重一起保留，并按上例将 repo ID 设置为 ``id_0``。 + +.. warning:: + + 该 checkpoint 仅在我们的内部测试环境（机器人位姿、相机、工作空间布局和物体）上训练，切换到不同的环境时预计表现会较差。若要部署到你自己的机器上，请使用 RLinf 采集示教数据并微调你自己的 checkpoint，参见 `RLinf 双臂 Franka 指南 `_ （采集 GELLO 示教数据、转换为 tcp_rot6d、运行 SFT，然后部署）。 + +.. _dual-franka-codex-isolation: + +机器人 Codex 运行配置隔离 +------------------------- + +在专用终端激活 ``.venv``，从仓库根目录设置独立的 Codex 配置与会话目录。每次新开机器人控制终端都需执行： + +.. code-block:: bash + + export CODEX_HOME="$PWD/.codex-rpent-live" + export CODEX_SQLITE_HOME="$CODEX_HOME" + export CODEX_SERVICE_TIER=fast + mkdir -p "$CODEX_HOME" + unset CODEX_API_KEY CODEX_BASE_URL OPENAI_API_KEY OPENAI_BASE_URL CODEX_ACCESS_TOKEN + export CODEX_BIN="$(python -c 'from codex_cli_bin import bundled_codex_path; print(bundled_codex_path())')" + +首次使用时，在 ``$CODEX_HOME/config.toml`` 中设置文件凭据存储，再用 SDK 自带的 CLI 登录： + +.. code-block:: toml + + cli_auth_credentials_store = "file" -``rotation_hold.max_drift_m`` 默认为 ``0.10`` m（10 cm），限制 TCP 相对本次旋转起始位置的欧氏距离；它与位置到位容差 ``move_tolerance_m`` （0.006 m）不同。漂移小于或等于上限时，旋转结束后仅在位置误差超过到位容差时复用 ``move_delta`` 回位。只有补偿成功（如需补偿），且最终位置和角度均达到各自容差，才返回 ``ok=True`` ，否则返回 ``False`` 。漂移超过上限时，旋转循环中或结束时的检查会抛出 RPC 错误，停止本次调用，不再继续平移补偿。 +.. code-block:: bash + + "$CODEX_BIN" login --device-auth + "$CODEX_BIN" login status -机器人自身的保护仍可能独立触发。 +若配置中已有 ``sqlite_home``，也应指向独立目录。使用自定义 API 时，重新设置本次部署的 ``CODEX_API_KEY`` 和 ``CODEX_BASE_URL``；其他登录方式见 `Codex 认证说明 `_。 + +RPent 不会自动启用上述隔离；它不隔离共享工作区或迁移旧会话。任务记忆另由 ``--memory-dir`` 选择。不要提交该目录或凭据。 启动双节点 Ray 集群 -------------------- -两个节点的角色不同（定义在 ``robots/dual_franka/runtime_config.py`` 中）： +启动 Ray 前，在两个节点分别激活安装好的 ``.venv``。``robots/dual_franka/runtime_config.py`` 的默认部署为： + +* 节点 ``0`` 是 Ray head：运行 RPent、环境 worker、左臂及夹爪控制器、base 相机和左腕相机。额外的 D455 由 RPent 在此本地打开；按需启动的本地 VLA、SAM3 服务也运行在这里。 +* 节点 ``1`` 是 Ray worker：运行右臂及夹爪控制器和右腕相机，需要安装环境并启动 Ray，但不需要另起 RPent runner。 -* 节点 ``0`` 是 Ray head 节点：运行双臂 Franka 环境 worker （全部相机、感知以及双臂和夹爪状态）和左臂的实时控制器。VLA 任务的本地 VLA 服务也运行在该节点上。 -* 节点 ``1`` 是 Ray worker 节点：只运行右臂的实时控制器，不接相机，也不运行 RPent 进程。 +RLinf 按配置的节点创建相机句柄，不会跨节点搜索被移动的 USB 相机。腕部相机应接在对应机械臂的控制节点，并按实际设备填写型号和序列号。 .. warning:: @@ -193,136 +309,106 @@ RPent 会将该机器人配置转换成内部双节点 RLinf cluster 和环境 ray stop --force ray start --address=HEAD_IP:6379 --node-ip-address=WORKER_IP -运行冒烟测试 +手动技能测试 ------------ -先按 :doc:`../guides/configure_planner` 配置规划器与模型服务。 - -任务 ``0`` 用于逐臂测试小幅平移、旋转和夹爪动作： +可先列出可调用的技能，再查看某个技能接受的参数及其类型。在仓库根目录运行： .. code-block:: bash - uv run --extra franka rpent --robot dual_franka --task-id 0 \ - --planner claude_code --model claude-opus-4-8 \ - --robot-config robots/dual_franka/config/example.yaml - -RPent 使用当前解释器启动 ``robots/dual_franka/env_server.py``，加载 RPent robot config 并生成内部 RLinf adapter config，然后连接 Ray，等待 ``healthz``，并将初始状态记录为 step ``0``。任务 ``0`` 不会加载 VLA。 + python robots/dual_franka/dual_franka_manual_call.py --list-primitives + python robots/dual_franka/dual_franka_manual_call.py --schema vla_right_grasp -VLA 抓取演示 -------------- +未指定服务地址时，脚本自动启动并连接所需服务，无需手动填写端口；退出时只关闭本次启动的服务。传入 ``--env-endpoint``、``--vla-endpoint`` 或 ``--sam3-endpoint`` 可复用已有服务。连接已有 Env 不会复位。若新启动 Env，手动动作前先请求场景确认并执行一次复位；显式调用手动 ``reset`` 也会等待确认。列出技能和查看参数不启动服务。 -RPent 提供了一个使用 VLA 抓取物品的演示。任务 ``1`` 提供 ``vla_right_grasp`` / ``vla_handoff`` / ``vla_left_place`` 工具，并可在本地启动双臂 Franka VLA 服务。``PI05_CHECKPOINT_PATH`` 指向训练好的 Pi-05 checkpoint，``DUAL_FRANKA_REPO_ID`` 是用于查找对应归一化统计的数据集 ID： +例如，先看 schema，再执行左臂小幅平移： .. code-block:: bash - export PI05_CHECKPOINT_PATH=/path/to/checkpoints/global_step_N - export DUAL_FRANKA_REPO_ID=org/dual-franka-tcp-rot6d - - uv run --extra franka rpent --robot dual_franka --task-id 1 \ - --cuda-device 0 \ - --planner claude_code --model claude-opus-4-8 \ + python robots/dual_franka/dual_franka_manual_call.py --schema move_delta + python robots/dual_franka/dual_franka_manual_call.py \ + --primitive move_delta --params '{"arm":"left","delta_xyz":[0.01,0,0]}' \ --robot-config robots/dual_franka/config/example.yaml -checkpoint 必须包含： +通过 ``--primitive NAME --params JSON`` 调用工具，不选择任务编号。示例配置的 ``vla.instruction`` 已填写 clean-desk 策略指令；更换 checkpoint 时可修改该值，或用 ``--instruction`` 覆盖。该配置只用于手动调用，任务 runner 仍使用已注册任务的指令。``--params`` 中的 ``prompt`` 记录技能意图，不替代策略指令。 -.. code-block:: text - - actor/model_state_dict/full_weights.pt - /norm_stats.json - -**预训练 checkpoint** - -ModelScope 上发布了一个可直接使用的 task ``1`` checkpoint： `Brunchlife/pi05-dualfranka-tcp-rot6d-clean-desk-532-delect-76000 `_。下载后将 ``PI05_CHECKPOINT_PATH`` 指向下载目录，并将 ``DUAL_FRANKA_REPO_ID`` 设置为包含 ``norm_stats.json`` 的子目录： +设置好上文的 VLA checkpoint 和归一化统计变量后，可以执行一次抓取技能： .. code-block:: bash - modelscope download \ - --model Brunchlife/pi05-dualfranka-tcp-rot6d-clean-desk-532-delect-76000 \ - --local_dir /path/to/pi05-dualfranka-clean-desk - - export PI05_CHECKPOINT_PATH=/path/to/pi05-dualfranka-clean-desk - -.. warning:: - - 该 checkpoint 仅在我们的内部测试环境（机器人位姿、相机、工作空间布局和物体）上训练，切换到不同的环境时预计表现会较差。若要部署到你自己的机器上，请使用 RLinf 采集示教数据并微调你自己的 checkpoint，参见 `RLinf 双臂 Franka 指南 `_ （采集 GELLO 示教数据、转换为 tcp_rot6d、运行 SFT，然后部署）。 + python robots/dual_franka/dual_franka_manual_call.py \ + --primitive vla_right_grasp \ + --params '{"prompt":"grasp the bowl","max_chunks":1}' \ + --robot-config robots/dual_franka/config/example.yaml -未设置 ``--vla-endpoint`` 时，RPent 会启动 ``rpent/robots/components/pi05_vla_server.py``，并只加载一次 ``pi05_dualfranka_tcp_rot6d``。 +独立 VLA 诊断控制台 +~~~~~~~~~~~~~~~~~~~ -也可以单独启动 VLA 服务： +该控制台独立运行 VLA，不经过智能体或注册任务。策略指令来自机器人配置中的 ``vla.instruction``，可用 ``--instruction`` 覆盖。请在源码仓库中运行；控制台按需启动环境和 VLA 服务，不启动 SAM3。 .. code-block:: bash - uv run --extra franka python -m rpent.robots.components.pi05_vla_server \ - --embodiment dual_franka \ - --model-path /path/to/checkpoints/global_step_N \ - --repo-id org/dual-franka-tcp-rot6d \ - --cuda-device 0 --transport http --host 0.0.0.0 --port 6000 - -然后向 ``rpent`` 传入 ``--vla-endpoint http://VLA_HOST:6000``。外部 endpoint 始终优先于本地自动启动。 - -连接外部环境服务 ----------------- + python -m tests.e2e_tests.dual_franka.dual_franka_vla \ + --robot-config robots/dual_franka/config/example.yaml -连接已经运行的双臂 Franka 环境服务： +启动后，在控制台输入： -.. code-block:: bash +* ``run N``：预测并执行 N 个动作块（每块是模型一次输出的一段连续动作）；不带 N 时持续运行，不在技能边界处停止。 +* ``reset``：等待操作员回复 ``done`` 后，在同一环境中复位双臂；回复 ``abort`` 则不执行复位。 +* ``quit``：退出控制台。 - uv run --extra franka rpent --robot dual_franka --task-id 0 \ - --env-endpoint http://ROBOT_HOST:PORT \ - --planner claude_code --model claude-opus-4-8 \ - --robot-config robots/dual_franka/config/example.yaml +启动时不会自动回位。运行中按 ``Ctrl+C`` 或创建日志目录下的 ``STOP`` 文件可停止；正在执行的 RPC 仍可能完成，这不是硬件急停。 +输入和预测保存为 JSON/NPZ。如果提示执行状态不确定，请检查现场后重启会话。 -工具与状态产物 --------------- +每块动作步数自动从 VLA 服务读取，也可用 ``--expected-action-steps`` 显式指定。 -双臂 Franka 扩展提供 ``view_env_state``、``view_camera_meta``、 ``move_delta``、``rotate_delta``、``open_gripper``、``close_gripper`` 和 ``vla_right_grasp`` / ``vla_handoff`` / ``vla_left_place``。每次解析式运动只会作用于一条臂（``left`` 或 ``right``）。所有会改变环境状态的工具都会在 RPent 统一的 ``EnvState`` 中保存每条臂的状态以及同步的 left-wrist、base 和 right-wrist 图像。 +运行一个任务 +------------ -安全要求 --------- +完成相机标定、机器人配置和 Ray 集群启动，并按 :doc:`../guides/configure_planner` 配置规划器。使用 Codex 前，先完成 :ref:`dual-franka-codex-isolation`。 -两条机械臂的急停按钮旁都必须有操作员值守。先使用极小幅度的单臂动作验证任务 ``0``，再尝试抓取。当相机与状态结果不一致、目标运动没有到位，或任何标定存在疑问时，应立即停止。 +设置好上面的模型变量后，以下以任务 ``2`` 为例运行；可通过 ``--task-id`` 选择其他已注册任务。RPent 自动启动所需的环境和模型服务，无需另开终端手动启动 VLA： -手动技能测试 ------------- +.. code-block:: bash -普通终端评测不加 ``--interactive`` 和 ``--dashboard``，在 TTY 中按提示完成人工确认。``--explore`` 不加 ``--interactive`` 也能按提示答复。需要通过终端对话指导探索时，添加 ``--interactive`` 或 ``-i``；Codex/Claude 的运行中输入和 API 的回复后追加输入方式见 :doc:`../guides/dashboard`。 + rpent --robot dual_franka --task-id 2 \ + --planner codex --model gpt-6-astra --reasoning-effort xhigh \ + --robot-config robots/dual_franka/config/example.yaml \ + --memory-profile local --memory-dir ./memory/dual_franka -使用 ``--dashboard`` 时，在 Dashboard 消息输入框中提交任务、消息和确认回复，无需 TTY，也不加 ``--interactive``；两个参数不能同时使用。Dashboard 支持 ``api``、``claude_code`` 和 ``codex`` （当前真机实验仅验证 Codex）。普通任务中，``/success``、``/failure`` 和 ``/continue`` 用于回答待处理的 ``request_operator_verdict``，随后智能体继续执行并调用 ``finish``。普通任务连接环境时复位，探索则等待场景确认后再复位。Dashboard 操作见 :doc:`../guides/dashboard`。 +连接环境会初始化控制器和夹爪，但不复位机器人姿态。智能体先调用 ``request_scene_reset``：处理夹持物、恢复桌面并确认双臂回位路径安全后回复 ``done``，工具才执行复位并采集新观测。未确认或复位失败时禁止运动，每次新运行或会话都需要重新确认。 -部署脚本位于 ``robots/dual_franka/``。在仓库根目录运行： +示例使用 Codex、``gpt-6-astra`` 和 ``xhigh`` 推理强度。也可将规划器和模型参数换为 ``--planner claude_code --model claude-opus-4-8``。模型与推理强度的配置见 :doc:`../guides/configure_planner`。 -.. code-block:: bash +在同一条命令中按需追加参数： - robots/dual_franka/run_manual_skill.sh --list-primitives - robots/dual_franka/run_manual_skill.sh --schema vla_right_grasp +* ``--explore``：探索并积累任务记忆，见下文。 +* ``--explore --interactive``：探索时在终端输入反馈；普通任务不支持单独使用 ``--interactive``。 +* ``--dashboard``：通过网页输入框提交任务、消息和确认；探索时同时加 ``--explore``。不能与 ``--interactive`` 同用。界面操作见 :doc:`../guides/dashboard`。 -通过 ``--primitive NAME --params JSON`` 调用工具。``--task-id`` 为命名 VLA 技能选择任务配置中的 ``vla_instruction``，规划器的阶段 prompt 单独记入日志。当前 clean-desk 任务继续使用 checkpoint 原来的训练指令。 ``--robot-config`` 选择机器参数，其中的 ``perception.calibration`` 映射指向 easy_handeye 手眼标定 YAML。本地 SAM3 需要安装 ``sam3`` extra；远端服务可通过 ``--sam3-endpoint`` 接入。 +查看结果 +-------- -机器人 Codex 运行配置隔离 -------------------------- +每次运行使用新的 ``--output-dir``。重点查看： -上述启动脚本使用独立的 ``RPENT_CODEX_HOME`` （默认仓库内 ``.codex-rpent-live``），不继承编程终端的 ``CODEX_HOME``。本地记忆默认放在该目录的 ``memory`` 中，Codex 状态数据库也使用独立目录。按需在独立目录创建私人 ``config.toml``，不要覆盖已有私人配置。 +* ``transcript_*.json``：规划器消息、工具调用与结束结果。 +* ``run.log`` 及各组件 ``*_server.log``：运行或启动错误。 +* ``states.json`` 及其引用的图像、深度和机器人状态文件：每步执行后的观测证据。 +* ``operator_events.json``：复位确认与人工判定。 -API 部署需显式设置 ``RPENT_CODEX_API_KEY`` 和可选的 ``RPENT_CODEX_BASE_URL``。脚本会清除继承的 ``CODEX_API_KEY``、``CODEX_BASE_URL``、``OPENAI_API_KEY`` 和 ``OPENAI_BASE_URL``。不使用 API key 时，应在独立目录下单独登录；可将凭据配置为保存在文件中。不要提交真实配置、凭据或日志。已有私人配置若使用系统钥匙串，须另行检查账号共享。 +探索模式的逐会话状态和人工记录位于 ``sessions/session_/``。真机任务成功以操作员判定为准，RPC 成功或规划器自己的 ``finish`` 文本不能替代该判定。通用目录说明见 :ref:`run-output-files`。 -模型、推理强度、服务档位分别通过 ``RPENT_CODEX_MODEL``、 ``RPENT_REASONING_EFFORT``、``RPENT_CODEX_SERVICE_TIER`` 设置，默认保持 ``gpt-5.5``、``medium``、``fast``。这些规则只适用于上述部署脚本，不改变直接调用 RPent CLI 的通用环境变量接口。建议通过启动脚本运行；手动 source 会修改当前终端的环境变量。 +工具与状态产物 +-------------- -目录隔离不是权限沙盒，也不隔离共享工作区文件。当前 planner 显式使用不请求交互审批、完整文件访问的运行参数。仅修改私人配置不能覆盖 planner 显式传入的权限；连接检测仍使用只读沙盒。 +双臂 Franka 扩展提供 ``view_env_state``、``view_camera_meta``、 ``move_delta``、``rotate_delta``、``open_gripper``、``close_gripper`` 和 ``vla_right_grasp`` / ``vla_handoff`` / ``vla_left_place``。每次解析式运动只会作用于一条臂（``left`` 或 ``right``）。所有会改变环境状态的工具都会在 RPent 统一的 ``EnvState`` 中保存每条臂的状态以及同步的 left-wrist、base 和 right-wrist 图像。 探索模式（人工复位与判定） -------------------------- -``dual_franka --explore`` 等待人工确认场景后复位机器人，并记录人工判定、相机及机器人状态。以下示例按前文配置规划器和 VLA 参数，使用终端确认运行任务 0；通过 Dashboard 消息输入框操作时，添加 ``--dashboard``： - -.. code-block:: bash - - rpent --robot dual_franka --task-id 0 --explore \ - --robot-config /path/to/robot.yaml \ - --memory-dir /path/to/memory/dual_franka \ - --explore-attempts-per-session 3 --explore-sessions 2 \ - --output-dir /path/to/new-run +在任务运行命令后追加 ``--explore``，即可启用探索模式。``--explore-attempts-per-session`` 设置每个规划器会话的尝试次数，``--explore-sessions`` 设置会话数；终端交互再加 ``--interactive``，网页交互改用 ``--dashboard``。 -每个规划器会话先通过 ``request_scene_reset`` 请求场景确认。处理夹持物体、恢复场景后回复 ``done``；机器人复位成功并取得相机和状态观测后开始一次尝试，复位失败时不允许运动。``--explore-attempts-per-session`` 限制每个会话的尝试次数。通过 ``--env-endpoint`` 接入的外部服务必须声明 ``explicit_reset_only=True``，才能支持先确认、后复位。 +每个规划器会话先通过 ``request_scene_reset`` 请求场景确认。处理夹持物体、恢复场景后回复 ``done``；机器人复位成功并取得相机和状态观测后开始一次尝试，复位失败时不允许运动。``--explore-attempts-per-session`` 限制每个会话的尝试次数。普通任务和探索模式通过 ``--env-endpoint`` 接入的外部服务都必须声明 ``explicit_reset_only=True``。 ``request_operator_verdict`` 请求 ``success``、``failure``、``continue`` 或 ``abort``，可附备注。其中 success/failure 答复判定当前尝试，``continue`` 继续尝试，``abort`` 停止探索。后续运动会清除先前判定；关闭终端输入也会停止探索。 @@ -347,30 +433,12 @@ API 部署需显式设置 ``RPENT_CODEX_API_KEY`` 和可选的 ``RPENT_CODEX_BAS 每条命令单独输入。不带斜杠的 ``success``、``failure`` 等词语是发给智能体的普通消息。已发送的机器人动作或 RPC 可能需返回后才能结束任务；这些命令不是急停。 -独立 VLA 诊断控制台 -~~~~~~~~~~~~~~~~~~~ - -原语测试复用 ``robots/dual_franka/run_manual_skill.sh``，VLA 诊断使用独立入口。不注册 task103/104，也不经过共享 runner 分发。``--task-id`` 选择已有 VLA 任务配置（默认 1），指令来自其 ``vla_instruction``，可用 ``--instruction`` 覆盖。请在源码仓库中运行以下命令。诊断只初始化环境和 VLA 组件，即使配置了 SAM3，也不会启动或连接其服务。 - -.. code-block:: bash - - python -m tests.e2e_tests.dual_franka.dual_franka_vla --task-id 1 \ - --robot-config /path/to/robot.yaml \ - --vla-model-path /path/to/checkpoint --vla-repo-id org/dataset - -支持 ``prompt <指令>``、``infer`` （不执行）、``step`` （重新推理并执行）、 ``run N`` （1–20 块）、``reset``、``quit``。初始化可能复位。执行前将输入与预测保存为 JSON/NPZ，拒绝无效观测或动作；执行结果不确定时禁止继续运动，需重启会话。 RPC 成功不代表任务成功。动作校验默认要求每块预测包含 20 步；如果 checkpoint 使用不同块长度，请通过 ``--expected-action-steps`` 显式指定匹配的值。外部模型服务使用标准 VLA 推理和健康检查 RPC。 - Flash 回放 ---------- -双臂使用共享的 :ref:`Franka Flash 流程 `。生成任务卡时使用 +双臂使用共享的 :ref:`Franka Flash 流程 `。以下以任务 ``0`` 为例；其他任务应同时替换生成和回放参数中的任务编号。生成任务卡时使用 ``--robot dual_franka --task dual_franka_t0``，回放时使用 ``--robot dual_franka --planner flash --task-id 0``，并提供双臂任务卡及配置。 平移标注使用具有深度和有效标定的 ``base``、``d455``、``left_wrist`` 或 ``right_wrist`` 相机。移动原语通过 ``arm: left`` 或 ``arm: right`` 指定机械臂； 左臂工作空间检查会把共享的右基座目标坐标转换到左基座坐标系。 - -停止运行 ------------- - -在终端按 Ctrl+C 请求结束 RPent。已经发送的机器人动作或正在执行的 RPC 需要返回后才能完成清理；紧急情况下应使用硬件急停按钮。停止后检查两条机械臂与夹爪状态，再按控制系统的停机流程关闭服务。