Rust-based robotics simulator with LLM-driven synthetic data generation and a Gymnasium-compatible training interface. Built on Bevy ECS and Rapier 3D physics, with Python training via TCP protocol.
- LLM-driven synthetic motion data — an LLM (GPT-5.4 by default) plans manipulation tasks, the simulator executes and validates them, producing labeled trajectory datasets at scale
- Behavioral cloning pipeline — train models from synthetic trajectories with automatic joint encoding, velocity/position prediction modes, and ONNX export
- Gymnasium-compatible training — PPO, SAC, or any RL algorithm via Stable Baselines3 over a TCP gym protocol
- Rust simulation engine — Bevy ECS plugin architecture with Rapier 3D rigid-body physics, URDF loading, domain randomization, and motor models
- Synthetic image generation — headless GPU rendering for camera and depth observations, with generative AI augmentation pipeline
- ONNX policy deployment — train in Python, export to ONNX, run natively in the simulator at full speed
Python (training) Rust (simulation)
----------------- ------------------
SB3 / PyTorch Bevy ECS
| |
Gymnasium env <---- TCP/JSON ----> clankers-gym server
| |
rewards.py +----+----+
terminations.py | Rapier |
joint_encoder.py | physics |
trajectory_dataset.py +----+----+
|
URDF loading
Motor models
Domain randomization
ONNX inference
3D visualization
Training happens in Python. Simulation happens in Rust. They communicate over TCP with JSON messages. Trained policies export to ONNX and run natively in the simulator at full speed.
crates/
+-- clankers-core System ordering, SimTime, Sensor/ActionApplicator traits
+-- clankers-noise Gaussian, uniform, bias, drift noise models
+-- clankers-actuator-core Motor model math (transmission, friction) -- no Bevy
+-- clankers-actuator Bevy plugin: PD control, joint components, dynamics
+-- clankers-env Episode lifecycle, sensors (joint, IMU, contact, camera)
+-- clankers-urdf URDF parsing and Bevy entity spawning
+-- clankers-physics Rapier 3D backend with configurable solver
+-- clankers-policy ONNX inference runner for trained policies
+-- clankers-domain-rand Per-episode physics randomization for sim-to-real
+-- clankers-gym TCP server, Gymnasium protocol, VecEnv support
+-- clankers-render Headless GPU rendering for image observations
+-- clankers-viz Interactive 3D visualization with egui
+-- clankers-teleop Manual control interfaces for debugging
+-- clankers-ik Inverse kinematics solver
+-- clankers-mpc Centroidal convex MPC + whole-body controller
+-- clankers-sim Top-level plugin, SceneBuilder
+-- clankers-record MCAP episode recording with provenance
+-- clankers-test-utils Shared test fixtures
python/
+-- clankers/ Training client library
| +-- client.py TCP client
| +-- gymnasium_env.py Full Gymnasium interface
| +-- sb3_vec_env.py Stable Baselines3 vectorized wrapper
| +-- joint_encoder.py Robot-agnostic joint position encoding
| +-- trajectory_dataset.py PyTorch Dataset for offline training
| +-- rewards.py Reward function templates
| +-- terminations.py Episode termination conditions
| +-- augmentation/ Synthetic image augmentation with diffusion models
| ...
+-- clankers_synthetic/ LLM-driven synthetic trajectory generation
| +-- pipeline.py End-to-end: plan -> compile -> validate -> package
| +-- compiler.py Skill execution through simulation
| +-- planner.py LLM plan generation (GPT-5.4 default)
| ...
+-- examples/ Training scripts
+-- train_ppo.py PPO training with SB3
+-- train_joint_bc.py Behavioral cloning from trajectories
+-- replay_policy.py Policy vs ground-truth comparison
...
examples/
+-- examples/ 22 Rust examples (pendulum, cartpole, arm, quadruped)
+-- urdf/ Robot models (pendulum, cartpole, 6-DOF arm, quadruped)
apps/
+-- clankers-app/ Example end-to-end simulation application
With Docker (nothing else needed):
docker compose run datagen # hardcoded pick plan, no API key needed
docker compose run datagen --n-plans 5 # LLM-planned dataset (set OPENAI_API_KEY first)Locally (Rust via rustup + Python 3.10+; the pinned
toolchain from rust-toolchain.toml installs automatically):
# Linux only: Bevy needs these system libraries
sudo apt-get install libasound2-dev libudev-dev libwayland-dev
pip install -e python -e python/clankers_synthetic
python -m clankers_synthetic --dry-run # auto-launches the simulator via cargoThe --dry-run flag executes a hardcoded pick-and-place plan through the
physics sim, runs the same validation gates as the LLM pipeline, and writes
dry_run_trace.json + dry_run_validation.json — no LLM involved, so it
works without an API key, and it exits non-zero if validation fails (usable
as a smoke test). Drop the flag (and set OPENAI_API_KEY) to have the LLM
(default gpt-5.4, override with --model) plan the manipulation tasks;
each plan is executed and physics-validated before it enters the dataset.
See python -m clankers_synthetic --help for scene, model, and server
options.
cargo test --workspace
cargo build --releaseRun a CartPole gym server, then train PPO from Python:
# Terminal 1: start the simulator
cargo run --release -p clankers-examples --example cartpole_gym
# Terminal 2: train
pip install -e "python[sb3]"
python python/examples/cartpole_train_ppo.pyRun the quadruped with MPC:
cargo run --release -p clankers-examples --example quadruped_mpc_viz| Robot | DOF | URDF | Examples |
|---|---|---|---|
| Pendulum | 1 | examples/urdf/pendulum.urdf |
pendulum_headless, pendulum_viz |
| CartPole | 2 | examples/urdf/cartpole.urdf |
cartpole_gym, cartpole_vec_gym, cartpole_policy_viz |
| 6-DOF Arm | 8 | examples/urdf/six_dof_arm.urdf |
arm_gym, arm_ik_viz, arm_pick_gym, arm_pick_replay |
| Quadruped | 12 | examples/urdf/quadruped.urdf |
quadruped_mpc, quadruped_mpc_viz, quadruped_mpc_bench |
Online RL — simulator runs a gym server, Python agent connects and trains:
cargo run --release -p clankers-examples --example arm_gym
python python/examples/train_ppo.py --port 9879
python python/examples/export_sb3_to_onnx.py # export to ONNX
cargo run --release -p clankers-examples --example arm_with_policy # run in RustOffline BC — train from recorded trajectory data:
# Generate the demo trajectory first:
python -m clankers_synthetic --dry-run --out output/arm_pick_dataset
python python/examples/train_joint_bc.py \
--trace output/arm_pick_dataset/dry_run_trace.json \
--scene python/clankers_synthetic/scenes/arm_pick_cube.json \
--mode velocity
python python/examples/replay_policy.py \
--model joint_bc.onnx \
--trace output/arm_pick_dataset/dry_run_trace.json --plotSynthetic data generation — LLM plans a manipulation task, simulator executes and validates:
# Uses the bundled arm_pick_cube scene and auto-launches the simulator;
# requires OPENAI_API_KEY (or pass --dry-run for the no-LLM demo plan).
python -m clankers_synthetic --n-plans 10 --out output/arm_pick_datasetAny robot's joints are encoded in alphabetical order for deterministic vector layout:
from clankers.joint_encoder import JointEncoder
encoder = JointEncoder(["wrist", "elbow", "shoulder"])
# Sorted: elbow, shoulder, wrist -> indices 0, 1, 2
vec = encoder.encode({"wrist": -0.3, "shoulder": 0.5, "elbow": 1.2})
# array([1.2, 0.5, -0.3]) -- always alphabetical
restored = encoder.decode(vec)
# {"elbow": 1.2, "shoulder": 0.5, "wrist": -0.3}Models trained with the encoder embed joint metadata in ONNX, so any model with matching input/output dimensions works.
- Bevy ECS for modularity — each feature is a plugin, compose what you need
- Rapier 3D physics — rigid-body simulation with configurable solver iterations
- TCP protocol keeps Python and Rust cleanly separated — no FFI, no shared memory
- ONNX export for policies — train in Python, deploy in Rust at native speed
- Alphabetic joint encoding for robot-agnostic DL — same model code works across robots
- MCAP recording for data provenance — replay any episode offline
MIT