Bi-Manual Remote Robotic Teleoperation

---

A minimal implementation of a bi-manual remote robotic teleoperation system using VR hand tracking and camera streaming.

✅ VR and browser visualization

✅ bi-manual hand gesture control

✅ camera streaming (mono + stereo)

✅ inverse kinematics

✅ Meta Quest Pro HMD + NVIDIA® Jetson AGX Orin™ Developer Kit

✅ .urdf robot model

✅ 3dof end effector control

✅ tested on real world robot

✅ record & playback trajectories

⬜️ debug 6dof end effector control

Setup

git clone https://github.com/kscalelabs/teleop.git && cd teleop
conda create -y -n teleop python=3.11 && conda activate teleop
make install-dependencies
pip install -r requirements.txt

Switching Embodiments

Currently, two motor types are supported - Bionic and Robstride motors. To switch between the two, modify the robot config (example in firmware repo / robot module). This is NOT the main config.yaml, which just specifies the specific meshes, links, and offsets for IK and visualization.

Scripts

demo.py - POC for integrating PyBullet with Vuer for real-time robot control

collect.py - Data collection script that interfaces with demo.py for video and position collection

Usage

Start ngrok on the robot computer.

ngrok http 8012

Open the browser app on the HMD and go to the ngrok URL (there will not be any content until you start the server on the robot computer).

Teleop

Start the server on the robot computer.

python demo.py --gui --firmware

Data Collection

Start the server on the robot computer.

python collect.py --task_name CONFIG_TASK_NAME --use_firmware True

Replaying Trajectories

Simply run the following command to replay the recorded trajectories. This currently only works with episodes split into hdf5 and mp4 files.

python replay.py --dataset_dir PATH/TO/DATASET --episode_idx EPISODE_IDX

Common Issues

Large Errors in Robot State and Actions

Check the units, offsets, and any other factors that might be causing the numbers being recorded to be different than you expect. Also, check that the motor configs are correct (signs, offsets, etc). The Robot class has some handy methods to test motors.

# Test motors
robot.test_motors()
# Calibrate robot
robot.calibrate_motors()

Robot Not Moving

Check that the robot is connected to the firmware and that the firmware is running. Check the status of the can bus and restart if needed.

sudo ip link show can0

Jerky Robot Movement

Depending on your setup (power supply, battery, etc), check that current draw is not too high. An abnormally high current draw is also indicative of large errors in motor state and actions since we are using a closed loop PD torque controller.

Useful tips

To debug Robstride motors, install the robstride cli to quickly check individual motors

pip install robstride

EXAMPLE:

robstride --interface socketcan --channel can[1/2/3/...] read [motor_id] loc_ref
robstride --interface socketcan --channel can[1/2/3/...] write [motor_id] spd_ref 2

Dependencies

• Vuer is used for visualization
• PyBullet is used for IK
• ngrok is used for networking

Citation

@misc{teleop-2024,
  title={Bi-Manual Remote Robotic Teleoperation},
  author={KScale Labs},
  year={2024},
  url={https://github.com/kscalelabs/teleop}
}