Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network

Venue: ICRA 2019

Author's mail : [email protected], [email protected]

This package produces visually similar robot hand poses based on depth images of the human hand in an end-to-end fashion, which is a collaborative work done by TAMS and Fucun Sun's lab of Tsinghua University.

The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and from simulated depth images of a robotic hand.

Please cite this paper (Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network), if you use our released code.

Video

Installation Instructions

OS

• [ROS Kinetic and Ubuntu 16.04]]
• [CUDA 9]

ROS Dependency

• bio_ik
• Moveit
• common_resources
• sr-config
• sr_interface
• sr-ros-interface-ethercat
• ros_ethercat
• sr_common
• sr_core
• sr-ros-interface
• sr_tools
• ros_control_robot

Python Dependency

• python2.7
• PyTorch
• mayavi
• numpy
• tensorboard
• matplotlib
• pickle
• pandas
• seaborn

Camera Drive

• librealsense

Setup

• Install necessary packages (kinetic branch) for Shadow Hand.
• Install Bio IK packages. Please follow the Basic Usage in README.md in bio_ik repository and set correct kinematics solver.
• Install RealSense Camera package:

  sudo apt install ros-kinetic-realsense-camera
  

• To simplify it, you can put above packages in one ros workspace.
• Download our package in same workspace, then build this package with catkin_make.

Dataset Generation

• Download (BigHand2.2M dataset). Put the lable file Training_Annotation.txt into ros/src/shadow_teleop/data/Human_label/. Building nine folders respectively called depth_shadow[1-8] saves robot depth images from nine viewpoint inros/src/shadow_teleop/data/.
• Generate robot mapping file by human hand keypoints from BigHand2.2M dataset. The generated file save in ros/src/shadow_teleop/data/human_robot_mapdata.csv.

  python ros/src/shadow_teleop/scripts/human_robot_mappingfile.py
  

• Run shadow hand in gazebo and use the our simulation world (./ros/src/teleop_motorhand/worlds/shadowhand_multiview.world).

  roslaunch teleop_motorhand shadow.launch
  

• Generate dataset by running the code:

  roslaunch shadow_teleop multi_shadow_sim_bio.launch
  

  Please note the location of saved depth images and the location of robot_joints_file.csv.
• Save robot_joints_file.csv as joint_all.npy by pandas.readcsv() or numpy.loadtxt().
• Crop human or shadow image into 100*100 (normalized to [0,255)):

    python utils/seg_depth.py
  

  Please change the location of original depth images and cropped depth images fl to your own dataset location.
• Last but not least, spilt joint_all.npy into training dataset and test dataset by yourself, and save as joint_train.npy and joint_test.npy at your own dataset location.

Model Training

• If you want to train the network yourself instead of using a pretrained model, follow below steps.

• Launch a tensorboard for monitoring:

  tensorboard --log-dir ./assets/log --port 8080

  and run an experiment for 200 epoch:

  python main.py --epoch 200 --mode 'train' --batch-size 256 --lr 0.01 --gpu 1 --tag 'teachnet' --data-path 'LOCATION OF YOUR TRAINING DATASET'
  

  File name and corresponding experiment:

  main.py                    --- Teach Hard-Early approach
  main_baseline_human.py     --- Single human
  main_baseline_shadow.py    --- Single shadow
  main_gan.py                --- Teach Soft-Early approach
  

Pretrained Models:

• Download pretrained models for real-time test.

RealsenseF200 Realtime Demo

• Launch camera RealsenseF200 (If you use the other camera which is suitable for close-range tracking, please use corresponding launch file). Or you can download the recorded example rosbag, and play the bag file:

  rosbag play [-l] example.bag
  

• Run Shadow hand in simulation or real world.

• Run the collision check service:

  rosrun shadow_teleop interpolate_traj_service
  

• Run the demo code.

  • Change the correct topic name based on your camera.
  • Limit your right hand to the viewpoint range of [30°, 120°] and the distance range of [15mm, 40mm] from the camera.

  python shadow_demo.py --model-path pretrained-model-location [--gpu 1]
  

Citation

If you use this work（collobrated with ）, please cite:

@inproceedings{li2018vision,
  title={Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network},
  author={Li, Shuang and Ma, Xiaojian and Liang, Hongzhuo and G{\"o}rner, Michael and Ruppel, Philipp and Fang, Bing and Sun, Fuchun and Zhang, Jianwei},
  booktitle={IEEE International Conference on Robotics and Automation (ICRA)},
  year={2019}
}