This repository was archived by the owner on Jan 6, 2026. It is now read-only.
This repository was archived by the owner on Jan 6, 2026. It is now read-only.
Integration with the Unity game engine #13
Description
I'm going to reproduce the hands movement in Unity. Started with Twinkle Twinkle Little Star example.
At first, I saved the scene with all the resources:
export_with_assets(
task.root_entity.mjcf_model,
out_dir="/tmp/robopianist/piano_with_shadow_hands",
out_file_name="scene.xml",
)
At second, I installed Unity plug-in from MuJoCo website.
At third, I changed scene assets from obj format to stl format like user pkrack wrote here
Finally, Unity has able to load this scene. But when I louched it I recived errors:
Element 'joint', line 1
NullReferenceException: Failed to create Mujoco runtime.
Mujoco.MjScene.StepScene () (at MuJoCo/Runtime/Components/MjScene.cs:345)
Mujoco.MjScene.FixedUpdate () (at MuJoCo/Runtime/Components/MjScene.cs:98)
The first occurrences of "range" in the xml debug file contain a comma in the values, not a period. Maybe that's the problem. But I do not know how to change the generation of this file)
<mujoco>
<worldbody>
<body pos="0 0 0" quat="-1 0 0 0" gravcomp="0" name="lh_shadow_hand/_0">
<body pos="0.4 -0.15 0.13" quat="-0.5 -0.5 0.5 0.5" gravcomp="1" name="lh_shadow_hand/lh_forearm_1">
<inertial pos="0 0 0.09" quat="-1 0 0 0" mass="3" diaginertia="0.0138 0.0138 0.00744" />
<joint type="slide" pos="0 0 0" axis="-1 0 0" ref="0" armature="0.0002" springref="0" springdamper="0 0" damping="67.4762" stiffness="0" solreflimit="0.02 1" solimplimit="0.9 0.95 0.001 0.5 2" solreffriction="0.02 1" solimpfriction="0.9 0.95 0.001 0.5 2" frictionloss="0.01" limited="false" margin="0" range="-0.4605 0.7605" name="lh_shadow_hand/forearm_tx_3" />
<joint type="slide" pos="0 0 0" axis="-1.192093E-07 0 1" ref="0" armature="0.0002" springref="0" springdamper="0 0" damping="67.4762" stiffness="0" solreflimit="0.02 1" solimplimit="0.9 0.95 0.001 0.5 2" solreffriction="0.02 1" solimpfriction="0.9 0.95 0.001 0.5 2" frictionloss="0.01" limited="false" margin="0" range="0 0.06" name="lh_shadow_hand/forearm_ty_4" />
<geom density="1000" type="mesh" mesh="mesh_6" pos="0 0 0" quat="-1 0 0 0" priority="0" contype="0" conaffinity="0" group="2" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_0_5" />
<geom density="1000" type="mesh" mesh="mesh_8" pos="0 0 0" quat="-1 0 0 0" priority="0" contype="0" conaffinity="0" group="2" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_1_7" />
<geom density="1000" type="mesh" mesh="mesh_10" pos="0 0 0" quat="-1 0 0 0" priority="0" contype="1" conaffinity="1" group="3" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_2_9" />
<geom density="1000" type="box" size="0.035 0.035 0.035" pos="0 -0.01000001 0.181" quat="0.9249092 0 0.3801881 0" priority="0" contype="1" conaffinity="1" group="3" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_3_11" />
<site type="box" size="0.001 0.001 0.001" pos="0 0 0" quat="-1 0 0 0" name="lh_shadow_hand/forearm_tx_site_12" />
<site type="box" size="0.001 0.001 0.001" pos="0 0 0" quat="-1 0 0 0" name="lh_shadow_hand/forearm_ty_site_13" />
<body pos="0 -0.01000001 0.21301" quat="-1 0 0 0" gravcomp="1" name="lh_shadow_hand/lh_wrist_14">
<inertial pos="0 1.490116E-08 0.029" quat="0.5 0.5 0.5 0.5" mass="0.1" diaginertia="6.4E-05 4.38E-05 3.5E-05" />
<joint type="hinge" pos="0 0 0" axis="-1.192093E-07 -1 0" ref="0" armature="0.0002" springref="0" springdamper="0 0" damping="0.5" stiffness="0" solreflimit="0.02 1" solimplimit="0.9 0.95 0.001 0.5 2" solreffriction="0.02 1" solimpfriction="0.9 0.95 0.001 0.5 2" frictionloss="0.01" limited="false" margin="0" range="-30,00002 10" name="lh_shadow_hand/lh_WRJ2_16" />
...
I attached this file to post, you may look it.
As a result, maybe it would be better to somehow copy the frame-by-frame hand positions to make an animated clip? Where can I get these values from an xml file generated by a neural network?
Importing the environment as it is in Unity seems to be a more confusing task.
debug.txt