Skip to content

fix(showcase): seat G1 on terrain surface so the Curriculum-Showcase stops NaN-ing - #155

Merged
KraHsu merged 2 commits into
devfrom
fix/curriculum-showcase-nan
May 24, 2026
Merged

KraHsu merged 2 commits into
devfrom
fix/curriculum-showcase-nan

Conversation

@KraHsu

@KraHsu KraHsu commented May 24, 2026

Copy link
Copy Markdown
Owner

Problem

genelab play GeneLab-Curriculum-Showcase-v0 --agent zero --steps 5 crashed with:

genesis.GenesisException: Invalid constraint forces causing 'nan'. Please decrease Rigid simulation timestep.

Reproduced on RTX 5060 Ti (local) and H200 (server) — not hardware-specific.

Root cause (dt was a red herring)

A no-step diagnostic showed the G1 spawned ~0.77 m buried in the RandomRough heightfield. terrain_levels_vel (which runs on every reset) and the runner's teleport both wrote the robot base to the terrain cell origin at z=0 (= terrain.pos[2]), ignoring the surface elevation and the robot's 0.76 m standing height. Feet landed at z=−0.72 under a +0.05 m surface. That interpenetration exploded the rigid contact solver:

  • dt=0.005 → instant NaN at step 1 (the reported symptom).
  • dt=0.002 → an unconverging multi-minute solver grind (_kernel_solve_graph).

The Terrain-Showcase shares the identical buried-spawn bug (out of scope here; see follow-ups).

Fix (one concept: seat the G1 on the terrain, not in it)

  • terrains/importer.py: add surface_height_at(xy) — world-frame heightfield surface lookup.
  • mdp/curriculums.py: terrain_levels_vel reseats the base at surface + init_pos.z + spawn_clearance instead of the bare cell origin (new optional spawn_clearance=0.1).
  • showcase runner.py: teleport reseats at standing height too, so the full 400-step demo survives its teleports.
  • showcase env_cfg.py: dt 0.005→0.002, decimation 4→10. The G1 is independently NaN-unstable at dt=0.005 even from a clean stance; this matches the known-good Velocity/Tracking-Flat configs. Control rate stays 50 Hz; the 2 s episode (100 control steps) is unchanged.
  • rl/runner.py: play_task caps scripted (zero/random) playback at simulation.steps, so play TASK --agent zero --steps N stops after N steps instead of looping forever headless (--steps was silently ignored on the RL play path; max_steps defaulted to None). Trained playback stays unbounded.

Verification

Check Result
play … --agent zero --steps 5 (exact repro) exit 0, no NaN (~12 s)
play … --steps 5 (showcase task.play() path) exit 0, no NaN
Full play … --steps 400 (all teleports) exit 0, no NaN; curriculum promotion fires (levels 0→4, histogram=[8,0,0,0,8])
No-step spawn diagnostic feet 0.14 m above surface (was 0.77 m buried)
uv run pytest -q 507 passed

Reference audit (CLAUDE.md)

grep across src/ examples/ tests/:

  • terrain_levels_vel — only non-test caller is the curriculum showcase cfg; signature change is additive (new optional spawn_clearance). No breakage. Behavior tests in tests/test_curriculum_terrain.py updated to the corrected (seated) spawn.
  • surface_height_at — new method; callers are terrain_levels_vel and the showcase runner (both in this PR).
  • reset_root_state_on_terrain — a reset-event approach explored mid-development was fully removed (the curriculum overwrites root pose after events, so the fix had to live in terrain_levels_vel); grep confirms zero references remain.

No plans/ files touched.

KraHsu added 2 commits May 24, 2026 14:01
…ps NaN-ing

The Curriculum-Showcase G1 spawned ~0.77 m buried in the RandomRough
heightfield: terrain_levels_vel (run on every reset) and the runner teleport
wrote the base to the cell origin at z=0 (= terrain.pos[2]), ignoring the
surface elevation and the robot's 0.76 m standing height. Feet ended at
z=-0.72 under a +0.05 m surface; that interpenetration blew up the rigid
contact solver — instant NaN at dt=0.005, an unconverging multi-minute grind
at dt=0.002. dt was a red herring.

- terrains/importer: add surface_height_at(xy) heightfield lookup.
- mdp/curriculums: terrain_levels_vel reseats the base at
  surface + init_pos.z + spawn_clearance instead of the bare cell origin.
- showcase runner: teleport reseats at standing height too, so the full
  400-step demo survives its teleports.
- showcase cfg: dt 0.005->0.002, decimation 4->10. The G1 is independently
  NaN-unstable at dt=0.005 even from a clean stance; this matches the
  known-good Velocity/Tracking-Flat configs. Control rate stays 50 Hz and the
  2 s episode (100 control steps) is unchanged.
- rl/runner: play_task caps scripted (zero/random) playback at
  simulation.steps, so `play TASK --agent zero --steps N` stops after N steps
  instead of looping forever headless (--steps was silently ignored on the RL
  play path). Trained playback stays unbounded.

Verify: `play GeneLab-Curriculum-Showcase-v0 --agent zero --steps 5` exits 0,
no NaN; full 400-step showcase exits 0 with curriculum promotion (levels
0->4). pytest: 507 passed.
@KraHsu
KraHsu merged commit a911bed into dev May 24, 2026
3 checks passed
@KraHsu
KraHsu deleted the fix/curriculum-showcase-nan branch May 24, 2026 06:06
KraHsu added a commit that referenced this pull request Jun 5, 2026
fix(showcase): seat G1 on terrain surface so the Curriculum-Showcase stops NaN-ing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant