Skip to content

Empty pruned_init files for three *_task tasks (libero_10_task t2, libero_spatial_task t3/t7) #246

Description

@xianhanglin

Summary

Three task-perturbation tasks ship pruned_init files with zero initial states. Any loader that picks an init state by index fails on them. As a result, these tasks can't be reset or evaluated. Results on these suites report a denominator of 90 or 80 instead of 100 per suite, which makes them hard to compare.

Suite Task id pruned_init file Size States
libero_10_task 2 KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it.pruned_init 364 B 0
libero_spatial_task 3 pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate.pruned_init 364 B 0
libero_spatial_task 7 pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate.pruned_init 364 B 0

Reproduction

With the package installed from any of the branches:

from liberopro.liberopro import benchmark

for suite_name, task_id in [("libero_10_task", 2), ("libero_spatial_task", 3), ("libero_spatial_task", 7)]:
    suite = benchmark.get_benchmark_dict()[suite_name]()
    print(suite_name, task_id, len(suite.get_task_init_states(task_id)))
# libero_10_task 2 0
# libero_spatial_task 3 0
# libero_spatial_task 7 0

The BDDL files for these tasks look inconsistent too

These may be why every init state got pruned:

  • libero_10_task/KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it.bddl:
    the language is "turn on the stove and put the pan on it" and the goal is
    (Turnon flat_stove_1) (On chefmate_8_frypan_1 flat_stove_1_cook_region), but
    the file name refers to the moka pot.
  • libero_spatial_task/pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate.bddl
    and .../pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate.bddl:
    both have the language "Pick the akita black bowl on the top of the cabinet
    and place it on the plate" and the goal (On akita_black_bowl_2 plate_1).
    They differ only in where akita_black_bowl_1 starts (cookies_1 vs.
    flat_stove_1_cook_region), so the two tasks are effectively duplicates.

Questions

  1. Can you publish regenerated init states (and
    corrected BDDL if needed)?

Activity

  1. akushonkamen commented on Oct 11, 2026

    @akushonkamen

    Confirmed the consumer-side root cause: robots/libero/env_server.py builds the init-state offset and trial count directly from len(suite.get_task_init_states(task_id)) with no guard for an empty set, so a task whose pruned_init file has 0 states silently contributes 0 trials and the suite denominator shrinks (100 → 90/80) without any warning.

    I'm preparing a fix that fails loudly (listing the affected task ids/suite) when an init-state set is empty, plus a doc note listing the currently known empty-state tasks. The underlying data regeneration (BDDL consistency + new pruned_init files) lives in the LIBERO-PRO data pipeline and is out of scope here.

    Note: this contribution was prepared with AI coding assistance, reviewed and will be tested by me.

  2. akushonkamen commented on Oct 11, 2026

    @akushonkamen

    PR opened: #267 — guards make_env() to raise a descriptive RuntimeError (suite, task id, full empty-task inventory) for zero-state pruned_init tasks instead of a bare ZeroDivisionError; docs and offline fake-suite tests included. 14 pre-existing main failures unchanged, zero new failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions