Skip to content

Commit 8a7a20c

Browse files
xueminchixueminchi
authored
Merge pull request OpenWAM-Official#24 from OpenWAM-Official/docs/robodojo-score-verification
docs: add reproducible RoboDojo score verification for issue OpenWAM-Official#23
2 parents dfaa377 + b1d1fd6 commit 8a7a20c

4 files changed

Lines changed: 592 additions & 0 deletions

File tree

‎OpenWAM/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,3 +213,6 @@ assets/*
213213
# Versioned repository images.
214214
!assets/repo_images/
215215
!assets/repo_images/**
216+
# Versioned RoboDojo score verification (see issue #23).
217+
!assets/robodojo_verification/
218+
!assets/robodojo_verification/**

‎OpenWAM/assets/robodojo_verification/README.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# RoboDojo score verification for OpenWAM-α
2+
3+
This directory answers [issue #23](https://github.com/OpenWAM-Official/OpenWAM/issues/23),
4+
which reported that averaging the 12 per-task Generalization cells published for
5+
OpenWAM-α gives 14.83 while the reported Gen-Std is 25.56.
6+
7+
Both numbers are correct. They are two different views of the same evaluation:
8+
14.83 is the merged 50-episode Generalization result, 25.56 is the standard-half
9+
result. Everything below is reproducible from the files in this directory.
10+
11+
The issue also observed that OpenWAM-α does not line up with the other models on
12+
the leaderboard. **That part is right**, and it is a real defect — but it lives in
13+
the leaderboard's per-task export, not in OpenWAM-α's scores. Section 5 covers it.
14+
15+
## Contents
16+
17+
| File | What it is |
18+
|---|---|
19+
| `openwam_robodojo_per_seed.json` | Per-seed results for all 42 tasks, plus the standard and random halves of the 12 Generalization tasks |
20+
| `verify_robodojo_scores.py` | Recomputes every published number from that file; `--leaderboard` also audits the live bundle |
21+
22+
```bash
23+
python verify_robodojo_scores.py # checks A-D, offline, no dependencies
24+
python verify_robodojo_scores.py --leaderboard # adds check E against the live leaderboard
25+
```
26+
27+
Exit code is 0 only if every check passes.
28+
29+
## 1. How RoboDojo evaluates a Generalization task
30+
31+
Each of the 12 Generalization tasks is a **paired** task: 25 episodes on the
32+
standard layout plus 25 on the `_random` layout, 50 in total. From the
33+
benchmark's own aggregation script,
34+
[`scripts/internal/summarize_result.py`](https://github.com/RoboDojo-Benchmark/RoboDojo/blob/main/scripts/internal/summarize_result.py):
35+
36+
> A task `X` that also has a sibling task `X_random` is a "paired" task: we take
37+
> the first 25 entries of `X` and the first 25 of `X_random` (matched by policy +
38+
> embodiment + seed) and merge them into 50.
39+
>
40+
> `_random` tasks are not reported on their own; they only feed their base.
41+
>
42+
> The Generalization section additionally lists each of the 12 tasks as separate
43+
> 25-episode standard and `_random` rows. **This is extra output only and does not
44+
> change the merged 50-episode scores used elsewhere.**
45+
46+
So one paired task yields three numbers, and the leaderboard publishes all three
47+
in different places:
48+
49+
| View | What it covers | OpenWAM-α SR |
50+
|---|---|---|
51+
| **Generalization** (per-task cell, dimension score) | merged 50 episodes | **14.83** |
52+
| **Gen-Std** | standard half only | **25.56** |
53+
| **Gen-Rand** | random half only | **4.11** |
54+
55+
They are tied together by construction: `(25.56 + 4.11) / 2 = 14.83`.
56+
57+
## 2. The per-task cells are merged values
58+
59+
For every one of the 12 tasks, at every seed, the published cell is exactly the
60+
mean of the two halves. Standard / random / published, per seed:
61+
62+
| Task | Standard | Random | (S+R)/2 | Published cell |
63+
|---|---|---|---|---|
64+
| stack_bowls | 88, 92, 84 | 4, 0, 4 | 46, 46, 44 | 46, 46, 44 |
65+
| push_T | 0, 0, 0 | 0, 0, 0 | 0, 0, 0 | 0, 0, 0 |
66+
| pack_objects_into_box | 4, 12, 12 | 0, 0, 0 | 2, 6, 6 | 2, 6, 6 |
67+
| fold_clothes | 92, 80, 88 | 16, 20, 36 | 54, 50, 62 | 54, 50, 62 |
68+
| hang_mugs | 16, 12, 4 | 0, 0, 0 | 8, 6, 2 | 8, 6, 2 |
69+
| sweep_blocks | 0, 8, 0 | 0, 0, 0 | 0, 4, 0 | 0, 4, 0 |
70+
| pour_liquid_into_cup | 56, 64, 44 | 16, 16, 12 | 36, 40, 28 | 36, 40, 28 |
71+
| make_toast | 0, 8, 4 | 0, 4, 0 | 0, 6, 2 | 0, 6, 2 |
72+
| arrange_largest_number | 0, 0, 4 | 0, 0, 0 | 0, 0, 2 | 0, 0, 2 |
73+
| sort_nesting_dolls_by_size | 8, 8, 12 | 0, 0, 0 | 4, 4, 6 | 4, 4, 6 |
74+
| store_laptop_and_headphones | 28, 36, 12 | 4, 4, 12 | 16, 20, 12 | 16, 20, 12 |
75+
| stack_blocks | 16, 8, 20 | 0, 0, 0 | 8, 4, 10 | 8, 4, 10 |
76+
77+
36 of 36 values reproduce. The 12 cells quoted in the issue — 45, 0, 5, 55, 5, 1,
78+
35, 3, 1, 5, 16, 7 — are the seed-averaged version of the fourth column, so their
79+
mean is the merged Generalization score, 14.83.
80+
81+
## 3. Where 25.56 comes from
82+
83+
Gen-Std is the standard half on its own. Per-seed means are 25.67 / 27.33 / 23.67,
84+
giving **25.56 ± 1.50**. The 12 standard-half task SRs are:
85+
86+
```
87+
stack_bowls 88, push_T 0, pack_objects_into_box 9, fold_clothes 87,
88+
hang_mugs 11, sweep_blocks 3, pour_liquid_into_cup 55, make_toast 4,
89+
arrange_largest_number 1, sort_nesting_dolls_by_size 9,
90+
store_laptop_and_headphones 25, stack_blocks 15
91+
```
92+
93+
Gen-Rand is the random half: per-seed 3.33 / 3.67 / 5.33 → **4.11 ± 0.87**.
94+
95+
The two exist precisely to separate in-distribution layouts from randomized ones,
96+
which is the comparison in §5.3.2 and Figure 16b.
97+
98+
## 4. How Avg is computed
99+
100+
Not a flat mean over tasks, and not a six-way mean that treats Gen-Std and
101+
Gen-Rand as separate dimensions. From `summarize_result.py::overall_seed_value`:
102+
103+
> Return (sr, score) at one seed as the mean of the per-dimension values.
104+
105+
There are five dimensions, and Generalization enters with its merged value:
106+
107+
```
108+
(14.83 + 9.25 + 25.33 + 9.11 + 1.08) / 5 = 11.92
109+
```
110+
111+
The same aggregation applied to Xiaomi-Robotics-1 gives its published 13.93:
112+
113+
```
114+
((28.00 + 6.00)/2 + 18.83 + 23.67 + 6.56 + 3.58) / 5 = 13.93
115+
```
116+
117+
A flat mean over the 42 tasks would give 12.33, not 11.92, because the dimensions
118+
hold 12 / 8 / 8 / 6 / 8 tasks — a dimension mean is not a task mean.
119+
120+
## 5. The real defect: the bundle's per-task field is not consistent across models
121+
122+
The issue is right that OpenWAM-α does not line up with the rest of the
123+
leaderboard. Running `verify_robodojo_scores.py --leaderboard` against the live
124+
bundle, for all 37 entries:
125+
126+
| What the per-task field holds | Models |
127+
|---|---|
128+
| the **standard half** | **31** |
129+
| the merged value | 0 |
130+
| neither reading matches the published Gen-Std | **1 — OpenWAM-α** |
131+
132+
So the exported per-task field carries the standard half for 31 models and the
133+
merged value for OpenWAM-α. Reading OpenWAM-α's row the way every other row must
134+
be read is what produces 14.83 against a published 25.56. **That inference was
135+
reasonable and the mismatch is real; it is an export inconsistency, not a scoring
136+
one.**
137+
138+
A cross-check that needs none of our data: Xiaomi-Robotics-1 has
139+
`stack_bowls` = 87 and `stack_bowls_random` = 16. If 87 were a merged value, the
140+
standard half would have to be `2 × 87 − 16 = 158%`. So for that model the field
141+
is unambiguously the standard half.
142+
143+
Two things follow, and both matter:
144+
145+
- **Every published aggregate we could check is correct**, for all 32 models with
146+
complete per-task data, OpenWAM-α included. The inconsistency is confined to
147+
which column the per-task API exposes.
148+
- **Figure 16b compares like with like.** Every value plotted there is a
149+
standard-half number: OpenWAM-α 25.56, Xiaomi-Robotics-1 28.00,
150+
GalaxeaVLA (G0.5) 20.00, DM0.5 18.00.
151+
152+
We have reported the export inconsistency to the RoboDojo maintainers. Until it
153+
is fixed, anyone auditing the leaderboard the same way will reach the same wrong
154+
conclusion, which is why this directory exists.
155+
156+
## 6. Why the gap looked anomalous
157+
158+
Under the reading in the issue, the apparent discrepancy for any model is forced
159+
to be exactly half its standard-to-random gap:
160+
161+
```
162+
Gen-Std − merged = Gen-Std − (Gen-Std + Gen-Rand)/2 = (Gen-Std − Gen-Rand)/2
163+
```
164+
165+
For OpenWAM-α that is `(25.56 − 4.11)/2 = 10.72`, matching the observed
166+
`25.56 − 14.83 = 10.73`. The policy with the widest in-distribution /
167+
out-of-distribution gap therefore shows the widest apparent discrepancy. Given
168+
OpenWAM-α has the largest such gap on the board (−21.44), it surfaces first —
169+
a property of the reading, not evidence about the scores.
170+
171+
## Raw data
172+
173+
`openwam_robodojo_per_seed.json` carries the full per-seed table this document is
174+
built from. The underlying `_result.json` files for every (task, seed) are
175+
available on request, and we are happy to walk through any individual row.

‎OpenWAM/assets/robodojo_verification/openwam_robodojo_per_seed.json‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
{
2+
"policy": "OpenWAM-α",
3+
"benchmark": "RoboDojo",
4+
"source_summary": "_summary_New_OpenWAM_RoboDojo_SFT_60k.md",
5+
"leaderboard_generated_at": "2026-09-05",
6+
"seeds": [0, 1, 2],
7+
"episodes": {
8+
"standalone_task": 50,
9+
"paired_half": 25,
10+
"paired_total": 50
11+
},
12+
"note": "generalization holds the two 25-episode halves of each paired task. merged_all_tasks holds the 50-episode value the leaderboard reports per task; for the 12 generalization tasks it equals (standard + random) / 2.",
13+
14+
"generalization": {
15+
"stack_bowls": {
16+
"standard": {"sr": [88, 92, 84], "score": [89.80, 93.20, 85.80]},
17+
"random": {"sr": [4, 0, 4], "score": [13.00, 6.60, 11.20]}
18+
},
19+
"push_T": {
20+
"standard": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]},
21+
"random": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]}
22+
},
23+
"pack_objects_into_box": {
24+
"standard": {"sr": [4, 12, 12], "score": [29.40, 30.60, 32.00]},
25+
"random": {"sr": [0, 0, 0], "score": [10.60, 15.80, 16.40]}
26+
},
27+
"fold_clothes": {
28+
"standard": {"sr": [92, 80, 88], "score": [92.00, 82.40, 90.40]},
29+
"random": {"sr": [16, 20, 36], "score": [19.20, 22.40, 38.40]}
30+
},
31+
"hang_mugs": {
32+
"standard": {"sr": [16, 12, 4], "score": [32.40, 31.00, 24.80]},
33+
"random": {"sr": [0, 0, 0], "score": [0.60, 4.80, 4.20]}
34+
},
35+
"sweep_blocks": {
36+
"standard": {"sr": [0, 8, 0], "score": [0.00, 8.00, 0.00]},
37+
"random": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]}
38+
},
39+
"pour_liquid_into_cup": {
40+
"standard": {"sr": [56, 64, 44], "score": [56.00, 64.00, 44.00]},
41+
"random": {"sr": [16, 16, 12], "score": [16.00, 16.00, 12.00]}
42+
},
43+
"make_toast": {
44+
"standard": {"sr": [0, 8, 4], "score": [12.00, 18.00, 18.00]},
45+
"random": {"sr": [0, 4, 0], "score": [7.00, 12.00, 8.00]}
46+
},
47+
"arrange_largest_number": {
48+
"standard": {"sr": [0, 0, 4], "score": [7.60, 4.60, 11.20]},
49+
"random": {"sr": [0, 0, 0], "score": [1.60, 2.80, 1.80]}
50+
},
51+
"sort_nesting_dolls_by_size": {
52+
"standard": {"sr": [8, 8, 12], "score": [8.00, 8.00, 12.00]},
53+
"random": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]}
54+
},
55+
"store_laptop_and_headphones": {
56+
"standard": {"sr": [28, 36, 12], "score": [44.00, 48.80, 40.00]},
57+
"random": {"sr": [4, 4, 12], "score": [9.60, 20.80, 20.00]}
58+
},
59+
"stack_blocks": {
60+
"standard": {"sr": [16, 8, 20], "score": [25.60, 20.60, 29.60]},
61+
"random": {"sr": [0, 0, 0], "score": [0.00, 3.00, 3.60]}
62+
}
63+
},
64+
65+
"merged_all_tasks": {
66+
"align_blocks": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]},
67+
"arrange_largest_number": {"sr": [0, 0, 2], "score": [4.60, 3.70, 6.50]},
68+
"build_tower": {"sr": [38, 42, 36], "score": [52.40, 54.80, 50.40]},
69+
"classify_objects": {"sr": [0, 0, 4], "score": [4.00, 5.40, 7.20]},
70+
"classify_objects_by_language": {"sr": [0, 0, 0], "score": [0.80, 2.40, 0.80]},
71+
"cover_blocks": {"sr": [16, 12, 22], "score": [22.00, 17.60, 27.10]},
72+
"deposit_coin": {"sr": [2, 2, 6], "score": [7.60, 7.20, 12.40]},
73+
"fasten_screws": {"sr": [6, 4, 2], "score": [25.00, 26.20, 22.80]},
74+
"fill_egg_holder": {"sr": [0, 0, 0], "score": [5.50, 4.70, 7.20]},
75+
"fill_pen_holder": {"sr": [2, 10, 6], "score": [12.90, 18.00, 15.20]},
76+
"fold_clothes": {"sr": [54, 50, 62], "score": [55.60, 52.40, 64.40]},
77+
"general_pickup": {"sr": [6, 8, 12], "score": [6.00, 8.00, 12.00]},
78+
"hang_mugs": {"sr": [8, 6, 2], "score": [16.50, 17.90, 14.50]},
79+
"imitate_sorting_sequence": {"sr": [2, 0, 0], "score": [5.10, 1.70, 1.90]},
80+
"insert_key": {"sr": [0, 0, 0], "score": [13.20, 14.10, 13.80]},
81+
"insert_tubes": {"sr": [8, 4, 8], "score": [24.80, 25.20, 28.80]},
82+
"make_kong": {"sr": [30, 36, 30], "score": [30.00, 36.00, 30.00]},
83+
"make_toast": {"sr": [0, 6, 2], "score": [9.50, 15.00, 13.00]},
84+
"match_and_pick_from_conveyor": {"sr": [42, 38, 30], "score": [42.00, 38.00, 30.00]},
85+
"organize_table": {"sr": [16, 8, 16], "score": [64.50, 57.50, 65.50]},
86+
"pack_objects_into_box": {"sr": [2, 6, 6], "score": [20.00, 23.20, 24.20]},
87+
"pick_from_conveyor_by_image": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]},
88+
"play_Xylophone": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]},
89+
"play_stacking_toy": {"sr": [0, 0, 0], "score": [0.20, 0.20, 0.00]},
90+
"play_tic_tac_toe": {"sr": [54, 52, 70], "score": [60.50, 57.70, 74.00]},
91+
"plug_in_charger": {"sr": [0, 4, 8], "score": [0.00, 4.00, 8.00]},
92+
"pour_balls_into_vase": {"sr": [16, 16, 20], "score": [16.00, 16.00, 20.00]},
93+
"pour_by_language": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]},
94+
"pour_liquid_into_cup": {"sr": [36, 40, 28], "score": [36.00, 40.00, 28.00]},
95+
"press_by_number": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]},
96+
"push_T": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]},
97+
"put_bottles_into_dustbin": {"sr": [98, 88, 88], "score": [98.00, 92.20, 91.90]},
98+
"solve_equation": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]},
99+
"sort_nesting_dolls_by_size": {"sr": [4, 4, 6], "score": [4.00, 4.00, 6.00]},
100+
"stack_blocks": {"sr": [8, 4, 10], "score": [12.80, 11.80, 16.60]},
101+
"stack_blocks_by_language": {"sr": [0, 0, 0], "score": [1.20, 0.80, 0.80]},
102+
"stack_bowls": {"sr": [46, 46, 44], "score": [51.40, 49.90, 48.50]},
103+
"store_laptop_and_headphones": {"sr": [16, 20, 12], "score": [26.80, 34.80, 30.00]},
104+
"store_tools_in_toolbox": {"sr": [0, 0, 0], "score": [1.00, 0.00, 0.00]},
105+
"swap_T": {"sr": [0, 0, 0], "score": [0.00, 0.00, 0.00]},
106+
"swap_blocks": {"sr": [0, 2, 0], "score": [0.00, 2.00, 0.00]},
107+
"sweep_blocks": {"sr": [0, 4, 0], "score": [0.00, 4.00, 0.00]}
108+
},
109+
110+
"published_aggregates": {
111+
"generalization_merged": {"sr": 14.83, "score": 20.71},
112+
"generalization_standard": {"sr": 25.56, "score": 33.16},
113+
"generalization_random": {"sr": 4.11, "score": 8.26},
114+
"precision": {"sr": 9.25, "score": 18.45},
115+
"long_horizon": {"sr": 25.33, "score": 34.93},
116+
"memory": {"sr": 9.11, "score": 10.41},
117+
"open": {"sr": 1.08, "score": 1.41},
118+
"average": {"sr": 11.92, "score": 17.18}
119+
}
120+
}

0 commit comments

Comments
 (0)