reward: BFCL multi-call grading misaligns candidate/gold calls, scoring correct answers 0

[philluiz2323]

Bug

_check_bfcl in src/trinity/orchestration/reward.py grades BFCL function-call answers. When there is more than one call, it aligns candidate and gold calls by sorting both on a shared json.dumps(..., sort_keys=True) key and then comparing positionally with zip:

key = lambda call: json.dumps({"name": call[0], "arguments": call[1]}, sort_keys=True, ...)
cand_calls_sorted = sorted(cand_calls, key=key)
gold_calls_sorted = sorted(gold_calls, key=key)
return all(_bfcl_call_matches(c, g) for c, g in zip(cand_calls_sorted, gold_calls_sorted))

But candidate args are concrete values ({"x": 1}) while gold args are allowed-value lists ({"x": [1, 2]}). Under sort_keys=True the JSON is keyed on arguments first, so the scalar-vs-list serialization sorts the two sides differently — the positional zip then pairs up mismatched calls and a correct multi-call answer scores 0.

Reproduce (offline)

import json
from trinity.orchestration import reward as R
gold = [{"f": {"x": [1, 2]}}, {"f": {"x": [1]}}]           # call A accepts 1 or 2; call B accepts 1
cand = [{"name": "f", "arguments": {"x": 1}}, {"name": "f", "arguments": {"x": 2}}]
R.score_text("bfcl_simple", json.dumps(cand), {"ground_truth": gold})
# -> 0.0   (a valid pairing exists: x=1->[1], x=2->[1,2], so it should be 1.0)

Single-call scoring is unaffected (sorting a 1-element list is a no-op).

Expected

A set of calls scores 1.0 iff each gold call can be paired with a distinct candidate call under _bfcl_call_matches — i.e. a bipartite perfect matching, independent of order.

Fix

Replace the sort-and-zip with a backtracking bipartite match (_bfcl_match_all_calls).

Scope note

The shipped bfcl_simple split is single-call, so this hardens the multi-call path rather than fixing a currently-observed eval regression — but the scorer already implements multi-call handling, and it is incorrect.

Location

src/trinity/orchestration/reward.py, _check_bfcl (~lines 681–694). Not related to the BFCL loading issues (#152 / #179).