Handle outputs segmented into multiple units (with ProbScalarizedModel only)#11

Merged

dennislwei merged 22 commits intomainfrom

Jan 6, 2026

Commits on Aug 26, 2025

clarify that output_orig can be a single unit or multiple units (but not a batch)
dennislwei
committed
handle segmented reference outputs (multiple output units) in ProbScalarizedModel
dennislwei
committed

Commits on Aug 29, 2025

update MExGen notebooks to save output token IDs and pass them to
dennislwei
committed

Commits on Sep 2, 2025

handle multiple output units/score columns in PerturbCurveEvaluator with ProbScalarizedModel only
dennislwei
committed

Commits on Sep 12, 2025

segment output text if desired
dennislwei
committed

Commits on Sep 13, 2025

merge non-alphanumeric output units into adjacent units
dennislwei
committed

Commits on Sep 16, 2025

Commits on Oct 2, 2025

find_unit_boundaries(): keep idx_token in range
dennislwei
committed

Commits on Oct 4, 2025

squeeze only axis=1 of importance score arrays
dennislwei
committed

Commits on Nov 2, 2025

_compute_log_probs_vllm: break large batches into multiple calls
dennislwei
committed

Commits on Nov 14, 2025

MExGenExplainer.segment_output: fix indenting
dennislwei
committed

Commits on Jan 5, 2026

a few comment clarifications and formatting fixes
dennislwei
committed