feat(data): add degenerate sample detection and governance (#217)

[HDM-xymm]

What does this PR do?

Add degenerate sample detection utilities in areno.api.data that identify empty, whitespace-only, special-token-only, no-trainable-token, and identical DPO preference branch samples. The detection is integrated into the SFT, DPO, and policy rollout paths with configurable SKIP/ERROR policy.

Key changes:

• Add DegenerateReason, DegeneratePolicy, DegenerateFilterConfig, SampleQualityReport dataclasses and 6 check_* detection functions
• Integrate detection into Trainer.load_prompt_batches, SFT, DPO, and PolicyOnly paths
• Add --degenerate-policy CLI flag and trainer_config field
• Add PromptBatch counters for skipped_degenerate and degenerate_reasons
• Add 23 CPU-only unit tests (no GPU/tokenizer dependency)
• Add user docs and design doc

Usage

Control degenerate sample handling via CLI:

#Default: silently skip degenerate samples with reason counters
areno train --ckpt Qwen/Qwen3.5-0.8B --degenerate-policy skip ...

#Fail fast: raise ValueError on the first degenerate sample
areno train --ckpt Qwen/Qwen3.5-0.8B --degenerate-policy error ...

Detection covers 5 categories across pre- and post-tokenization stages:

Reason	Stage	Example
empty	pre-tokenization	"" prompt or response
whitespace_only	pre-tokenization	" \n\t "
special_tokens_only	post-tokenization	all tokens are <|...|>
no_trainable_tokens	post-tokenization	every position is in the prompt prefix
identical_preference_branches	pre-tokenization	DPO chosen == rejected

Example output

With --degenerate-policy skip (default), degenerate rows are filtered and counted:

batch = trainer.load_prompt_batches(dataset, batch_size=4, max_prompt_tokens=512)
#batch.skipped_degenerate == 2
#batch.degenerate_reasons == {"empty": 1, "whitespace_only": 1}

With --degenerate-policy error, the first degenerate sample raises:

ValueError: degenerate sample detected (pre_tokenization): prompt is an empty string

Related issue

Fixes #217

Type of change

• ✨ New feature

How was it tested?

pytest tests/test_degenerate_sample_cpu.py -v

23 CPU-only unit tests covering:

check_prompt_text — empty and whitespace-only prompt detection
check_response_text — empty and whitespace-only response detection
check_tokenized_prompt — zero-length and all-special-token detection
check_trainable_tokens — all-prompt-mask (no trainable token) detection
check_preference_pair — identical DPO chosen/rejected branch detection
apply_degenerate_policy — SKIP returns True, ERROR raises ValueError, disabled config passes through
record_degenerate_reason / format_degenerate_reasons — reason counting and formatting
No GPU or tokenizer dependency. Existing tests pass with default skip policy (backward-compatible).

Checklist

• The PR title summarizes the contribution.
• Linked the related issue in the description (if any).
• Existing tests pass (pytest tests/ -k cpu).
• New behavior is covered by tests.
• Described the test commands run and any hardware limitations.
• Public API / CLI changes are additive and backward-compatible (see CONTRIBUTING.md).

##Update
-Update (2026-08-24): Rebased onto latest main to resolve merge conflicts with the MLX backend and multimodal optimizer features. Fixed lint errors (unused variable, import ordering, formatting) and
updated test mocks to include the new degenerate_policy field and degenerate_filter_config() method.