accept signal only at tail of output, to handle self-corrected emissions

[iatsiuk]

problem

during the first claude review phase, claude can emit a signal marker, immediately realize the mistake in plain text, and keep working correctly - but ralphex has already captured the premature signal and terminates the run as failed.

real log excerpt (opus 4.7, first claude review):

[16:21:39] 0 errors remaining (warnings are pre-existing). running tests once more to confirm:
[16:21:44] 5/5 pass. committing the fixes:
[16:21:58] <<<RALPHEX:TASK_FAILED>>>
[16:21:58] wait - issues were found AND fixed, so per path b i should NOT output any signal. let me correct that.
[16:21:58] issues were found and fixed. stopping here per path b - the external loop will run another review iteration to verify the fixes.

failed: 2026-04-20 16:21:59 (9m23s) - runner: first review: review failed (FAILED signal received)

what actually happened: claude found 3 issues, fixed them, ran tests (5/5 pass), committed, then mistakenly emitted TASK_FAILED, then self-corrected in the next blocks quoting "path b" from the prompt verbatim. per path b in review_first.txt:83-87, no signal should have been emitted at all - another review iteration should run to verify the fixes. ralphex aborted the whole run instead.

prior art

the existing approach on similar bugs is known:

• Claude evaluation emits CODEX_REVIEW_DONE despite codex reporting findings #92 "claude evaluation emits CODEX_REVIEW_DONE despite codex reporting findings" - fixed with prompt only, code-level guard explicitly rejected as "ralphex second-guessing the llm"
• Review phase emits REVIEW_DONE signal multiple times #93 "review phase emits REVIEW_DONE signal multiple times" - fixed with prompt only
• claude review loop doesn't exit when no issues found #69 "claude review loop doesn't exit when no issues found" - mixed fix, added HEAD hash fallback for a different scenario

in this case the prompt is already explicit (review_first.txt:83-87 describes path b in plain text, claude even quotes it back verbatim), yet opus 4.7 still emits the signal first and self-corrects after. further prompt tuning may reduce the frequency but cannot rule out mid-output mistakes - the model has no way to "unsay" a marker it already emitted.

proposed approach (non-intrusive code fix)

instead of the current stream-time behavior in pkg/executor/executor.go:401-403:

if sig := detectSignal(text); sig != "" {
    signal = sig
}

which captures the first signal marker and never releases it, scan the accumulated output from the tail after claude exits:

1. find the last occurrence of any <<<RALPHEX:...>>> marker in the full output.
2. inspect the text that follows the marker until EOF.
3. if the trailing text is empty, whitespace, or very short (e.g. < 50 chars), accept the signal.
4. if substantial prose follows the marker, treat it as a self-correction and ignore the signal (same effect as no signal - the outer loop decides next step, typically another iteration).

this does not parse semantics, does not second-guess the llm's decision, and does not introduce keyword heuristics. it only answers one mechanical question: "did the model stop talking right after emitting the marker, or did it keep going?" a signal that is not the end of the output was, by observation, never intended as the final verdict.

why this is different from #92

#92's rejected guard re-evaluated the codex output in go (re-parsing findings) to override claude's decision. the proposed tail check does the opposite: it trusts whatever claude said last, and treats mid-output markers as incomplete drafts. there is no re-evaluation of the review itself.

edge cases

• valid pattern "signal is last line": still accepted.
• valid pattern "short punctuation/whitespace after signal": accepted via small threshold.
• pathological case where claude legitimately writes a long postamble after a valid signal: the outer loop runs one extra iteration, which is the same cost as the current false-positive on the other side.
• could be combined with a prompt tweak requiring the signal to be the final non-empty line, which makes the tail check the enforcement mechanism rather than a heuristic.

[umputun]

I don't like this approach. It addresses a model-level problem at the wrong level and effectively changes the signal contract from "marker appears in output" to "marker must be at the tail", which is a real behavior change with real risks. I've seen plenty of runs where claude emits a proper, intended signal and then follows it with a short explanation or summary — with the tail check that whole class of runs starts getting rejected as "self-corrected". And on top of that, custom review scripts and codex wrappers that do signal-then-summary would break silently.

Also, any threshold (50 chars, "substantial prose", etc.) is a heuristic we'd have to live with. #92 and #93 were intentionally fixed at the prompt level to avoid exactly this kind of second-guessing, and I'd rather stay consistent with that.

I'd prefer to push back on the model first. The Path B instruction is already explicit, but opus 4.7 still misfired — so the prompt clearly isn't strong enough for it yet. Worth experimenting with:

• making the final-line rule explicit: "if you emit a signal marker it MUST be the final non-empty line of your output, nothing after it"
• if prompt tightening isn't enough, we could introduce a new explicit cancel signal (doesn't exist today — would be something like <<<RALPHEX:CANCEL>>>). Narrow, explicit, no heuristics. But only worth doing if plain prompt wording doesn't hold
• tightening the Path B wording so the model doesn't reach for a signal reflexively after fixing

If prompt-only tweaks don't move the needle on opus 4.7 across a few runs, we can revisit. But I don't want to ship a tail-position check as the primary defense.

[umputun]

transfering to discussions

[iatsiuk]

fair points, and i get the reasoning. keeping the signal contract clean and pushing fixes to the prompt layer is consistent with #92/#93.

my concern is that prompt-level fixes assume stable instruction-following from the model. anthropic's models have been drifting week-to-week on that. what holds on opus 4.7 today may not hold after the next silent update. the current path b wording is already pretty explicit. the model even quotes it back verbatim in the same response where it misfires. that suggests the failure mode isn't "prompt too vague" but "model emitted before finishing its own reasoning", which is harder to close with wording alone.

on frequency: this only happened a few times and only on opus 4.7, which may itself be a bad sign. a newer model regressing on something sonnet opus 4.6 handled fine points to an unstable baseline rather than an isolated prompt issue.