Skip to content

fix(analyzer): detect progress from committed work and embedded RALPH_STATUS#181

Open
enlego wants to merge 1 commit intofrankbria:mainfrom
enlego:fix/progress-detection-committed-work
Open

fix(analyzer): detect progress from committed work and embedded RALPH_STATUS#181
enlego wants to merge 1 commit intofrankbria:mainfrom
enlego:fix/progress-detection-committed-work

Conversation

@enlego
Copy link

@enlego enlego commented Feb 17, 2026

Summary

  • Fix false-negative progress detection when Claude commits work atomically after each task
  • git diff only sees uncommitted changes; projects using atomic commits trigger the circuit breaker with no real issue
  • Also fixes inconsistency where EXIT_SIGNAL was parsed from the embedded RALPH_STATUS block but TASKS_COMPLETED_THIS_LOOP and FILES_MODIFIED were not

Root cause

In analyze_response (JSON path, lib/response_analyzer.sh), progress is detected only via git diff --name-only. When Claude commits after each loop (a common and encouraged pattern), git diff returns 0 → has_progress = false → circuit breaker opens after CB_NO_PROGRESS_THRESHOLD consecutive loops.

Additionally, the JSON path already extracts EXIT_SIGNAL from the embedded ---RALPH_STATUS--- block in the result field, but ignores TASKS_COMPLETED_THIS_LOOP and FILES_MODIFIED from the same block.

Changes

lib/response_analyzer.shanalyze_response, JSON path (~line 343):

  1. Recent-commit detection: After git diff, check git log --oneline -1 --since="20 minutes ago". If a recent commit exists, set has_progress=true and populate files_modified from git diff-tree or git diff HEAD~1.

  2. Embedded RALPH_STATUS parsing: Extract TASKS_COMPLETED_THIS_LOOP and FILES_MODIFIED from the result field's ---RALPH_STATUS--- block, consistent with the existing EXIT_SIGNAL extraction. Also normalizes literal \n escape sequences to real newlines before parsing so grep/cut work correctly in both production and test environments.

tests/unit/test_json_parsing.bats — 4 new tests (section PROGRESS DETECTION WITH COMMITTED WORK TESTS):

  1. Detects progress from TASKS_COMPLETED_THIS_LOOP in embedded RALPH_STATUS
  2. Detects progress via recent git commit when git diff is zero
  3. Extracts FILES_MODIFIED from RALPH_STATUS when git diff is zero
  4. Does not flag progress when TASKS_COMPLETED_THIS_LOOP is zero

Test plan

  • npm test passes 100% (456/456 tests)
  • New tests cover: RALPH_STATUS-based progress, recent-commit detection, FILES_MODIFIED extraction, zero-task no-progress case
  • No regression in existing json_parsing, exit_detection, or circuit_breaker tests

🤖 Generated with Claude Code

…_STATUS

When Claude commits work atomically after each task, git diff returns zero
uncommitted changes. Ralph incorrectly treats this as no progress and opens
the circuit breaker after CB_NO_PROGRESS_THRESHOLD loops.

Two complementary fixes in the JSON analysis path:
- Check git log --since for recent commits, not just git diff
- Parse TASKS_COMPLETED_THIS_LOOP and FILES_MODIFIED from the embedded
  RALPH_STATUS block in the result field (consistent with existing
  EXIT_SIGNAL extraction added in Bug frankbria#1 Fix)

Also fixes literal \\n handling: normalize escape sequences from the jq
result field before grep/cut so parsing works in both production (where
jq decodes JSON escapes to real newlines) and test environments.

All 456 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@macroscopeapp
Copy link
Contributor

macroscopeapp bot commented Feb 17, 2026

Detect progress in lib/response_analyzer.sh::analyze_response from recent commits within 20 minutes and from embedded RALPH_STATUS fields

Add progress detection that checks git log --oneline -1 --since="20 minutes ago" and parses TASKS_COMPLETED_THIS_LOOP and FILES_MODIFIED from an embedded RALPH_STATUS in JSON .result; populate files_modified from git diff HEAD~1 --name-only or the embedded value; add unit tests for these cases in tests/unit/test_json_parsing.bats.

📍Where to Start

Start in analyze_response within lib/response_analyzer.sh, focusing on the JSON-handling path that parses RALPH_STATUS and checks recent commits.


Macroscope summarized e82400e.

Copy link
Owner

@frankbria frankbria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @enlego — thanks for tackling this, it's a real problem. When Claude commits atomically after each task, git diff shows zero changes and the circuit breaker trips falsely. Appreciate you digging into it.

That said, there are a few things I'd like to see addressed before we can merge:

1. Link to an issue

Please open a GitHub issue describing the bug and link it to this PR. We track all changes against issues for traceability.

2. The 20-minute heuristic is fragile

git log --oneline -1 --since="20 minutes ago" has edge cases:

  • If a loop iteration takes longer than 20 minutes, it misses the commit
  • If loops are fast, it can pick up commits from a previous iteration and falsely report progress

A more robust approach would be to use the loop's start timestamp (which Ralph already tracks) instead of a hardcoded window. For example, git log --after="$loop_start_timestamp" would precisely capture commits made during the current iteration.

3. The sed 's/\\n/\n/g' workaround is brittle

The comment explains this well, but having production code include a workaround for test environment behavior is a maintenance risk. Consider either:

  • Adjusting the test fixtures to produce real newlines (matching production behavior)
  • Using a helper function that abstracts the newline handling

4. claude-review CI is failing

Please investigate — could be flaky CI or something in the diff it flagged.

The core idea is solid. Happy to re-review once these are addressed!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants