fix(review): guarantee newlineChunks forward progress under high overlap

[jsdevninja]

Summary

• Floor the next start in newlineChunks at start + 1 so newline snap + near-budget chunkOverlap cannot stall the loop (end - overlap <= previous start).
• Clarify the incomplete #rag-verify clamp comment: chunkOverlap < chunkChars alone does not guarantee forward progress once newline snap shrinks end.
• Add regression coverage for the exact hang repro (chunkChars: 100, chunkOverlap: 99), a table sweep of boundary overlaps, and the chunkJsTs oversized-unit fallback path through newlineChunks.

Closes #7447

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

• Focused src/review/rag.ts defensive fix. Ran npx vitest run test/unit/rag.test.ts -t "#7447|#rag-verify infinite-loop|newlineChunks newline-boundary|#4996" (12 pass). Production call sites never pass opts (defaults CHUNK_CHARS/CHUNK_OVERLAP stay outside the stall region); remaining test:ci surface will run in CI.

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
• API/OpenAPI/MCP behavior is updated and tested where needed.
• UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
• Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
• Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

N/A — backend RAG chunker defensive fix only; no visible UI change.

Notes

• Approach: loop-side Math.max(start + 1, end - chunkOverlap) rather than tightening the overlap clamp further, so every chunkChars >= 1 / clamp-allowed overlap pair has a hard forward-progress proof even after newline snap.

Made with Cursor

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.40%. Comparing base (c18b123) to head (552134c).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7496      +/-   ##
==========================================
- Coverage   91.41%   91.40%   -0.02%     
==========================================
  Files         717      717              
  Lines       73035    73035              
  Branches    21635    21637       +2     
==========================================
- Hits        66765    66754      -11     
- Misses       5227     5234       +7     
- Partials     1043     1047       +4     

Flag	Coverage Δ
shard-1	31.89% <0.00%> (-0.05%)	⬇️
shard-2	35.96% <100.00%> (-0.01%)	⬇️
shard-3	31.44% <0.00%> (-0.01%)	⬇️
shard-4	43.91% <0.00%> (-0.02%)	⬇️
shard-5	37.05% <0.00%> (ø)
shard-6	35.84% <0.00%> (-0.09%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/review/rag.ts	99.60% <100.00%> (ø)

... and 2 files with indirect coverage changes

[loopover-orb]

Tip

✅ LoopOver review result - approve/merge recommended

_{Review updated: 2026-07-20 13:25:16 UTC}

2 files · 1 AI reviewer · no blockers · readiness 95/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
This fixes a real infinite-loop hazard in newlineChunks: when newline snap shrinks `end` below start + chunkChars, `end - chunkOverlap` could be <= the previous `start`, stalling the window and hanging on large files with near-budget overlap. Flooring `start` at `start + 1` guarantees at least 1 char of forward progress every iteration, which is correct and minimally invasive. The regression tests directly reproduce the reported hang (chunkChars:100, chunkOverlap:99) and sweep boundary overlap ratios plus the chunkJsTs oversized-unit fallback path, which is good coverage of the actual failure mode rather than a fabricated one.

Nits — 4 non-blocking

• The `Math.max(0, …)` removal claim in the comment (src/review/rag.ts:298) is fine since `start` is loop-invariant >=0, but worth double-checking no other call site can pass a negative initial `start` into newlineChunks.
• The it.each table in test/unit/rag.test.ts doesn't assert forward-progress bounds as tightly as the single regression test (e.g., no exact chunk-count assertion), so it's more of a sanity sweep than a precise regression guard — acceptable but could be tightened.
• Consider asserting exact or near-exact chunk counts in the it.each sweep (test/unit/rag.test.ts:255-269) to make the regression more precise rather than just bounding by text.length.
• The updated comment in src/review/rag.ts:195-198 is a clear improvement in explaining why overlap<chunkChars alone wasn't sufficient — good documentation of the root cause for future readers.

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#7447
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 193 registered-repo PR(s), 120 merged, 38 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	jsdevninja; Gittensor profile; 193 PR(s), 38 issue(s).
Improvement	✅ Minor	risk: clean · value: minor · LLM: significant

Linked issue satisfaction

Addressed
The diff adds a `Math.max(start + 1, end - chunkOverlap)` floor in newlineChunks guaranteeing strict forward progress regardless of newline-snap interaction, directly matching the issue's requested fix approach, and adds the exact repro test plus a boundary-value sweep and a chunkJsTs fallback-path test as required.

Review context

• Author: jsdevninja
• Role context: outside_contributor
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: not available
• Official Gittensor activity: 193 PR(s), 38 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Triage stale or unlinked PRs.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[loopover-orb]

LoopOver approves — the gate is satisfied and CI is green.