Harvester heading extraction misidentifies Markdown fence boundaries #1138

Issue

What is the issue?

On upstream main 00a7b8cb5b03d5ea1c487cb86d65e7d720bfc247, HeadingExtractor._toggle_fence() toggles its code-block state whenever a line begins with three backticks or tildes after lstrip().

A shorter fence or a different marker can therefore close an open block incorrectly. Indented code lines can also toggle the state and hide valid headings. This changes the headings and line ranges used by the harvester to partition documents into sections.

Expected Behaviour

A closing fence should use the opener's marker, contain at least as many markers, have at most three leading spaces, and have only spaces/tabs after it. Code-block contents should remain excluded from heading extraction.

For the reproduction below, the extracted headings should be ['Before', 'After'].

Actual Behaviour

The extractor returns ['Before', 'Fake']: the three-backtick line incorrectly closes a four-backtick block, exposing a code line as a heading. The later four-backtick line then reopens the state, hiding the actual heading.

Steps to reproduce

From the repo root with development dependencies installed:

python - <<'PY'
from application.utils.harvester.heading_extractor import HeadingExtractor
text = "# Before\n````\n```\n# Fake\n````\n# After\n"
print([heading.text for heading in HeadingExtractor().extract(text)])
PY

Proposed scope

Track the opening marker and length, validate matching closing fences, and preserve the existing heading-range calculation. No new dependencies or schema changes are needed. Full Markdown container parsing is outside this scoped correction.

I found an existing local candidate patch and checked it against current upstream. Upstream fails 11 assertions/subtests in the candidate's 14 heading tests; the candidate passes all 14. The harvester suite ran 135 tests successfully (2 skipped). Black, a focused strict mypy check of heading_extractor.py, and patch whitespace checks pass. Independent source review found no blockers. These are local checks, not upstream CI results.

I searched existing issues and PRs for the fence-boundary defect and did not find a dedicated pending fix. Would this scoped correction be useful, and is anyone already working on it?

AI assistance

Codex helped inspect the existing local patch, prepare the reproduction, run checks, independently review the scope, and draft this report.