Submission: henrybrewer00-dotcom - #86
Open
henrybrewer00-dotcom wants to merge 3 commits into
Open
Conversation
Offline, CPU-only document adjudication pipeline. No LLM, no network, no GPU. evaluate.py on data/train: 143.49/150 (extraction 45.27, classification 79.04, calibration 19.19, zero catastrophic false approvals). That figure is in-sample. Out-of-fold with a full refit per fold: 130.10/150. Solution repo: https://github.com/henrybrewer00-dotcom/mib-doc-solution
Measured at the end: 28 of 40 scan pages carry per-row horizontal displacement with a median 54 px spread. Undoing it via the printed page border recovers true field values OCR could not read (25 -> 27 across 18 pages, none made worse). It explains why four separate resolution-axis fixes measured -50, +0.09, -3 and -0.53. Documentation only -- no code, predictions or artifact change.
Two further runs of the de-shift lab moved the BASELINE (25, 25, 22), which it should not, since no repair is applied to it. The lab picks the page to measure using the estimator, so changing the estimator changes the page set. Each run is a valid paired comparison on its own pages; the three are not comparable. What is left is two pages out of eighteen, and one page regressed in a run made after the +2 claim was written, so 'no page made worse' was wrong. The displacement itself is still real and large. The recovery is not resolved at this sample size, and the +0.3 extraction / +0.9 total projection is withdrawn.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Offline, CPU-only document adjudication pipeline. No LLM, no network, no GPU.
Solution repository: https://github.com/henrybrewer00-dotcom/mib-doc-solution (public, includes
Dockerfileand MITLICENSEat root)Scores
scripts/evaluate.pyoverdata/trainthrough the Docker contract: 143.49 / 150 — extraction 45.27, classification 79.04, calibration 19.19, 0 catastrophic false approvals, 0 missing cases.That number is in-sample: the adjudicator was fitted on those labels. The honest forecast is 130.10 / 150 out-of-fold, 5-fold with a full refit per fold across seeds 0-3, at 22 catastrophic false approvals per 1,000.
Approach
Visibility is decided by rendering rather than parsing: a span is evidence only if a human looking at the page would see marks there. Hidden text, out-of-crop text and barcode payloads never supply a field value, and their presence is not used as a feature either. Two OCR engines resolve raster pages; a precedence chain resolves conflicts; a three-member calibrated ensemble produces the posterior, and an expected-value rule turns it into a decision against the published payoff matrix.
Where the ceiling is
Of the 4.73 extraction points lost, none is a parsing bug — 3.38 are in no text layer and 1.39 exist only in hidden answer-key text, which
EVALUATION.mdmarks unrecoverable. Fed perfect fields the same adjudicator scores 77.77/80 against 67.79 on fields actually read, so classification is extraction-limited rather than model-limited.MEMO.mdrecords every improvement attempt including the ~25 that measured null or negative, and the four train/serve bugs found and fixed along the way.