Opus 5 reader: dense-render comprehension lower than expected in our runs — are we misconfiguring something?

[akumrazor]

Hi — first, thanks for pxpipe. We've been running a fork of it in front of Claude Code for a few weeks and the token savings are real and substantial.

We wanted to sanity-check the reader quality after claude-opus-5 was promoted to a default reader in #147, and our numbers came out lower than we expected. We're posting them because we suspect we may be holding it wrong, and we'd rather ask than assume.

What we ran

Both existing eval levels against claude-opus-5, on a corpus extracted from our own Claude Code sessions with eval/extract-corpus.mjs (20 text blocks, 10 sessions).

The one change we made to the harness: the reflow/aa variants render at 100 columns, so we added arms that resolve geometry from the runtime itself rather than restating constants —

const PROD = resolveGptProfile('claude-opus-5');   // 312 cols, 728px cap, profile style
renderTextToPngs(reflow(src) ?? src, PROD.stripCols, PROD.style, PROD.maxHeightPx)

This follows the note already in render-bridge.mjs about scoring a model through the geometry the runtime would actually choose. (#159 has the Windows fixes we needed to run the harness at all; these arms are separate and not in that PR.)

L1 — transcription fidelity

variant	mean char accuracy	vs baseline
baseline	95.29%	—
prod-5x8 (= DENSE_RENDER_STYLE)	59.88%	-35.41pp
prod-7x10 (same, +2px pitch/axis)	78.13%	-17.16pp

Split by block size, prod-5x8 is 83.1% at <=1500 chars and 36.6% above; baseline stays flat (97.6% / 93.0%), so it doesn't look like "long text is just harder".

L2 — comprehension

Added a fourth arm at the same production geometry:

arm	mean judge score	pass / borderline / fail
production	38.7%	1 / 4 / 5
reflow	37.7%	1 / 3 / 6
aa	36.4%	1 / 2 / 7

Image savings held at 61.5% fewer images across all three — the compression itself works fine.

We'd expected comprehension to hold up better than transcription, given #147's note that Opus 5 "reads pxpipe images well enough for gist". In our runs the three compressed arms land in the same band, which suggests we're measuring something other than what that note describes.

Where we think we could be wrong

Genuinely unsure about several of these, which is the main reason for the issue:

• Our L1 baseline differs from prod-* on three axes at once — reflow, aa, and page cap (1932 vs 728) — so the gap isn't attributed to any single one. A cleaner isolation might tell a different story.
• Our corpus never reaches production density. DENSE_CONTENT_CHARS_PER_IMAGE is 28,080; our largest L1 block is 4,000 chars (a 1568x112 strip) and L2 histories render at ~8,000. Everything we measured is thinner than what ships.
• The L2 judge is the same model reading answers derived from images it read poorly, which caps how much the absolute number can carry.
• 2 of 10 L2 production judgments failed to parse and parseJudge defaulted them to 0.5. Excluding them the mean is 35.9% over n=8 — the default quietly turns a harness failure into a mid-range score, which might be worth handling separately regardless of this issue.
• Corpus composition. Ours is our own work (mixed Spanish/English, heavy on code and tooling discussion). Possible it's unrepresentative.

What would help

Mostly: does this reproduce for you? If these numbers look wrong against your own corpus, we'd like to know what we're doing differently. And if they look roughly right, is the current density a deliberate tradeoff we should be reading differently — e.g. is the intended contract that imaged history is lossy-by-design and the text-preserving path added in #147 is what carries correctness?

Happy to share the redacted aggregates, the added eval arms, or re-run with different parameters if that's useful. We didn't attach raw results because they quote our own project content.

🤖 Generated with Claude Code

[teamchong]

As #147 pointed out with the 2/15 score, Opus 5's transcription recall is bad

the reason I kept it as default anyway is because in my own sessions, I watched the model act like it's nearsighted: when it can't read the dense image, it just uses grep like wearing glasses to grab the exact text it needs

It's kind of like how other tools using RAG pretend to be "lossless" (which I don't like)

I'll probably revert it as default after running a few more tests