Head-to-head results for Word and Excel, with the improvement list; compact batch output; Word contents; no false picture pages for Word

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Author: ReadycodeAI

benchmark/head-to-head.md

@@ -0,0 +1,54 @@
+# Head-to-head: standard AI vs Reader
+
+Each AI answered the same questions about the same file twice, in fresh chats: once with its normal built-in tools (terminal, scripts, file search), once using only Reader. The prompts didn't say what Reader was. Answers were checked against a key computed by code from the file. Reader's side is also confirmed by its own local log.
+
+## Word: Australian Universities Accord Final Report (263,487 tokens, 11 questions)
+
+| | Codex, standard | Codex, Reader | Cursor (Grok 4.7 low), standard | Cursor (Grok 4.7 low), Reader |
+|---|---|---|---|---|
+| Correct | 11/11 | 11/11 | 11/11 | 11/11 |
+| Tool calls | 4 (one failed on quoting) | 2 | 5 (one search missed the file) | 2 |
+| Time | about 3–4 min (1 m 17 s of work shown) | about 30 s end to end; Reader answered all 11 in about 1 s | about 2–3 min | about 1 min |
+| Text brought into the chat | many search dumps (not measured) | 14,439 tokens of evidence (5.5% of the report) | about 50–70k characters of search output (estimated) | 15,291 tokens of evidence (5.8%) |
+| Citations | none | section and paragraphs on every answer | none | section and paragraphs on every answer |
+| The 2 questions the report can't answer | "Not in the document" ✓ | ✓ | ✓ | ✓ |
+
+Neither app shows its exact billed tokens for these runs, so the token comparison is between what each method brought into the chat, not the bill.
+
+**What the AIs themselves said:**
+
+- Codex: "For a large, mostly textual report and a set of factual questions, readycode-reader is the better default." It put the time saving at roughly 80–95%, and Reader's output at 60–75% smaller than its own search dumps.
+- Cursor: "The reader was the better shape for this job. One call took all 11 questions, each answer came with a section and paragraph, an answerable score, and a clear not_in_document."
+
+**Where Reader was weaker:**
+
+- **"List everyone" questions.** For "who else was on the panel", Reader's answerable score was only 0.53. With Reader, Codex still found the full list (including ex-officio members Tony Cook and Ben Rimmer, from Appendix B). With Reader, Cursor gave only the six names in the covering letter, while standard search found all eight. Questions whose answer is a complete list need more than the top passages.
+- **Output size in Codex.** The 11-answer reply was large enough for Codex to cut off part of it on screen. Fixed after this run: batch output is now compact JSON, with the document details given once.
+- **Misleading picture warning.** Reader reported "23 picture-only pages" for the Word file; those were only short paragraphs. Fixed: only PDFs can have picture-only pages.
+- **Pictures aren't read.** The report has 44 pictures; anything that exists only inside a picture can't be found, so "not in the document" means "not in its text".
+- **Forensic work.** Both AIs noted that direct file inspection is better for exact string counts, formatting, metadata and revision history.
+
+**What this shows:** strong agents with a terminal get these answers right on their own, by searching. Reader's advantage on a report like this is **speed (2–4× faster), fewer steps (2 instead of 4–5), cited answers and clean "not in the document" verdicts**, rather than a large token cut. The large token savings appear when an agent would otherwise read a lot of the file: in the NASA PDF run, Codex used 913,468 tokens normally and 41,997 with Reader.
+
+**A Codex setup note:** Codex exposes MCP tools through tool search. Twice it said Reader wasn't available until the prompt asked it to use tool search to find `readycode-reader`.
+
+## Excel: 100 MB workbook (1.1 million rows, 11 questions)
+
+| | Codex, standard | Codex, Reader | Cursor (Grok 4.7 low), standard | Cursor (Grok 4.7 low), Reader |
+|---|---|---|---|---|
+| Lookups and the missing name (questions 1–8) | 8/8 | 8/8 | 8/8 | 8/8 |
+| Counts and ranking (questions 9–11) | 3/3 (9, 869, Roob Inc 286) | declined (needs_calculation) | 3/3 | declined (needs_calculation) |
+| Work | 7 commands plus writing a parser script | 2 calls, no code | 7 steps plus a parser script | 2 calls, no code |
+| Time | about 73 s of tool time (the scan itself about 26 s) | about 46 s of tool time | 2–3 min (the scan about 16 s) | 2–3 min, mostly waiting for the load |
+| Text returned | about 6–8k tokens (estimated) | 2,323 evidence tokens plus metadata (about 3–4k) | about 48 KB of full rows | 2,323 evidence tokens |
+
+**The honest reading:** agents with a terminal can write a streaming parser and answer everything, including exact counts, in a few minutes. Reader answered every lookup with sheet-and-row citations in two calls and no code. It grouped duplicate records, kept "Nat Becker" apart from "Prof. Nat Becker II", and refused to guess counts. For spreadsheets, Reader wins on ease and setup, not on raw speed, and it loses on counts until exact calculations exist.
+
+**Improvements both AIs asked for, in priority order:**
+
+1. **Exact calculations:** count, distinct, top N and sums over whole columns. The person chooses all sheets or one, and rows or unique people; the answer comes back as a number with the sheet and column.
+2. **Exact field matches first:** exact name or email matches ahead of partial word matches, with near-matches clearly separated.
+3. **Return only the relevant columns:** a phone question needs the name, the phone and the source, not long Text and Description cells.
+4. **Load progress:** `still_reading` should say how far along it is.
+5. **Explain the token figures:** `document_tokens` estimates the file's size as text (about 3.5 characters per token); it is not model usage.
+6. **Coverage report:** say which sheets and how many rows were read, and anything skipped, so an absence result can be trusted.

benchmark/run.cjs

@@ -49,7 +49,7 @@ async function run(spec, file) {
     pass += ok;
     return { ok, question: q.q, verdict: a.verdict, evidence_tokens: a.evidence_tokens };
   });
-  return { document: spec.document.title, correct: `${pass}/${rows.length}`, document_tokens: res.answers[0].document_tokens, evidence_tokens: res.total_evidence_tokens, load_seconds: Number(loadSeconds.toFixed(1)), ask_seconds: Number(askSeconds.toFixed(1)), check_cost_usd: Number(cost.toFixed(4)), rows };
+  return { document: spec.document.title, correct: `${pass}/${rows.length}`, document_tokens: res.document_tokens, evidence_tokens: res.total_evidence_tokens, load_seconds: Number(loadSeconds.toFixed(1)), ask_seconds: Number(askSeconds.toFixed(1)), check_cost_usd: Number(cost.toFixed(4)), rows };
 }
 
 if (require.main === module) {

src/core.cjs

@@ -518,9 +518,10 @@ const noul = (a) => (a && Number.isFinite(Number(a.noul)) ? Number(a.noul) : nul
 // are plain data and can be cached; a store is attached after loading.
 function recordFromPages(id, kind, pages) {
   return {
-    version: 6, id, kind, outline: pages.outline || [], pictures: pages.pictures || 0, pages: pages.length,
+    version: 7, id, kind, outline: pages.outline || [], pictures: pages.pictures || 0, pages: pages.length,
     textPages: pages.filter((p) => p.text.length >= 40).length,
-    pictureOnlyPages: pages.filter((p) => p.text.length < 40).map((p) => p.page),
+    // Only a PDF page can be picture-only; a short Word or text part is just short.
+    pictureOnlyPages: kind === ".pdf" ? pages.filter((p) => p.text.length < 40).map((p) => p.page) : [],
     passages: passagesOf(pages), printed: kind === ".pdf" ? printedPages(pages) : {},
   };
 }
@@ -726,7 +727,8 @@ function createReaderCore({ apiKey = "", log = null } = {}) {
     if (list) {
       list = list.slice(0, LIMITS.maxQuestions);
       const out = await Promise.all(list.map((q) => askOne(q, rec)));
-      return { answers: out.map((r, i) => ({ question: list[i], ...r })), total_evidence_tokens: out.reduce((s, r) => s + r.evidence_tokens, 0) };
+      // The document and its size are stated once, not repeated in every answer.
+      return { document: rec.id, document_tokens: rec.tokens, total_evidence_tokens: out.reduce((s, r) => s + r.evidence_tokens, 0), answers: out.map((r, i) => { const { document: _d, document_tokens: _t, ...rest } = r; return { question: list[i], ...rest }; }) };
     }
     if (!question) throw new Error("A question (or a questions list) is required.");
     return askOne(question, rec);

src/engine.cjs

@@ -87,7 +87,7 @@ async function open(file, id, ext, hash) {
   const cached = path.join(CACHE, `${hash}.json`);
   let rec = null;
   try { rec = JSON.parse(fs.readFileSync(cached, "utf8")); } catch (_) { rec = null; }
-  if (!rec || rec.version !== 6) {
+  if (!rec || rec.version !== 7) {
     rec = core.recordFromPages(id, ext, await readPages(file));
     fs.mkdirSync(CACHE, { recursive: true });
     fs.writeFileSync(cached, JSON.stringify(rec));

src/server.cjs

@@ -66,7 +66,7 @@ async function handle(line) {
   if (method === "tools/call") {
     try {
       const out = await callTool(params && params.name, params && params.arguments);
-      return send({ jsonrpc: "2.0", id, result: { content: [{ type: "text", text: JSON.stringify(out, null, 1) }] } });
+      return send({ jsonrpc: "2.0", id, result: { content: [{ type: "text", text: JSON.stringify(out) }] } });
     } catch (e) {
       return send({ jsonrpc: "2.0", id, result: { isError: true, content: [{ type: "text", text: String((e && e.message) || e).slice(0, 400) }] } });
     }