Skip to content

Commit bfce503

Browse files
ReadycodeAIclaude
andcommitted
Exact spreadsheet calculations, exact matches first, only the needed columns
- Counts, distinct values, top N (by rows or by different people), sums, averages, min, max and lists, computed by code over every row. ask_document turns plainly worded questions into a plan that Jev checks against the question (verdict "calculated"); the new calculate tool takes plans directly. - Spreadsheet lookups: rows holding the exact value asked about come first (match: exact / partial), show only the columns the question needs, and count identical records exactly. - Load progress while still_reading, a coverage report, and a note that document_tokens is an estimate. - "List everyone" questions in Word files also return the headings of neighbouring sections; the disagreement flag needs two relevant passages. - Benchmarks: PDF 8/8, Word 11/11 (panel question now needs all 7 members), Excel 11/11 with the 3 counts exact (9, 869, Roob Inc 286). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 40fd44c commit bfce503

14 files changed

Lines changed: 871 additions & 96 deletions

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,5 @@ node_modules/
22
web/dist/
33
*.log
44
.DS_Store
5+
.claude/
6+
launch/

‎README.md‎

Lines changed: 41 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ ReadyCode Reader is a free, open-source [MCP](https://modelcontextprotocol.io) s
77
- **Not in the document:** if the file can't answer, your AI is told `not_in_document` instead of getting passages to guess from.
88
- **Passages disagree:** if two passages give different values, your AI is told to report both.
99
- **Hidden instructions:** passages that try to instruct the AI reading them are flagged and treated as data only.
10-
- **Needs a calculation:** spreadsheet questions like "how many…" or "list every…" are flagged instead of being estimated from a few rows.
10+
- **Exact spreadsheet calculations:** "how many…", "which … appears most often", totals, averages and "list everyone who…" are computed by code over every row and come back as numbers, with the sheets, column and conditions used. A model never estimates them.
11+
- **Exact matches first:** a spreadsheet row holding the exact name you asked about comes before look-alikes ("Nat Becker" before "Prof. Nat Becker II"), and rows show only the columns the question needs.
1112

1213
## Standard AI vs Reader
1314

@@ -17,22 +18,32 @@ Same questions, same AI, fresh sessions: once the normal way (the AI reads the f
1718
|---|---|---|
1819
| **Codex**, 200-page NASA PDF, 8 questions | 8/8 correct · **913,468 tokens** · about 3 min | 8/8 correct · **41,997 tokens** · about 20 s |
1920
| **Grok 4.7 in Cursor**, same PDF | 8/8 correct · read the whole book (about 45,000 tokens) · about 4 min | 8/8 correct · **4,693 tokens** of evidence · about 2 min |
21+
| **Codex** and **Grok 4.7 in Cursor**, 263,000-token Word report, 11 questions | 11/11 · 4–5 tool calls · 2–4 min · no citations | 11/11 · 2 calls · 30 s–1 min · section and paragraph on every answer |
22+
| **Codex** and **Grok 4.7 in Cursor**, 100 MB Excel workbook (1.1 million rows), 11 questions | 11/11 · wrote its own parser script · 7 steps | 8/8 lookups in 2 calls, no code; declined the 3 counting questions (since fixed, see below) |
2023

21-
Token counts for Codex come from Codex's own session log. That's **95% fewer tokens and about 9× faster** for the same answers. Most of the saving comes from steps: an agent re-sends its whole conversation at every step, and with Reader it needed 2 steps instead of 17.
24+
Token counts for Codex come from Codex's own session log. On the PDF that's **95% fewer tokens and about 9× faster** for the same answers. Most of the saving comes from steps: an agent re-sends its whole conversation at every step, and with Reader it needed 2 steps instead of 17. On the Word report the gain was speed, fewer steps and citations rather than tokens. Full results: [`benchmark/head-to-head.md`](benchmark/head-to-head.md).
2225

2326
## Measured on large files
2427

2528
Every expected answer below was checked by code against the file, and every "not in the document" question was checked to be truly absent. Scripts and question files are in [`benchmark/`](benchmark/).
2629

2730
| File | Size as text | Correct | Tokens returned | Time for all questions | Check cost |
2831
|---|---|---|---|---|---|
29-
| NASA *Earth at Night* (PDF, 200 pages) | 42,282 tokens | 8/8 | 3,487 | 2.3 s | $0.0027 |
30-
| Australian Universities Accord Final Report (Word) | 263,487 tokens | 11/11 | 15,790 | 1.7 s | $0.0042 |
31-
| Sample contacts workbook (Excel, 100 MB, 1.1 million rows) | 271,424,159 tokens | 11/11 | 2,323 | 0.9 s | $0.0038 |
32+
| NASA *Earth at Night* (PDF, 200 pages) | 42,282 tokens | 8/8 | 3,487 | 1.6 s | $0.0027 |
33+
| Australian Universities Accord Final Report (Word) | 263,487 tokens | 11/11 | 14,991 | 1.1 s | $0.0044 |
34+
| Sample contacts workbook (Excel, 100 MB, 1.1 million rows) | 271,424,159 tokens (estimate) | 11/11 | 2,494 | 2.4 s | $0.0022 |
3235

33-
The Word report is larger than many AI context windows, and the spreadsheet is more than a hundred times larger than any. Each set includes questions the file cannot answer; Reader said `not_in_document` every time. On the spreadsheet it also declined 3 counting and ranking questions rather than guess.
36+
The Word report is larger than many AI context windows, and the spreadsheet is more than a hundred times larger than any. Each set includes questions the file cannot answer; Reader said `not_in_document` every time.
3437

35-
Honest limits of these results: they are our own runs on three files, not an independent study, and head-to-head runs against a standard AI exist so far only for the PDF.
38+
On the spreadsheet, 8 questions are lookups. Each is answered with only the name and the column asked about, 13–40 tokens each. The other 3 are exact calculations over all 1.1 million rows:
39+
40+
- "How many people work at Smith-Hickle?": 9 rows, and Reader adds that they hold only 1 different name.
41+
- "List everyone whose job title is Floor Layer": 869 rows, 150 different people, listed.
42+
- "Which company appears most often?": Roob Inc, 286 rows.
43+
44+
In the Word set, "Who else was on the panel?" needs all 7 other members, including two ex-officio members found only through section headings.
45+
46+
Honest limits of these results: they are our own runs on three files, not an independent study. The head-to-head runs above were made before exact calculations existed and have not been re-run since.
3647

3748
## Install (MCP)
3849

@@ -64,7 +75,7 @@ args = ["-y", "@readycode/reader"]
6475

6576
**Claude Desktop**: add the same entry as Cursor under `mcpServers` in `claude_desktop_config.json`.
6677

67-
Restart your tool and check that `readycode-reader` shows three tools: `load_document`, `ask_document` and `list_documents`.
78+
Restart your tool and check that `readycode-reader` shows four tools: `load_document`, `ask_document`, `calculate` and `list_documents`.
6879

6980
## Use
7081

@@ -76,9 +87,22 @@ Your AI loads the file, sees its contents, asks Reader specific questions and an
7687

7788
- `load_document(path)` reads a PDF, Word (.docx), Excel (.xlsx), TXT, Markdown, CSV or JSON file. It returns the file's size and **its contents**: a PDF's bookmarks with page numbers, the Word headings, or a spreadsheet's sheets, columns and an example row. That tells the AI what the file covers before it asks anything.
7889
- `ask_document(questions: [...])` asks up to 20 questions in one call. Each answer carries its passages with page numbers (and printed page numbers), Word sections and paragraphs, or spreadsheet sheet and row.
90+
- `calculate(operation, column, where, sheet, unique_by, n)` runs an exact calculation over a loaded spreadsheet: `count`, `distinct`, `top` (most or fewest), `sum`, `average`, `min`, `max` or `list`. It takes conditions (equals, contains, not equals, above, below), one sheet or all, and can count people instead of rows (`unique_by: "Name"`). You rarely need it: `ask_document` turns plainly worded questions into these plans, has each plan checked against the question, and answers with verdict `calculated`.
7991
- `list_documents()` lists what is loaded.
8092

81-
Very large files keep reading in the background: `load_document` may answer `still_reading`, and `ask_document` waits for the file to finish.
93+
A calculated answer looks like this:
94+
95+
```json
96+
{ "question": "How many people in the file work at Smith-Hickle?", "verdict": "calculated",
97+
"answer": "9 rows where Company is \"Smith-Hickle\" in all 4 sheets, holding 1 different Name value.",
98+
"plan": "Count the rows where Company is \"Smith-Hickle\" in all sheets, and count the different Name values among them.",
99+
"calculation": { "operation": "count", "unique_by": "Name", "where": ["Company is \"Smith-Hickle\""], "sheets": "all",
100+
"rows_checked": 1116271, "matching_rows": 9,
101+
"matching_rows_by_sheet": { "Worksheet (2)": 0, "Worksheet (3)": 0, "Tablo3": 1, "Worksheet": 8 },
102+
"result": 9, "unique": 1, "method": "Computed by code over every row; nothing estimated." } }
103+
```
104+
105+
Very large files keep reading in the background: `load_document` may answer `still_reading` with how far it has got, and `ask_document` waits for the file to finish. `load_document` also reports what was read (every sheet and row, and anything skipped), so a "not in the document" answer can be trusted. `document_tokens` is an estimate of the whole file's size as text (about 3.5 characters per token), for comparison; it is not model usage.
82106

83107
## Try it in your browser
84108

@@ -94,17 +118,18 @@ The same engine runs in a web page: choose a file, add your OpenRouter key, ask.
94118
- is there enough here to answer it?
95119
- do these passages disagree?
96120
- does any passage try to instruct the AI?
97-
- for spreadsheets, does it need a calculation over every row?
121+
- for spreadsheets, does it need a calculation over every row, and does the calculation plan Reader made fit the question?
98122

99123
Nothing is set up per document. The check costs about $0.0004 a question on your key and takes about a second.
100-
5. **Return** only the passages that answer, most relevant first, with the verdicts and token counts.
124+
5. **Calculate** (spreadsheets): when a question needs every row, code runs the plan over the whole workbook. Values that differ only in capitals or spacing count as one, and rows can be counted as rows or as different people. A 1.1-million-row workbook takes about 1–2 seconds.
125+
6. **Return** only the passages that answer, most relevant first, with the verdicts and token counts. For "list everyone" questions in Word files, the headings of neighbouring sections come too, since a list often runs over several sections.
101126

102127
## Limits (read these)
103128

104129
- **Picture-only pages** (scans, images) have no text to search and are listed as such; pictures inside Word files are counted, not read. Reading them needs a vision model; that's planned.
105130
- **Summaries of a whole document** are a different job. Reader answers specific questions.
106-
- **Spreadsheet calculations** (counts, totals, averages, rankings, "list every row that…") are flagged, not answered. Exact calculations are planned.
107-
- **The "passages disagree" check** sometimes fires when passages don't really disagree. Treat it as "double-check these sources".
131+
- **Spreadsheet calculations** cover counts, distinct values, rankings, totals, averages, lowest and highest values, and lists, with simple conditions. Formulas, pivot tables, date ranges and combining sheets with different columns are not supported yet. When Reader can't turn a question into a plan it is sure of, it answers `needs_calculation` rather than guess, and your AI can call `calculate` directly.
132+
- **The "passages disagree" check** now needs two relevant passages about the same thing, but it can still fire when passages don't really disagree. Treat it as "double-check these sources".
108133
- **Old formats** (`.doc`, `.xls`, `.xlsb`) aren't supported; save them as `.docx` or `.xlsx`.
109134
- **It depends on Jev.** If the decision model is unavailable, Reader still returns the top search results, marked `unchecked`.
110135
- **Check the passages.** Every answer comes with its sources so you can.
@@ -120,4 +145,6 @@ The same engine runs in a web page: choose a file, add your OpenRouter key, ask.
120145

121146
Apache-2.0. Built by [ReadyCode](https://readycode.ai). If you build on it, keep the NOTICE file, and we'd love a link back. "ReadyCode" is our name; please don't use it for forks.
122147

123-
**More tools from ReadyCode, and early access to the hosted version:** [readycode.ai](https://readycode.ai)
148+
**Source:** [github.com/ReadycodeAI/readycode-reader](https://github.com/ReadycodeAI/readycode-reader). Issues and ideas are welcome there.
149+
150+
**News about Reader and new ReadyCode tools, and early access to the hosted version:** [readycode.ai/reader](https://readycode.ai/reader)

‎benchmark/README.md‎

Lines changed: 19 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,21 +17,33 @@ A question passes when:
1717

1818
- **answer questions:** Reader's verdict is `answer_from_passages` (or `low_confidence`) and the passages it returned contain every expected phrase;
1919
- **not-in-the-document questions:** the verdict is `not_in_document`;
20-
- **calculation questions** (spreadsheets): the verdict is `needs_calculation`.
20+
- **calculation questions** (spreadsheets): the verdict is `calculated` and the number matches the one the benchmark computes itself over every row (the matching row count, or the top value and its row count).
21+
22+
For a question that asks for a complete list, the headings Reader returns in `nearby_sections` count as returned text: they are how a list that runs over several sections is found.
2123

2224
## The files
2325

2426
- **NASA *Earth at Night*** (2019, 200-page PDF, NP-2019-07-2739-HQ): free from nasa.gov. Questions: [`nasa-earth-at-night.json`](nasa-earth-at-night.json), 6 answerable and 2 that the book cannot answer.
25-
- **Australian Universities Accord – Final Report** (Australian Government, Word .docx, about 263,000 tokens): published by the Department of Education. Questions: [`universities-accord-word.json`](universities-accord-word.json), 9 answerable and 2 that the report cannot answer.
26-
- **A 100 MB contacts workbook** (1.1 million rows of fake people generated with the Faker library, 4 sheets, columns Name, Email, Phone, Address, Company, Text, Description, Job Title). [`excel-lookups.cjs`](excel-lookups.cjs) picks rows at random (fixed seed) and computes each expected answer over every row: 5 lookups by name, 2 reverse lookups (by email and by phone), 1 name that isn't in the file, and 3 counting or ranking questions. It works on any workbook with those columns.
27+
- **Australian Universities Accord – Final Report** (Australian Government, Word .docx, about 263,000 tokens): published by the Department of Education. Questions: [`universities-accord-word.json`](universities-accord-word.json), 9 answerable and 2 that the report cannot answer. "Who else was on the Review Panel?" needs all 7 other members, two of them ex-officio.
28+
- **A 100 MB contacts workbook** (1.1 million rows of fake people generated with the Faker library, 4 sheets, columns Name, Email, Phone, Address, Company, Text, Description, Job Title). [`excel-lookups.cjs`](excel-lookups.cjs) picks rows at random (fixed seed) and computes each expected answer over every row: 5 lookups by name, 2 reverse lookups (by email and by phone), 1 name that isn't in the file, and 3 counting, listing or ranking questions whose answers it also computes itself. It works on any workbook with those columns.
2729

2830
## Results (30 September 2026, Jev `typesafe/jev-1.13`)
2931

3032
| File | Correct | Document tokens | Tokens returned | Load | All questions | Check cost |
3133
|---|---|---|---|---|---|---|
32-
| NASA PDF | 8/8 | 42,282 | 3,487 | 0.6 s | 2.3 s | $0.0027 |
33-
| Word report | 11/11 | 263,487 | 15,790 | under 1 s (cached) | 1.7 s | $0.0042 |
34-
| Excel workbook | 11/11 | 271,424,159 | 2,323 | 14.3 s | 0.9 s | $0.0038 |
34+
| NASA PDF | 8/8 | 42,282 | 3,487 | under 1 s (cached) | 1.6 s | $0.0027 |
35+
| Word report | 11/11 | 263,487 | 14,991 | under 1 s (cached) | 1.1 s | $0.0044 |
36+
| Excel workbook | 11/11 | 271,424,159 (estimate) | 2,494 | 16.2 s | 2.4 s | $0.0022 |
37+
38+
On the workbook, the 8 lookups return only the columns each question needs (13–40 tokens each). The 3 calculations are exact:
39+
40+
- 9 rows at Smith-Hickle, which hold 1 different name.
41+
- 869 Floor Layer rows, 150 different people, listed.
42+
- Roob Inc as the most frequent company, with 286 rows.
43+
44+
The list is 1,934 of the 2,494 tokens.
45+
46+
Earlier results on the same day, before exact calculations, column trimming and the list change: Word 15,790 tokens returned; Excel 2,323 tokens returned, with the 3 calculation questions declined (`needs_calculation`) rather than answered.
3547

3648
## Standard AI vs Reader (NASA PDF)
3749

@@ -44,4 +56,4 @@ The same 8 questions were given to each AI twice, in fresh sessions: once readin
4456

4557
Most of Codex's saving is cached tokens, which cost less, so the cost saving is smaller than the token saving: about 85% rather than 95%.
4658

47-
Head-to-head runs for the Word and Excel files haven't been done yet.
59+
Head-to-head runs for the Word and Excel files, against Codex and Grok, are in [`head-to-head.md`](head-to-head.md).

‎benchmark/excel-lookups.cjs‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,9 +33,12 @@ const { run } = require("./run.cjs");
3333
const company = pick().Company, title = pick()["Job Title"];
3434
questions.push({ q: `How many people in the file work at ${company}?`, calc: true, truth: where("Company", company).length });
3535
questions.push({ q: `List everyone whose job title is ${title}.`, calc: true, truth: where("Job Title", title).length });
36-
questions.push({ q: "Which company appears most often in the spreadsheet?", calc: true });
36+
const counts = new Map();
37+
for (let r = 0; r < N; r++) { const c = row(r).Company; if (c) counts.set(c, (counts.get(c) || 0) + 1); }
38+
const [topValue, topRows] = [...counts.entries()].sort((a, b) => b[1] - a[1])[0];
39+
questions.push({ q: "Which company appears most often in the spreadsheet?", calc: true, top: { value: topValue, rows: topRows } });
3740
const r = await run({ document: { title: `${file.split(/[\\/]/).pop()}: ${N.toLocaleString("en")} rows` }, questions }, file);
38-
for (const row of r.rows) console.log(`${row.ok ? "PASS" : "FAIL"} ${row.question}\n ${row.verdict}, ${row.evidence_tokens} tokens`);
41+
for (const row of r.rows) console.log(`${row.ok ? "PASS" : "FAIL"} ${row.question}\n ${row.verdict}, ${row.evidence_tokens} tokens${row.answer ? `\n ${row.answer}${row.expected ? ` (expected ${row.expected})` : ""}` : ""}`);
3942
console.log(`\n${r.document}\n${r.correct} correct | document ${r.document_tokens.toLocaleString("en")} tokens | evidence ${r.evidence_tokens.toLocaleString("en")} tokens | load ${r.load_seconds} s | ${r.rows.length} questions ${r.ask_seconds} s | checks $${r.check_cost_usd}`);
4043
process.exitCode = r.rows.every((x) => x.ok) ? 0 : 1;
4144
})().catch((err) => { console.error(err.message); process.exit(1); });

0 commit comments

Comments
 (0)