-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathproofloop-prod-proxy-benchmark-matrix.html
More file actions
34 lines (34 loc) · 3.32 KB
/
Copy pathproofloop-prod-proxy-benchmark-matrix.html
File metadata and controls
34 lines (34 loc) · 3.32 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
<!doctype html>
<meta charset="utf-8">
<title>ProofLoop Prod Proxy Benchmark Matrix</title>
<style>
body { font-family: Inter, system-ui, sans-serif; margin: 32px; color: #17201a; }
table { border-collapse: collapse; width: 100%; margin: 18px 0 28px; }
th, td { border-bottom: 1px solid #dde5dd; padding: 8px 10px; text-align: left; vertical-align: top; }
th { color: #526154; font-size: 12px; text-transform: uppercase; letter-spacing: .04em; }
code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; }
.callout { border: 1px solid #dde5dd; background: #f8fbf8; border-radius: 6px; padding: 14px 16px; margin: 18px 0; }
.bar { width: 100%; min-width: 160px; height: 12px; border-radius: 3px; background: #eef3ef; overflow: hidden; }
.bar span { display: block; height: 100%; background: #2e9e6b; }
</style>
<h1>ProofLoop Prod Proxy Benchmark Matrix</h1>
<div class="callout">
<strong>4/1354 task targets have prod live-browser proof.</strong>
<p>All-task winner: <code>none yet</code>. Current adapter-smoke winner: <code>poolside/laguna-xs-2.1</code>.</p>
</div>
<h2>Families</h2>
<table><thead><tr><th>Family</th><th>Tasks</th><th>Prod passed</th><th>Runnable now</th><th>Blocked</th><th>Coverage</th></tr></thead><tbody><tr><td><code>spreadsheetbench-v1-full-912</code></td><td>912</td><td>0</td><td>912</td><td>0</td><td><div class="bar"><span style="width:2%"></span></div></td></tr>
<tr><td><code>spreadsheetbench-v2-full-321</code></td><td>321</td><td>0</td><td>321</td><td>0</td><td><div class="bar"><span style="width:2%"></span></div></td></tr>
<tr><td><code>bankertoolbench-full-100</code></td><td>100</td><td>0</td><td>100</td><td>0</td><td><div class="bar"><span style="width:2%"></span></div></td></tr>
<tr><td><code>accounting-live-proofloop</code></td><td>4</td><td>0</td><td>4</td><td>0</td><td><div class="bar"><span style="width:2%"></span></div></td></tr>
<tr><td><code>notion-live-proofloop</code></td><td>4</td><td>0</td><td>4</td><td>0</td><td><div class="bar"><span style="width:2%"></span></div></td></tr>
<tr><td><code>proximitty-underwriting-pr0</code></td><td>4</td><td>0</td><td>4</td><td>0</td><td><div class="bar"><span style="width:2%"></span></div></td></tr>
<tr><td><code>finch-prod-proxy-task</code></td><td>1</td><td>1</td><td>1</td><td>0</td><td><div class="bar"><span style="width:100%"></span></div></td></tr>
<tr><td><code>finauditing-prod-proxy-task</code></td><td>1</td><td>1</td><td>1</td><td>0</td><td><div class="bar"><span style="width:100%"></span></div></td></tr>
<tr><td><code>workstreambench-prod-proxy-task</code></td><td>1</td><td>1</td><td>1</td><td>0</td><td><div class="bar"><span style="width:100%"></span></div></td></tr>
<tr><td><code>noderoom-multi-user-conflict</code></td><td>6</td><td>1</td><td>6</td><td>0</td><td><div class="bar"><span style="width:17%"></span></div></td></tr></tbody></table>
<h2>Models</h2>
<table><thead><tr><th>Model</th><th>Adapter smoke</th><th>Est. cost</th><th>Measured cost</th></tr></thead><tbody><tr><td><code>z-ai/glm-5.2</code></td><td>3/3</td><td>$0.2454</td><td>$0.2450</td></tr>
<tr><td><code>deepseek/deepseek-v4-flash</code></td><td>1/3</td><td>$0.0683</td><td>$0.0690</td></tr>
<tr><td><code>poolside/laguna-xs-2.1</code></td><td>3/3</td><td>$0.0326</td><td>$0.0330</td></tr>
<tr><td><code>qwen/qwen3.7-plus</code></td><td>3/3</td><td>$0.1994</td><td>$0.1990</td></tr></tbody></table>