Benchmark Comparison: JSONL vs. TOON output for json-render efficiency #33
Description
Hi @ctate and team,
I've been experimenting with json-render and ran a benchmark comparing the default JSONL format to TOON.
Key findings from the benchmark:
- Cost savings: Up to 89% reduction in cost with TOON.
- Latency improvement: 74% faster response times.
- Trade-offs: TOON doesn't support streaming/hot-loading out of the box, which I know is core to json-render's demo. However, I've mitigated this by implementing a polling mechanism that loads updates every X ms once the full TOON output is translated to JSONL. This allows for efficient batch processing while simulating incremental updates. For non-streaming use cases or with adaptations like this, the efficiency gains are massive.
I documented the full benchmark in my repo here. It includes the code, raw data, and charts for reproducibility.
Additionally, I've shared two demo videos in this tweet comparing JSONL and TOON outputs for the same prompt "Create a UI to modify users data (picture,username, descripcion, mail)" . They highlight the practical differences in cost and time.
I'm sharing this not as a proposal to change the project (unless it aligns with your vision), but to spark a discussion. Maybe it could inspire an optional mode or further optimizations? I'd love to hear your thoughts on this.
Thanks for the great work on json-render!