Skip to content

tokenizer: reuse buffers in peeking reads - #4

Open
mvzink wants to merge 1 commit into
mainfrom
perf-tokenize-word-buffer
Open

mvzink wants to merge 1 commit into
mainfrom
perf-tokenize-word-buffer

Conversation

@mvzink

@mvzink mvzink commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

peeking_read_while appends to a caller's buffer and leaves the first non-matching character unread. Use it for word, numeric-prefix and dollar-value scans; peeking_take_while shares the same loop. This avoids temporary suffix strings and copies, complementing apache#2226.

Compared with the word-only change cc080fdd, the measured helper took 1.9% more tokenizer time with 8-character identifiers and 2.4% less with 128-character identifiers. The word-only change took 10.5%/9.2% less time than the original baseline; the helper was not measured directly against that baseline in the same batch.

These are medians of paired mean-time ratios from ten alternating pairs of the committed MySQL identifier benchmarks, with spans and output destruction included. Both comparisons used Linux on a Ryzen 5 3600, Rust 1.98.1, a fixed CPU and the performance governor. Within each comparison, both arms used identical locks and default features. Adapter throughput was not measured.

Original baseline: 1ea2488a632762e85f2cdb1c8bbe08f80800338e; word-only: cc080fdd3cccc379622370c13073b3fe98fdcfaf; measured helper: 3a150f27b5650585ca1b99a252870f29111b310f. Copy the added benchmark into the original baseline.

Related: apache#2036, apache#1557. AI-assisted.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@mvzink
mvzink force-pushed the perf-tokenize-word-buffer branch 2 times, most recently from 946d8a2 to fab9561 Compare October 21, 2026 00:08
@mvzink mvzink changed the title All dialects: Reuse the tokenizer word buffer tokenizer: reuse the word buffer Oct 21, 2026
@github-actions github-actions Bot added the MySQL label Oct 21, 2026
@mvzink
mvzink force-pushed the perf-tokenize-word-buffer branch 2 times, most recently from f7b4489 to cc080fd Compare October 21, 2026 17:57
@mvzink mvzink changed the title tokenizer: reuse the word buffer tokenizer: reuse buffers in peeking reads Oct 23, 2026
Append matching characters directly to existing strings with
peeking_read_while, keeping the first rejected character unread. Share the
loop with peeking_take_while and use it for word, numeric and dollar-value
scans to avoid temporary suffix allocations and copies.
@mvzink
mvzink force-pushed the perf-tokenize-word-buffer branch from 3a150f2 to 15e05d5 Compare October 23, 2026 05:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants