Repository navigation
Conversation
|
|
mvzink
force-pushed
the
perf-tokenize-word-buffer
branch
2 times, most recently
from
October 21, 2026 00:08
946d8a2 to
fab9561
Compare
mvzink
force-pushed
the
perf-tokenize-word-buffer
branch
2 times, most recently
from
October 21, 2026 17:57
f7b4489 to
cc080fd
Compare
Append matching characters directly to existing strings with peeking_read_while, keeping the first rejected character unread. Share the loop with peeking_take_while and use it for word, numeric and dollar-value scans to avoid temporary suffix allocations and copies.
mvzink
force-pushed
the
perf-tokenize-word-buffer
branch
from
October 23, 2026 05:43
3a150f2 to
15e05d5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
peeking_read_whileappends to a caller's buffer and leaves the first non-matching character unread. Use it for word, numeric-prefix and dollar-value scans;peeking_take_whileshares the same loop. This avoids temporary suffix strings and copies, complementing apache#2226.Compared with the word-only change
cc080fdd, the measured helper took 1.9% more tokenizer time with 8-character identifiers and 2.4% less with 128-character identifiers. The word-only change took 10.5%/9.2% less time than the original baseline; the helper was not measured directly against that baseline in the same batch.These are medians of paired mean-time ratios from ten alternating pairs of the committed MySQL identifier benchmarks, with spans and output destruction included. Both comparisons used Linux on a Ryzen 5 3600, Rust 1.98.1, a fixed CPU and the performance governor. Within each comparison, both arms used identical locks and default features. Adapter throughput was not measured.
Original baseline:
1ea2488a632762e85f2cdb1c8bbe08f80800338e; word-only:cc080fdd3cccc379622370c13073b3fe98fdcfaf; measured helper:3a150f27b5650585ca1b99a252870f29111b310f. Copy the added benchmark into the original baseline.Related: apache#2036, apache#1557. AI-assisted.