Replace tree-sitter
bindings with tree-house
#12972
Open
+2,350
−4,263
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR drops our dependency on the upstream
tree-sitter
Rust bindings and moves us to a new highlighter crate we've been working on:tree-house
.The main benefit of the new highlighter is that it fixes correctness issues, especially since the reverse query precedence PR. It solves longstanding issues #1151 and #6491, and updates us to the latest tree-sitter C source (v0.22 -> v0.25), which will become important as more grammars adopt the new ABI. It's also a slight performance boost in my benchmarking (5-10%). We've hoped to separate out our highlighting machinery for a while so that other applications could take advantage and the hope is that
tree-house
is the right way to expose it (\cc @hadronized).The new highlighter crate makes some abstractions clearer. The types to care about are already established in the
syntax
module:Loader
, an ECS-style holder of all languages (pub struct Language(pub u32)
). This stores the language configuration as well as any grammar and query information and allows looking up a language by file-type, language name, shebang, etc.. Instead of being stored onSyntax
this is now stored separately onDocument
and passed toSyntax
for updates. In the long run we can eventually store a singleArcSwap<syntax::Loader>
onEditor
instead of many clones ofArc<ArcSwap<syntax::Loader>>
.Syntax
, a wrapper of theSyntax
type fromtree-house
. It holds a tree of trees of the injection layers of a document and provides logarithmic lookups to determine an injection layer (including the layer's language, config, tree and queries) for a given byte range. It also has a query iterator that works across injections which can support indents, textobjects, etc. in the future.There are some differences compared to
master
that are less significant:helix_core::syntax::config
. This is not strictly necessary for this change but it seems like a good opportunity to move these largely unrelated types away from the syntax module.u32
s instead ofusize
. This causes some changes inhelix-core
and the commands modules but is otherwise an implementation detail.tree_sitter::Point
in the indents module has been replaced with byte indices. Generally we can avoid the point types and functions.syntax::generate_edits
for example now produces zero points.TreeCursor
type that acts over injection layers is built intotree-house
and is basically free to create - it doesn't require any extra sorting of layers. That type intree-house
uses theTreeCursor
type from the C library and so it should be more efficient.Highlight
type comes fromtree-house
now and is represented as a non-max u32 (enables the null pointer optimization forOption<Highlight>
).... and some other changes that are worth reviewing:
syntax::merge
in favor of a cursor-like API that matches the highlighter. See theOverlayHighlights
andOverlayHighlighter
types in the rewrittensyntax
module.cargo xtask query-check
can now point out unknown properties like(#set! priority ..)
and unknown predicates like(#has-ancestor? ..)
and has a nice visual output. This especially benefits the indent query which previously used runtime panics for the same effect, seeIndentQuery
.Example query compilation errors...
Fixes #1151
Fixes #3391
Fixes #6491