Expose Tantivy 0.26.0 features: aggregations, tokenizers, regex, sort, bytes, count, MLT

[nyo16]

Summary

• Switch Tantivy from git pin (commit 51f340f) to crates.io 0.26.0 release
• Extend FieldDef tuple from 4 to 6 elements to support fast and tokenizer options
• Add 7 new features across schema, query, and analytics layers
• 229 tests (54 new), 0 failures, clippy clean

New Features

Schema

Feature	API	Description
Bytes field	Schema.add_bytes_field/3	Binary data storage/retrieval via rustler::Binary
Custom tokenizers	add_text_field("f", tokenizer: "en_stem")	Per-field tokenizer: default, raw, en_stem, whitespace
Fast fields	add_u64_field("f", fast: true)	Columnar storage for sort & aggregations

Queries

Feature	API	Description
Count	Searcher.count/3	Lightweight doc count without retrieval
Regex	Searcher.search_regex/4	Programmatic RegexQuery on text fields
MoreLikeThis	Searcher.search_more_like_this/3	Find similar documents by term distribution
Sort by field	Searcher.search_query_sorted/5	Sort by fast field value instead of BM25 score

Aggregations

Feature	API	Description
JSON pass-through NIF	Searcher.aggregate/5	Single DirtyCpu-scheduled NIF for all aggregation types
Builder DSL	Aggregation.Bucket / Aggregation.Metric	Terms, range, histogram, filter + avg, sum, min, max, stats, cardinality, percentiles

Example Usage

# Custom tokenizer + fast field
schema = Schema.new()
  |> Schema.add_text_field("title", stored: true, tokenizer: "en_stem")
  |> Schema.add_text_field("category", stored: true, tokenizer: "raw", fast: true)
  |> Schema.add_f64_field("price", stored: true, fast: true)

# Count without fetching docs
{:ok, count} = Searcher.count(searcher, "elixir", ["title"])

# Sort by price descending
{:ok, results} = Searcher.search_query_sorted(searcher, "*", ["title"], "price", reverse: true)

# Aggregation: avg price per category
alias Muninn.Aggregation
alias Muninn.Aggregation.{Bucket, Metric}

aggs = Aggregation.new()
  |> Aggregation.add("by_category",
    Bucket.terms("category", size: 10)
    |> Aggregation.sub("avg_price", Metric.avg("price"))
  )

{:ok, results} = Searcher.aggregate(searcher, "*", ["title"], aggs)

Test Plan

- All 175 existing tests pass (backward compatible)
- 54 new tests covering bytes, tokenizers, count, regex, MLT, sort, aggregation builder, aggregation integration
- cargo clippy clean
- mix format + cargo fmt clean