[FEATURE] Evaluate `gigatoken` as tokeniser

[MacLemon]

Problem

gigatoken is a new tokeniser claiming speed improvements in orders of manitude. A significantly faster tokenizer means better performance overall.

• gigatoken

llamacpp is already working on this:

• llamacpp: Research: Tokenizer with faster string search
  #21575 concludes the use of gigatoken.
• llamacpp: Feature Request: Gigatoken Tokenizer #26139
  This enhancement is expected to trickle down to Ollama eventually.

vllm-project i working on this as well:

• vllm-project: [RFC] [Feature] [Experimental] Support GigaToken Accelerated Tokenizer Mode
  #49411

Proposal

Evaluate the possibility to incorporate gigatoken as tokeniser to further improve performance.

[youssofal]

interesting. tokenizer wall-time isn't the bottleneck on our agent lane today (prefill + decode dominate), but orders-of-magnitude claims are cheap to bench. queued as a research item, not a release item.