Add Simple Token Counter

[mhpenta]

Adding a simple token counter would allow a number of quality of life improvements in Crush and other applications using Fantasy:

1. Pre-flight validation to avoid context errors.
2. Tool call / result warnings, such as alerting when MCP tool outputs are large.
3. Opening the door to different summarization/compaction patterns.

Proposal

This requires a simple token counter - although, in theory, we could also use something like github.com/tiktoken-go/tokenizer.

In practice, we simply don't need to be that accurate. Additionally, creating a very simple approach in Fantasy allows for easily evolving to more accurate approaches in the future.

  // tokens.go

  // TokenEstimator provides fast approximate token counts.
  type TokenEstimator struct{}

  // NewTokenEstimator creates a new token estimator.
  func NewTokenEstimator() *TokenEstimator {
      return &TokenEstimator{}
  }

  // Estimate returns an approximate token count for the given text.
  // Uses ~4 bytes per token heuristic, not runes, scales better for foreign languages 
  func (e *TokenEstimator) EstimateTokens(text string) int {
      return (len(text) + 3) / 4
  }

Could be interfaced with:

type TokenEstimator interface {
	EstimateTokens(text string) int
}

Why Fantasy versus Crush

Since this is a general LLM utility, it seems more correct to put it on Fantasy itself. Besides, I can envision making small additions to this over time to make its estimates more accurate.

Alternatives

The alternative would be to use github.com/pkoukk/tiktoken-go but, in my view, this just isn't necessary and is an unnecessary dependency.

Opencode itself does something similar to this.

Summarization / Compaction

In Crush, if we can estimate tokens, we can potentially follow different compaction patterns, allowing conversation turns over 80% context or allowing the user to configure when summarization / compaction begins.