Description
UniqToken currently evaluates English, Python code, Indic (Hindi/Telugu/Tamil/Bengali), CJK, Arabic, and European agglutinative languages (Turkish, Finnish). We want to expand our empirical benchmark suite to include African agglutinative and morphologically rich languages like Swahili and Yoruba.
Scope of Work
- Add raw text evaluation samples for Swahili and Yoruba to
BENCHMARK_CORPORA in benchmarks/benchmark_suite.py.
- Run
python benchmarks/benchmark_suite.py to calculate Bytes/Token and Fertility metrics.
- Verify zero fallback regressions.
Getting Started
- Files to edit:
benchmarks/benchmark_suite.py
- Test command:
python benchmarks/benchmark_suite.py
Description
UniqToken currently evaluates English, Python code, Indic (Hindi/Telugu/Tamil/Bengali), CJK, Arabic, and European agglutinative languages (Turkish, Finnish). We want to expand our empirical benchmark suite to include African agglutinative and morphologically rich languages like Swahili and Yoruba.
Scope of Work
BENCHMARK_CORPORAinbenchmarks/benchmark_suite.py.python benchmarks/benchmark_suite.pyto calculate Bytes/Token and Fertility metrics.Getting Started
benchmarks/benchmark_suite.pypython benchmarks/benchmark_suite.py