Description
When running python cli.py train --corpus data.txt --vocab-size 32000 on multi-gigabyte files, terminal feedback is currently logged on EM iteration boundaries. Adding a dynamic progress bar will improve the developer experience.
Scope of Work
- Integrate
rich or tqdm in cli.py and unigram_trainer.py.
- Display real-time throughput (MB/s), EM step convergence $\Delta\mathcal{L}$, and current vocabulary pruning candidate count.
Getting Started
- Files to edit:
cli.py, unigram_trainer.py
- Test command:
python cli.py train --corpus tests/sample.txt --vocab-size 500
Description
When running
python cli.py train --corpus data.txt --vocab-size 32000on multi-gigabyte files, terminal feedback is currently logged on EM iteration boundaries. Adding a dynamic progress bar will improve the developer experience.Scope of Work
richortqdmincli.pyandunigram_trainer.py.Getting Started
cli.py,unigram_trainer.pypython cli.py train --corpus tests/sample.txt --vocab-size 500