Add JUnit 5 support and initial unit test for Tokenizer

[ayush0407]

This PR adds initial testing support to the project by integrating JUnit 5 and creating the first unit test for the Tokenizer class.

Changes:

• Added JUnit 5 dependency in pom.xml
• Created TokenizerTest.java in src/test/java/llama
• Added a simple test to validate the tokenization of a basic prompt

This sets the foundation for adding more tests across the codebase.

[CLAassistant]

All committers have signed the CLA.

[Copilot]

Pull Request Overview

This PR integrates JUnit 5 into the project and adds the first unit tests for the Tokenizer class to establish a testing foundation.

• Added JUnit 5 dependency in pom.xml
• Created TokenizerTest.java with basic tokenization tests

Reviewed Changes

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

File	Description
pom.xml	Added JUnit 5 dependency for unit testing support
src/test/java/com/example/tokenizer/impl/TokenizerTest.java	Introduced initial test class covering core tokenizer methods

Comments suppressed due to low confidence (1)

src/test/java/com/example/tokenizer/impl/TokenizerTest.java:39

• [nitpick] Add tests for edge cases (e.g., empty or null inputs) to encodeOrdinary to ensure the tokenizer handles them without errors.

void testEncodeOrdinary() {

[mikepapadim]

Thank you for your contribution @ayush0407

I have a few comments:

• the changes in pom file are missing
• we need to have some instructions how to run the tests
• we need to wait for #17 to be merged first as it refactors a bit the functionality of Tokenizer class to support multiple tokenizers

[ayush0407]

Thank you for your contribution @ayush0407

I have a few comments:

• the changes in pom file are missing
• we need to have some instructions how to run the tests
• we need to wait for [model] Add support for Mistral models #17 to be merged first as it refactors a bit the functionality of Tokenizer class to support multiple tokenizers
  Sure I forgot to Push the pom file with the JUnit dependency.

[ayush0407]

Ill fix the Rest by Eod today thanks for the Suggestion

[ayush0407]

Hi @mikepapadim, Review changes completed.

[mikepapadim]

hello @ayush0407, it still missing instructions on how to run the test and the expected output, also a test for mistral tokenizer. Can you possibly add these? thanks

[ayush0407]

hello @ayush0407, it still missing instructions on how to run the test and the expected output, also a test for mistral tokenizer. Can you possibly add these? thanks

sure.

[ayush0407]

Hi @mikepapadim, please review the updated PR.

[mikepapadim]

Thank you @ayush0407

[orionpapadakis]

Nice work, @ayush0407! 👍

To make the JUnit support more complete and meaningful, I suggest the following improvements:

1. Add unit tests for LlamaTokenizer methods.
2. Include more targeted checks, prioritizing the encode and decode methods for both LlamaTokenizer and MistralTokenizer.

This will help ensure correctness and consistency across tokenizer implementations.

[orionpapadakis]

I would expect a check based on the expected result rather than checking if it's not null and empty.

[orionpapadakis]

regexPattern() should not return null

[orionpapadakis]

This TokenizerTest covers the static replaceControlCharacters() methods which are implemented in the Tokenizer interface, not LlamaTokenizer. In order this junit support to make sense I would suggest to add coverage for LlamaTokenizer as well.

[ayush0407]

sure