This is an LLM that I created from scratch.

How to use this code:

If you want to simply test out the end result, download "LLM_Inference" and both ".pt" weights files in "Weights" in the Releases section. Make sure to change the file paths for the weights accordingly in the "LLM_Inference" Jupyter Notebook. Afterwards, simply run the Jupyter Notebook Cells in order.

If you want to run the training loop, download "LLM_Training" and all the data files in the "Data" Releases section. You will need to run the following Linux command once you get into the section where the weight files are stored on your computer: "cat TinyStoriesV2-GPT4-train.txt.part-* > TinyStoriesV2-GPT4-train.txt" in order to include the entire dataset. Make sure to change the file paths for the weights accordingly in the "LLM_Training" Jupyter Notebook.

If you want to view my original codebase, download the file "LLM"

Information about the model:

Context Length: 256 Vocabulary Size: 512 Hidden Dimension: 512 Number of Layers: 8 Number of Heads: 8 Batch Size During Training: 16