A PyTorch implementation of Meta's LLaMA 3 architecture. This repository contains code for training and evaluating a smaller version of the LLaMA 3 language model.
This project implements the LLaMA 3 architecture from scratch using PyTorch. It includes:
- Tokenizer implementations trained on the Tiny Shakespeare dataset
- Model training code and pre-trained weights
- Inference and evaluation utilities
The implementation is based on insights from the LLaMA 3 research paper and inspired by similar efforts to reproduce transformer-based language models.
llama3-implementation/
│
├── models/ # Trained model checkpoints
│ ├── Llama3_2024-10-05_23-12-27.json # Model configuration
│ └── Llama3_2024-10-05_23-12-27.pth # Model weights
│
├── tokenizers/ # Tokenizer models with different vocabulary sizes
│ ├── tiny_shakespeare_tokenizer_128.model
│ ├── tiny_shakespeare_tokenizer_256.model
│ ├── tiny_shakespeare_tokenizer_512.model
│ └── tiny_shakespeare_tokenizer_1024.model
│
├── tiny_shakespeare_tokenizer.py # Tokenizer implementation
├── input.txt # Sample input text
└── llama3_implementation.ipynb # Training and evaluation notebook
- Python 3.8+
- PyTorch 2.0+
- Jupyter Notebook (for running the included notebook)
- Clone the repository:
git clone https://github.com/prathamesh-mandavkar/llama3-implementation.git
cd llama3-implementation- Install the required dependencies:
pip install torch numpy tqdm matplotlib jupyterExecute the training code in llama3_implementation.ipynb or use the provided pre-trained models in the models/ directory.
The model was trained on the Tiny Shakespeare dataset, following architectural principles from the LLaMA 3 paper. The implementation includes attention mechanisms, positional encodings, and feed-forward networks characteristic of modern transformer-based language models.
This implementation draws inspiration from:
- LLaMA 3 Research Paper
- Andrej Karpathy's "Let's reproduce GPT-2 (124M)"
- Let's build the GPT Tokenizer
- Tunadorable
This project is available under the MIT License - see the LICENSE file for details.
Contributions are welcome! Please feel free to submit a Pull Request.
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add some amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request