A PyTorch-based reproduction of OpenAI's GPT-2 model, inspired by the original research paper and popular tutorials by Andrej Karpathy, Vukrosic & Tunadorable.
This repository is a massive overhaul of Modded-NanoGPT, NanoGPT, TinyLlama, Meta's Lingua, and GPT Lab, designed to be an accessible, from-scratch base for amateurs to conduct cost-effective and rapid LLM research at a scale sufficient for ArXiv-worthy experiments. Existing lightweight GPT implementations often suffer from outdated architectures, narrow use cases, hidden dependencies, high computational costs, or complexity that hinders quick iteration—this project strikes a unique balance, offering a clean, modular, and affordable platform without sacrificing performance or educational clarity.
Under the hood, this project faithfully reproduce OpenAI's GPT-2 transformer architecture in PyTorch, paired with a custom BPE tokenizer, end-to-end training scripts, and inference tools. The goal is to enable anyone with consumer-grade hardware or low-cost cloud GPUs to experiment with large language models and contribute novel ideas to the field.
- GPT-2: Language Models are Unsupervised Multitask Learners (Radford et al., 2019)
Read the paper - Andrej Karpathy’s Tutorials
- Reproducing GPT-2 (124M): https://youtu.be/l8pRSuU81PU?si=5FuXWc6VJghGZBr0
- Building the GPT Tokenizer: https://youtu.be/zduSFxRajkE?si=FtokVhpIZZPZSzOR
- GPT Lab by Evintunador: https://github.com/evintunador/gpt-lab
- Tunadorable
- Talking thru GPT-Lab w/ @vukrosic : https://www.youtube.com/live/WDxdXn3tsNw?si=0VXl624Qjqshu3kY
- Train your own GPT experiments easy & cheap : https://youtu.be/4cvBgHMDISs?si=CFcfR72ahrS5CRFD
-
Fork or Template
Fork this repository or use it as a GitHub template to start your own project. -
Clone locally
git clone https://github.com/prathamesh-mandavkar/gpt2-implementation.git cd gpt2-implementation -
Virtual environment (recommended)
python3 -m venv venv source venv/bin/activate -
Install dependencies
pip install -r requirements.txt
Train a BPE tokenizer on the FineWeb corpus. Adjust --samples (total characters, split evenly across GPUs) and --vocabsize (excluding special tokens).
- Single GPU
python tokenizer/train_tokenizer.py \ --samples 100000 \ --vocabsize 1000 \ --name readmetokenizer \ --demo
- Multi-GPU (replace
Gwith GPU count)torchrun --nproc_per_node=G tokenizer/train_tokenizer.py \ --samples 100000 \ --vocabsize 1000 \ --name readmetokenizer \ --demo
For a detailed walkthrough of BPE tokenizers on CPU, see Andrej Karpathy’s video.
-
Download FineWeb dataset (versions:
10B,100B,10Bedu,100Bedu)python scripts/download_fineweb.py \ --version 10B \ --num_shards 1 \ --shard_size 10000000 \ --tokenizer readmetokenizer_v1000_n100000.pkl
shard_size: number of tokens per shardnum_shards: total training shards (validation uses 1 shard by default)
-
Download benchmarks
python scripts/download_hellaswag.py
Train GPT-2 on prepared shards. Vocabulary size = tokenizer size + special tokens (e.g., 1000 + 1).
- Single GPU
python train_gpt.py \ --model_name ReadmeGPT \ --tokenizer readmetokenizer_v1000_n100000.pkl \ --vocab_size 1001 \ --model_dim 128 \ --num_heads 4 \ --num_layers 6
- Multi-GPU (replace
Gwith GPU count)torchrun --nproc_per_node=G train_gpt.py \ --model_name ReadmeGPT \ --tokenizer readmetokenizer_v1000_n100000.pkl \ --vocab_size 1001 \ --model_dim 128 \ --num_heads 4 \ --num_layers 6
Warning: Using
--save_modelwill create a.ptcheckpoint. By default,.ptfiles are ignored by.gitignoreto prevent large uploads.
Generate samples from a trained checkpoint:
python generate.py \
--checkpoint experiments/ReadmeGPT/<latest>.pt \
--prompt "Once upon a time" \
--max_length 100 \
--temperature 0.9 \
--top_k 50Contributions welcome! Please open issues or PRs for bugs, features, or performance improvements. Follow PEP8, include tests, and document new functionality.