🤖 transformer-from-scratch

A full Encoder-Decoder Transformer built from first principles using PyTorch.
Every component — from token embeddings to cross-attention — implemented manually, zero shortcuts.

Understanding the core concepts of the Transformer model and implementing it from first principles.

🗂️ Project Structure

transformer-from-scratch/
│
├── transformer/                # Core package
│   ├── __init__.py
│   ├── layers.py               # All building-block modules
│   ├── encoder.py              # Encoder stack
│   ├── decoder.py              # Decoder stack
│   ├── transformer.py          # Full Transformer model
│   └── utils.py                # Causal mask generation
│
├── docs/                       # Handwritten derivation notes (PDF)
│   └── transformer-notes.pdf
│
├── run.py                      # Training script
├── test.py                     # Inference / testing script
├── pyproject.toml
└── README.md

---

🧱 Layers (layers.py)

Module	Role
TokenEmbedding	Learned embeddings scaled by √d_model
PositionalEncoding	Sinusoidal position encodings (sin/cos)
InputBlock	TokenEmbedding + PositionalEncoding combined
SelfAttention	Single-head scaled dot-product attention
MultiHeadAttention	Multi-head self-attention + linear projection
MaskedMultiHeadAttention	Causal masked MHA (decoder self-attention)
CrossAttention	Decoder queries attend to encoder keys/values
FeedForward	Position-wise FFN: Linear → ReLU → Linear (4× expansion)

---

⚙️ Setup

Requires Python >=3.11 and uv

git clone https://github.com/anikchand461/transformer-from-scratch
cd transformer-from-scratch
uv sync

Run training:

uv run run.py

Run inference/test:

uv run test.py

---

📝 Implementation Notes

• Positional Encoding — standard sinusoidal: sin for even indices, cos for odd indices across d_model dimensions
• Causal Mask — masked positions filled with -1e9 before softmax, effectively zeroing future token attention
• Cross-Attention — decoder hidden states → Q, encoder output → K and V
• Residual + LayerNorm — applied after every sub-layer (Post-LN, matching the original paper)
• Constraint — d_model must be divisible by num_heads

---

📚 Documentation

Detailed handwritten derivations, mathematical walkthrough, and step-by-step code explanations are in docs/transformer-notes.pdf.

---

📖 References

Attention Is All You Need — Vaswani et al., 2017  |  The Illustrated Transformer — Jay Alammar