π BOOK β Build an LLM from Scratch (Raschka, 2024)
Rating: 9/10 | Beginner to Intermediate AI/Software Engineers

This book walks through the full lifecycle of building a large language model: from raw text tokenization all the way to classical fine-tuning examples.
I enjoyed this read: although it's introductory, it has great explanations, great pictures (visual learners winning here), and fun Python exercises. I read it in 3 days, and it helped me refresh some key concepts in my mind.
The journey is structured as the following:
| Stage | Focus |
|---|---|
| 1 | Core mechanics: data, attention, architecture (attention, transformers), pretraining |
| 2 | Foundations: training loop, evaluation, loading pre-trained weights |
| 3 | Fine-tuning: classification or instruction-following |
As I read through the material, below are my highlights β with a moderate attempt at coherence β for my personal inventory (and perhaps as a way to connect with other friendly AI scientists and builders out there who love talking about the same things). As always, please remember our rules:
π€ If these notes look cool to you, it's your β¦moral dutyβ¦ to read the original resources.
π€ Just another blog, but not your usual blog: they're living notes that I change as I fancy.

The Pipeline for Working with Text Data
1. Tokenization; split text into tokens
2. Byte Pair Encoding (BPE); the tokenization scheme used by GPT-2/3 (using Python's tiktoken)
- start with individual characters and iteratively merges the most frequent adjacent pairs into subwords and words
3. Token ID; each token maps to an integer in the vocabulary
4. Embeddings; token IDs are looked up in a learned weight matrix to produce continuous vectors the network can process
- a fancy lookup table for which the lookup weights get optimized during training
Attention
Self-attention is the heart of the transformer. For each token in the sequence, compute how much it should "attend to" every other token, then blend those tokens' information accordingly. The mechanics use three learned projections per token (just as in database lingo):
- Query (Q) β "what am I looking for?"
- Key (K) β "what do I offer?"
- Value (V) β "what information do I carry?"
Attention scores are computed as dot products between queries and keys, scaled by the square root of the embedding dimension (to prevent gradients from vanishing with large values), then passed through softmax to produce weights that are normalized.
GPT Model Architecture
The full GPT model stacks transformer blocks repeatedly:
- Masked multi-head attention
- Masking ensures the model can only attend to past tokens during training (masked positions are set to negative infinity before softmax)
- Multi-head attention runs several attention operations in parallel with different learned projections, then concatenates the results
- The smallest GPT-2 has 12 heads over a 768-dimensional embedding space
- Layer normalization (applied before each sub-layer)
- Feed-forward network with GELU activation.
- GELU is smoother around zero than ReLU, and allows small negative values, which improves gradient flow during training
- Shortcut (residual) connections
- Deep networks suffer from vanishing gradients β by the time error signals propagate back through many layers, they shrink toward zero
- Residual connections give gradients a shortcut, preserving signal and enabling much deeper networks to train
GPT-2 model sizes for reference (their weights are open):
| Variant | Parameters | Attention Heads | Embedding Dim | Context Length |
|---|---|---|---|---|
| Small | 124M | 12 | 768 | 1,024 |
| Medium | 355M | 16 | 1,024 | 1,024 |
| Large | 774M | 20 | 1,280 | 1,024 |
| XL | 1.5B | 25 | 1,600 | 1,024 |
Pretraining && Evaluation

Pretraining is conceptually straightforward: minimize cross-entropy loss between the model's predicted token probabilities and the actual next token.
In practice:
- Cross-entropy loss (measuring divergence between predicted and true token distributions)
- Perplexity = exp(loss) (as a more interpretable metric)
- AdamW optimizer (as the standard choice for better regularization)
- Temperature scaling and probabilistic sampling introduce controlled randomness, trading coherence for diversity
At inference time, GPT generates one token at a time:
1. Feed the input sequence through the model
2. Convert the final output logits to probabilities via softmax
3. Sample or greedily select the next token (greedy decoding always picks the highest-probability token)
4. Append it to the sequence and repeat
Classification Fine-Tuning
The classical spam detection classification example:
- Replace the final output layer with a small classification head
- Freeze most of the model's layers
- lower layers encode broadly useful representations (syntax, semantics)
- upper layers are more task-specific and benefit from adaptation
- Only train the last transformer block, final LayerNorm, and the new classification head
- Use the same cross-entropy loss as pretraining
- Don't need to fine-tune the whole model (just the top layer(s) on a small labeled dataset)
What This Book Does Well
- Demystifies LLMs by building everything from first principles in PyTorch
- Bridges the gap between intuition and implementation (concepts like attention and tokenization become much clearer once you've coded them)
- The progression from "random weights β pretrained β fine-tuned" mirrors how real production models are built