π BOOK β AI Engineering (Huyen, 2024)
Rating: 8/10 | Beginner to Intermediate AI/Software Engineers
A good end-to-end overview of the field.
As I read through the material, below are my highlights β with a moderate attempt at coherence β for my personal inventory (and perhaps as a way to connect with other friendly AI scientists and builders out there who love talking about the same things). As always, please remember our rules:
π€ If these notes look cool to you, it's your β¦moral dutyβ¦ to read the original resources.
π€ Just another blog, but not your usual blog: they're living notes that I change as I fancy.
The AI Engineering Stack
- The stack has three layers: Application Development (prompts, context, evaluation, interfaces), Model Development (modeling, training, finetuning, dataset engineering), and Infrastructure (model serving, compute, monitoring).
- AI engineering differs from traditional ML engineering in three key ways:
1. You use a pre-trained model instead of training your own.
2. Models are bigger, consume more compute, and have higher latency.
3. Models produce open-ended outputs, making evaluation much harder.
- In short: AI engineering is less about model development and more about adapting and evaluating models.
Model Adaptation
- Adaptation techniques fall into two categories: prompt-based (no weight updates) and finetuning (weight updates required).
- Prompt engineering is easier to start, requires less data, and lets you experiment across many models β but may not be enough for complex tasks.
- Finetuning can significantly improve quality, latency, and cost, but is more complicated and data-hungry.
- Training phases:
1. Pre-training: trains from scratch; most resource-intensive (up to 98% of compute for InstructGPT).
2. Finetuning: continues training a previously trained model; requires fewer resources.
3. Post-training: conceptually the same as finetuning, but done by model developers (e.g., OpenAI) to align behavior before release.
- Post-training typically consists of two steps:
1. Supervised Finetuning (SFT) β optimizes for conversations instead of text completion.
2. Preference Finetuning β aligns outputs with human preferences via RL (RLHF, DPO, RLAIF).
Transformer Architecture
- The transformer was created to address two problems with seq2seq + RNNs: (1) outputs were generated using only the final hidden state; (2) sequential processing was slow for long inputs.
- The attention mechanism lets the model weigh the importance of different input tokens when generating each output token.
- Attention uses three vectors: Query (Q) (current decoder state), Key (K) (token identifiers), Value (V) (token content).
- Attention is almost always multi-headed, allowing the model to attend to different groups of tokens simultaneously.
- Inference for transformer-based LLMs consists of two steps:
1. Prefill: processes all input tokens in parallel β **compute-bound**.
2. Decode: generates one output token at a time β **memory bandwidth-bound**.
- A transformer model's size is determined by: model dimension, number of transformer blocks, feedforward layer dimension, and vocabulary size.
- The embedding module converts tokens and positions into vectors; it determines max context length.
- The output layer (unembedding layer / model head) maps output vectors into token probabilities.
Model Sizing & Compute
- To estimate inference memory: multiply number of parameters Γ bytes per parameter (e.g., 7B params Γ 2 bytes = 14 GB minimum).
- Sparse models have a high percentage of zero-value parameters, enabling more efficient storage and computation.
- Mixture-of-Experts (MoE): model is divided into expert groups; only a subset of experts is active per token (e.g., Mixtral uses 46.7B total params but only 12.9B active per token).
- FLOPs measure compute requirement for a task; FLOP/s measures hardware peak performance (e.g., NVIDIA H100 NVL delivers ~60 TeraFLOP/s).
- Model performance depends on both model size and dataset size β bigger means more compute, and compute costs money.
- The number of training tokens β the number of tokens in a dataset (training for 2 epochs on 1T tokens = 2T training tokens).
- KV cache is used only during inference (not training); its memory grows linearly with sequence length and batch size.
- Formula:
2 Γ B Γ S Γ L Γ H Γ M
- Formula:
Alternative Architectures
- SSMs (State Space Models) show promise for long-range memory as an alternative to transformers.
- Evolution of SSMs: S4 β H3 β Mamba (scales to 3B params; inference scales linearly vs. quadratic for transformers) β Jamba (hybrid TransformerβMamba; 52B total / 12B active params, fits in a single 80 GB GPU, supports 256K context length).
Sampling & Output Generation
- A language model outputs a logit vector; each logit corresponds to one token in the vocabulary.
- Temperature controls creativity: higher temperature β rarer tokens more likely β more creative but less coherent; lower temperature β more consistent but potentially boring.
- Constrained sampling filters the logit vector to only tokens that meet defined constraints, used for structured outputs.
- Perplexity = exponential of entropy/cross-entropy; measures the model's uncertainty when predicting the next token.
1. More structured data β lower perplexity.
2. Bigger vocabulary β higher perplexity.
3. Longer context β lower perplexity.
- Entropy measures the average information carried by a token; higher entropy = more bits needed to represent a token.
Evaluation
- Evaluation is described as the biggest bottleneck to AI adoption.
- Evaluation is needed throughout: model selection, benchmarking progress, deployment decisions, and production monitoring.
- Evaluation-Driven Development: successful enterprise applications tend to be those with clear, measurable evaluation criteria (recommender systems, fraud detection, code generation, classification tasks).
- The four-step evaluation workflow: (1) filter by hard attributes, (2) use public benchmarks to shortlist, (3) run internal experiments, (4) monitor in production.
- Evaluation criteria fall into buckets: domain-specific capability, generation capability, instruction-following capability, cost, and latency.
- Data contamination detection methods: n-gram overlapping and perplexity (unusually low perplexity suggests the model has seen the data before).
- When designing an evaluation pipeline: check for variance across runs, avoid redundant correlated metrics, and account for latency/cost overhead.
Prompt Engineering & In-Context Learning
- Teaching models via prompts is called in-context learning.
- Each example in the prompt is a "shot" β few-shot, zero-shot, 5-shot, etc.
- Prompt engineering adjusts model behavior without changing weights.
- How much prompt engineering is needed depends on the model's robustness to prompt perturbation.
Prompt Security
- Three main types of prompt attacks: prompt extraction, jailbreaking / prompt injection, and information extraction.
- Risks include: remote code/tool execution, data leaks, social harms, misinformation, service interruption, and brand risk.
- Adversarial purposes include: data theft, privacy violations, and copyright infringement.
RAG (Retrieval-Augmented Generation)
- RAG enhances generation by retrieving relevant information from external memory (databases, chat history, the internet).
- Think of RAG as dynamic context construction β equivalent to feature engineering in classical ML.
- A RAG system has two components: a retriever (indexing + querying) and a generator.
- Retrieval approaches:
1. Term-based: lexical matching; generally faster.
2. Embedding-based (semantic): ranks by meaning alignment; requires vector database + ANN search.
3. Hybrid search: combines both for production systems.
- Vector search is framed as a nearest-neighbor search problem; large datasets use Approximate Nearest Neighbor (ANN) algorithms.
- RAG quality should be evaluated both component-by-component and end-to-end (retrieval quality, embedding quality, final outputs).
- Retrieval tactics: chunking strategy, reranking, query rewriting, contextual retrieval.
- In short: finetuning is for form; RAG is for facts.
Memory in AI Models
- AI models have three memory mechanisms:
1. Internal knowledge β encoded in weights during training; doesn't change unless the model is updated.
2. Short-term memory β the context window; fast but capacity-limited; doesn't persist across queries.
3. Long-term memory β external data sources via retrieval (e.g., RAG); persistent and deletable without retraining.
Finetuning (Deep Dive)
- Finetuning = adapting a model by further training all or part of it.
- Backpropagation powers training: forward pass (compute output) + backward pass (update weights via gradients).
- Training memory = model weights + activations + gradients + optimizer states.
- Full finetuning of a 7B model with Adam optimizer in FP16 requires ~56 GB β exceeds most consumer GPUs (12β48 GB).
- PEFT (Parameter-Efficient Finetuning) methods reduce trainable parameters:
- Adapter-based (additive): adds modules to the model.
- LoRA (Low-Rank Adaptation): decomposes weight matrices into two smaller matrices; merges back without extra inference latency.
- LoRA is built on low-rank factorization (e.g., a 9Γ9 matrix β 9Γ1 and 1Γ9; 81 β 18 params). Higher rank = more information preserved, but more parameters.
- LoRA simplifies serving multiple finetuned models due to its modularity.
- Task vectors / delta parameters: subtracting the base model from a finetuned model captures the "essence" of the task; can be linearly combined for model merging.
- Distillation: finetune a small model to imitate a larger model using data generated by the large model.
- Catastrophic forgetting: neural networks can forget previous tasks when sequentially finetuned on new ones.
- Finetuning hyperparameters:
1. Learning rate: step size toward the goal β too small = slow convergence; too large = overshooting.
2. Batch size: fewer than 8 can cause unstable training; larger batches produce more stable updates.
3. Gradient accumulation: accumulate gradients across batches when memory limits batch size.
4. Epochs: large datasets may need 1β2 epochs; small datasets may benefit from 4β10.
5. Prompt loss weight: response tokens should contribute more to the loss than prompt tokens during instruction finetuning.
Data & Dataset Engineering
- Data-centric AI improves performance by enhancing data quality and diversity; contrasts with model-centric AI (better architectures, bigger models).
- Data needed for finetuning depends on: finetuning technique (full vs. PEFT), task complexity, and base model's starting performance.
- Deduplication methods: pairwise comparison (exact/n-gram/fuzzy/semantic), hashing (MinHash, Bloom filter), dimensionality reduction + comparison.
Inference Optimization
- Autoregressive models generate tokens sequentially β 10 ms/token Γ 100 tokens = 1 second total latency.
- Latency metrics:
1. TTFT (Time to First Token): duration of the prefill step; important for chatbots.
2. TPOT (Time Per Output Token): ~120 ms/token (6β8 tokens/second) is sufficient for most reading speeds.
3. Total latency = TTFT + TPOT Γ (number of output tokens).
- GPU utilization (nvidia-smi) measures % of time the GPU is active β not how efficiently it's working. 100% utilization doesn't mean 100% efficiency.
- Four computation optimization techniques: vectorization, parallelization, loop tiling, operator fusion (the most architecture-specific).
- Major AI accelerators beyond NVIDIA: AMD GPUs, Google TPU, Intel Habana Gaudi, Graphcore IPU, Groq LPU, Cerebras QPU.
AI Application Architecture
- A production AI system evolves incrementally:
1. Add external data sources and tools for richer context.
2. Add guardrails to protect users and the system.
3. Add model router and gateway for complex pipelines and security.
4. Add caching to optimize latency and cost.
5. Add complex logic and write actions for maximum capability.
Miscellaneous
- Lindy's Law: the future life expectancy of a technology is proportional to its current age β if it's been around a while, it'll likely stick around.
- Embeddings: numerical representations that capture the meaning of data.
- Quantization: reduces the precision of model weights β technically changes weight values but is not considered training.
- Tasks are either compute-bound (bottlenecked by computation speed) or memory bandwidth-bound (bottlenecked by data transfer rate); the distinction matters for optimization strategy.