📚 BOOK → LLM Engineer (Chaumond et al., 2024)

Rating: 9/10 | Beginner to Intermediate AI Engineers

Mid-weeks are a nice time to read a good book, and I like to review the ML/AI engineering “classics” because:

1️⃣ They are usually nice' n 'easy' n 'fun.
2️⃣ Even though a great part of the readings are just repeating or rephrasing knowledge I already know, I always learn something new, which expands my toolkit.
3️⃣ Finishing a good book (and they usually take me 1-3 days) has a gratifying feeling of getting things done.
4️⃣ It's always entertaining to get inside the minds of smart people in the field.
5️⃣ Every time I read a book, I take note of the parts I find interesting, then when I finish the book I go over the notes. This has been my learning technique my entire life and I enjoy it a lot. Repetition ➡️ Memorization.


Today I finished reading this book from the co-founders of HuggingFace and ZenML, which covers useful topics such as MLOps (DevOps) pipelines, RAG pipelines, fine-tuning pipelines, evals approaches, and inference optimizations.


As I read through the material, below are my highlights — with a moderate attempt at coherence — for my personal inventory (and perhaps as a way to connect with other friendly AI scientists and builders out there who love talking about the same things). As always, please remember our rules:

🤖 If these notes look cool to you, it's your ✦moral duty✦ to read the original resources.
🤖 Just another blog, but not your usual blog: they're living notes that I change as I fancy.







DATA ENGINEERING


  • Introduction and several examples of how to use ZenML for orchestrating ML pipelines, storing and versioning ML pipelines as outputs, and attaching metadata to artifacts for better observability (an artifact is any file produced during the machine learning lifecycle, such as datasets, trained models, checkpoints, or logs).
  • FTI pipeline: 1) Feature (Pandas, Polars, Spark, DBT, Flink, Bytewax); 2) Training (PyTorch, TensorFlow, Scikit-Learn, XGBoost, Jax); 3) Inference (PyTorch, TensorFlow, Skit-Learn, XGBoost, Jax).
  • Use Qdrant to store the data from MongoDB after it's processed and transformed (vector similarity search engine and vector database).
  • Use SageMaker for a fully managed machine learning service (AWS’s ML platform used to train and deploy models).



RAG FEATURES PIPELINE


  • Fundamental problems that RAG solves: 1) Hallucinations, 2) Old or private information -> Any LLM is bound to understand the data it was trained on (parameterized knowledge). Even if the LLM can perfectly answer what happened in the past, it won't have access to the newest data or any other external sources on which it wasn't trained.
  • A RAG system is composed of three modules independent of each other: 1) Ingestion pipeline (batch or streaming pipeline used to populate the vector DB), 2) Retrieval pipeline (queries the vector DB and retrieves relevant entries to the user's input), and 3) Generation pipeline (uses the retrieved data to augment the prompt and an LLM to generate answers - by the top K's most similar entries, comparing the embeddings from the vector storage with the user's input vector and by using a distance metric to compare two vectors (Euclidean, Manhattan, or cosine distance).
  • One-hot encoding converts categorical variables into a binary matrix representation. Each category is represented as a unique binary vector. For each categorical variable, a binary vector is created with a length equal to the number of unique categories, where all values are zero except for the index corresponding to the specific category, which is set to one. It can lead to a high-dimensional feature space if the categorical variable has many unique values, making the method impractical.
  • Feature hashing (hashing encoding): used to convert categorical variables into numerical features by applying a hash function to the category values. Compared to one-hot encoding, the method is not bound to the number of unique categories, but it reduces the dimensionality of the feature space by mapping categories into a fixed number of bins or buckets. Thus, it reduces the dimensionality of the feature space, which is particularly useful when dealing with high-cardinality categorical variables. This makes it efficient in terms of memory usage and computational time. There is a risk of collisions, where different categories might map to the same bin, leading to a loss of information. Difficult to understand the relationship between the original categories and the hashed features. Embeddings help us encode categorical variables while controlling the output vector’s dimension.
  • Change data capture (CDC): strategy that allows you to optimally keep two or more data storage types in sync without computing and I/O overhead. It captures any CRUD operation done on the source DB and replicates it on a target DB. Optionally, you can add preprocessing steps in between the replication.



SUPERVISED FINE-TUNING (SFT)


  • Post-training pipeline: data curation -> data deduplication -> data decontamination -> data quality evaluation -> data exploration.
  • Calculating an ideal number of samples -> difficult task -> both the quality of the data and the size of the model can have a dramatic impact. For large models (~ 70 billion parameters), this number can be as low as 1k high-quality samples.
  • The most popular approach to fuzzy deduplication is MinHash deduplication. It maintains high accuracy while significantly reducing computational complexity. MinHash operates by generating compact representations, or signatures, for each data item. These signatures serve as fingerprints that capture the essence of the data while drastically reducing its dimensionality. These signatures can be compared using similarity measures like Jaccard similarity to efficiently identify near-duplicates.
  • The LLM-as-a-judge strategy -> Different LLMs have different levels of performance across tasks, and their evaluations often align more closely with those of non-experts. With domain-specific datasets -> use domain-specific models instead of general-purpose LLMs.
  • Memory usage depends on several factors, including model size, training techniques, and optimization methods. At its simplest, using a single-GPU setting, the memory required can be estimated using the following formula: Memory = parameters + gradients + optimizer states + activations.
  • For a basic setup using 32-bit floating point (fp32) precision, we can estimate: 1) Parameters: Learnable weights and biases (weights in the attention mechanisms, feed-forward layers, and embedding layers - cost: 4 bytes/parameter (FP32) or 2 bytes/parameter (FP16/BF16)); 2) Gradients: the partial derivatives of the loss function with respect to each model parameter (cost: 4 bytes/parameter); 3) Optimizer states: additional values maintained by optimization algorithms like Adam or AdamW (help in adapting the learning rate for each parameter and navigating the loss landscape more effectively - cost: 8 bytes/parameter); 4) Activations: the intermediate outputs of each layer in the neural network during the forward pass(e.g., the outputs of attention mechanisms, feed-forward layers, and normalization layers - cost: variable, but often negligible for small batch sizes).
  • Several techniques can be employed to reduce memory usage during LLM fine-tuning: 1) Model parallelism spreads the workload across multiple GPUs, though it adds some overhead; 2) Gradient accumulation enables larger effective batch sizes without proportional memory increase; 3) Memory-efficient optimizers like 8-bit Adam can reduce the footprint of optimizer states; 4) Activation checkpointing trades computation for memory by recalculating certain activations. Using mixed precision with model parallelism might reduce costs to around 14-15 bytes per parameter, compared to the 16-byte baseline.
  • LoRA -> parameter-efficient technique for fine-tuning LLMs -> address the computational challenges associated with adapting massive neural networks. Introduces trainable low-rank matrices that modify the behavior of the model without changing its original parameters. Key advantages: 1) Dramatically reduced memory usage during training; 2) Faster fine-tuning process; 3) Preservation of pre-trained model weights (non-destructive); and 4) Ability to switch between tasks efficiently by swapping LoRA weights.
  • To implement LoRA effectively, we need to select the correct hyperparameters and target modules. LoRA comes with two hyperparameters: 1) Rank: Determines the size of the LoRA matrices (larger ranks capture more diverse tasks but could lead to overfitting); and 2) Alpha: scaling factor applied to the LoRA update.
  • Initially, LoRA was primarily focused on modifying the attention mechanism, specifically the query (Q) and value (V) matrices in transformer layers. However, LoRA dan also be applied to other key components of the model: 1) Key (K) matrices in attention layers; 2) Output projection layers (often denoted as O) in attention mechanisms; 3) Feed-forward or Multi-Layer Perceptron (MLP) blocks between attention layers; and 4) Linear output layers.
  • When fine-tuning LLMs, the learning rate is the most important hyperparameter -> control much the model's parameters are updated during training. It typically ranges from very small values like 1e-6 to larger values like 1e-3. A common starting point for transformer models is often around 1e-5. If the learning rate is too low, training progresses slowly and may get stuck in suboptimal solutions. Conversely, if it's too high, training can become unstable or diverge, leading to poor performance. The learning rate scheduler adjusts the learning rate throughout the training process. The two most common types of schedulers are linear and cosine.
  • The batch size determines the # of samples processed before the model's weights are updated. Typical sizes for fine-tuning range from 1 to 32, with common values being 1, 2, 4, 8, or 16. Larger batch sizes lead to more stable gradient estimates and can improve training speed, as they provide a better approximation of the true gradient of the entire dataset (however, they also require more memory, which can be a limiting factor on GPUs with less VRAM).
  • To overcome memory constraints from larger batch sizes -> use gradient accumulation can be used -> performs multiple forward and backward passes with smaller mini-batches, accumulating the gradients over these steps before applying a single update to the model’s parameters -> particularly useful when working with large models or limited GPU memory (e.g., if you want to achieve an effective batch size of 32 but your GPU can only handle 8 samples at a time: set the gradient accumulation steps to 4 -> you’ll process 4 mini-batches of 8 samples each, accumulating the gradients).
  • The number of epochs -> represents the number of complete passes through the entire training dataset -> for fine-tuning, the typical range is 1 to 10 epochs, with many successful runs using 2 to 5 epochs (more epochs allow the model to refine its learning, improving performance).
  • Optimizers adjust the model's parameters to minimize the loss function AdamW 8-bit performs comparably to the 32-bit version while using less GPU memory. It combines adaptive learning rates with weight decay regularization, often leading to better training stability and model performance.
  • Weight decay works by adding a penalty for large weights to the loss function, encouraging the model to learn simpler, more generalizable features. This helps the model avoid relying too heavily on any single input feature, which can improve its performance on unseen data.
  • Gradient checkpointing -> reduces memory consumption during training by storing only a subset of intermediate activations generated in the forward pass (can quickly become impractical due to hardware limitations, especially on GPUs with limited memory capacity).
  • Metrics important to monitor: 1) Training loss: measures how well the model is performing on the task it’s being trained for - it should continuously decrease on average; 2) Validation loss: measures the loss using the validation set instead of the training set; a well-fitted model typically shows both training and validation losses decreasing and eventually stabilizing, with a small gap between them; and 3) Gradient norm: represents the magnitude of the gradient vector during training (large gradient norms can indicate training instability like overfitting, especially if accompanied by a divergence between training and validation losses).



FINE-TUNE WITH PREFERENCE ALIGNMENT


  • Addresses the shortcomings of SFT by incorporating direct human or AI feedback into the training process.
  • RLHF works by iteratively improving both a reward model and a policy: 1) Reward model learning; 2) Policy optimization; and 3) Iterative improvement.
  • Direct preference optimization (DPO) -> reformulation of the preference learning problem. Unlike RLHF, which typically involves training a separate reward model and then using reinforcement learning algorithms like PPO to fine-tune the language model, DPO takes a more direct approach.
  • DPO has several advantages over traditional RLHF methods: 1) it simplifies the preference learning pipeline, reducing the engineering complexity associated with RLHF methods; 2) when trained with adapters (LoRA, QLoRA), the frozen and trained models don't have to be separated; 3) DPO often matches the performance of more complex RLHF methods; 4) It tends to be more stable during training and less sensitive to hyperparameters.
  • Both RLHF and DPO benefit significantly from the integration of synthetic data.



EVALS LLMS


  • LLM evaluation is a crucial process used to assess the performance and capabilities of the models -> it can take multiple forms, such as multiple-choice question answering, open-ended instructions, and feedback from real users.
  • While general-purpose evaluations are the most popular ones, with benchmarks like Massive Multi-Task Language Understanding (MMLU) or LMSYS Chatbot Arena, domain- and task-specific models benefit from more narrow approaches. This is particularly true when dealing with entire LLM systems (as opposed to models), often centered around a retrieval-augmented generation pipeline.
  • Three key differences in how these models work, which impact the evaluation process: 1) Numerical metrics: accuracy, precision, recall, or mean squared error; 2) Feature engineering: LLMs are designed to handle raw text data directly, reducing the need for manual feature engineering; and 3) Interpretability: requesting explanations during the generation process can give insights into the model's decision-making process.
  • The most straightforward metrics are low-level and correspond to how models are trained: 1) Training loss: Based on the cross-entropy loss, measures the difference between the model's predicted probability distribution and the true distribution of the next token; 2) Validation loss: Calculates the same loss as training loss, but on a held-out validation set to assess generalization; 3) Perplexity: Exponential of the cross-entropy loss, representing how surprised the model is by the data (lower is better); and 4) Gradient norm: Monitors the magnitude of gradients during training to detect potential instabilities or vanishing/exploding gradients.
  • After pre-training, it is common to use a suite of evaluations to evaluate the base model, for instance: 1) MMLU (knowledge, Tests models on multiple-choice questions across 57 subjects, from elementary to professional levels); 2) HellaSwag (reasoning, Challenges models to complete a given situation with the most plausible ending from multiple choices); 3) ARC-C (reasoning, Evaluates models on grade-school-level multiple-choice science questions requiring causal reasoning); 4) Winogrande (reasoning, Assesses common sense reasoning through pronoun resolution in carefully crafted sentences); and 5) PIQA (reasoning; Measures physical common sense understanding through questions about everyday physical interactions).
  • Benchmarks targetting capabilities connected to the ability of fine-tuned models to understand and answer questions: 1) IFEval (instruction following, Assesses a model’s ability to follow instructions with particular constraints); 2) Chatbot Arena (conversation, A framework where humans vote for the best answer to an instruction); 3) AlpacaEval (instruction following, Automatic evaluation for fine-tuned models that is highly correlated with Chatbot Arena); 4) MT-Bench (conversation, Evaluates models on multi-turn conversations, testing their ability to maintain context and provide coherent responses); and 5) GAIA (agentic, Tests a wide range of abilities like tool use and web browsing).
  • Classification tasks also benefit from classic metrics: 1) Accuracy: refers to the proportion of correctly predicted instances compared to the total instances; 2) Precision: The ratio of true positive predictions to the total positive predictions made by the model; 3) Recall: The ratio of true positive predictions to the total actual positive instances; and 4) F1 Score: The harmonic mean of precision and recall, used to balance both metrics.
  • Retrieval-Augmented Generation Assessment (Ragas) -> open-source toolkit designed to provide developers with a comprehensive set of tools for RAG evaluation and optimization -> designed around the idea of metrics-driven development (MDD). Metrics include: 1) Faithfulness: measures the factual consistency of the generated answer against the given context; 2) Answer relevancy: evaluates how pertinent the generated answer is to the given prompt; 3) Context precision: evaluates whether all the ground-truth relevant items present in the contexts are ranked appropriately; and 4) Context recall: measures the extent to which the retrieved context aligns with the annotated answer (ground truth).
  • ARES (an automated evaluation framework for RAG systems) -> tool designed to evaluate RAG systems, combining synthetic data generation with fine-tuned classifiers to assess various aspects of RAG performance, including context relevance, answer faithfulness, and answer relevance.



INFERENCE OPTIMIZATION


  • Most of the LLMs, like GPT or Llama, are powered by a decoder-only Transformer architecture. The decoder-only architecture is designed for text-generation tasks (it predicts the next word in a sequence based on preceding words, making it effective for generating contextually appropriate text continuations). An encoder-only architecture, like BERT, focuses on understanding and representing the input text with detailed embeddings (it excels in tasks that require comprehensive context understanding, such as text classification and named entity recognition). The encoder-decoder architecture, like T5, combines both functionalities (the encoder processes the input text to generate a context-rich representation, which the decoder then uses to produce the output text. This dual structure is particularly powerful for sequence-to-sequence tasks like translation and summarization, where understanding the input context and generating a relevant output are equally important).
  • To predict the 100th token in a sequence, a model needs the context of tokens 1 through 99. When predicting the 101st token, it again needs the information from tokens 1 through 99, plus token 100. The key-value (KV) cache addresses this issue by storing key-value pairs produced by self-attention layers.
  • Batching (processing multiple inference requests simultaneously) -> approach to achieve high throughput. Larger batch sizes spread out the memory cost of model weights and transfer more data to the GPU at once, better saturating its parallel compute capacity. Decoder-only models pose a particular challenge due to the high variability in input prompt lengths and desired output lengths.
  • Speculative decoding (assisted generation) -> even with continuous batching, the token-by-token generation process fails to fully saturate the parallel processing capabilities of the accelerator. It aims to use this spare compute capacity to predict multiple tokens simultaneously, using a smaller proxy model: 1) Apply a smaller model, like a distilled or pruned version of the main model, to predict multiple token completions in parallel; 2) Feed these speculative completions into the full model to validate which predictions match what the large model would have generated; and 3) Retain the longest matching prefix from the speculative completions and discard any incorrect tokens. If the small model approximates the large model well, multiple tokens can be generated in a single step. A 90% match could result in a 3–4X speedup.
  • The Transformer architecture is based on the attention mechanism, which scales quadratically with the number of input tokens (or sequence length) -> inefficient for longer sequences, where the size of the KV cache can blow up.
  • PagedAttention (Kwon, Li, et al. 2023) —> addresses memory challenges by drawing inspiration from virtual memory and aging in operating systems -> it partitions the KV cache into blocks, eliminating the need for contiguous memory allocation -> each block contains the keys and values for a fixed number of tokens -> during attention computation, the paged attention kernel efficiently fetched these blocks, regardless of their physical memory location —> the partitioning allows for near-optimal memory utilization -> this is useful for batching more sequences together, which increases throughput and GPU utilization —> this block-based approach naturally supports memory sharing across multiple output sequences generated from the same prompt.
  • FlashAttention-2 (Tri Dao, 2023) -> key innovations are designed to address the quadratic runtime and memory constraints in traditional attention —> by dividing input and output matrices into smaller blocks, it ensures that these blocks can fit into the GPU’s on-chip SRAM, which is much faster than high-banditdh memory —> reduces the frequency of data transfers between the GPU's main memory and its processing units —> this is combined with online softmax, which computers the softmax function independently for each block of attention scores matrix, rather than for the entire matrix at once.
  • Model parallelism allows to distribute the memory and compute requirements of LLMs across multiple GPUs —> this enables the training and inference of models too large to fit on a single device, while improving performance in terms of throughput (tokens per second) —> there are 3 main approaches to model parallelism, each involving splitting the model weights and computation into different ways: data parallelism, pipeline parallelism, and tensor parallelism.
  • Pipeline parallelism (Huang et al., 2023) -> distributing the computational load of training and running large neural networks across multiple GPUs —> unlike DP that replicates the entire model on each GPU, PP partitions the model's layers across different GPU —> this approach allows each GPU to handle a specific portion of the model, reducing the memory burden of each GPU —> pipeline bubbles" the bubbles arise when some GPUs are idle, waiting for activations from preceding layers -> this idle time can reduce the overall efficiency of the process.
  • Megatron-LM (2019) -> distribute the computation of LLM layers across multiple devices —> in contrast to pipeline parallelism, it splits the weight matrices found in individual layers —> enables simultaneously computations, significantly reducing memory bottlenecks and increasing processing speed. Large matrices, such as the weight matrices in MLPs or the attention heads in self-attention layers are portioned across several GPUs - each GPU holds a portion of these matrices and perform computations on its slide.
  • In the context of self-attention layers, TP is particularly efficient due to the inherent parallelism of attention heads — each GPU can computer a subset of these heads independently, allowing the model to process large sequences more effectively - this makes TP more efficient Etna pipeline parallelism, which requires waiting for the completion of previous layers.
  • Data, tensor, and pipeline parallelism are orthogonal techniques that can be combined: 1) Pipeline parallelism provides the greatest memory reduction but sacrifices efficiency due to pipeline bubbles; 2) If low latency, prioritize tensor parallelism and accepting a larger memory footprint; and 3) A model may be split depth-wise into a few pipeline stages, with tensor parallelism used within each stage.


Model Quantization

  • By default weights are stored in a 16-bit or 32-bit floating point format (FP16, FP32) which provides high precision but comes at the cost of increased memory usage and computation complexity.
  • Quantization represents the weights and activations of a neural network using lower-precision data types —> focuses on reducing the precision of the model's weights and activations and reduced memory footprint while accelerating inference.
  • Larger models with 30Bi parameters can outperform smaller models (7b-13b) in terms of quality when quantized to 2- or 3- bit precision.
  • Post-training quantization (PTQ): 1) Weights of a pre-trained model are directly converted to a lower precision format without retraining; and 2) Easy to implement, may result in performance degradation.
  • Quantization-Aware Training (QAT): 1) Quantization during the training or fine-tuning stage, allowing the model to adapt to the lower precision weights; and 2) Better performance but requires additional computation resources and representative training data.
  • Active-aware eight quantization: 1) Identifies and protects the most important weights which are determined based on activation magnitude instead of weight magnitude; and 2) Apply optimal per-channel scaling to these salient weights, without relying on back propagation or reconstruction, ensuring that the LLM does not overt the calibration set.



RAG INFERENCE PIPELINE


  • You can split software architectures into 3 modules: retrieval, augment the prompt, and generate the answer.
  • At the retrieval step, not when calling the LLM, you write most of the RAG inference code. This step is where you have to wrangle your data to ensure that you retrieve the most relevant data points from the vector DB.
  • Architecture: 1) User query; 2) Query expansion (Generate multiple queries that reflect different aspects or interpretations of the original user query); 3) Self-querying (Extract useful metadata from the original query, Metadata will be used as filters for the vector search operation, eliminating redundant data points from the query vector space); 4) Filtered vector search (Embed each query and perform a similarity search to find each search’s top K data points); 5) Collecting results (Aggregate the results of all the xN searches ending up with a list of N x K results containing a mix of articles, posts, repositories, chunks); 6) Reranking (Keep only the top K most relevant results from the list of N x K potential items by filtering further); 7) Build the prompt and call the LLM (Map the final list of the most relevant K chunks to a string used to build the final prompt, Create the prompt using a prompt template, the retrieved (After the LLM processes the prompt, the RAG logic finishes by sending the generated response to the user).
  • Vector search is pivotal in retrieving relevant information based on semantic similarity -> a plain vector search, however, can introduce challenges that affect both the accuracy and latency of information retrieval (it operates on the numerical proximity of vector embeddings without considering the contextual or categorical nuances that might be crucial for relevance) -> as the size of the dataset increases, plain vector search can suffer from scalability issues (the lack of filtering means the search algorithm has to compute similarities across the entire vector space, which can significantly increase latency) -> Filtered vector search can be a solution by filtering after additional criteria, such as metadata tags or categories, reducing the search space before computing vector similarities.



INFERENCE PIPELINE DEPLOYMENT


  • Requirements: 1) Throughput (Number of inference requests a system can process in a given period, Requests per second); 2) Latency (Time for a system to process a single inference request from when it is received until the result is return); 3) Data; and 4) Infrastructure (For high throughput -> scalable infrastructure to manage large data volumes and high request rates, possibly through parallel processing, distributed systems, and high-end GPU).
  • Inference deployment types: 1) Online real-time inference (HTTP requests, with a REST API or gRPC server); 2) Asynchronous inference (Queued HTTP requests, handled asynchronously); and 3) Offline batch transforms (Manually ran or scheduled job).



MLOPS AND LLMOPS


  • Model registry: a centralized repository for storing trained ML models (HuggingFace, Comet ML, MLflow, ZenML).
  • Feature store (Preprocessing and storing input data as features for both model training and inference pipelines, e.g., Hopsworks, Tecton, Featureform).
  • ML metadata store (Tracks information related to model training, such as model configurations, training data, testing data, and performance metrics, e.g. Comet ML, MLflow).
  • ML pipeline orchestrator (Automating the sequence of steps in ML projects, e.g., ZenML, airflow, prefect, dragster).
  • Input guardrails: Protect against exposing private information to external APIs and executing harmful prompts that could compromise your system (model jailbreaking). Output guardrails: Catch failed outputs that don't respect your application's standards.
  • Prompt monitoring (Opik and Langfuse): when generating an answer with LLM, don’t for the whole answer to be generated, but stream the output token by token. Tracking the latency of generating answer -> Time to first token, Time between tokens, Tokens per second, Time per output token, and Total latency.
  • Continuous Training (CT) pipeline (ZenML, Metaflow, Dagster, Airflow): Leverages the code managed by the CI/CD pipeline to automate your data, training, and model-serving process, where the data and model dimensions are present only in the AI world.
  • Monitor: 1) Model configuration (Consider both the LLM and other models used within the RAG layer, Log model IDs and temperature); 2) Total number of tokens; and 3) Duration of each step.
  • Model tests: 1) The shapes of the input and model output tensors; 2) That the loss decreases after one batch or more of training; 3) Overfit on a small batch, and the loss approaches zero; 4) The training pipeline works on all support devices, such as CPU and GPU; and 5) Early stopping an d checkpoint logic works.
  • Logs: 1) Document the system configurations; 2) Record the query, results, and intermediate outputs; 3) Log when a component begins, e ends, crashes; and 4) Ensure that each log entry is tagged and identified in a way that clarifies its origin within the system.
  • Drifts: proxy metrics that help detecting potential issues with the prediction model in time without requiring any ground truths/labels. Types: 1) Data drift -> or covariate shift - when the distribution of the production data deviates from that of the training data; 2) Target drift -> shifts in output distribution - the shift could involve changes in the shape of the distribution or the additional and removal of classes in categorical tasks; and 3) Concept drift —> concept drift —> makes our model ineffective because the patterns it learned to associate inputs with outputs become outdated.
  • Drifts can be detected and measured by 1) A reference window (the collection of data points used as a baseline to compare against the production data distributions); or 2) A test windows.


⬛️