🧠 IDEA → WebLLM, WebGPU, Open-Source Tiny Inference
Beginner to Advanced AI Scientists and Engineers | Today's Word: Threshold · Schwelle · きょうかい · סַף
🎼 "In the face of the Sublime, we feel a shiver, a foretaste of death itself, something too large for our minds to encompass. And for a moment it shakes us out of our smugness and releases us from the deathlike grip of habit and banality." — Robert Green, on The Laws of Human Nature
🎶 "The rest is silence." — William Shakespeare, on Hamlet

In another of my side quests, I've been looking at small, distilled LLMs that can be run locally in the browser (or are tiny enough to be submitted as very fast, free inference jobs in the cloud).
I built an open-source e2e demo that you can play with (and you should check the source code, as it's pretty good 😏).
The inference stack we'll be learning about is:
1️⃣ WebLLM
2️⃣ WebGPU
3️⃣ MLC and MLC-LLM

As always, please remember the rules:
🤖 If you see a 👾, it means I found something particularly cool or learned something new.
🤖 If you see a ✨, it means things that are so cool that they need a lil glitter around them.
🤖 If these notes look cool to you, it's your ✦moral duty✦ to read the original resources.
🤖 Just another blog, but not your usual blog: they're living notes that I change as I fancy.
🏯 WebLLM, The In-Browser LLM Engine

- WebLLM is a JavaScript/TypeScript inference engine for running LLMs directly in the browser, using MLC-compiled WebGPU kernels running inside a Web Worker.
- Zero Server Dependency (and privacy): Once the model weights are downloaded and cached in your browser, everything runs locally.
WebLLM: A High-Performance In-Browser LLM Inference Engine, Ruan et al. (arXiv:2412.15803)

With machine learning compilers MLC-LLM and Apache TVM, WebLLM leverages optimized WebGPU kernels, overcoming the absence of performant WebGPU kernel libraries. Evaluations show that WebLLM can retain up to 80% native performance on the same device with room to close the gap further.
Open-weight providers now routinely ship capable small models in the 1-8B-parameter range, and techniques such as quantization have made real-time local inference common on consumer hardware (Abdin et al. (2024), Gemma Teamet al. (2024), Liu et al. (2026)).
- Enabling LLM-based features in web applications poses three challenges:
1️⃣ A standardized API that web applications can easily incorporate
2️⃣ Adaptation to the browser’s runtime environment
3️⃣ efficient GPU acceleration.
- WebLLM’s architecture addresses these challenges by dividing the system into three corresponding parts:
1️⃣ ServiceWorkerMLCEngine: lightweight, endpoint-like object the web developer instantiates in the page. It takes OpenAI-style JSON requests and streams back OpenAI-style JSON responses.
2️⃣ MLCEngine running inside Web Workers / Service Workers: the actual computation happens off the main thread, so a long generation doesn't freeze page scrolling or input.
3️⃣ Ahead-of-time-compiled WebGPU kernels: produced by MLC-LLM/TVM. Plus a WebAssembly layer (compiled from C++ via Emscripten) for CPU-side work that doesn't belong on the GPU: the grammar engine for structured/JSON generation, paged-KV-cache bookkeeping, and tensor-manipulation glue for launching kernels.
The lightweight frontend engine
ServiceWorkerMLCEnginethat is exposed to the web application, and a backend engine MLCEngine in the worker thread that actually computes the LLM workload. The two engines communicate via message-passing, and the messages are simply OpenAI-style requests and responses.
Using WebGPU is not enough since LLM inference requires non-trivial computation on the CPU. WebAssembly (WASM) is a portable low-level bytecode that can be compiled from C++ code and run in a JavaScript runtime with near-native performance. Thus, instead of re-implementing CPU workload in JavaScript, WebLLM leverages Emscripten to compile high-performance subsystems written in C++ into WebAssembly for various CPU workloads in LLM inference, including a grammar engine for structured generation, sequence management in the paged KV-cache and tensor manipulation for launching kernels.
Web Workers

- LLM inference can take a long time, therefore WebLLM supports Web Workers.
- It utilizes background threads so that running heavy AI tasks doesn't freeze or lag the website's user interface.
🏯 MLC (Machine Learning Compilation)

What is ML compiler? A machine learning compiler is a specialized compiler that transforms high-level ML models into optimized code that can efficiently run on various hardware platforms. It bridges the gap between ML frameworks and hardware backends, enabling models to run faster and use less memory across different devices from cloud servers to edge devices.
- The user-facing browser layer. It runs supported open LLMs entirely in the browser, uses WebGPU for local acceleration, uses WebAssembly for CPU-side runtime work, and presents an API designed to resemble OpenAI’s chat-completion APIs.
- A compiler discipline and toolchain, built on Apache TVM, that takes a model written in Python and compiles it into optimized GPU kernels and portable binaries for a target backend (optimized implementations for different hardware and runtime targets; WebGPU included). It's the translator.
- The deeper problem MLC solves is hardware fragmentation.
- A conceptual example is:
import { CreateMLCEngine } from "@mlc-ai/web-llm";
const engine = await CreateMLCEngine("a-supported-MLC-model-id");
const stream = await engine.chat.completions.create({
messages: [
{ role: "system", content: "You are a concise assistant." },
{ role: "user", content: "Summarize this note." }
],
stream: true
});
for await (const chunk of stream) {
process.stdout?.write?.(chunk.choices[0]?.delta?.content ?? "");
}
MLC-LLM
WebLLM leverages machine learning compilation libraries MLC-LLM and Apache TVM to compile performant WebGPU kernels. MLC-LLM ingests any open-source model’s implementation in Python, which can use techniques such as the aforementioned PagedAttention and FlashAttention, and compiles the model’s computation into the backend of interest (in this case, WebGPU). Besides compiling to the specified target, MLC-LLM provides both graph-level optimizations (e.g. kernel fusion) and operator-level optimizations (e.g. GEMM tiling) to improve kernel performance. MLC-LLM converts open-source models into two artifacts: converted weights and a WASM library. The WASM library contains both WebGPU kernels and non-kernel functions in WebAssembly.
- An LLM deployment engine built on machine-learning compilation that provides optimized inference across GPUs, browsers, mobile devices, and servers.
- It compiles and runs code on MLCEngine: a unified high-performance LLM inference engine across the above platforms.
- MLC-LLM also runs as a server: MLCEngine exposes an OpenAI-compatible REST endpoint, Python API, JavaScript API, and native iOS/Android bindings — all sharing the same compiled core, so a WebLLM app in the browser and a native iOS app can, in principle, be running weights produced by the exact same compilation pipeline.
Apache TVM
A deep learning compiler that enables access to high-performance machine learning anywhere for everyone.

TensorIR

Why compilation is needed
- A model checkpoint is not automatically a high-performance program for every device.
- It encodes parameters and an architecture, but efficient execution still requires decisions such as:
1️⃣ How to store weights and activations in memory.
2️⃣ Which numerical precision and quantization scheme to use.
3️⃣ Which operations can be fused to avoid intermediate memory traffic.
4️⃣ How to tile matrix multiplications.
5️⃣ How to schedule work across GPU threads and workgroups.
6️⃣ How to target the constraints and capabilities of WebGPU, Metal, Vulkan, CUDA, CPUs, or mobile devices.
- Apache TVM’s compilation model explicitly supports transformations at both the graph and tensor-program levels, then produces a deployable module for the target device.
MLC LLM artifacts for the web
- For WebLLM deployment, MLC LLM produces two key artifacts:
1️⃣ Converted model weights, often quantized (e.g., 4-bit) and packaged in MLC format.
2️⃣ A binary kernel library — a .so/.dylib/.dll for native targets, or a WebAssembly .wasm module containing WebGPU kernels for the browser, containing execution logic, including the compiled WebGPU kernels and non-kernel runtime functions.
- WebLLM downloads and uses both of these artifacts in the browser.
- The model identifier must correspond to an artifact WebLLM supports and can access; an actual production integration should also display loading progress, verify WebGPU availability, handle cancellation, and offer a graceful server or CPU fallback.
JIT and ahead-of-time compilation
- MLC LLM can compile model libraries at runtime (JIT) or generate them explicitly for deployment (ahead of time).
- Ahead-of-time compilation is particularly relevant for web and mobile deployments because it lets an application distribute a target-specific artifact rather than asking each client to perform model compilation during startup.
- The WebLLM paper compared browser WebLLM with native MLC LLM on the same Apple MacBook Pro M3 Max, using Chrome Canary 133.
- In that experiment:
| Model | WebLLM generation | speed Native MLC LLM speed | Browser performance retained |
|---|---|---|---|
| Llama 3.1 8B | 41.1 tokens/s | 57.7 tokens/s | 71.2% |
| Phi-3.5 Mini (3.8B) | 71.1 tokens/s | 89.3 tokens/s | 79.6% |
🏯 WebGPU, The Hardware Accelerator

- A standardized low-level browser API that exposes modern GPU graphics and general-purpose computation to web applications (JavaScript). It's the hardware doorway.
- It acts as the performance backbone, allowing WebLLM to achieve close to native inference speeds.
- The catch is that WebGPU ships with no ML kernel library — no cuDNN- or cuBLAS-equivalent — so every matrix multiply, attention op, and quantization kernel has to be hand-written or generated by a compiler (That gap is exactly what MLC fills).
- WebGPU exposes compute shaders and storage buffers directly, which is the same programming model native ML kernel authors already use. The shader executes thousands of parallel operations.
- A shader is the portion of a WGSL program that executes a shader stage in a pipeline. A shader comprises:
1️⃣ An entry point function.
2️⃣ The transitive closure of all called functions, starting with the entry point. This set includes both user-defined and built-in functions.
3️⃣ The set of variables and constants statically accessed by all those functions.
4️⃣ The set of types used to define or analyze all those functions, variables, and constants.
- An application begins with:
const adapter = await navigator.gpu.requestAdapter();
const device = await adapter.requestDevice();
WebGL

- Prior to WebGPU, browser GPU compute meant abusing WebGL's graphics pipeline (packing tensors into textures) — workable but awkward and slow.
| Feature | WebGL | WebGPU |
|---|---|---|
| Primary origin | Graphics | Graphics + compute |
| Modern GPU model | Limited | Yes |
| General compute | Awkward | First-class |
| Compute shaders | No native equivalent | Yes |
| GPU memory model | Older abstraction | Modern explicit resources |
| ML suitability | Possible but awkward | Much better |
| Shader language | GLSL | WGSL |
| Modern GPU APIs | Less aligned | Designed around modern GPU architecture |
- Feb 2017 — Apple's WebKit team proposes a W3C community group after showing an early "WebGPU" proof-of-concept based on Metal; Google and Mozilla had parallel prototypes ("NXT" and an Obsidian/Vulkan-based design, respectively).
- 2018 — Chrome announces intent to implement the emerging standard.
- Feb 2020 — Google ships Tint, and the shading language work converges on WGSL.
- April 2023 — Chrome 113 and Edge ship WebGPU by default on Windows/ChromeOS/macOS (Android followed in Chrome 121).
- June–July 2025 — Safari (Safari 26, tied to macOS Tahoe/iOS 26) and Firefox (Firefox 141) ship WebGPU, completing coverage of all four major engines.
- As of late 2025 — the spec sits at W3C Candidate Recommendation (CR) status — mature and implemented everywhere, but formally still short of final "Recommendation."
WGSL (WebGPU Shading Language)

- The native shader language designed for WebGPU.
- The syntax is similar to Rust.
- If you're trying to optimize or implement an LLM GPU kernel yourself, WGSL becomes very relevant.
- WebGPU issues a unit of work to the GPU in the form of a GPU command.
- WGSL is concerned with two kinds of GPU commands: 1) a draw command executes a render pipeline in the context of inputs, outputs, and attached resources, 2) a dispatch command executes a compute pipeline in the context of inputs and attached resources.
🏯 Other Considerations
The competitive landscape
WebLLM is not the only way to run models in a browser; it's worth knowing where it sits:
➡️ Transformers.js (Hugging Face) — a JS port of the transformers pipeline API, backed by ONNX Runtime Web, which can itself target either WebAssembly or WebGPU depending on browser support. It covers a much broader task surface (embeddings, classification, audio, vision) than WebLLM, and is arguably the easiest on-ramp for non-LLM tasks, but for pure autoregressive LLM decoding, benchmarks generally show it in the same performance ballpark as WebLLM when both use WebGPU, with WASM fallback for smaller/simpler models.
2️⃣ wllama / llama-cpp-wasm — WebAssembly bindings for llama.cpp itself. Historically WASM/CPU-only (SIMD-accelerated, no GPU needed at all, which is a real advantage for reach), wllama added WebGPU support in its v3 release. It works from GGUF files directly, so it plugs into the same model ecosystem as desktop llama.cpp/Ollama users already use.
3️⃣ WebNN (Web Neural Network API) — a different W3C standard, lower-level and NPU/accelerator-aware rather than GPU-compute-aware. It's designed to let the browser hand inference off to whatever dedicated silicon a device has (Apple Neural Engine, Qualcomm/Intel NPUs, etc.), not just the GPU. It reached an updated W3C Candidate Recommendation in January 2026 after what the spec calls "over 100 significant changes," including new transformer-support operators and a buffer-sharing MLTensor API — clearly being pushed toward LLM-scale workloads, but implementation is still catching up to WebGPU's browser coverage.
- The rough mental model: WebGPU/WebNN are the standards bodies' answer to "how does a web page reach hardware acceleration," while WebLLM/Transformers.js/wllama are competing (and partly complementary) engineering answers to "how do I actually get a chat model running well on top of that."
Practical limitations
➡️ First-load cost: multi-gigabyte weight downloads and WebGPU shader compilation add real latency (roughly 1–5 seconds of shader compile time alone is commonly reported, on top of the download) before the first token appears. Caching (IndexedDB/OPFS/Cache API) is what makes repeat visits fast.
2️⃣ Hardware floor: WebGPU support on lower-end Android devices and older GPUs remains uneven — this is a driver/vendor coverage problem, not something WebLLM or WebGPU-the-spec can fix unilaterally.
3️⃣ The native-performance gap means anything latency-critical at scale is still likely to prefer server-side inference; WebLLM's value proposition is privacy, offline capability, and zero server cost — not raw speed.
4️⃣ Security surface: WebGPU is comparatively young as a spec and its sandboxing is still being actively hardened; running untrusted model weights or untrusted WGSL is not risk-free.
Lastly, some food for thought...
Performance depends on:
- GPU driver
- browser implementation
- WebGPU backend
- model size
- quantization
- context length
- KV cache
- memory bandwidth
- shader efficiency
- browser overhead
- thermal limits
- available GPU memory
- CPU performance for tokenization and auxiliary operations