Over the past few weeks I built several AI tools that run entirely inside the browser — a webcam 3D-avatar tracker, an AI image upscaler, a video converter. What they share is that there is no server: the user’s browser is the inference engine.
The upside is clear, but building these surfaces a completely different set of traps than server-based AI. This is the web AI development playbook I put together — what to run it on, how to design it, and where it breaks, in order.
Why run AI in the browser
- Data never leaves the device — camera, photos and documents aren’t sent to a server, so privacy is the default.
- No install, no sign-up — usable instantly from a single link.
- Zero inference cost — the user’s device does the compute, so traffic can grow without growing your GPU bill.
- Effortless scale — it’s just static files, so a CDN spreads it for you.
- Works offline — once the model is cached, it runs without internet.
What you run it on — the core tech
- WebAssembly (WASM) — runs inference engines written in C++/Rust at near-native speed in the browser. Most web AI ends up on WASM.
- WebGPU / WebGL — GPU acceleration. WebGPU is the newest and most powerful; unsupported environments fall back to WebGL or WASM.
- SIMD + multithreading — WASM SIMD speeds up vector math and SharedArrayBuffer enables threads — but only when the page is “cross-origin isolated” (more below).
Choosing a runtime — it depends on the task
- Vision (face, hands, pose, segmentation) → MediaPipe (Tasks Vision). Models and preprocessing are packaged together, so it is the fastest to wire up.
- General ONNX models → ONNX Runtime Web. Train in PyTorch, export to ONNX, infer in the browser.
- Custom / experimental → TensorFlow.js. You can even train in the browser.
- LLMs / transformers → Transformers.js (Hugging Face), WebLLM. Small models now run in the browser too.
A development playbook — step by step
1. Look at “download size” first
On a server, model size is a disk concern; in the browser the user downloads it every time. First load equals bounce rate. Prioritize int8/fp16 quantization, pruning and smaller variants. Trading a little accuracy to cut tens of MB usually feels better in practice.
2. Loading strategy — CDN vs self-hosting, and caching
A CDN (jsDelivr, etc.) is convenient but brings cross-origin constraints. Self-hosting is faster and more stable and keeps CORS simple. Cache model files once with the Cache API or a Service Worker, and always show a progress indicator — a blank screen during a tens-of-MB download is a bounce.
3. Secure context — HTTPS is not optional
The camera and microphone (getUserMedia) and some APIs only work in a secure context (HTTPS or localhost). Opening an HTML file as file:// blocks both the camera and module/WASM loading. Even local development needs a localhost server, and deployment must be HTTPS.
4. Cross-origin isolation — the prerequisite for threads
To use WASM multithreading (SharedArrayBuffer) the page must be cross-origin isolated. Turn it on with response headers:
Cross-Origin-Opener-Policy: same-origin Cross-Origin-Embedder-Policy: require-corp
The catch: the moment you enable require-corp, every external resource must satisfy CORP/CORS or it is blocked. So your CDN must send these (jsDelivr does):
Access-Control-Allow-Origin: * Cross-Origin-Resource-Policy: cross-origin
5. Don’t block the main thread — Web Workers
Inference is heavy. Run it on the UI thread and the screen freezes. Move heavy runtimes into a Web Worker and pass only the result back via postMessage. For real-time work like a camera, put it on a requestAnimationFrame loop and drop frames when you fall behind.
6. Split large inputs — tiling/chunking
Push a high-resolution image to the GPU all at once and memory blows up. Tile it, infer per tile, and stitch the results (the upscaler does exactly this). Dispose tensors the moment you are done — this matters most on mobile.
7. Always ship a fallback
User environments vary wildly. Layer fallbacks: WebGPU → WASM (SIMD) → single thread. ONNX Runtime Web steps down automatically if you pass an execution-provider array:
ort.InferenceSession.create(modelUrl, {
executionProviders: ['webgpu', 'wasm']
});
Pitfalls you will definitely hit
- COEP blocks the CDN — with require-corp on, resources that don’t send CORP/CORS are blocked. Use a friendly CDN or self-host.
- file:// doesn’t work — modules, WASM and the camera are all blocked. Always serve over https/localhost.
- Memory limits — especially on mobile. Tile, and dispose tensors immediately.
- iframe embedding needs matching COEP — to embed a tool in a cross-origin-isolated parent, the child must use the same COEP or it’s blocked with “refused to connect.”
- Preprocessing format — get input normalization, channel order (NCHW/NHWC) and color space wrong and results silently break.
Deployment
A web AI tool is ultimately just static files. Put them on static hosting, or embed them into a content site (like WordPress) via an iframe. Keeping model files on the same domain simplifies caching and CORS.
When not to use it
- Multi-GB models — downloading them on every visit is unrealistic.
- Training / fine-tuning — a server or dedicated hardware is the right place.
- When you must hide model weights, or consistency/auditing matters — server inference is safer.
Wrapping up
The essence of browser AI is “using the user’s device as the inference server.” It is powerful on cost and privacy, but you must design around web-specific constraints — secure context, isolation, memory, loading — from the start. Start small: get it working with a small model and a WASM fallback, then scale up to WebGPU and multithreading.