The Quiet Rise of On-Device AI (Your Browser Is Getting Smarter)

Everyone's watching the giant models: the ones that live in data centers, drink rivers of electricity, and answer through an API. Meanwhile, with much less fanfare, something else has been happening: genuinely useful AI has started running on the machine in front of you. In your laptop. In your phone. Increasingly, right inside a browser tab.

This is the AI trend I'd actually bet on for everyday tools, and it's worth understanding why: including the honest limits, because on-device AI is having a moment and the moment comes with some overselling.

What changed

Five years ago, running a neural network in a browser was a party trick. Three things quietly ended that era.

The hardware caught up. Modern laptops and phones ship with dedicated neural processors, and even ordinary graphics chips can churn through the matrix math that models are made of. The computer you already own is a serious inference machine that spends most of its life idle.

The browser grew real muscles. Two technologies matter here. WebAssembly lets near-native code, the kind image and PDF engines are written in, run safely inside a tab at real speed. WebGPU gives web pages honest access to your graphics hardware. Together they mean a web page can do work that used to require installed software.

Models learned to be small. The research world figured out distillation and quantization: ways of compressing a model to a fraction of its size while keeping most of its ability. A model that fits in a phone today would have been state-of-the-art-adjacent a few years ago. Whisper transcribes speech on a laptop. Small language models draft and summarize entirely offline. Background-removal and upscaling models fit in a browser tab.

Why anyone should care where the model runs

Because where the computation happens decides three things you feel directly.

Privacy stops being a promise and becomes physics. When processing happens on your device, your file doesn't go anywhere, there's no server to trust, no retention policy to squint at, no terms-of-service question about what happens to the thing you pasted. "We can't see your data" beats "we promise not to look" every single time. This is the entire philosophy our tools are built on, it's why browser-based tools are the future, and AI workloads don't change the argument. They strengthen it, because AI inputs (your documents, your photos, your prompts) are the most sensitive data of all.

Latency collapses. No upload, no queue, no download. For interactive work (cleaning up audio, editing images, transcribing a meeting) the difference between "instant" and "spinner" changes what a tool feels like.

Cost goes to zero. Cloud AI bills someone for every request, which is why so many AI products meter you, upsell you, or quietly harvest something. Work done on your own silicon is free at any scale. That economic fact shapes product design more than any philosophy does.

What on-device AI genuinely can't do (yet)

Here's the honest half of the picture. The frontier chatbots (the ones you have long reasoning conversations with) are not running in your browser any time soon. Models like that have hundreds of billions of parameters. Even compressed, they don't fit in a tab, and the small models that do fit are noticeably less capable at open-ended reasoning. If you need deep research help or subtle judgment, the data center still wins, with all the judgment caveats that come with it.

There are practical frictions too. Models must be downloaded before first use: tens or hundreds of megabytes, which is real on a metered connection (browsers cache them after that). A five-year-old laptop will be slower than a rented H100, always. And browser support for WebGPU, while good now, still has rough edges across devices.

So the near future isn't "local replaces cloud." It's a split: open-ended intelligence in the cloud, bounded skills on the device. Transcription, translation, image cleanup, format conversion, summarizing a document you're reading (bounded, well-defined tasks) migrate to your hardware. The stuff that needs a giant brain stays remote.

What shipping on-device AI taught us

We ship this, so here is field experience rather than punditry. Our Video Intelligence workspace runs Whisper entirely in the browser through transformers.js and ONNX Runtime: the model downloads once (the small English model is tens of megabytes), caches locally, and then transcribes with no server involved. What we learned: first-run download time is the real UX cost, WebGPU support decides how fast inference feels, and users genuinely care that a recording of their own voice never leaves their machine. The capability ceiling is lower than a datacenter model, and for a defined job like transcription it is more than enough.

Where we stand

People sometimes assume a site with "AI" in the air must be shipping prompts to a server farm. So, plainly: our tools run in your browser, and we don't dress things up as AI when they aren't. The heavy lifting in our image, PDF, audio, and video tools is classic engineering (WebAssembly codecs, canvas pipelines, real parsers) because for bounded jobs like compressing an image or rebuilding a scanned PDF's text with OCR, well-built conventional engines are fast, predictable, and private. Where a genuine model earns its place: the ESRGAN network behind the image upscaler, the OCR engine reading pixels into text. It downloads to your machine and runs there. Your files never leave the tab either way.

That's the quiet part of the on-device story: it isn't really about AI at all. It's about an old principle (your data is yours, your hardware is capable, and software should come to you) getting a new generation of very good tools. The giant models will keep making headlines. The local ones will keep making themselves useful, one browser tab at a time.

Sources

• transformers.js, runs Whisper inside the browser
• openai/whisper, the open speech model