Local inference, also called edge inference, client-side inference, or on-device inference, means running AI model inference directly on the end user’s device rather than on a centralized server.
Client-side inference has four massive advantages over server-side inference:
- Zero network latency: There’s no communication overhead, saving tens or even hundreds of milliseconds.
- Independence: There’s no dependency on internet connection or impact from high server traffic or downtime.
- Improved privacy: The end user’s data never leaves their device.
- Cost: Datacenter GPUs are expensive, while edge inference is free for the developer, unlocking new business models.
Local inference sounds perfect in theory. In practice, there’s a reason most inference happens in the cloud. There are four weaknesses to local inference that limit its applications:
- Hardware capabilities: Even high-end prosumer desktops offer a fraction of the speed and power of datacenter GPUs.
- Thermal constraints: Local devices have worse cooling than datacenters, further limiting their speed and power.
- Fragmented support matrix: Endless combinations of hardware and software make standardization challenging.
- Battery life: Inference is a demanding workload that quickly drains the batteries of laptops and smartphones.
When building with local inference, it’s important to keep your audience in mind. An AI enthusiast may have the latest and greatest phone and a powerful computer, but a median user is more likely to have older devices with less powerful components.
Local inference is turning the corner from experimentation to production, with a strong ecosystem across hardware and software and a vibrant community closely affiliated with the world of open models.
3.5.1 Desktop Inference
The classic local device is a workstation or gaming PC equipped with one or two high-end consumer GPUs from NVIDIA or AMD. While researchers and enthusiasts do use these setups, they’re a small portion of the desktop inference market.
Increasingly, Apple is the leader in the desktop inference market. Apple’s custom M-series CPUs and GPUs draw from a single unified memory, giving inference on GPUs access to far more memory, albeit at slower speeds.
The highest-end options from Apple and NVIDIA currently available on the market illustrate the tradeoff between memory capacity and speed.
| Hardware | NVIDIA RTX 5090 | Apple M3 Ultra |
|---|---|---|
| Memory | 32 GB | 512 GB |
| Bandwidth | 1,792 GB/s | 819 GB/s |
| Cost (full computer) | $5,000 | $10,000 |
This trend continues through midrange hardware at more reasonable price points. Low-end computers like Chromebooks are not equipped to run any meaningful local inference.
The enthusiast-led open-source ecosystem focuses on running frontier open models on desktops and laptops. Today, running aggressively quantized 100B+ parameter models on high-end personal hardware is possible using tools like Ollama and llama.cpp.
The increased popularity of Mixture of Experts is also a tailwind for desktop inference. These models have fewer active parameters, meaning that an individual user’s single request only touches a fraction of the model’s total weights.
Image generation is also quite popular on personal computers, especially via ComfyUI, a tool for assembling multiple image model components together into a single pipeline.
Smaller language models and other modalities like speech are also feasible to run on midrange computers, and the industry is rapidly developing browser inference libraries like WebLLM and other cross-platform standards to bring these capabilities from early adopters to the mainstream.
3.5.2 Mobile Inference
Local inference on mobile devices represents the majority of on-device workloads today. Both major operating systems offer tooling for developers to add edge inference to applications:
- Android: Google’s AI Edge SDK and ML Kit GenAI APIs interface with Gemini Nano and OSS Gemma models.
- iOS: Apple’s Foundation Models and Core ML frameworks provide APIs for models across modalities.
Mobile devices have extremely limited hardware capabilities and battery capacities, making inference even more challenging. Even high-end phones struggle to run models with more than one or two billion parameters.
Still, some modalities are well-suited for edge inference on phones. For example, transcription and speech synthesis models are latency-sensitive, and some models are small enough to run in real time on modern phones. Other discrete tasks, like translation, can be handled on edge devices by small fine-tuned models.
Like any other software, the future of inference isn’t local or cloud, it’s both working together to power fast and seamless user experiences. Small models and quick queries will run on end-user devices, while more demanding workloads will remain on datacenter GPUs in the cloud.