All else being equal – hardware, runtime, optimizations, architecture – inference on a smaller model with fewer parameters will be faster and cheaper than inference on a larger model with more parameters.
That’s why the most important decision in model performance optimization isn't the runtime engine or speculation algorithm, it’s which model you choose to work with in the first place.
AI engineers iterating on early-stage products should just use prebuilt pay-per-token APIs for powerful frontier models like Kimi and DeepSeek (or even closed models like GPT and Gemini). Before product-market fit, it’s not worth spending time or money on doing your own inference.
But when it’s time to scale, the opposite advice applies. Find – or create – the smallest, easiest-to-run model that’s smart enough to handle the task at hand. In many cases, this is still going to need to be a trillion-parameter frontier model. But it’s always worth checking if a smaller, cheaper, faster model can get the job done.
Which model you pick also affects what inference optimizations are available to you. Inference engines vary in the depth of support for different model architectures. Stick with popular model architectures to ensure you’ll find robust support across the performance tooling landscape.
1.3.1 Model Evaluation
Model evaluation, or evals, is the practice of systematically measuring model intelligence.
High conviction model evaluation is a prerequisite for inference engineering. Evals help inference engineers:
- Spend time wisely: Before investing in making a model fast, evals ensure the model is useful.
- Establish a baseline: Some performance optimization techniques risk reducing model quality, requiring a baseline to compare against.
Unlike standard intelligence benchmarks, which measure model capabilities against common tasks like MMLU or SWE-bench, evals are tailored to specific products, domains, and tasks.
Intelligence benchmarks are useful for shortlisting models, but they have become saturated or even gamed. Goodhart's Law states that "when a measure becomes a target, it ceases to be a good measure," and this applies to frontier labs’ heavy incentive to show new world-record intelligence benchmarks with each model release.
While there are better ways to gauge overall model intelligence, like Elo rating on head-to-head win rate versus other models, there is no substitute for directly measuring how a model performs for your application.
A few tips for doing useful model evaluation work:
- Look at your data: Check eval results against your intuition for the product and problem space.
- Be precise: Have a clear idea of the hardest problems a model needs to solve and focus evaluation there.
- Use tools: Don’t reinvent the wheel on one of the fundamental problems in AI engineering.
Appendix B includes recommendations for tooling and further reading about evals.
1.3.2 Fine-Tuning for Domain-Specific Quality
Fine-tuning is the practice of taking a pre-trained foundation model and adapting it to a specific use case by introducing new data.
If you can fine-tune a small model to pass your evals, you set yourself up for an easier time hitting your latency and cost targets for inference.
A great example of a domain where fine-tuning is effective is translating English into SQL, a language used to query databases.
General-purpose coding models are good at writing SQL, but these models are hundreds of billions of parameters. SQL is a relatively constrained language, so for an application that only needs to generate SQL queries from natural language prompts, a tiny fine-tuned model of just a few billion parameters can reach equivalent performance on this specific task.
Text-to-SQL is an extreme example – many domains won't support such a vast reduction in model size – that illustrates what’s possible with a cleanly scoped domain, a strong set of evaluation criteria, and high-quality labeled data to use for fine-tuning.
1.3.3 Distillation
What if you could retain most of the intelligence of a large model at a fraction of the size? That’s the idea behind distillation.
Distillation is the process of using a large “teacher” model to train a smaller “student” model to emulate the larger model’s behavior. Unlike fine-tuning on synthetic data, where the model is trained on input-output pairs, distillation shows the student model the teacher model’s actual probability distributions, not just its final answers.
Where fine-tuning teaches a model to perform better in a specific domain, distillation teaches the model how to emulate the behavior – good and bad – of a larger model.
Distillation sees substantially less real-world use than fine-tuning.
When a frontier lab releases a family of models of different sizes, the smaller models are generally not distilled from the larger ones. Instead, the models are independently trained to prevent the biases of the large models from artificially limiting the smaller models. But if the lab only trains a large model, distillation can make that model more accessible.
In January 2025, open model research lab DeepSeek released their then-flagship reasoning model DeepSeek-R1. As the model was so large (671B parameters), they also released distilled versions of the model on top of the most popular open model architectures at the time: Llama 3 and Qwen 2.5.
These distilled models showed similar reasoning behavior to the main DeepSeek-R1 model, albeit with worse intelligence benchmark scores, but the distilled models were relatively small and could take advantage of existing performance work for the Llama and Qwen architectures.
At publication, these DeepSeek-R1 distills are still among the most popular distilled models on Hugging Face, along with distills of models like Whisper for audio transcription and some image generation models.