Published
As a Solutions Architect at Baseten, internal coding use cases have been among the most frequent I’ve encountered, especially since the bill for frontier, closed-source AI is rising every day.
The problem that many engineers and engineering leaders encounter is that they don’t know which model to use, or how exactly to tell if a model meets their criteria. They hear the noise on social media and the news about a new open-source model and think they should just roll with that. This blog is an attempt to explain how to start your journey and how to evaluate models for coding use cases, without just falling for the hype.
Three questions before you deploy: how you're serving the model, whether you're comparing candidates under the same conditions, and whether the tasks represent your work.1) How to use leaderboards
A leaderboard score tells you how a model performed in a particular setup. That includes the serving engine, precision, chat template, harness, prompt, and reasoning settings. Before treating a leaderboard as a reference, check how that setup differs from yours. Models are optimized for different settings, so their performance may vary significantly given your configuration.
Despite their wide acceptance and popularity, benchmarks can also be flawed. For instance, when OpenAI audited SWE-Bench Pro, it found that an estimated 30% of the public tasks had problems, like expecting outputs that were never asked for in the task description, rejecting valid fixes, or accepting incomplete ones. Not surprisingly, OpenAI withdrew its recommendation to use Pro after also raising concerns about SWE-bench Verified. So while these benchmarks can be useful, just make sure to check why a task failed before blaming the model.
For me, a leaderboard is useful in helping narrow down the list of candidate models to two or three. Then I want to see how those models perform on the team's work, using the setup they plan to deploy.
Narrow down the task(s) before picking a benchmark
“Coding use cases” can describe many different types of work: autocomplete, fixing a bug, answering a question about a repo, writing tests, or reviewing a PR. Each of these needs to be evaluated differently.
For instance, if your target use case is fixing bugs, the model (or agent) has to locate the relevant code and make a change without breaking something else. If the benchmark you’re using asks it to write a function, it’s only testing part of your actual use case.
Before starting your evaluation, write down the evaluation contract you’ll reference throughout this process. The evaluation contract should include:
the workflows, repos, and languages you’ll focus on,
the tools the agent can call,
parameters like context and output limits, timeouts, and retries, and
how much autonomy a coding agent gets.
In terms of autonomy, the contract should specifically include whether the agent can run tests, push changes, or use your network.
Match the benchmark to the decision
Too often, teams lean on public benchmarks that don’t serve the tasks they want to measure. It might sound obvious, but if you want to evaluate repository bug-fixing performance via a benchmark, it needs to include tasks that require finding and changing code in a repository. Similarly, for terminal work, you need tasks that exercise that tool and environment.
Depending on your resources, 30-100 tasks can be a manageable start. Be sure to save the task IDs, dataset version, and split. Again, don’t take failures at face value: a bad task description or grader can give misleading results.
For repo work (like bug fixes), I'd consider a reviewed SWE-bench subset; for terminal use, selected Terminal-Bench tasks; and for editing, an Aider Polyglot sample. LiveCodeBench is popular for code generation specifically (but not necessarily suitable for other tasks, like bug fixes).
Each coding benchmark measures a specific kind of work; choose the ones that match what you expect the model to do. LiveCodeBench, for example, tests isolated code generation but not navigating a full repository (let alone your team’s specific repositories). Only a private task set built from your team's workflow shows how the model will perform on your work.Each coding benchmark measures a specific kind of work; choose the ones that match what you expect the model to do. LiveCodeBench, for example, tests isolated code generation but not navigating a full repository (let alone your team’s specific repositories). Only a private task set built from your team's workflow shows how the model will perform on your work.
2) Evaluate the system, not just the model
Many different pieces can play into a failed run: the serving layer may have parsed a tool call incorrectly, your agent may have dropped useful context, or failed to apply an edit. Since all of this sits between the model and the result you get from it, it belongs in the evaluation, too.
That’s why it’s important to use the same scaffold you intend to deploy in production during your evaluation. If you haven't chosen one, SWE-agent or mini-SWE-agent are well-documented starting points, just be sure to note the version and keep that scaffold consistent across candidate runs.
When in doubt, rule out the plumbing first
Take one request and follow it end-to-end.
Check formatting, truncation, and tool-call round trips. If a tool result is missing, resolve that before using the run to judge the model. For models with separate reasoning output, check how the server and scaffold handled it. Also verify that the reasoning controls you send are supported; accepting a parameter isn't proof that it took effect. Then run a task needing several tool calls and check whether the agent can recover from a failed call without losing the context it needs.
When a documented recipe is available, try to reproduce it with the same checkpoint, precision, prompt, reasoning settings, and limits. Start with a subset so you can inspect configuration differences and failed traces before spending on the full run. Record what you reproduced and what you couldn't.
Set the run budget at the same time: task count, inference spend, and concurrency. For self-hosting, include GPU type, GPU-hours, and elapsed time so another team can interpret the cost and timing.
Disentangle routing from your evaluation
If routing is enabled in any capacity, this requires a separate, careful inspection. Since an “auto” router mode can always select a smaller model for tasks it deems to be simpler (or retry on a different model given an error), it’s a simple way to overlook whether you’re really judging your target model.
I’ve run into this issue while using a popular harness. While I thought the main model was handling the run, almost 60% of the tasks were going to a smaller model because the auto routing mode classified them as “simple.” I had to inspect the gateway logs to find the routing, since it wasn’t surfaced via the UI.
This happens regardless of the harness you’re using, and applies wherever routing or fallbacks are enabled. Make sure to log the served model for every request.
3) Use your current setup as a baseline
What’s your goal in finding an open-source coding model? The one I see the most is cost reduction, but even if cost is driving the evaluation, your open-source candidate model still needs to meet your desired quality threshold. Decide what tradeoff you’re willing to accept to replace the incumbent solution.
I'd start the evaluation with the incumbent model and one or two challengers. A broader search means more to inspect, and most use cases realistically come down to 1-3 models anyway.
Use the same tasks, container, permissions, scaffold, timeout, and retry policy, then set and document resource budgets. Each model needs a supported chat template, tool-call format, and reasoning controls. Record those differences, but don’t force them to align when they shouldn’t; forcing an unsupported interface on a model would make the comparison disingenuous.
Run each model three times as a starting point, and report average success and consistency. You can then try model-specific edit formats or context strategies, but report those tuned runs separately so the reader knows which system produced each result.
Your bugs are the best benchmark
Ultimately, a private task set is where you test fit for your own team. Use problems from the repos you’d set the model or agent to work on, with the conventions and dependencies it will encounter. A small set of representative bugs can be more useful for this decision than a larger set of public benchmark tasks.
Choose tasks with acceptance criteria you can check; fixed bugs, merged PRs, and resolved tickets give you a starting point. Include different languages, task types, and levels of complexity. For each task, save the starting commit, description, evaluation tests, and regression suite. Before the task goes in the set, confirm it fails for the reason you think it does, and that the original fix passes your grader. A task that fails for the wrong reason, or a reference patch the grader rejects, will mislead every candidate equally.
Keep the solution commit, PR discussion, and held-out evaluation tests out of the agent's reach. Version the task set, image, dependencies, scaffold, prompt, and grader. If you tune against these results, use a separate holdout set for the final decision.
4) Reporting success (and failure) for open-source coding models
First, count fully resolved tasks: the target tests pass, and the regression checks still pass. Report the count with the percentage (e.g., report “20 out of 30” in addition to reporting a 67% pass rate), so the evaluation size is visible.
Break out severe regressions and consistency across attempts. Tag timeouts, malformed tool calls, and context exhaustion for diagnosis while still counting them as system failures. Agree on how to handle infrastructure failures before the run. Review a sample of patches yourself, record the review time, and state how many you inspected.
Record tokens, tool calls, attempts, elapsed time, and cost separately from the quality score. The budgets and timeouts remain part of the evaluation protocol because they affect whether a task gets completed. Once you establish quality, compare cost per resolved task, including the cost of retries.
Measure speed over the workflow. Time to first token (TTFT) matters in an interactive session; time to a completed patch matters when an agent is working through a bug. Engineers' impressions are useful too, especially for adoption, but report them separately from measured latency.
Read the failures before you trust the score
When a coding model misses a task, record the stage where it failed and the most likely cause to check next. A wrong file selected points to repository search or missing context, while a malformed tool call points to the schema, template, or parser. Each pairing is just a starting hypothesis, so label a failure "unknown" when the evidence doesn't support one.Read the trace for each failed task, including tool outputs, the patch, and test results. You need enough detail to distinguish a wrong fix from a failed edit, missing context, or a problem with the grader.
I use two labels: where the attempt failed and what might fix it. The first could be discovery, diagnosis, editing, validation, execution, context, infrastructure, or grading. The second could point to the serving layer, configuration, scaffold, missing context, model, or task. Leave it unknown when the trace doesn't support a diagnosis.
Compare effort on the same task. If one candidate makes 3x as many tool calls as the others, inspect what it kept trying. It may be searching poorly, repeating an error, or struggling with the interface. If all candidates fail, inspect the task itself as well. Neither pattern proves a cause.
Test your diagnosis before acting on it. A skipped test could come from the prompt, tool access, or the model. If you find an invalid task, document the issue and apply the same correction or exclusion to every candidate. Update the denominator. Failing on one model is not a reason to remove a task.
Tie the results back to your initial contract
Okay, now you have results; how do they compare to the contract you started with?
First, check the blocking requirements: critical workflows, regressions, and repeatability. Break out the results you obtained by any important distinguishing category (like programming language) and include a task count for each. This will help uncover a weak score in any one area, despite a high score overall. For instance, with 30 tasks and a single run, the 95% confidence interval is roughly ±17 points; one task moves the score by 3.3 points. So if one model resolves 16 tasks and the other 18, that isn't a ranking. Inspect the tasks where the two disagree, then decide whether you need more runs (consistency) or more tasks (coverage).
On your evaluation card, be sure it includes data like:
Model: checkpoint, precision, serving engine, chat template
Agent: scaffold and version, prompt, tool schema, reasoning settings
Tasks: dataset version and split, container image
Run: limits, sampling, retries, run date, infrastructure failures
Links to the configs and traces
This will help teammates (or you in the future) check if your results are reproducible (they should be).
5) Making a decision
You ran your tests and documented your results: now, which model will you choose? Once you land on a candidate open-source coding model, the ultimate test is a pilot with the engineers who will end up using it.
Request-level telemetry will show which models handle the work and whether routing follows the intended policy. In addition to token costs, compare total spend, since cached token hit rate will heavily impact cost overall.
Finally, subjective tasks are often the most powerful: ask engineers which tasks they would move over, or when/why they preferred their existing setup. This will ultimately be the biggest barrier to adoption, since a model can pass all of your evaluations but still feel awkward when applied.
The final checklist
Before making the deployment decision, do a final assessment for these problems:
Skipping smoke tests
Not defining network access
Mixing cost into the quality score
Writing the gates after seeing the numbers
Comparing your own run against somebody else's published number
Counting success on retry when production would not retry that way
No evaluation card, so nobody can reproduce or challenge what you found
Answering a software engineering question with an algorithmic benchmark
Assuming any public benchmark's tasks are valid without examining failure cases
Trusting the harness’ model label instead of checking the served model per request
Forcing the wrong chat template or tool format on a model just to have identical settings
A private task set with no tests, no check that the reference fix passes the grader, or leaked solutions
Your first evaluation will probably take the longest, but by thoroughly documenting everything, you’ll be able to reuse it when the next model arrives.
For hands-on help choosing the best open-source coding model for your use case, reach out to talk to our engineers.
