Coding agents have become one of the fastest-growing line items for engineering organizations. Open models now perform close to frontier closed models at a fraction of the cost. The harder questions are which tasks they're great for, how to test them in your own codebase, and how to roll them out without impacting your engineer's productivity.
In this session, Alex Ker and Molly Effers of Baseten walk through a practical framework for evaluating and piloting open models for agentic coding, drawn from work with engineering teams of every size. They'll cover:
Different objectives in rolling out open models for coding, including cost, quality, and ease of adoption, and particular examples of how to reduce cost levers beyond picking the lowest cost per token.
Which open models are worth shortlisting today, and how to map them from the closed models your engineers already use
How to scope a pilot that keeps your existing harnesses, such as Claude Code and Codex.
How to build a benchmark from your own merged PRs and measure cost per successful task
Important performance and reliability metrics to check before rollout
You'll leave with a clear process for deciding where open models fit in your stack, along with a detailed template to run your own evaluation.
