Company overview
Parallel is built on a single premise: AI agents will use the web a thousand times more than humans ever have. Parallel powers web research for AI-native teams at companies like Harvey, Granola, Hex, and Dropbox. Parallel builds the infrastructure agents need to retrieve the most relevant and high-quality information when they need it. Rather than having agents scrape, extract, and compress raw web content themselves, Parallel’s products resolve multi-hop research in a single call, delivering structured, LLM-ready results. This lowers both end-to-end agent cost and latency.
Impact highlights
2x latency improvement with Baseten speculative decoding and KV cache-aware routing
3x cost savings when moving from closed-source to open-source models
Inference for multi-turn agentic pipelines
Challenge
Parallel utilizes structured agent pipelines to power their Web Tool APIs. Single jobs can involve multiple model calls to reason, generate search queries, and synthesize results. To support the tens of thousands of requests per hour generated by these pipelines, Parallel needed throughput-optimized inference capable of handling significant load and bursty traffic, since each agentic pipeline often requires multiple turns. To adeptly serve such a significant load, Parallel needed inference that could cut prefill time with KV cache-aware routing. Higher throughput enables Parallel's web agents to run longer and across many turns, enabling them to return results that no human researcher can replicate at scale.
Two weeks before launching their Search API, Parallel was serving thousands of multi-hop agentic searches per hour on closed-source models. However, given their projected long-term scale, the costs of relying on closed weight models post-launch would be unsustainable. To solve this, Parallel identified an open model that met their strict quality standards and began searching for an inference partner capable of providing the performance and reliability their workload required.
"Our workloads can be very spiky. We need to be able to manage the peaks, but we also don't want to pay for the valleys. Pay-per-token with Model APIs enabled us to easily make the transition to open-source until our scale merited dedicated deployments optimized for throughput and flexible autoscaling."
Solution
Two weeks before their Search API launch, Parallel turned to Baseten's Model APIs to handle their agentic pipeline. With Baseten’s Model APIs, Parallel utilized the latest open-source models with pay-per-token pricing with infrastructure that could the unpredictable traffic spikes ahead of their upcoming launch. Parallel’s Search API launch blew past demand expectations, all of which was served on Baseten Model APIs.
Post launch, it was soon apparent that Parallel had enough traffic to merit a dedicated deployment of the open-source model they were utilizing on Model APIs. Dedicated unlocks greater performance stability (no API neighbors) and customizability (custom autoscaling settings and performance optimizations). The Baseten model performance team optimized the open-source model Parallel selected utilizing speculative decoding with DFlash to cut latency in half.
Since the first deployment of a dedicated open-source model on Baseten, Parallel has explored additional models to further cut down on costs without a loss of quality.
"We're seeing open-source models get cheaper and better while the frontier models just keep getting more expensive. Open-source models can give us the same quality as small closed-source models on certain use cases at about a third of the cost. But to actually run those workloads properly, we needed dedicated deployments with KV cache-aware routing — otherwise every turn in the conversation resets and you lose the whole benefit."
Result
Baseten’s proprietary runtime cut latency by 50% and increased throughput by 3x through KV cache-aware routing and speculative decoding. Within a few months of their initial deployment, Parallel scaled traffic with Baseten ~100x. That growth was made possible by the flexibility of the Baseten product portfolio across multi-tenant and dedicated infrastructure. Model APIs serve unpredictable early demand, while dedicated deployments provide the stability and control to scale with confidence.
Baseten’s throughput and latency optimizations delivered 3x savings versus the equivalent closed-source model. Eliminating redundant processing made it economically viable to run the kind of long, multi-turn agent workflows that deliver meaningfully better results than a single-shot query. As open-source models continue closing the quality gap, Parallel now has the infrastructure to move workloads quickly, without taking on meaningful operational overhead.
"We have a lot of tokens we want to use to continually test new models. But we’re a very lean team and need minimal ops overhead. As new models comes out, we want to minimize the time to test, optimize, and deploy it into production. Baseten is the infrastructure that lets us do that."
Closing the retraining loop on a single platform
Challenge
For Parallel, the economics of agentic inference only worked when smaller, fine-tuned models can take over work that larger general-purpose models are doing today. A fine-tuned, small open-source model can match the quality of closed-source models on a narrow task at a fraction of the cost. To enable this, the Parallel team needed to fine-tune, deploy, shadow-test, and iterate quickly enough to keep up with their own model roadmap.
Rapid iteration between training and testing models requires an integrated infrastructure. Splitting training and inference across different providers creates constant bottlenecks, forcing teams to transfer weights, rebuild configs, and retest optimizations after every run. Parallel needed training, fine-tuning, dedicated deployments, and runtime optimization to live on the same platform.
Solution
Parallel runs multi-node training jobs and interactive development for rapid experimentation. Parallel stood up several fine-tuned production models within weeks of onboarding to training, leveraging Baseten’s on-demand compute to run multi-node training jobs when needed while only paying for what they used.
Their retraining loop runs entirely on Baseten. Fine-tuned weights deploy with one click from a completed training run into a dedicated deployment. From there, Parallel routes shadow traffic against the new model, validates accuracy, and shifts production traffic over without ever moving data between vendors. The most impactful lever so far has been migrating certain search workloads from frontier models onto a fine-tuned 4B model, compounding the cost savings already captured from the closed-to-open-source migration.
"Running training jobs and deploying them on the same platform has enabled us to iterate faster. My team can spin up a new fine-tune, ship it to a dedicated deployment, shadow test it against production traffic, and deploy to production all with Baseten. That feedback loop has unlocked a flexibility to constantly iterate and improve model quality and economics"
Result
Parallel stood up multiple fine-tuned production models within weeks of onboarding to training, and is now operating a continuous retraining cycle where workloads progressively migrate from large general-purpose models to smaller, use-case-specific fine-tunes.
The broader effect is on team velocity. Parallel can now treat every new open-source release as a candidate for a same-week shadow deployment, with no infrastructure migration tax. A closed training to inference loop ensures Parallel can continuously iterate to increase quality and control economics as the number of SOTA open-source models rapidly grows.
