Switching from a closed source ecosystem where you consume ML models from API endpoints to the world of open source ML models can seem intimidating. But this checklist will give you all of the resources you need to make the leap.
The biggest advantage of the open source ecosystem in ML is the sheer number and variety of models to choose from. But that amount of choice can be overwhelming. Here are some alternatives to closed-source models to get you started:
Large language models (LLMs):
Text embedding models:
Speech to text (audio transcription) models:
Text to speech (audio generation) models:
Inference for most generative models like LLMs requires GPUs. Picking the right GPU is essential: you want the least expensive GPU powerful enough to run the model with acceptable performance.
For a 7 billion parameter LLM like Mistral 7B, you usually want an A10. A10s also give great performance for Whisper and Bark, but these smaller models can also fit on the less-expensive T4, though with longer generation times. And text embedding models don’t need a GPU at all, though a T4 can accelerate inference.
Here are some buyer’s guides to GPUs:
If you’re just experimenting with open source models or you need to get something in production yesterday, you can skip this step. But one of the most powerful things that switching to open source models unlocks is the ability to optimize a balance of latency, throughput, quality, and cost to align with your use case.
Get started with:
Once you have your model and hardware configuration, it’s time to deploy. You can deploy a curated selection of models from our model library in just a couple of clicks or use Truss, our open source model packaging framework, to get any model up and running behind an API endpoint.
Dive into deployment with:
Open source models in the Baseten model library.
A quickstart guide for Truss, an open source model packaging framework.
Baseten has guides for:
Another great way to build with LLMs is to use a tool like LangChain as an abstraction on top of your model endpoint, which helps with switching between models, APIs, and providers.
If you want to dive deeper, check out our guide to open source alternatives for ML models. Wherever you are in your journey from evaluation to adoption for open source ML models, we’re here to help at firstname.lastname@example.org.