Changelog

Our latest product additions and improvements.

🍮 Weekly round-up: Deploy Flan-T5 XL on Baseten

November 28, 2022

Deploy Flan-T5 XL on Baseten

Flan-T5 XL is an open-source large language model developed by Google. Flan-T5 is an instruction-tuned model, meaning that it exhibits zero-shot-like behavior when given instructions as part of the prompt. You can learn more about instruction tuning on Google's blog.

The model also comes with a starter app so that you can experiment with instruction tuning. You can give it a try here!

A screenshot of the Flan-T5 XL starter app

If you want to fine-tune and build with state of the art models like Flan T5, check out what we are working on with Blueprint and join the waitlist for early access.

🎞️ Weekly round-up: Live reload Truss in Docker dev environment

November 14, 2022

Live reload Truss in Docker dev environment

The latest release of Truss, version 0.1.5, introduces a live reload mechanic to improve developer velocity when working with Docker.

Docker is great because it makes your development environment nearly identical to your production environment. But that comes at the expense of rebuilding your environment when you make changes to your Truss. With live reload, you can now make changes to your model code and keep the same Docker container running, which can save several minutes every time you change your code.

To enable this feature, install the latest version of Truss and set live_reload = True in your Truss config file.

💺Weekly round-up: Python environment per application

November 7, 2022

Python environment per application

Until today, applications on your Baseten account shared a single Python environment. Now, you can install Python packages from PyPi or system packages like ffmpeg on an app-by-app basis. What’s more, draft and production versions of the same application also run in different environments.

This means that you can:

  • Install or upgrade a Python package without affecting applications in production
  • Run different versions of the same package in different applications
  • Publish and manage your code and dependencies in sync

Baseten’s application builder is designed for making apps to handle real production use cases, and this change gives you an even more flexible, robust developer experience.

🎃 The pumpkin patch

This week’s small-but-mighty changes to bring more magic to your models!

Use more keyboard shortcuts: Accelerate your workflows with a dozen new view builder keyboard shortcuts, listed here. My favorite: nudge components around the view with arrow keys.

Copy-and-paste improvements: Multiselect and copy-and-paste between views now work together, and pasting multiple components preserves their relative layout.

❗Special issue: Deploy MLflow models on Baseten

November 2, 2022

Deploy MLflow models on Baseten

Baseten now supports MLflow models via Truss. MLflow is a popular library for model experimentation and model management with over ten million monthly downloads on PyPi. With MLflow, you can train a model in any framework (PyTorch, TensorFlow, XGBoost, etc) and access features for tracking, packaging, and registering your model. And now, deploying to Baseten is a natural extension of MLflow-based workflows.

Deploying an MLflow model looks a bit like this:

import mlflow
import baseten
 
model = mlflow.pyfunc.load_model(MODEL_URI)
baseten.deploy(model, "MLflow model")

For a complete runnable example, check out this demo on Google Colab.

Baseten uses MLFlow's pyfunc module to load the model and packages it via Truss. To learn more about packaging MLflow models for deployment, consult the Truss documentation on MLflow.

❗Special issue: Use Stable Diffusion instantly as an API

November 1, 2022

Deploy Stable Diffusion instantly

What if instead of painstakingly configuring Stable Diffusion to run locally or paying for expensive cloud GPUs, you could deploy it in a couple of clicks? And better still, it would be instantly available as an authenticated API?

Baseten has added Stable Diffusion to our model library so you can do exactly that. Simply deploy the pre-trained model on your Baseten account then use the starter app or built-in API to use the model.

Stable Diffusion model page

Deploy Stable Diffusion today and build awesome tools for generating everything from avatars to Zoom backgrounds.

🇲🇽 Weekly round-up: Wait less. Build more!

October 25, 2022

Explore models with guidance

Often, the hardest part of a project is getting started. And when you’re getting started with an unfamiliar model, there are a few things you want to do: try it on a variety of inputs, parse its output to a usable form, and tweak its configuration to meet your needs.

A screenshot of the Whisper model's new README

Baseten’s library of models now features comprehensive updated READMEs for many of our most popular models, with more coming soon. 

Load Baseten up to ten times faster

Baseten power users are filling their workspaces with powerful models and dynamic apps. And we found that as the number and size of deployed systems grew on an account, load times shot way up. So we refactored the user interface to load much faster.

But saying “the website is way faster” is hardly useful information. Here’s a table showing how much loading time is saved:

A chart of average loading times

Saving time on your MLOps isn’t just about removing clunky hours-long deploy processes. We also care about saving you seconds at the margin.

🪵 Weekly round-up: Deploy OpenAI Whisper instantly

October 17, 2022

Deploy OpenAI Whisper instantly

We added Whisper, a best-in-class speech-to-text model, to our library of pre-trained models. That means you can deploy Whisper instantly on your Baseten account and build applications powered by the most sophisticated transcription model available.

A screenshot of the Whisper starter app

You can deploy Whisper from its model page in the Baseten app. Just sign in or create an account and click “Deploy.” The model and associated starter app will be added to your workspace instantly. Or, try the model first with our public demo.

Review improved model logs

In a comprehensive overhaul, we made model logs ten times shorter but way more useful. Here’s what we changed:

  • Build logs are now separated into steps for easier skimming
  • Model deployment logs are surfaced just like build logs
  • Model OOMs are now reported
  • Many extraneous log statements have been deleted

OOM logging is a particularly important improvement. An OOM, or out-of-memory error, is a special lifecycle event that we monitor for on Kubernetes. This error means that the model is too big for the infrastructure provisioned for it. Existing logging solutions don’t capture these errors, resulting in frustrating debugging sessions, so we built a special listener to let you know about OOMs right away.

A screenshot showing an OOM error in entries 4 and 5

🌰 Weekly round-up: Select multiple components in the view builder

October 11, 2022

Select multiple components in the view builder

In the view builder, you can now select multiple components at the same time and move them as a single block. You can also bulk duplicate and bulk delete multiple selected components.

Selecting multiple components, duplicating them, then deleting them

To select multiple components, either use Command-click on each component you wish to select, or drag your cursor over an area of the screen to select everything within its path.

🎃 The pumpkin patch

This week’s small-but-mighty changes to bring more magic to your models!

Set image empty state: You can now specify custom text to appear in an image component when no image is present.

The new empty state placeholder field

Remove canvas frame: You can hide the canvas frame in your application to give the published views a consistent all-white background.

The same app with and without a canvas border

🍂 Weekly round-up: Deactivate and delete model versions

October 4, 2022

Deactivate and delete model versions

Baseten supports deploying multiple versions of the same model, so you can iterate, test, and experiment to your heart’s content. Now, you can either deactivate or delete model versions when they are no longer useful.

A screenshot showing the options available on a deployed model version

A deactivated model version cannot be invoked or used in applications, but can be re-activated. A deleted model version is permanently gone. Either way, neither deactivated nor deleted model versions count against your deployed model limit.

Record audio with new microphone component

When building UI models like Whisper and wav2vec that process audio, you’ve been able to let users upload audio clips with the file upload component. With the new microphone component, you can instead let users capture audio directly in the app.

A screenshot showing the new microphone component

🎃 The pumpkin patch

This week’s small-but-mighty changes to bring more magic to your models!

Share state between views: If you’re building a complex application on Baseten with multiple views, you might want to share state between those views. This is useful for building interactions like clicking on a row in a table and going to a detail page pre-populated with that row’s information.

New account profile and API key pages: Go to Settings in the main sidebar and you’ll find Account settings broken out from Workspace settings for easier access

Set object fit in image components: Select from five options to set the object fit that works best in the context of your application.

A gif of the five different image fit options

❗Special issue: Build stateful web pages with Python

September 30, 2022

As a data scientist, you can do a lot with Python: train models, build servers, write scripts. Now, in the Baseten app builder, you can do something special with Python: build dynamic front-end web interactions that run in the browser.

The Python code in this screenshot runs directly on the end user's browser

JavaScript’s claim to fame is that it is the language of the web browser. But building a dynamic front-end experience shouldn’t require learning a new programming language. So we now support Python as the default way to write front-end functions in the view builder. JavaScript is still supported as well.

Before, with Javascript:

let nums = [1, 2, 3, 4, 5]
let evens = nums.filter(function (num) {
  return (num % 2 == 0);
});
return evens

Now, with Python:

nums = [1, 2, 3, 4,5]
return [n for n in nums if n % 2 == 0]

Python in the view builder isn’t just for basic operations. You can import many packages, including all Python built-in packages and popular data science libraries like numpy. Unfortunately, due to limitations in the technology powering Python in the web browser, you can’t install your own packages like you can in the back end.

A screenshot of an app build with Python in the view builder

To help you get started with this exciting new capability, we prepared a tutorial on building a stateful web app with Python in the view builder. Follow along to build an auto loan payoff calculator in fifteen minutes using drag and drop components, state, and a Python function. Read the tutorial here.

🏈 Weekly round-up: Deploy models with any interface

September 27, 2022

Deploy models with any interface

By default, when you deploy a model to Baseten, like in this XGBoost classifier example, the deployed model expects to receive a dictionary with the key inputs and a list of input values, and will return a dictionary with the key predictions and a list of model results.

model_input = {"inputs": [[0, 0, 0, 0, 0, 0]]}
model_output = {'predictions': [0.21339938044548035]}

Until now, that default behavior has not been changeable. All models deployed to Baseten had to follow that spec. However, this interface was too inflexible, so as of the most recent version of the Baseten Python package (0.2.7), you can set your own interface for your models.

Setting your model interface

Baseten uses Truss under the hood for model deployment. You can customize your model’s interface by editing the predict function in models/model.py in Truss. The auto-generated predict function for the aforementioned XGBoost example looks like this:


def predict(self, request: Dict) -> Dict[str, List]:
    response = {}
    inputs = request["inputs"]
    dmatrix_inputs = xgb.DMatrix(inputs)
    result = self._model.predict(dmatrix_inputs)
    response["predictions"] = result
    return response

Modify this function to parse whatever request and response you want, and remember that Truss also supports pre- and post-processing functions that can further modify input and output when more complicated parsing is needed.

Backwards compatibility

This change does not modify the behavior of existing deployed models, nor the default behavior of future models. However, it does change how deployed models are invoked through the Baseten Python client.

Previously, the predict() function wrapped its argument in a dictionary with the inputs key. Now that said key is not required, the predict() function passes its inputs as-is, which means you have to enter the entire model input yourself.

Before (1.0 spec):

baseten.predict([[0,0,0,0,0,0]])

Now (2.0 spec):

baseten.predict({"inputs": [[0,0,0,0,0,0]]})

The syntax for invoking a model via an API call has not changed.

However, to prevent breaking existing scripts using baseten.predict, a new spec_version flag is now included in Truss. This parameter is set to 2.0 by default for all new models, so they will use the new input spec, but all existing models will continue to function exactly as they have been on the 1.0 spec. You can upgrade your model to the latest interface spec by changing the flag in the config.yaml file in Truss.

Enjoy this unrestricted interface by installing the newest versions of Truss and the Baseten client.

pip install --upgrade baseten truss

Pass a Truss to baseten.deploy()

We also cleaned up the deployment experience in the Baseten Python client. You no longer have to use different functions to deploy an in-memory model versus a model packaged as a Truss. Whatever you have, toss it into baseten.deploy() and we’ll take care of it.

Plus, when you deploy an in-memory model, the deploy function now gives you insight into what is happening to package that model, including the path to where the auto-generated Truss folder lives. This is useful if you want to change something about your deployed model’s behavior (like updating the interface) after deploying the model, just edit the truss and pass it into baseten.deploy() to ship a new version!

INFO Autogenerating Truss for your model,
find more about Truss at https://truss.baseten.co/
INFO You can find your auto generated Truss at
/root/.truss/models/lightgbm-MMVJD0

🎃 The pumpkin patch

This week’s small-but-mighty changes to bring more magic to your models!

Tab complete bindings: In the view builder, you can create a binding tag by typing “{{“ and you can now close the binding by pressing “Tab” after selecting the data you want to use in the binding.

Model building banners: When you deploy a new version of a model to your Baseten workspace, you’ll see a banner on other versions of the model letting you know that the new version is building.

A screenshot of the model deployment banner showing a new version of the bert-base-uncased model building

🕵️ Weekly round-up: Search your resources

September 21, 2022

Search your resources

Where did I leave that cutting-edge machine learning model? Never lose track of your application resources again with full workspace search. Just type Command-k (Ctrl-k on Windows) or click the search bar and you’ll be able to find the resource you're looking for: view, data table, code file, anything!

A screenshot of the resource search UI overlaying the view builder

Search results are segmented by scope: the application you're in your Baseten workspace as a whole. Click on any resource and you’ll be taken directly to it!

🎃 The pumpkin patch

This week’s small-but-mighty changes to bring more magic to your models!

Copy data explorer values: Use values from the data explorer with a convenient copy button, just like worklet outputs. The copy button appears per value on hover and copies the information as JSON.

A screenshot showing the copy button on data explorer values

📠 Weekly round-up: Back up your apps to GitHub

September 6, 2022

Back up your apps to GitHub

This week, a feature for workspaces that use our starter or business plans: back up your Baseten apps to GitHub. Paired with the draft environment, this gives you a safety net while developing your production applications. Get started by following the GitHub sync docs.

An example GitHub repository storing apps built on Baseten

Once GitHub sync is turned on in your workspace, you can enable it app-by-app to back up applications to GitHub. Applications will be backed up immediately, and a new version will be committed to the repository each time you publish.

🎃 The pumpkin patch

This week’s small-but-mighty changes to bring more magic to your models!

Set multiline secret values: In your workspace settings, you can now store multiline secret values. This is useful for storing things like PGP and SSH keys.

Setting a multiline secret

Reorder event handlers: In the view builder, you can add multiple event handlers to each component. Event handlers are executed sequentially. If you want to change the order that event handlers run in, you can now do so by dragging and dropping them into place.

Reordering event handlers

📧 Weekly round-up: Model deployment emails

August 29, 2022

Model deployment emails

Watching code build is the digital equivalent of watching paint dry. Model deployment on Baseten generally runs in less than five minutes, but that's enough time to:

  • Make yourself a cup of coffee
  • Listen to your favorite song
  • Run a mile (if you're Usain Bolt)

So take a microbreak — we'll send you an email once the model deployment is finished!

An email confirming successful model deployment

We also send a notification email if a deployment fails or if an active model goes down. And you can click the link in any model status email to be taken directly to the deployed model’s page.

🎃 The pumpkin patch

This week’s small-but-mighty changes to bring more magic to your models!

Set state with math: When using the “set state” action, the value field will automatically evaluate mathematical expressions.

Setting state with a mathematical expression

App & model tags: Starter applications and pre-trained models are now tagged in their respective tables. These resources do not count against your workspace limits.

Starter applications tagged in the applications list

🎺 Weekly round-up: Include secrets in deployed models

August 22, 2022

Include secrets in deployed models

The word “secret” is derived from Latin, secreto, which literally translates to “don’t store your API keys and AWS access tokens in your source code.” Amazing what insights linguistics offers!

For models that use secrets to access file stores, databases, external APIs, or other resources, you need to store secrets to access these resources. Secrets also pop up often in pre- and post-processing code to store model inputs and outputs. When you use Truss to deploy your model on Baseten, you can add secrets to your model’s config as follows:


secrets:
    my_api_key: null

Then, you can access the secrets in the Truss' model file by passing them into the init function:


def __init__(self, secrets) -> None:
    self._api_key = secrets["my_api_key"]

To set secrets in production, use Baseten’s secret management feature. Get started with secrets by following the documentation to add and reference secrets in your model.

Connect to external Postgres databases with SSL

Baseten supports data connections to external sources. Now, data connections to PostgreSQL tables come with an optional additional layer of security: connecting with SSL. Simply check the SSL option and paste in a client key to connect via SSL. You also have the option to provide a self-signed certificate.

Options for adding an SSL certificate to PostgreSQL data connections

Let us know if you need SSL for another data connection like MySQL and we will expand the functionality based on user requests!

🎃 The pumpkin patch

A handful of small-but-mighty changes to make Baseten more joyful to use:

  • Workspace settings are now organized as full-page tabs rather than a modal
  • The app builder tour is dismissible for those who prefer to explore on their own