Please rotate your device

To continue exploring this website, please rotate your device to a vertical orientation

Changelog

Startup cache

Date

Use the startup cache to cut the time later replicas spend recompiling kernels. A deployment's first replica compiles its torch.compile, Triton, CUDA JIT, and autotuner output before it's ready, and the cache restores those artifacts on every replica that scales up, wakes from scale to zero, or restarts, so those replicas become ready sooner.

The cache needs no config.yaml setting or code change, and it works with single-node Truss deployments and Baseten Inference Stack engine deployments, including multi-node ones. Contact us to turn it on for your organization, then push or promote a new deployment to start using it.

For more information, see our docs.