GPU temperature and utilization metrics

You can now track three additional GPU metrics in the Metrics tab for your Dedicated Inference deployments:

• GPU temperature: Temperature in degrees Celsius.

• GPU tensor core utilization: Tensor core activity as a percentage, on GPUs with profiling support.

• GPU memory bandwidth utilization: Memory read and write activity as a percentage.

Compare these graphs with request load and latency to investigate performance bottlenecks. Use Customize view in the Metrics tab to show the graphs you need.

For more information, see our docs.