NVIDIA’s leadership in the AI hardware market has made it the world’s most valuable company. But it is far from the only company to make hardware that can run AI inference.
From fellow industry giants like Amazon and Google to a massive crop of startups, competitors are pouring billions of dollars into developing and manufacturing alternatives to NVIDIA GPUs.
While this book focuses on optimizing inference on NVIDIA GPUs, here’s a short list of the other notable hardware options for running model inference.
| Company | Stage | Flagship Product |
|---|---|---|
| AMD | Public | MI350 GPU: A datacenter GPU with competitive specs on AMD’s own software stack. |
| AWS | Public | Inferentia and Trainium: A pair of chips purpose-built for inference and training, respectively, on AWS. |
| Cerebras | Startup | WSE-3: A wafer-scale chip with extremely high memory bandwidth to remove decode bottlenecks. |
| Etched | Startup | Sohu: An Application-Specific Integrated Circuit (ASIC) for the transformer architecture. |
| Furiosa | Startup | RNGD: A power-efficient accelerator designed for tensor contraction operations. |
| Public | TPU: A Tensor Processing Unit is an AI-specific ASIC built for inference and training. | |
| Groq | Startup | LPU: A composable language processing unit relies on SRAM for high memory bandwidth |
| Qualcomm | Public | Cloud AI 100 Ultra: A full-sized GPU composed of multiple power-efficient mobile GPUs. |
| Sambanova | Startup | RDU: A Reconfigurable Dataflow Unit with large memory allocation for trillion-parameter models. |
GPUs are fairly general-purpose accelerators. Every hardware company competing to win inference workloads from NVIDIA is betting on a specific edge where their product can win:
- Memory bandwidth: Startups like Cerebras and Groq achieve high token per second scores for LLMs by accelerating decode on ultra-high-bandwidth memory.
- Power efficiency: Companies like Furiosa and Qualcomm design chips for lower power consumption, which leads to cheaper operating costs.
- Platform integration: Enterprises like Amazon and Google build deep integrations with their cloud service platforms and proprietary closed models.
While each of these accelerator options have their winning use cases, they share common challenges:
- Software: Without CUDA, hardware providers have to rebuild the entire inference stack for their accelerators.
- Manufacturing: Companies need to assemble the most complex object humankind has ever created.
- Distribution: After manufacturing a chip, providers need it installed and brought online to put capacity on the market.
Competition accelerates innovation. A market with more hardware options in the datacenter is good for every inference engineer. Competition in this space will only grow more robust as inference workloads become more valuable, as will exploration of options outside the datacenter such as local inference.