Inference engineering draws on many technologies, from CUDA to Kubernetes. But there’s one critical area that’s often overlooked when optimizing for latency and building for scale: client code.
There are two sides of a call to an inference service:
- Client: The browser, agent, or application making a request to the inference engine.
- Server: The inference service that handles the client request and returns the model results.
The industry standard for client code is the OpenAI SDK, which supports a wide range of compatible providers in addition to OpenAI’s own models. Popular AI engineering frameworks and libraries like LangChain, Vercel AI SDK, LiteLLM, LlamaIndex, and dozens more can also serve as clients.
Whether you’re using an existing library or your own code, there is the potential for latency overhead or throughput bottlenecks. And for real-time applications, you may need a protocol other than HTTP, like WebSockets, to deliver a continuous connection.
7.5.1 Client Latency Overhead
Depending on the client’s internet connection and the protocol used, establishing a session between a client and server takes a few dozen milliseconds.
In a high-performance system with a 300-millisecond P95 end-to-end latency SLA, a TLS handshake costs at least ten percent of that latency budget before inference even starts. Future requests from the same client should save time by re-using existing sessions.
Session re-use is not a new idea by any means, and tools like the OpenAI SDK provide it silently under the hood. However, when building your own client for non-standard modalities, follow best practices like session re-use.
7.5.2 Asynchronous Inference
Some systems are built for throughput, not latency. Use cases like bulk document processing and corpus embedding are not latency sensitive, so it makes sense to switch to asynchronous jobs.
Asynchronous requests are a “fire and forget” approach to executing inference.
Ordinary synchronous inference requests have a timeout, generally of a few minutes, after which the request will fail. Asynchronous jobs fix this by immediately acknowledging the request and later returning the result of the asynchronous job to a webhook supplied in the original request.
Asynchronous jobs still have time limits, but these requests are usually measured in hours, not minutes. Along with strong server-side queuing, asynchronous requests make high-throughput, latency-insensitive systems more robust and efficient.
7.5.3 Streaming and Protocol Support
Streaming makes applications feel instant. For language models, streaming text output over HTTP is sufficient. But for other modalities, especially live voice and video, both input and output streams need to be able to carry more data.
The two most common bi-directional streaming client-server connection protocols are:
- Websockets: For streaming use cases where strong schema enforcement is not required.
- gRPC: For well-defined service-to-service communication.
WebSockets are useful for transmitting unstructured and real-time data, like audio, where the server receiving the request can parse it and process it downstream. With WebSockets, a server can support up to a fixed, developer-configurable number of clients; when that concurrency is reached, new connections cannot be established and must wait until either a slot is free or another replica scales up.
Similar to WebSockets, gRPC enables bi-directional streaming support, but for structured data. Requests transmitted via gRPC must follow a predefined schema, which takes away the load of having to parse the input. This additional validation layer makes gRPC slightly slower than WebSockets.
