vLLM & Performance Tuning

Let’s apply our knowledge to a concrete scenario: you need to serve granite-3.3-8b-instruct that is integrated with a chat application. Your primary goal is to minimize inference latency for a given number of concurrent users - 32 in this case - with an expected generation length <2048 tokens. As you have limited flexibility to scale GPU resources, you want to understand the key vLLM parameters and strategies to maximize performance before simply adding more hardware.

Before diving into tuning, it’s important to understand the key performance considerations in vLLM:

Latency: The time it takes for a single request to complete. Of particular importance here is going to be tracking the Time To First Token (TTFT), which will ensure the user can experience snappier responses.

Throughput: The number of requests (or tokens) processed per unit of time. While we’re optimizing for latency, higher throughput often means better resource utilization, which can indirectly help avoid latency spikes due to queuing.

Granite-3.3-8b-instruct is a popular, powerful small-size dense model. Here are the primary avenues for optimization.

Quantization is arguably the most impactful change you can make for latency, especially with vLLM’s efficient kernel implementation for w8a16 or w4a16.

Why? Reducing precision directly shrinks the model’s memory footprint and enables faster arithmetic on modern GPUs.

What to try (highly dependent on available hardware):

FP8: If you have access to NVIDIA H100 GPUs or newer (e.g., B200), FP8 (E4M3 or E5M2) is a game-changer. These GPUs have dedicated FP8 Tensor Cores that offer significantly higher throughput compared to FP16. This provides a direct path to lower latency per token without significant accuracy loss for Llama 3 models.

INT8 (e.g., AWQ): Starting with A100 or even A6000/3090 GPUs, INT8 quantization is an excellent choice. It reduces the model to 8B * 1 byte = 8GB, halving the memory footprint and enabling faster integer operations.

INT4: If you’re pushing for absolute minimum latency and can tolerate a small accuracy trade-off, INT4 (e.g., via AWQ or other 4-bit methods) can reduce the model to 8B * 0.5 bytes = 4 GB. This is extremely memory-efficient and, on some hardware, can offer further speedups. Test accuracy thoroughly with your specific use case, as 4-bit can sometimes be more sensitive. Similarly, check out FP4 versions when Nvidia Blackwell hardware is available.

Please refer to the compatiblity chart https://docs.vllm.ai/en/latest/features/quantization/supported_hardware.html for up to date quantization support in vLLM.

Your chart should look similar to the one below.

Up to 2x speedup!

Following the same principle as quantization, serving a smaller model (when accuracy on task is acceptable) will enable faster response times as less data is moved around (model weights+activations) and less sequential computations are involved (generally fewer layers). For this particular use-case, consider ibm-granite/granite-3.1-2b-instruct.

While Granite 3 is a strong dense model, for certain latency-sensitive scenarios, considering a Mixture-of-Experts (MoE) model like Mixtral 8x7B could be a compelling alternative.

Why MoE for Latency? MoE models have a large total number of parameters (e.g., Mixtral 8x7B has 47B total parameters), but critically, they only activate a sparse subset of these parameters (e.g., 13B for Mixtral 8x7B) for each token generated. This means the actual computational cost per token is significantly lower than a dense model of its total parameter count. Which is especially true when sharding experts over multiple GPUs with MoE especially with vLLM’s optimized handling of MoE sparsity.

Trade-offs: While MoE models can offer lower inference latency per token due to their sparse activation, they still require enough GPU memory to load the entire model’s parameters, not just the active ones. So, Mixtral 8x7B will demand more VRAM than Llama 3 8B, even if it’s faster per token. You’ll need sufficient GPU memory (e.g., a single A100 80GB or multiple smaller GPUs with tensor parallelism) to fit the full 47B parameters.

vLLM has strong support for MoE models like Mixtral, including optimizations for their unique sparse compute patterns and dynamic routing.

Consider When: Your application might benefit from the increased quality often associated with larger (total parameter) MoE models, combined with the per-token speed advantages of their sparse computation.

Speculative decoding is a powerful technique to reduce the generation latency, particularly noticeable for the Time To First Token (TTFT). Speculative decoding is fundamentally a tradeoff: spend a little bit of extra compute to reduce memory movement. The extra compute is allocated towards the smaller draft model and consequent proposer verifying step. At low request rates, we are memory-bound, so reducing memory movement can really help with latency. However, at higher throughputs or batch sizes, we are compute-bound, and speculative decoding can provide worse performance.

The graph here from this blog highlights the tradeoffs of speculative decoding at low request rate vs bigger batch sizes.

Take away message: As long as the number of requests is bound to use a non-intensive amount of GPU resources (lower req/s), speculative decoding can provide a nice speedup.

What to try: You’ll need to specify a smaller draft model. A good starting point for Llama/Granite might be a smaller Llama/Granite variant or as in this example a speculator trained specifically for our use-case. Let’s redeploy with the following engine arguments:

What arguments would you change for a batch processing job that analyzes 100,000 customer reviews every night?

Optimization is an iterative process. As you tune vLLM, continuously monitor key metrics: