Optimizing Inference Performance for the Open-Weight LLM GLM-5.1

See how Fixstars engineers optimized inference for the 754B-parameter open-weight LLM GLM-5.1, cutting P90 Time-to-First-Token from 59 seconds to 3.8 seconds (about 15x) and roughly tripling KV-cache capacity on a single 8-GPU NVIDIA H200 node. Download the white paper for the full approach, including DP Attention, KV-cache-aware routing, and results on NVIDIA B200.

White Paper Overview

GLM-5.1 is a 754-billion-parameter open-weight LLM, and it powers Fixstars Vega, our secure on-premises AI appliance. Running a model this large inside a single server is difficult. When we first deployed it on one node of eight NVIDIA H200 GPUs, the model loaded but ran far too slowly to be practical.

This paper explains how our engineers fixed that. We start by looking at why GLM-5.1 is hard to optimize: its MLA attention, DeepSeek Sparse Attention, and 256-expert MoE layers do not fit well with the usual tensor-parallel approach. From there, we describe three changes that did most of the work. DP Attention removes the KV-cache duplication that tensor parallelism forces, which frees up far more cache capacity. KV-cache-aware routing, combined with CPU cache offloading, keeps the system from recomputing the prefill. And instead of a synthetic benchmark, we built our test set from real coding-agent sessions so the numbers reflect actual use.

The effect was substantial. P90 Time-to-First-Token fell from 59 seconds to 3.8 seconds, about 15 times faster. Total KV-cache capacity per node grew roughly 3.5 times, and the node held a stable concurrency of more than 50 against an original target of 30, all on the same hardware. We also report results on NVIDIA B200 GPUs and share what we learned about choosing hardware, quantization format, and parallelization strategy for large MoE models.

If you are deploying open-weight models on your own infrastructure or evaluating on-premises AI hardware, this paper gives you a concrete, reproducible account of how to make it work. Download this white paper to learn how Fixstars optimized GLM-5.1 inference for faster response times and greater capacity on a single GPU node.