The semiconductor research firm SemiAnalysis has published a social media post offering high praise for NVIDIA's performance optimizations in its vLLM inference engine, while also clearly pointing out that Advanced Micro Devices (AMD) still has significant ground to cover in achieving vLLM support for certain models.
This assessment comes just two weeks after the same firm suggested NVIDIA's "CUDA moat is slowly eroding," citing growing competitive pressure from hyperscale cloud providers and AI model companies developing their own custom ASICs for specific training or inference tasks.
These two contrasting viewpoints refocus the AI chip competition debate from "who has the superior hardware" to a more fundamental question: in the era of large-scale inference deployment, the depth of a software ecosystem may prove more decisive than a single generation of GPU hardware leadership.
For AI enterprises handling billions of inference calls daily, the chasm between "good support for some models" and "stable optimization for all models" is magnified exponentially by scale effects.
Against a backdrop of intensifying hardware competition and custom ASICs gradually capturing inference market share, NVIDIA's accumulated depth in its inference software stack is emerging as a more durable competitive barrier than raw hardware specifications.
The vLLM Gap Reveals an Ecosystem Divide
vLLM is currently the most widely used open-source engine for large language model inference.
SemiAnalysis's choice of vLLM as a benchmark itself conveys a judgment: the open-source inference ecosystem is becoming a key battleground for measuring the real-world performance of AI chips.
The gap is particularly pronounced in the case of the Kimi K2.5 model, a hybrid expert (MoE) model with hundreds of billions of parameters.
NVIDIA's GB200 NVL72 platform connects 72 GPUs via rack-scale NVLink, supporting wide expert parallelism (Wide EP) at a scale of 8 to 16.
This architecture significantly reduces the expert weights each GPU must handle, thereby lowering HBM bandwidth pressure and enabling All-to-All communication to occur within the high-speed NVLink domain instead of over the slower InfiniBand network.
The result is a per-GPU throughput exceeding 12,000 tokens/second. In contrast, the Advanced Micro Devices (AMD) MI355X performed notably worse in the same test, primarily due to its inability to achieve comparable expert parallelism and rack-scale interconnectivity.
On the software side, NVIDIA's Dynamo distributed inference framework deeply integrates vLLM, incorporating optimizations specifically for MoE models like disaggregated serving for prefill and decode, efficient KV cache transfer, and dual-batch overlapping.
This framework can fully leverage the hardware potential on the NVL72 platform, whereas Advanced Micro Devices (AMD) currently relies mainly on standard vLLM and its DISAGG version, lacking the deep optimizations for massive MoE models and wide-parallelism scenarios.
The Implication Behind "Some Models"
SemiAnalysis's qualification that Advanced Micro Devices (AMD) is behind on "some models" contains two layers of information.
First, AMD is not lagging across the board. In general-purpose computing scenarios, its MI series GPUs already possess competitive strength, a fact validated by Meta's recent massive procurement order.
Second, the word "some" precisely highlights the nature of the current gap: the absence of comprehensive software ecosystem coverage.
In AI inference scenarios, enterprise customers seek stable, predictable deployment experiences. The cost of maintaining two separate software stacks to cover different models is often a decisive factor in decision-making.
For Advanced Micro Devices (AMD) to bridge the gap from "leading in some areas" to "fully viable everywhere," the required software engineering investment may be more time-consuming than hardware iteration.
This is not merely a matter of writing more drivers and adaptation layers; it means establishing broad compatibility and trust across thousands of model architectures, continuously evolving open-source frameworks, and a fragmented developer community.
In the Inference Era, Software is the True Moat
As the AI industry's focus shifts from training to inference, the strategic value of software versus hardware is being structurally reassessed.
Training tasks are centralized and controllable, where hardware performance gaps can be bridged with capital investment. Inference, however, is distributed and continuous; microsecond-level latency differences are magnified across billions of daily calls, directly translating into diverging cost structures.
NVIDIA's software ecosystem barrier is built upon three overlapping layers: the CUDA toolchain with its two decades of development and roughly 4 million developers; priority adaptation for all mainstream machine learning frameworks; and deep integration with optimization libraries like cuDNN, TensorRT, and NCCL.
The switching costs generated by the combination of these three factors far exceed the gap of any single hardware specification. This perspective aligns with SemiAnalysis's analysis from two weeks prior.
At that time, the firm noted that Anthropic had established a multi-platform compute architecture using Google TPUs, Amazon Trainium, and NVIDIA GPUs, with much Claude model training running on TPUs and Claude Code inference increasingly deployed on Trainium, slowly eroding NVIDIA GPU's share.
However, the recent positive assessment of vLLM performance indicates that NVIDIA's lead in deep optimization of the inference software stack has not narrowed in sync with the evolving hardware competitive landscape.
Comments