Google is breaking down the walls around its custom tensor processing units, opening up its in-house AI chips to outside clients for the first time on a significant scale.
Independent research firm SemiAnalysis has published the first third-party inference benchmarks for the TPUv7 Ironwood, revealing that Google's latest chip beats NVIDIA Corp (NASDAQ: NVDA) B200 and B300 by up to 50% on a cost-per-dollar performance basis under matched conditions. This marks a formal challenge to NVIDIA Corp (NASDAQ: NVDA)'s dominance in the AI inference chip market.
SemiAnalysis data shows that at an interactive speed of 100 tokens per second per user, Ironwood's inference cost is roughly $0.181 per million tokens, coming in 19% cheaper than the B200's $0.222 and 34% cheaper than the B300's $0.276.
Under heavier concurrent workloads and using Google's internal total cost of ownership model, Ironwood's dollar-for-dollar performance edge expands to 76.7% over the B200 and an even more striking 130.2% over the B300. In parallel, Google is pushing forward with TorchTPU, a next-generation native PyTorch backend slated for an October open-source release. This initiative aims to replace the TorchAX solution, which had significant limitations, and lower the barrier for external developers looking to build on TPU hardware.
This development directly threatens NVIDIA Corp (NASDAQ: NVDA)'s market position. Anthropic has committed to purchasing over one million TPUs, becoming Google's largest external TPU customer, with projections suggesting its TPU usage will surpass DeepMind's internal consumption by 2029. SemiAnalysis believes that as optimizations like prefill/decode disaggregation and speculative decoding roll out, TPUv7's competitiveness will continue to climb, potentially putting it in direct competition with NVIDIA Corp (NASDAQ: NVDA)'s GB200 and GB300 NVL72 systems across the full performance curve.
Cost Efficiency: Ironwood Takes the Lead Over Blackwell in Most Workloads
SemiAnalysis benchmarked TPUv7 Ironwood against NVIDIA Corp (NASDAQ: NVDA)'s B200 and B300 using a Qwen3.5 397B FP8 model in aggregated serving mode, conducting a head-to-head comparison.
Ironwood's cost advantage shines brightest in low-concurrency, high-interactivity scenarios. At an interactive speed of 20 tokens per second per user, Ironwood achieves a per-chip throughput of 9,364 tokens per second, outperforming the B200's 8,903 and the B300's 8,925 by roughly 5%. Combined with lower per-hour chip costs, Ironwood delivers 50.4% more tokens per dollar than the B200 and a substantial 96.0% more than the B300.
On end-to-end latency, Ironwood remains competitive. At a median response time of 20 seconds, Ironwood's cost per million tokens sits at approximately $0.098, undercutting the B200's $0.106 by 8% and the B300's $0.132 by 25%.
However, the results are not universally favorable. In a localized range around a 30-second median response time, the B200 still manages to edge ahead on cost-per-dollar performance. Additionally, when NVIDIA Corp (NASDAQ: NVDA) GPUs leverage FP4 precision, they retain an edge in that format since Ironwood lacks native FP4 support—a gap expected to close with the next-generation TPUv8i, which will feature native FP4 acceleration.
In a non-apples-to-apples comparison where GB300 NVL72 runs disaggregated serving while TPUv7 uses aggregated serving, the GB300 holds a roughly 30% cost-per-dollar advantage at mid-range end-to-end latency levels. SemiAnalysis suggests this gap will vanish once TPUv7's disaggregated serving optimizations are fully implemented.
TorchTPU: Building a Native PyTorch Ecosystem to Replace TorchAX
The software stack has long been the central obstacle to Google's TPU externalization efforts. Previously, external developers had to translate PyTorch models into JAX execution via TorchAX—a path fraught with issues in low-level optimization, paged attention mechanisms, and compatibility with vLLM working models.
The new TorchTPU approach leverages PyTorch's PrivateUse1 backend extension point to expose TPU as a native PyTorch device (device="tpu"). Developers can simply invoke .to("tpu") calls while retaining familiar distributed interfaces including DDP, FSDP2, and DTensor. On the compilation side, TorchDynamo and AOTAutograd generate FX computational graphs, which TorchTPU lowers to StableHLO and then XLA converts into TPU-executable code. The underlying Pallas kernels continue to handle performance-critical operations.
The practical significance of this architecture shift is that vLLM and SGLang can now reuse more upstream model code, schedulers, and API logic without rebuilding across the PyTorch-to-JAX framework boundary. SemiAnalysis notes that Inferact, RadixArk, and Red Hat are all collaborating with Google to establish TorchTPU as a first-class supported backend for vLLM and SGLang.
TorchTPU remains in private testing and is slated for open-source release during the October PyTorch conference. Google plans to extend support to Kimi K3, GLM5.3, and Google's own open-source Gemma4 models after completing initial adaptation for Qwen3.5 397B. SemiAnalysis projects that once a handful of models are optimized, the marginal cost of adding new model support will drop significantly, potentially giving TPU a "Day 0" support position in vLLM and SGLang.
Kernel Optimization: Hundreds of Engineering Hours for Throughput Gains
To back up these performance numbers, Google's engineering team has invested heavily in targeted optimizations for TPU inference kernels, covering attention mechanisms, MoE routing, and hybrid model state management.
On attention parallelism, addressing the uneven distribution in Qwen3.5's GQA layers (32 query heads with only 2 KV heads), the TPU backend now supports a combination of 8-way attention data parallelism (DP8) with 8-way expert parallelism (EP8), eliminating unnecessary All-to-All communication overhead.
For communication, Google merged expert IDs and routing weights into a single All-Gather operation, saving roughly 80 microseconds per layer in DeepSeek-V3 tests—approximately 4.64 milliseconds per forward pass across 58 layers. Additionally, ReduceScatter collective operations have been moved to the SparseCore, paired with double buffering to overlap computation and communication. This yields throughput improvements of 4.1% to 14.2% for 8k1k workloads at concurrencies ranging from 64 to 512.
In MoE routing kernels, shifting irregular token rearrangement to the SparseCore, implementing triple buffering for expert weights, and introducing dedicated permutation paths for small batch scenarios resulted in 7.3% throughput gains at concurrency 64 and 5.1% at concurrency 128 for 8k1k serving.
For hybrid model state management, reducing recurrent state storage precision from FP32 to BF16—while maintaining FP32 arithmetic precision within VMEM—halved HBM footprint. This produced a 15% throughput improvement for 1k8k at concurrency 512. By optimizing KV cache page layouts (placing sequence dimensions on the 128-lane axis), available KV pages jumped from 5,141 to 10,283, delivering a 16.5% throughput increase and a 95% reduction in median TTFT for 8k1k tests at concurrency 128.
Ironwood Architecture: Co-Design as the Root of Cost Advantage
SemiAnalysis attributes Google's inference cost advantages to the synergistic co-design of chip, interconnect, and compiler rather than simple single-chip compute stacking.
The TPUv7 Ironwood breaks from the "MegaCore" design pattern of the TPU v4 and v5p era, instead integrating two independent compute dies on a single chip connected via high-bandwidth die-to-die links. Frameworks like JAX expose these as two separate logical devices. Each chip features 2 TensorCores and 4 third-generation SparseCores, with HBM capacity roughly six times that of the previous Trillium generation, and introduces native FP8 hardware support for the first time.
For the matrix multiply unit, Ironwood adopts a 256×256 systolic array capable of 65,536 multiply-accumulate operations per cycle—four times the previous 128×128 design. However, this brings stringent model shape requirements: matrix dimensions must be padded to multiples of 256, otherwise MXU utilization suffers. For instance, Llama 3 8B's attention head dimension of 128 caps MXU utilization at only 50% for two attention matrix multiplications on Ironwood.
On interconnect, Ironwood maintains the 3D Torus topology, with the basic building block being a 4×4×4 cube of 64 chips. Via optical circuit switches, this scales up to a 9,216-chip superpod delivering 42.5 FP8 exaflops of total compute. The ICI network bypasses the host CPU, enabling direct exchange of activations and gradients between chips and providing near-NVLink-level bandwidth across the entire pod.
Looking Ahead: TPUv8i and the Software Roadmap
Google's newly announced eighth-generation TPU splits training and inference into two separate chips for the first time: the TPU 8t for training and the TPU 8i for inference.
The TPUv8i (codenamed Boardfly) replaces the 3D Torus topology with a flattened hierarchical network built on high-radix switches. At scales of 1,024 to 1,152 chips, network diameter drops from roughly 16 hops to approximately 7—a reduction of more than 50%—which helps lower tail latency in MoE routing and multi-turn agent workloads. The TPUv8i also comes equipped with 19.2 Tb/s of ICI bandwidth (doubling the prior generation) and 384 MB of on-chip SRAM (tripling the prior generation), designed specifically to cache KV caches for inference and agent models on-chip. Additionally, TPUv8i adds native FP4 compute, which SemiAnalysis believes positions it for direct competition with NVIDIA Corp (NASDAQ: NVDA)'s Rubin NVL72.
On the software roadmap, Google's near-term externalization priorities include speculative decoding optimizations, prefilling-decoding disaggregation, KV cache DRAM offloading, and support for multi-turn agent workloads under the AgentX initiative. Google has already open-sourced TPU-Sync (formerly TPU-raiden), a KV cache transfer library, and plans to support the industry-standard Mooncake Store offloading solution.
SemiAnalysis concludes that Google's decades of software engineering experience and mature quality-driven culture set it apart from AMD, suggesting the external TPU software stack will mature significantly faster than competitors'. "Rome wasn't built in a day, but TPU externalization is advancing at remarkable speed."
Comments