At the recent Hot Chips technology conference, Sangwook Han from Samsung's DRAM design team delivered a presentation that formally disclosed the company's three-stage roadmap for HBM evolution. The ultimate destination of this roadmap is a novel architecture called zHBM, which involves stacking DRAM vertically and directly onto compute chips such as GPUs and TPUs, thereby completely eliminating the 2.5D interposer layer.
According to reports from August 23, Samsung has stated that compared to standard HBM4E, the zHBM approach claims to achieve a 70% reduction in power consumption, a 230% increase in DRAM bandwidth, and a savings of 100W per DRAM module, while also freeing up an additional 8.3% of power headroom for the GPU.
Understanding HBM and its Bottlenecks
HBM, or High Bandwidth Memory, is one of the most critical storage components in current AI training and inference systems. It is composed of two types of chips: the C-die (core chip), which contains DRAM memory cells that can be stacked vertically up to 16 layers high, and the B-die (base die), located at the bottom of the entire stack, which is responsible for handling various DRAM control functions and communicating with compute chips like GPUs via the PHY (physical interface layer). These two components are connected through TSVs (Through-Silicon Vias), which are vertical conductive channels that penetrate the chip.
Per reports, current HBM4 stacks already exceed 3TB/s in bandwidth, with HBM4E pushing into the 4TB/s range, and HBM5 doubling the bandwidth of HBM4 while exceeding 60GB in capacity. However, the continuous expansion of bandwidth is facing two major hard constraints: the physical limits of TSV count and spacing, and the maximum I/O count and speed of the PHY interface within the base die. At the same time, the process node gap between the B-die and the compute chip (xPU SoC) is narrowing with each generation, which is both a challenge and an entry point for Samsung's roadmap.
Stage One: Making Room for Compute Chips
The primary goal of the first stage of Samsung's roadmap is to "reclaim" silicon area from the xPU (compute chip). Samsung has already applied its D1c and 4nm logic processes to the base die of HBM4, with the main objectives being to reduce power consumption and shrink effective area. This marks the true starting point for the convergence of DRAM and advanced logic processes.
Specific measures include replacing traditional HBM PHY with D2D interfaces to shorten channel lengths, directly improving energy efficiency while freeing up valuable silicon space for the XPU. Additionally, Samsung plans to offload the memory controller from the XPU to the B-die of cHBM, which is expected to free up 5% to 10% of XPU area, corresponding to a 10% to 20% performance improvement. The company is also introducing an SRAM-based fine-grained repair scheme (Near-MC SRAM-Based Cell Repair), which utilizes idle space on the B-die to deploy SRAM repair resources. To address the hotspot issues that arise as a side effect of area reduction, Samsung has introduced Heat Path Block (HPB) technology, built upon the cHBM4 solution, which can reduce peak temperatures by over 35% and cover 50% of the PHY area.
Stage Two: The B-die Begins to "Grow" Computing Power
The focus of the second stage shifts from "making space" to "adding functionality," which Samsung defines as the functional expansion phase. As AI large model context windows expand dramatically, the demand for KV cache (Key-Value Cache, the memory area used to store intermediate states during model inference) is growing exponentially. Samsung plans to integrate memory expansion controllers and PHY on the idle silicon area of the base die, expanding the system's usable memory capacity through external solutions like LPDDR or HBM.
The company also proposes integrating partial Processing Elements (PE) onto the base die, offloading some computations that would normally be executed on the xPU to the memory side. This reduces D2D bandwidth requirements, power consumption, and thermal burden, a configuration termed AHBM (Advanced HBM). Furthermore, this stage includes integrating advanced RAS (Reliability, Availability, Serviceability) sensors with real-time telemetry capabilities, as well as on-chip self-test (ATIP) features on the base die, to enhance yield and test coverage.
Stage Three: zHBM - Eliminating the Interposer, DRAM Directly on Compute Chips
The third stage represents the ultimate form of the entire roadmap: zHBM. Current mainstream AI systems adopt a 2.5D packaging architecture where GPUs and HBM are placed side-by-side on the same interposer, transmitting data through horizontal interconnects. The zHBM concept is to "verticalize" this structure, placing the DRAM stack directly on top of the xPU chip to achieve true 3D vertical integration.
Key features of zHBM described by Samsung include distributed I/O to minimize data transmission distances within the HBM stack, a 3D structure that eliminates traditional 2D interfaces for significant system efficiency gains, an I/O power target of approximately 0.5 pJ/bit achieved by removing redundant modules like SerDes, and a bandwidth improvement of over 2.3 times with a system thermal headroom of 100W. Samsung's demonstration involved four zHBM stacks on top of a single XPU. To achieve these goals, the company is developing two key packaging technologies: WoW (Wafer on Wafer) and HCB (Hybrid Cube Bonding), aiming for ultra-high I/O density and ultimately building a unified SoC-DRAM co-design system.
The Core Logic Behind the Roadmap
The central narrative of Samsung's presentation is to reposition the HBM base die from a "passive data transfer station" to an "intelligent partner with active computing capabilities." The logic across the three stages is clear: first, compress area and release xPU space through process upgrades (Stage One); second, utilize the freed space to integrate more functions, expand capacity, and add computing power (Stage Two); and finally, completely restructure the system architecture through 3D vertical integration, achieving simultaneous breakthroughs in power consumption, bandwidth, and thermal management (Stage Three). In the presentation summary, Samsung stated, "By mastering advanced packaging and unified SoC-DRAM co-design, we will overcome the power, area, and capacity bottlenecks constraining AI systems, paving the way for higher efficiency, higher performance, and greater scalability in the years to come."
Comments