What Is HBM? Why High-Bandwidth Memory Is AI's Real Bottleneck

An AI accelerator is only as fast as the data it can reach. A processor that finishes its arithmetic and then waits for the next batch of numbers is expensive hardware doing nothing.

High-bandwidth memory, or HBM, is the memory built to prevent that wait. A single stack of Micron's HBM3E is rated at more than 1.2 terabytes per second (see Sources, [2]) — and the way that number is achieved explains why memory, not only compute, now shapes how AI chips are designed and how quickly they can be supplied.

What HBM actually is

Conventional DRAM sits on modules spread across a motherboard, connected to the processor through a comparatively narrow interface.

HBM takes a different approach. It stacks multiple DRAM dies vertically and connects them with through-silicon vias (TSVs) — electrical connections that pass directly through the silicon — to deliver very high bandwidth to GPUs and AI accelerators and reduce the time compute devices spend waiting for data (see Sources, [4]).

Scientific American describes stacks built 12 or even 16 layers high, and compares the design to adding lanes to a highway: more parallel paths between the processor and its memory, rather than one faster lane (see Sources, [3]).

The stacked package is then placed very close to the processor. How that close integration is manufactured — TSMC's CoWoS advanced packaging — is covered in our companion explainer, Nvidia AI Chip Packaging: How HBM and TSMC CoWoS Work.

Where the bandwidth numbers come from

HBM's headline figures are not mysterious. They are the product of two things: how fast each data pin runs, and how many pins the stack has.

When JEDEC, the standards body for memory, published the HBM3 standard, it defined data rates of up to 6.4 gigabits per second per pin, which it described as equivalent to 819 GB/s per device (see Sources, [1]). That matches a 1,024-bit interface: 6.4 Gb/s × 1,024 pins ÷ 8 bits per byte ≈ 819 GB/s.

The next generation pushes the pin speed higher. Micron rates its HBM3E pins at more than 9.2 Gb/s and the resulting bandwidth at more than 1.2 TB/s per placement, in 8-high 24 GB and 12-high 36 GB versions (see Sources, [2]). Scientific American reports Micron saying its HBM4 chips exceed 2.8 TB/s and are designed for Nvidia's Vera Rubin GPUs (see Sources, [3]).

These are peak ratings. They describe what one stack's interface can carry at full speed, not the sustained throughput a real workload achieves. They are also per stack: an accelerator package can place several stacks around its compute dies, which is how GPU-level memory bandwidth reaches far higher totals.

The 18.75× HBM-versus-DDR5 figure, explained

A comparison circulating in coverage of HBM — and used in our video — says data moves 18.75 times faster with stacked memory than with DDR5. It comes from a theoretical example published by Kings Research (see Sources, [5]).

The example imagines moving one dataset of 500 terabytes in a single pass:

The arithmetic is straightforward. 500,000 GB ÷ 64 GB/s ≈ 7,812 seconds, and 500,000 GB ÷ 1,200 GB/s ≈ 416 seconds. The ratio between the two bandwidths, 1,200 ÷ 64, is 18.75.

Read that way, the comparison is still useful. It shows why the memory attached directly to an AI accelerator is HBM rather than the DDR5 used as general system memory: the stacked design moves data at an order of magnitude more bandwidth per device.

Why AI workloads need that bandwidth

Training a large model runs across huge clusters of accelerators, and serving it — chatbots, coding tools, AI agents — keeps data moving constantly. As Scientific American puts it, "a GPU waiting for data is wasted hardware" (see Sources, [3]).

That makes memory bandwidth an economic question as well as an engineering one. Accelerators are among the most expensive components in a data center, and every cycle they spend idle while data arrives lowers the return on that hardware. The same logic drives the GPU-rental businesses described in The Billion-Dollar Business of Renting Nvidia GPUs: compute only earns money while it is doing work.

Capacity is the second constraint. Scientific American reports Columbia professor Keren Bergman's assessment that, as models grow, the memory available close to the processor falls short of what is needed by one to two orders of magnitude (see Sources, [3]).

Why HBM is hard to supply

Only a few companies make HBM at scale. Scientific American names Micron, the largest memory maker in North America, alongside SK Hynix and Samsung in Asia (see Sources, [3]).

Each stack is a demanding product: many DRAM layers, thousands of vertical connections, and a package that must sit beside the processor. And because the memory is integrated so closely, a constraint in HBM output — or in the advanced packaging that joins it to the compute dies — limits how many finished accelerators can ship, regardless of how many compute chips are available.

The same article also records the counterweights: warnings about an AI investment bubble from the heads of Google and OpenAI, stalled data center construction, and lenders' concerns about debt (see Sources, [3]). Strong demand for AI memory today is not a guarantee that every planned build-out happens on schedule.

What the evidence supports

The sources support three conclusions.

First, HBM's advantage is structural: stacking dies and connecting them through the silicon gives each stack a far wider, faster interface than conventional memory — 819 GB/s per device for HBM3 and more than 1.2 TB/s per stack for Micron's HBM3E (see Sources, [1] and [2]).

Second, the widely quoted 18.75× figure is a theoretical illustration of that interface gap, not a benchmark (see Sources, [5]).

Third, because AI accelerators depend on memory bandwidth and capacity, and because only a few manufacturers produce HBM, memory supply is one of the factors that decides how much AI hardware can be built (see Sources, [3]).

What the sources do not establish is how sustained bandwidth compares across real systems, or how quickly HBM supply will expand. Those are the numbers to watch next.