LLMs and AI accelerators hit a bottleneck because memory bandwidth has grown far more slowly than compute — a ~6,500x compute growth versus ~7x bandwidth growth over 2006–2026, the memory wall — so extra flops go unused when data cannot reach the chip fast enough.
The IBM AI newsletter is an IBM newsletter covering AI industry news, events, and insights. The video describes it as a recurring source of AI updates, while IBM's subscription page identifies the broader Think newsletter as also covering cloud, security, and sustainability.
SGLang is an open-source serving framework and inference engine for large language models and multimodal models, developed by the SGLang project. It provides an alternative serving stack for deployed models, with runtime and kernel optimizations for inference workloads. Its documented capabilities include RadixAttention, a zero-overhead batch scheduler, cache-aware load balancing, structured-output decoding, speculative decoding, disaggregated prefill and decode, model parallelism, and support for GPU and TPU backends. The project also publishes integrations and optimizations for current open models and multimodal, image, and video-generation workloads.
TensorRT-LLM is an LLM-serving system that uses batching, including continuous batching, to improve inference performance and efficiency.
vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.
Searchable transcript of The AI Memory Wall: Why LLMs & AI Accelerators Hit a Bottleneck — IBM Technology (12:21). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by IBM Technology. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:00 Hello everyone, over the past two decades, the compute capabilities of single chips have grown about 6,000 times. You would expect that workloads would be six times as fast, but this is not the case. Often engineers, when they profile their workloads, they don't see much performance difference. And these chips are sitting idle most of the time waiting for data from memory.
00:25 The memory wall. If you're in the business of building, training or serving models, this idea will explain why your workloads or your AI models don't get twice as fast when you chip, chips flops is, is twice fast. It also explains why batching can help insert workloads and it doesn't in others. And it also explains why AI accelerators are designed the way they are.
00:59 So, over the past two decades, there are two quantities that have grown at two different rates. So here we have the growth. In log scale. And then 2006 to 2026 is the period we're interested in. So the first quantity is computes. And here compute is measured in terms of plots per second. So, for example, in video GPUs in the early 2006, they were performing at roughly a few hundred billion operations per second, while we're looking at the B200 in 2025, so this chip is able of performing like 2,000 trillion operations
01:44 per second. So you can see kind of how much the growth around 6,500x growth. At the same time, during the same period, the other quantity that have been growing is the memory bandwidth measured at bytes per second. And you can see here, this has grown much slower compared to the compute capabilities. And the gap that you see between these two is actually keeps widening.
02:35 So, and it's roughly 7x. And this gap is what we call the memory wall. So, a, a faster chip does not necessarily mean faster workload, especially if we cannot feed it data as fast as, you know, it needs. And that is the whole idea behind the memory wall. Additional compute does not automatically improve performance, and the reason for that is a single ratio that we call arithmetic intensity.
03:09 Arithmetic intensity is defined as how many math operations you perform per unit bytes or flops per byte. And the idea here is if you're able to do many, many math operations per byte that you move from memory, you're actually in what we call in the compute bound regime. However, if you are not able to many operations per bytes you move or you do far less math operations per bytes of data that you moved from memory, you're in what we call the memory bound regime So let's draw here a model that we call the roofline
03:37 model. And what we measure here is on the X axis we call it the arithmetic intensity. And as I mentioned, it's something that we measure by flops per byte or number of flops that you have divided by bytes. And then on the Y axis here we have the attainable bandwidth measured as flops per second. I'm gonna draw here two quantities, the same quantities that we drew before, which is the memory bandwidth, draw, you know, measured in bytes per second, and then we have here the peak compute.
04:59 And then the peak compute is, again, flops per second, per second. Right? So note here what we are measuring in the X axis, it's something that is workload property. And these two graphs, basically they intersect at this point that we call the ridge points. So, for example, for a B200 GPU, the rich value is 281 flops per second. How did we measure this number?
05:38 This is actually a property of the chip. Here, we're in this regime, we are talking about the peak compute bandwidth that a chip can do. And this is basically your memory bandwidth. So the idea is if you improve your arithmetic intensity, you're also improving your attainable bandwidth. How much you can do in the chips. How many flops you can do per second.
06:05 But as you move closer to the rich point, you started hitting the limits of what the chip can do physically. Basically, that's the peak bandwidth of your chip. And basically, when you compute the arithmetic intensity, which is a workload-dependent characteristics, you need to compare that against or see where it fits with the rich points. So let's first see where this 281 came from.
06:34 On a B, B200 GPU, from the specs, the B200 GPU has about 2.25 petaflops per second in terms of its peak bandwidths. And then for the memory, it can do eight terabytes per second. So the way I calculate the ridge is basically I do the 2.25 petaflops divided by the eight terabytes per second, and that gives me about 281 flops per second. And note here, just a small caveat here, on the eight, you know, the NVIDIA GPU specs, it tells you 36 petaflops per second, but they're talking about eight GPUs.
07:27 And this is also for the BF16 sparse models. So what we do here is you have that, and then you divide that by eight, and that gives you roughly 2.25, that you need to look for a single GPU chips. And that's where we're getting basically this 281. So what this means for my workload, if my workload's arithmetic intensity is in this regime, I am memory bound, meaning that I am not able to process as many operations as I want per byte of memory moved.
08:05 If my arithmetic intensity is greater than the rich point, in this case, the 281 number, I am compute bound. And this also explains to me what is the bottleneck that I am facing for my particular workload. Let's look at two examples, the decode and prefill stages in a large language model. In a decode stage, what we need to do is we need to read large amounts of weights from memory to generate one token at a time if I'm using about size of one.
08:42 So, the arithmetic intensity of the decode is usually in this regime, far left, you know, to the ridge point. And that places me, you know, in the memory bound regime. So this is the decode workload. However, if I look at the prefill, prefill we're doing large GEM operations. And in this case, I'm doing more than a 1,000 flops, flops per second. My arithmetic intensity is more than 1,000 in this case.
09:23 While here in the decode, my arithmetic intensity is about one flop per second. So notice here that this is way greater than my reach point, and this places me in the compute bound regime. So in this case, if I am in memory bound, even if I throw more compute at my workload, it's not gonna help much. I need to, to get the data faster to the chip. And one technique that is used to help this problem is what we call batching.
09:53 Batching, what it does is it helps basically move your arithmetic intensity, increases the arithmetic intensity and moves it, moves it closer to the rich point. What does batching do? Batching in this case, it basically allows you to serve multiple requests at the same time. And in this case, you are actually reusing the same weights that you've loaded in memory multiple times to do multiple token generations.
10:19 So here, effectively, you are doing many more math operations per the same amount of memory that you moved. And that basically improves your arithmetic intensity and gets you closer to the rich point and improves your performance. And this is the reason why many mature LLM-serving environments like vLLM and SGLang and TensorRT LLM, they all use batching and continuous batching to basically help improve the performance.
10:51 So what you see here is the chip or the hardware has stayed the same. What has changed is the workload. And that is a very important point to notice here. So the arithmetic intensity here, it's chara … it gives you a characteristics of the workload and gives you an idea of where your workload sits with respect to the rich points. And also it helps explain where are your bottlenecks coming from.
11:16 You need to throw more compute or you need to improve your memory. So, to sum up, this is the memory wall. And, you know, the key idea here is, to get our workloads faster, we need to move data faster to the chips. And there are several ways that we need to understand. So first, before you benchmark, get an estimate of your workload using the arithmetic intensity to help you understand.
11:43 And also get, basically from the spec sheet of your hardware, what is the peak memory bandwidth? What is the, the peak compute that you can get? So, to get a rough idea of where you sit using a roofline modeling technique like this. And, you know, this way you understand where you need to optimize your, you know, spend more time to optimize your workload. And thank you for watching.