The same model runs at different speeds on different chips because the KV cache sits in different memory architectures under different orchestrators: GPUs leave allocation and access to the serving runtime plus kernels, TPUs let the compiler tile and pipeline HBM-to-VMEM movement, and IBM's Spyre uses compiler-generated explicit loads and stores to manage a per-core 2 MB scratchpad, and these choices shape decode efficiency and throughput.
The IBM AI newsletter is an IBM newsletter covering AI industry news, events, and insights. The video describes it as a recurring source of AI updates, while IBM's subscription page identifies the broader Think newsletter as also covering cloud, security, and sustainability.
Memory Systems in Dataflow Architectures is an IBM-offered Coursera course about memory systems for dataflow architectures used in AI computing. It examines how memory capacity, bandwidth, and data movement affect LLM inference, including how GPUs, TPUs, and other AI accelerators store and move KV caches.
vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.
Searchable transcript of Why AI Models Run Faster on Some Chips: The KV Cache Effect — IBM Technology (10:01). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by IBM Technology. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:00 You have seen a model type out and answer one word at a time. Here is the part that is easy to miss. Every single one of those words has to look back over the whole conversation before it can be chosen. So, the longer you talk, the more there is to look back through. And doing that from scratch every time would be enormously wasteful. So, the model keeps notes on what it has already read.
00:23 Those notes have a technical name, the KV cache. So, where does the model keep its notes? So think of these squares as tokens that have been previously generated. And processed so far. At each attention layer, earlier tokens leave behind keys and values that can be reused. So now, when I generate a new token, then, the model still does new computation.
01:08 But it does not need to recreate all of those historical keys and values. So, the keys and the values. And so the model, the mental model here is very simple. So we want to keep the notes, the KV notes. And then reuse the paths which is part keys and values. As the context grows, the notebook grows too. So we want enough capacity the..to hold the history.
01:42 But, computation benefits from a much smaller, fast, local working sets. So, here we need to have a large memory, large capacity. However, the computation benefits from local and fast memory. Characteristics of this large capacity is large, but slow. And this linked local memories characteristics is fast, but small. So, the system's problem is not just storing the KV cache, it is moving the KV data needed for attention efficiently between memory levels and the compute.
02:31 So let's look at a few examples. We start with the GPU. So the GPU the KV starts first starts in the on-device memory. This is the large on-device memory. And it's typically the HBM. Then we have the on-chip fast memory. Which basically serves the compute units. So, the key question is who orchestrates the data movements? And in the case of the GPU, we have the runtime.
03:22 Plus the kernels. And the data moves in both directions across these different levels. So typically on a data center GPU, the KV cache typically resides in the GPU device memory, often the HBM. And the serving runtime decides how each request KV storage is allocated, for example vllm. So rather than reserving one big slab per request, it hand-outs small fixed-size blocks on demand and keeps a table of where they they went in memory.
03:56 that is the same trick an operating system uses for virtual memory. The attention kernel determines the access pattern and how data is stitched through on-chip resources and the hardware caches. This is something that is managed and optimized together. So here, if you look at who orchestrate, it is actually the runtime plus the kernel. And this is not a claim that the hardware is passive.
04:22 This is actually our first answer here for who decides. So at the server level, mainly the runtime, and the kernels for the case of the data center GPUs. So now now let's look at another example the TPU from Google or the transfer processing unit. And here you get the same capacity versus locality problem, but a different layer owns the orchestration.
04:46 So TPU. Same thing we start with the large on-device memory here, the HBM. And then we have the fast on-chip memory that Google calls vector memory or VMEM, for short. Then, of course, the compute um..units. That the fast on-chip serves directly. And the question again here. Who...who orchestrates? And in the case of the TPU, as opposed to the GPU, here we have the compiler, plus the kernels.
05:40 And in this case we have tiles. Plus pipelines. As opposed to pages and access patterns. So, in this case, the large buffers sit on...in the device memory of the site, and the computation ru...runs out of a much faster memory, which is the VMEM. So here, the compiler has to orchestrate and to work out ahead of time how to cut the the data into tiles, and how to fetch the next tile while it is still computing on the current one.
06:20 And that is all the tiling and the pipelining means in this case. Tiles and pipelines. The kernel you wrote shapes those choices and the hardware's copy engine carry out the moves. So, in this case, we see here two examples the CPU...the GPU and the TPU and different orchestration layers. So now, let's look at another example. Spyre chip or Spyre Accelerator from IBM, which is an example of a dataflow accelerator.
06:49 And in this case with Spyre, so, we start with this large LPDDR5 memory that has 128 GB per card. And then, Spyre has per core... so, a card has three, two cores, and then, each core has 2 MB scratchpad. This is the fast SRAM memory that serves the compute units. And, of course, data flows in each direction across, you know, these different levels. And the same question we can back to here who orchestrates the data movements?
07:40 In this case, it's the compiler. And here we have explicit loads plus stores. So Spyre makes the software managed local memory storage especially visible. A card has 128 GB of LPDDR5 and 32 active cores, and each core has a 2 MB programable SRAM scratchpad. Unlike a conventional hardware cache that automatically decide what to retain locally, the Spyre's scratchpad is explicitly managed, and the compiler partitions the work and schedules the loads and stores needed to keep the scratchpad fed.
08:26 So the orchestration answer here is the most explicit of the three. Compilers schedule data movement. So in the GPU it was the serving runtime and the kernels that largely orchestrate the KV allocation and the access with hardware caches and the memory system underneath. For the case of the TPU, it was the compiler and the kernel that shape the HBM to VMEM, or the vector memory tiling and pipelining.
08:52 In the case of Sypre, it is compiler-generated scheduling that explicitly manages the scratchpad's movement. So as you can see here, we have the same KV cache state, but a different memory pad and the different control module underneath. And this is why two accelerators running the same model can behave very differently during generation. The reusable attention state is conceptually the same, but where it lives, how it reaches the compute and which layer orchestrate that movement all defer here.
09:32 For lower context decode, those choices can strongly shape efficiency and throughput. So next time when you compare accelerators, do not ask only how much math they can do. Ask how they move the model's memory states.