← All transcripts

AI Model Debugging: Find the First Divergence Across AI Accelerators Transcript, AI Summary & Key Points

IBM Technology · 2 days ago · Education · 14:30 · EN

Watch on YouTube

AI Summary

A model that produces different results on a new AI accelerator — same weights, same input, no crash or error message — can be debugged by finding the first observed divergence between executions. The method: run controlled inputs through a validated reference and the target accelerator, compare intermediate tensors at selected boundaries using numerical tolerances (shape, dtype, max absolute difference, relative error, cosine similarity), isolate the operation where the difference exceeds tolerance, build a minimal reproducing test, then keep it as a regression test in CI/CD. Numerical variation is not automatically a bug — precision changes and reordered floating-point operations cause expected differences, and the job is to separate expected variation from real correctness problems. The first divergence is not the root cause; it narrows a model-wide failure into a small region to investigate.

Key Points

  • Moving a model to a new accelerator changes the whole software stack — model, compiler, runtime, accelerator and its back end — so the problem can appear at any level: operations and shapes at the model level, lowering and fusion at the compiler level, scheduling and buffering at the runtime, precision and layout at the accelerator.
  • Small numerical differences are not automatically bugs: changed precision or floating-point operation order produces expected variation, and the task is distinguishing expected variation from a correctness problem.
  • Small differences propagate: if the final output is wrong, the fault may have started early in the model and grown through later computations.
  • Crashes can point to the wrong place — on asynchronous devices the error may only surface when the host synchronizes, after the operation that caused it.
  • Localize before diagnosing: run the same controlled inputs on a validated reference (CPU, GPU, or another trusted accelerator) and the target, saving tensors at boundaries such as layers or operators.
  • The reference must be validated, not assumed correct just because it is the reference.
  • The first observed divergence is the boundary between the last matching checkpoint and the first failing one — it does not prove that operation is broken, since the cause could be in the operation, its inputs, the compiler, or elsewhere between checkpoints.
  • Comparing tensors: check shape and dtype first, then numerical measures — maximum absolute difference, relative error, cosine similarity — with tolerances chosen for the operation and precision; no single tolerance works for every model and operation.

Tools & resources

2 items

Links mentioned

🔒 Full analysis locked

Unlock more videos and the full analysis

A credit unlocks one video's full analysis for good — the build steps, the tools and how each was used, the methods behind every use case. Pro opens the whole library instead, and raises how many videos you can analyse a day.

Unlock full analysis — free

Transcript

Searchable transcript of AI Model Debugging: Find the First Divergence Across AI Accelerators — IBM Technology (14:30). Search for a phrase, then click its timestamp to jump straight to that moment in the video.

Captions sourced from the original video on YouTube, published by IBM Technology. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.

00:00 Hello. Imagine you have a model that works on your reference machine. You move it to a new AI accelerator using the same weights and the same input. It compiles, it runs, but the answer is different. Sometimes the difference is small. Other times the model produces something clearly wrong and there may be no crash or useful error message. The final output tells you that something changed.

00:23 It does not tell you where. So how do we find the part of the execution where the two machines first stop agreeing? Debugging on a new accelerator. Moving the model to a new accelerator changes more than the hardware. It also changes the software stack that translates and executes the model. That gives us several possible places to investigate. Before we start guessing, we need a way to narrow the search.

01:14 The model passes through a compiler, a runtime, and finally the accelerator. So in a typical software stack, we have the, your model. Underneath that you have the compiler stack. Then we have the runtime. And then we have the accelerator. And then its back end. The compiler translates model operations into work that the device understands and can execute.

01:52 The runtime changes, manages that work and the data it uses, and the accelerator executes the compilation. So at the level of the model, we have things like the operations, the shapes that we have to worry about. At the compiler level, we worry about things like lowering, fusion, etc. At the runtime, there are scheduling decisions that happen. There is also buffering.

02:26 And then at the accelerator, there are things like precision that needs to be determined, layout decisions. And this is just a much simplified version. And the problem can appear at any of these levels here. But a numerical difference is not automatically a bug that you could observe. Some differences are expected, especially when precision or the order of floating point operation changes.

02:54 Our job is to distinguish expected variations from an actual correctness problem. So if I take a model, I can basically draw it maybe as a simple sequence of computations. So let's say my model has the, uh, embedding. Then I have the attention block. Then I have the normalization and then I have my logits, which gives me the, the, the result, uh, the results, or my prediction.

03:39 So here we're thinking of the model as a sequence of computations. And here I'm just showing a simplified example: embedding, attention, normalization and the final logits. And the real model has many more operations and repeated blocks. So if the final output that I'm seeing is wrong, the problem might be near the end, but it also might have started much earlier, and the small difference can propagate throughout in later computations and become much larger.

04:21 A crash can also point to the wrong place. Typically on an asynchronous device, an error may only be reported when the host synchronizes after the operation that caused it, so the final results is basically too late. And in this case, it's not enough information to locate the problem. The approach I use is to colocalize the difference before trying to diagnose it.

04:55 First we localize. Then we diagnose. The idea is to run the same controlled inputs through a validated reference and the accelerator, then compare the intermediate results at selected boundaries. So I have my reference machine. And then I have my target accelerator. So I mentioned boundaries. And those boundaries might be layers, operators or other useful points in the execution.

05:42 We are not trying to explain the bug yet. We are trying to reduce the amount of code and computation we need to investigate. So let's say on my reference machine, we begin with the reference implementation that we have already validated. We might run on a CPU, a GPU or another trusted system or accelerator. The important point is that we have evidence that behaves, that it behaves as expected, we do not assume it is correct simply because it is the reference.

06:17 So for the same input, we run in basically these layers, right? We have, for example, from our example we have the embedding, we have the attention, we have the normalization and then we have the logits. So for the same input we save the implementation tensors at our chosen boundaries. Here I'm using embedding, attention, normalization and logits as my boundaries.

06:41 This is just a simple example. And now we run the same model on the new accelerator and compare the same boundaries. The embedding. And in this case you know it agrees. We keep going. Attention also agrees. Normalization. Here we notice that the difference is larger than our allowed tolerance. And that is what we call the first mismatch that we see.

07:24 So the later outputs are now suspect. Because, you know, here, these later outputs started seeing divergence. So they may differ simply because the input they received has already started changing. And instead of investigating the whole model, we focus on this region here between the last matching checkpoint and the first failing one. We call this the first observed divergence.

08:05 This is first observed divergence. And so this is really matters because we only compared selected checkpoints. The actual difference we may have started, you know, may have started somewhere else between them. This does not prove that normalization itself is broken. The cause could be in the operation itself, its inputs, the compiler, or another part of the execution.

08:33 But we have made progress. We now have a much smaller region to investigate. So what do we mean here by agreeing? What does it mean for two tensors to, to agree? For floating point computations, we often do not require every bit to be identical. Different hardware may change the order of operations, and lower precision can, can also introduce expected differences.

09:01 So we compare using tolerances appropriate for the operation and the precision we're using. First we check the shape. We check, you know, the dtype. We also look at numerical measures such as maximum absolute differences. We can also look at something like the relative error. We can also look at cosine similarity. For example, say that I have a reference that is, say, 1.00, 2.00 and say, maybe, uh, 3.00 in my tensor.

09:46 So this is my reference. And then I have my target. I see something like 1.00, 2.01, 3.00. Whether that is acceptable depends on the test. So and there is no single tolerance that works for every model and every operation. So now we have found the suspect region. We can inspect it with purpose, and we can turn this into a methodological approach that can we can use to diagnose my problems at a deeper level.

10:40 Once we have found the suspect region, we can inspect it further, and we can turn these boundaries into diagnosis. So we first compare the inputs to the operation. So compare the inputs. Next we isolate the operation where we started seeing the divergence. Next we can do a minimal test. And then we can also inspect things like precision, layout, lowering, etc.

11:42 And then we rerun. And after we rerun, we go again and compare the results again. Are the shapes, the types and values what we expect? And this reduces the problem into a small test that we have here that can help us reproduce the difference. For normalization mismatch, for example, we might check the precision used for accumulation, the reduction behavior, the broadcasting, the tensor layout, or how the compiler translated the operation.

12:26 This small test can help us separate those probabilities. So once we identify and fix the, or isolate the cause, we rerun again and compare and check whether, you know, the divergence disappears or maybe moves somewhere else. \So when we have a small test that reproduces the problem, we keep it as a regression test. So this could turn into what we call a regression test.

13:06 And this can be used in my CI/CD for example. That way if the same failure returns after a future compiler or runtime change, we have a better chance of catching it early instead of debugging the entire model again. We also rerun the full model to make sure the fix works in the original execution. So when a model works on a reference machine that produces different results on a new accelerator, start by comparing the executions.

13:38 Use a validated reference, compare the intermediate outputs and find the earliest boundary where the difference exceeds the expected tolerance. That gives you a smaller region to investigate. From there, check the inputs, isolate the operation, reproduce the problem, and test the fix. The first observed diversion does not tell you the root cause. It tells you where to start looking, and that can turn a model-wide failure into a much more manageable engineering problem.