The Inference Frontier: 10x Faster Models to Self-Optimizing AI — Philip Kiely & Ali Taha, Baseten Transcript, AI Summary & Key Points

00:00 JLM52 is very very good at writing GPU kernels. It was very funny internally. We had a GLM52 endpoint that we're using to like that we plugged in in our cloud code harness. So every engineering team uses like our JLM52 and it will do a forward pass on the JLM52 instance of the you know the node and then it will get the profile trace and it will analyze it and it will find the kernels that are the bottlenecks in SLM and then it will write the new kernels and it will do another profiling trace and when it's done it

00:30 uploads the image to our thing and then we can pull that image down and repeat the cycle. Some of the GPU kernels that we're on JM52 within our inference engine is written by JM52. Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content.

00:52 We've been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the [clears throat] only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring the Inspace to you each and every week.

01:17 If you do it, I promise you, we'll never stop working to make the show even better. Now, let's get into it. Okay, we're here in the studio with uh Philip uh old friend from from uh inference engineering the book as well as base 10 and uh everything that you've done you and I have done before as well as Ali. Welcome. >> Pleasure to meet >> Waterlue intern.

01:41 >> Waterl intern always. >> When did you get Waterlue intern as a >> as a handle? I think the rebranding happened like mid-March. When I saw it was open I was like I have to take it off for grabs. >> The problem is that Ali is really good at his job and is not going to be an intern much longer. So we have to figure out, you know, who's going to get the handle.

01:58 >> Pass the torch over. >> Oh, okay. It can be like you just pass it to another water. >> It's another water lantern. >> Yeah. Enter. >> Enter. Yeah, >> you got to get from >> Yeah, we got to get >> They have to But it could come from base 10. So it's like whoever base 10 gets from Waterlue, he has the title of >> Water. They have to pass [laughter] halfway through.

02:17 You either get it or you're out. >> You should also do like a big graduation ceremony where you change the handle. just say [laughter] >> I mean you guys are good at ceremonies clearly you know we had a nice launch of the book very successful but before we get into all that I want to start off with a fun question for you okay you're expert inference engineer what happens when I send a long query say 200,000 tokens into base 10's inference what's the process of query through GPU model routing balancing all that what is

02:46 all the stuff that we don't think about >> with a long query specifically the first thing that I'm going to ask is, have you sent me this query before or at least part of it? Um, and I really hope you have because it's going to be a lot easier for me and a lot cheaper for you. So, the first thing that we're going to look at is some kind of cashware routing where we're going to see we probably have a number of instances, a number of replicas up serving whatever model you're hitting.

03:13 uh we want to send this one to something with number one available prefill workers and number two ideally some cached input already there so that we can skip prefill on at least part of these 200,000 tokens. Um if you're doing 200,000 tokens it's probably coding or a multi-turn agent or something where you would expect to have that cached. Um if you don't we're going to have to send it to a prefill worker.

03:39 Um we've at least on certain models disagregated prefill and decode. Um, so you're going to have one set of GPUs that's solely going to process the input, create that KV cache, and get you your first token, and then that's going to be passed over to a separate set of GPUs, which is going to run decode. We're going to iteratively make those tokens. Um, we're probably going to have some kind of speculator model.

04:01 In front of that, I'm going to assume that you're doing coding. And because of that, our speculator model, which assumes you're doing coding, is going to have a high uh draft token acceptance rate. If I'm wrong and you're asking me to summarize every Harry Potter book, uh it's going to be slower. And then we stream that output to you and uh account for it.

04:19 Um charge you, you know, some number of couple of pennies and say, uh, hey, would you like to send another one? >> Except they send isn't charged by pennies. Well, yeah, we we charge we I'm assuming that we're talking about the public model APIs. If you are, you know, setting up a a dedicated deployment, then yeah, it's not pennies. >> Yeah. I mean, one of the key differentiators when I was talking with uh Bayen initially was that uh actually people who want very very high volume just need to rent by the box cuz then

04:53 it's up to you to figure out how to saturate the box. And more often than not, it's like way cheaper if you're pushing like millions of tokens per hour if you just pay per hour instead of paper token. >> Yeah, they they do. I think that we've increasingly seen a lot of demand for the sort of paper token APIs just because everyone wants to try open models and then once they find a use case that's really sticky, uh, then they they move over to dedicated.

05:15 >> Is there a best practice on when it's time to swap over? >> Couple reasons. Yeah, reliability. That's a big one, right? Like if they have a very specific use case, they want you to train something specifically for them. Like they want their own spec for instance for their own traffic. >> Speculative decoding. >> Specting. Yeah. Yeah. Um [laughter] sorry like speculative the way like like basically if you have a huge model, right?

05:35 And so the model is going to be generating one token at a time, every single turn, every single forward pass. So we attach like this little like kind of parasite like this this layer that goes on top of the model. And this model just has to predict it does three very fast ultra reagressive over passes and it will predict like you know three certain tokens and then you do one forward stage over the entire original model in order to see if those predictions were correct or not and then you accept them or you reject them.

05:59 Now this draft model is is traffic specific. So if you like like um you know Philip said if you're summarizing Harry Potter books I can train exclusively that draft model on Harry Potter books and I can guarantee you that I'm going to accept the three tokens every single time. And so with that case I increase your decode speed. I wouldn't be able to provide this to you if you want to share endpoint because I have no idea if you're doing Harry Potter, if you're doing coding, if you're doing English like we don't know.

06:21 Um also there's a thing in the book that mentioned that if they really cared about a specific threshold in chapter 4 I think. Do you remember that? >> Yeah. The the things that you can do is you you can set you know a specific like batch sizing a specific like parallelism strategy if you're trying to optimize for like throughput versus uh latency. Um, you can, you know, maybe maybe a NVFP4 quant doesn't pass your benchmarks and you want to run a model at higher precision, you could do that.

06:49 Um, there's just a bunch of reasons why you might want to have your own endpoint. And and the the biggest one, of course, just being like you don't have to deal with someone else throwing 100 million of tokens of benchmarking traffic at the endpoint when you happen to be trying to serve your users. Yeah, I think one thing that is that is a classic journey, you know, like it's basically viv is asking the what happens when you type google into the browser.

07:13 Um tool calling is that just you know you're you're generating JSON or is there more complication beyond that? Certain customers that we have they have their own post-trained models and so they they demand tool calling that's not just like you know parsifile or you know go find the weather. It's it's something that's very specific and you have to do post training on this and if the post training on the model is not good or if the quantization after the post training to get the inference to be fast the model will

07:43 struggle reading the JSON file and reading the tool calling but it doesn't it doesn't require its own like sandbox. It's not like it's going to use that tool calling to like like escape a sandbox or like it doesn't have to be contained. It can just be a normal dedicated deployment. The challenge with tool calling more and more seems to be that the companies want certain tool calling which which is a very sensitive thing to train and because you're dealing with all of the JSON outputs if it doesn't like close the end of

08:06 the request in a very certain manner you end up with a model that did the tool calling and like the thinking and so as a result of that it didn't see the result and just hallucinated the result as it decoded. That seems to be the most challenging thing with tool calling not really the sandboxes more. Yeah, that's a challenge on the training side. And then on the inference side, there's work that you can do to scope the possible output.

08:29 So we published this actually at this point close to 2 years ago. Um the the solution to this problem which is you basically make a state machine and you use that to constrain the output to a specific format. So this is the structured output problem. If you remember back in >> Yeah. The specific grammar is GML had this thing. >> Yeah. So it's it's like the the old school like make sure this is only JSON return only JSON or my grandma's going to die type >> BNF grammar.

09:04 At some point the openi released a thing that was like yeah if you want to constrain your output write BNF grammar uh back >> in our inference system it's just a yeah specified output format. um and and you get the the guarantee that your output's going to be structured along that format. Um and so applying that to tool calls can like help cut down on obviously you can still call the wrong tool or call node tool.

09:29 It doesn't solve the the certainty problem, but it at least solves the output structuring problem within tool calls >> and MCP is just another form of tool, right? I guess as far as there's no special thing there. The thing I'm always like explaining to people is the LLM is actually not capable of doing anything. It's only capable of making suggestions of what to do and then if those suggestions are formatted in a certain way and applied to a system that knows what to do with them, then an action occurs.

09:56 Yeah, >> part of the fun stuff is, you know, this is solved outside of tool calling, too. Like in an agent loop, >> if the output is not correct, or you're right, like reasoning, uh, tool calling was done in the reasoning state. Just be like, oh, I don't know what to do. Let me just try again. And, you know, it might get there after a few tries. Um, and on your point of training, sometimes this is harder in smaller models.

10:15 So, you don't have the same exact quality output when you just swap from a big model, right? >> Yeah. I I will say that um before we I I think we need to go back to inference engineering [laughter] proper but um I had expected that something would replace JSON because it's hard to stream JSON cuz JSON must be complete and you must have open and close brackets and everything.

10:38 So it's hard to parse something or validate something while it's being streamed. Uh so people invented all sorts of things that are like I I forgot the the name of some of these alternatives but it's basically something like tommo something like YAML. Um, but JSON CS3 dominant still >> the JSON outputs aren't that long, right? Like I guess you could have a long cuz tool calls also contain the arguments in them and and perhaps for a certain tool you might pass like a very long argument, but my impression of the sort of

11:08 median tool call is that it's a relatively small number of tokens, right? So I would expect that speculators are generally fairly good at something as formatted as JSON. Um, and so you would have like a pretty fast decode step there and and that the streaming wouldn't be as valuable. But may maybe I'm wrong about that. >> You're also bounded by the software that the model is going to integrate with if the software is built with JSON for the tool calls or if the company that you're you know if your customer says that

11:32 this is how our software works and our tools are interface with JSON. You can ask them to like you know change their their software and and say like yeah this is going to be better for the model. But like with the wet training shouldn't be that much of a a difference. It's also more profitable if it outputs more tokens. Probably >> depends on your business model.

11:50 You know, it really depends. Uh but I will say that, you know, as a writer who like experienced a lot with AI generated output, I do try to move from text to JSON text, which is very long JSON, right? Like like there's paragraphs in every field because I'm trying to structure it, right? I want you to first make factual statements, then make opinions, then make bullet point summaries, have dates, have entity references, have your sources for references, all these things.

12:17 Anyway, so so these are things that like I think people who really really experiment with structural output have to really care about. But um let's let's sort of recurse up the stack a little bit. Before we started recording, you actually mentioned something which is really cool, which is that there's a lot of engineering that inference engineering that goes on when a new model provider releases a new model, right?

12:37 So let's call it GLM 5.2 Kimmy K3 I had pre previously assumed especially if it's like well GM GLM 5 to 5.1 to 5.2 to like you know that you've supported them before. Is it that much work? >> It's a lot of work. >> Yeah. Okay. So like you know a lot of people all you guys right whenever a new model launch like people rush to say like oh hugging face supports this, fireworks supports this, space 10 supports this and I'm like yeah of course we support it but what what goes into that?

13:04 What goes >> I think it's [clears throat] more than just supported too right? uh it benefits the consumer a lot like I think it was with uh Kimmy K2.5 or GLM52 the latest there was sort of an inference war right X provider is at 90 tokens a second the next day we're at 150 then I kind of kicked that off with the GLM52 I wrote a Twitter article about it got like half a million views um >> piston being number one >> yeah [laughter] >> yeah which uh which then got everyone really excited about hey how can we you know

13:35 benchmark acts a little bit further. And um there's a difference between support the model as in like I can make a token out of this model and support a model as in I have a production ready API from this model. >> Getting to the point of I can make a token out of this model is not that hard because generally the um open- source inference engines, your VLMs, SGLANS of the world often times even receive weights ahead of time.

14:04 maintainers do or the people making the model merge PRs to ensure support. So you generally can you know just kind of get it working on the standard open- source stack without too much pain in most cases. The challenge is you know every inference company is going to have our own proprietary stack. um you know some open source components, some some in-house stuff and for any arbitrary model there's going to be some new stuff.

14:34 Sometimes you get lucky like K uh 25 to 26 was like pretty pretty similar. >> Yeah, it was pure continued post train if I remember correctly. >> Even in those cases there's still stuff you have to do. You have to redo the quantization work. You're taking the model from generally these models are not released in NVFP4. um and we want them to be in NVFP4 for maximum blackwell compatibility.

14:56 So we have to perform that quantization um and you know calibrate the quantization to make sure that we're not causing any kind of regression in the model's intelligence and then we also have to train the speculator as we've talked about generally we have obviously we have ZDR zero data retention on our model API so we don't know exactly the traffic that people are sending us but we know what's popular we know that coding use cases are popular uh we know that agents agentic use cases are popular so we can get public

15:27 data sets that are representative of that kind of traffic and and train gen general speculators. Now with speculators today, you need to train the speculator using the base model itself because you're getting hidden states out of the model from running inference on these specific prompts and that is the training data you use to create the speculator.

15:46 So there's that process which you need the the real model weights for. Uh and then there's of course just the process of you know standing up all the infrastructure behind it, loading all the stuff in, testing it. Um and then when there's a new model with a newer architecture, I think that like obviously the deepseek models tend to be the most challenging as they have like the most novel architectural stuff going on uh model over model.

16:11 But every new model has something. I mean Kimmy K2 had or sorry uh GLM52 had >> spars. >> Yeah. The the DSA >> which was brought from Deepseek. Yeah. Yeah. And and you know we >> so you can copy paste and do it. >> I don't know how this works [laughter] you know. So like we had to like build support for that into our runtime. Um and and you're right like it actually is really interesting the way that all of these open source labs borrow from each other.

16:36 For example like GLM 5.2 doesn't have vision. So something that uh Harry a guy on our team if if we could take a look at this um he like kind of grafted the Kimmy um vision encoder onto GLM 5.2. too. >> We training the projector. >> Exactly. So, if you think about like the encoder, there's the encoder which is the part that looks at the image and and turns it into latent information and then there's the projector which kind of >> spaces.

17:03 Okay. [laughter] >> And then there's the projector that that maps it onto uh the the model itself. And then there's the model weights. You don't want to mess with the model weights because you run a chance of making the model dumber at something else for the purpose of giving it vision. So instead, Harry started with just a projector which is only a handful of millions of parameters.

17:27 >> Yeah. And >> can you show the the training room? Like the way it groed is very very maybe Ali, you should you should take it from here. You've got a better understand than I did. >> Yeah. You can see like he the way he trained this is is really really cool. At the beginning he was training it using just like here's a picture of a mountain. Can you describe what's in this mountain?

17:43 And that caused it just like the first the first you know learning walls. Like here you can see this all we're trying to teach it is to translate the encoded like it's it's already taken the encoder from chemik it's taken the the image it's frozen frozen frozen frozen so the understanding the brain is frozen and the eyes are frozen it's just we're trying to interconnect between the eye and the brain right so the projector and so you take the tokens and then he's like oh can you describe what's in this image and it's

18:07 like oh it's it's a mountain or it's a person or it's a human whatever the case is >> but that didn't cause complete understanding so he changed it such that every image was associated with a data set of questions like, "Does this image have a white male? Does this image have birds in the top corner? Does this image have a scientist in it?" All that stuff.

18:24 And it would have to answer questions correctly. And using not just training and describing an image, but being able to answer question answer question, answer question answer over time. Like you can see like groing, which is like genuinely insane that retrofitting vision into a large LLM can learn to that extent. And even for images that it doesn't perform well on, for instance, if you ask it a picture of like Stephen Hawking, who is this?

18:44 Maybe it doesn't get it but it will say something like this is Albert Einstein like it still understands this is a scientist who is a man who has you know done significant achievements all that stuff. So that's like really really cool. >> Yeah. So uh we've covered howen before who was the author of the lava paper that did this uh a while ago and I think that that's very foundational work for anyone who hasn't done vision work before.

19:06 >> Same with the clip and meta clip where you go from just captioning to building out questions off the image and how much better you can get performance. >> Right. Right. Right. Yeah. But what's what's so exciting about this is if you look at a model like this now, obviously this is a little bit more of a research project. It's not you it got to 56% on MMO Pro, I think.

19:24 So not not quite Frontier, but if you're running this model, you haven't suffered any loss on your GLM52 quality. If you don't have an image, it'll just behave exactly the way it used to. And >> which in the inference code, you literally do not include the other part, right? >> Yeah. I mean you would just skip the encoder if you don't have an image input.

19:46 >> Okay. >> Um just confirmation. >> Does it affect a lot on the overall inference side? Like you're not adding much. You're adding a very small vision encoder. These are typically like less than a billion parameters. >> Yeah, it's I mean there's a little bit less standardization among vision encoders. Um so the sort of support matrix can be a little bit uh sparser but overall um yeah it's a pretty it's a pretty minor component of the overall system.

20:11 And and ultimately what you get out of the system is all of a sudden you have Kimmy vision, GLM weights and DeepSeek attention all in one model. And that's I think a lot of the power and beauty of open source is that you can take all of these different components and and combine them together into a system that's better than anyone can be individually.

20:30 >> People used to say that you would also do Franken merges where you would take like layers from each model. No. Does anyone do that anymore? Well, to to your point previously when you were mentioning like um the work that goes into supporting a model when it first comes out like JM52 or Miniax M3 or whatever the case is. Sometimes you do have to like you do have to switch out some things.

20:52 Like for instance, the minax M3 head uses full attention and with full attention you end up with this like insane bottleneck inspected cuz you're doing auto reggressive token generation for three tokens and you're doing this like like n squ over all of the tokens that are in your sequence. Um your KV cache is like very large because it's not sparse.

21:11 It's not top k. So we find it better to like okay we're going to replace this you know we're going to replace this layer with a layer from another model that's using like GQA for instance and then just with the right training you can get it to have the same acceptance rate. So it is it is very possible to to retrofit layers from other models and very much needed actually if a layer is like inefficient the training just becomes the challenge like how do you ensure that you train it properly which again to earlier point

21:35 is like the the mesh between training and inference as in like like you need very good training in order to do fast inference that's like I feel like more and more becoming true. >> Yeah. Anything else on the support side when you when you say like get it to fully production ready? >> Yeah. I think that there's also a question of just, you know, we can test a model to a pretty extensive degree, but we're trying to get it out quickly and then you see a bunch of other people test it and you get interesting results.

22:08 There was an issue with um GLM briefly where we had some like mode collapses where it would just output the same token over and over again for certain prompts on certain temperatures. Like once you expose an endpoint to to the real world, there's going to be, you know, so many more varieties of of things given to it that that you're able to, you know, discover and and patch things.

22:32 So it's not just a, you know, day zero process. It's then like for the first week, for the first month, if a model remains popular, like how do you both fix bugs and then continue to push the envelope on performance? >> What do you mean you don't want your model outputting SSS? [laughter] >> Is there loop detection on that stuff? By the way, it still happens like quite a lot, which is surprising.

22:52 We have like we in our in our endpoint like if a model was to output the same exact token like four plus times we just call the generation we say like oh sorry this like try again or like we will reprocess the request cuz we know then like if it like yeah it's four times the same token it's probably collapsed. >> Yeah. Is there a way to opt out in case I really actually want that?

23:12 >> You're actually [laughter] >> I think I think there's a way that we have to handle it. I'm not exactly certain. I feel like in certain models like when they output something like you can imagine like a table for instance. So they want they want to draw like 12 dashes and 12 dashes. Yeah, I think there's a way for that to happen. I think we only do it on certain tokens like we exclude certain special characters.

23:31 >> So we only do it on like certain like like S is the most common almost JL F2. >> Um and I think it was DSV4 as well >> like you just have like looping issues where like >> Yeah. Is there special something special about S? >> No, just seems to be the one token. >> Yeah. and it's and it's only temperature zero or >> even other.9 or whatever it will still it will still collect weird it's an inference it's an inference problem to be honest it's like a software problem like often times um the image you run like in Nvidia

24:00 will release an image for instance and if we will upstream the changes from their latest image into our stack we'll find that it it fixes it or often times this will only happen in an inference engine that you're using like silang but if you were to switch to VLM that isn't the case so it seems to be like an extremely like nondeterministic kind of software issue and not really a model issue.

24:19 It's not like a weights problem. Like people say like, "Oh, it's a problem with the quantity. We did PTQ wrong, right?" But that isn't that doesn't make sense because the same exact weights used with a different inference engine does not repeat this the problem. And sometimes it's um the kernels that are being used in the back end have like these very subtle sometimes race conditions where if you were to use this model hosted on one cluster, you will never get this problem.

24:43 >> Oh my god. >> But if you host it on a different cluster, you will. And the reason is the KV cache transfer from a node to node in that one cluster is using a slower interconnect than the node to node in another cluster. So that exposes the race whereas in another cluster it doesn't. So then you end up just like okay this model is not going to be hosted on this cluster.

24:58 We're going to host it on you know another cluster because that cluster exposes that problem. But then it ends up with like okay is it the software is it the model weights or is the hardware. >> There is a thing about this with temperature zero still not being deterministic right most say because of hardware. Even at temperature zero same model you won't always get the same output.

25:15 M >> even but I'm surprised by the race condition one because uh I thought PyTorch was a graph that like guarantees that you at least you know execute things in the right order. >> Well, [laughter] true like like I guess I guess I'm not I'm not saying that this like I guess well you have things like PDL optimizations where like you can you can start a kernel before the end of the previous kernel and that's like you want to do that because there's no Exactly.

25:37 Exactly. But but you don't do it cleanly like you you overlap a little bit of the execution. No, I guess it is very possible that the kernel itself like that one block that is supposed to be running in this instance of time that kernel itself has a race condition. For instance, like a missing barrier like often if you're designing a kernel and you want it to be very fast.

25:51 If you don't test it extensively, you'll you'll have certain threads access data points from registers before they've been written to by other threads for example because like your barrier is wrong or your synchronization is wrong. But yeah, like the testing itself is is very very difficult in most like and >> and there's no like borrow checker for uh like Rust like that that if you if you're trying to have like memory safety if it's it's it sounds like a comparable problem.

26:16 >> Well, yes, but you're working in CUDA, right? In Nvidia GPU, it's like >> you just need a higher level language like modular. >> Maybe that's what modular is supposed to do. I don't know. [laughter] >> How do you see keeping quality of the model? So you talked about all these steps of okay you got to do quantization train your own speculative decoder run on different hardware um looking at other model providers okay you kicked off a inference speed race on the consumer end um what goes into keeping quality the same

26:43 across them right sure you can run benchmarks but like how do you determine how much quantization are there standards what actually goes into >> there's a few things on quality most influence optimizations are lossless less KV caching for example you are just recomputing or preventing recmputing the same values speculation of course if a draft token is wrong it gets rejected the main lossy optimization is quantization um and that really comes down to number one data format uh number two uh which parts of the model you

27:19 choose to quantize which layers and number three like doing a lot of calibration on the quant weights uh to ensure that you're sort of preserving all the outliers. There's other sort of tricks that you can do though. A big one is long context cuz one thing you asked at right at the beginning is oh what's going to happen if I send a 200,000 token request in.

27:41 So obviously with a long input sequence you need to you know store a lot more information uh you need to process a lot more tokens and so even if a model has a context of a certain length you might as an inference provider choose to build an API with a shorter context length and of course a full length one as well because if someone doesn't need the full million token context for example you can get them better performance performance.

28:11 I don't know if that's exactly like quality of the model. The the way that I think about quality is to what degree are we faithfully serving the original model. If you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close are we getting to that you know 100% fidelity of the model.

28:34 You can also of course think about quality from the training side and how do you push yourself past 100%. But when I think about purely inference optimizations, it's getting faster while staying as close to that 100% fidelity mark as as possible. And certainly our standard internally is that like you should not be able to tell the difference between our API and a you know sort of official API.

29:01 I think Kimmy in particular does a good job of vendor benchmarking here where they >> they released an actual vendor. Exactly. Because they accused [clears throat] uh some people Amazon there was some provider that was not doing very well on Kimy's benchmark. >> Yeah. So [laughter] reflect it would reflect before >> this was a long time ago right? >> No like [clears throat] like three four five months ago.

29:21 >> This also happened with I don't remember which model but they they pulled out quite a few and then they started a whole chart about this. It might have been >> Jimmy vendor verifier. >> Yeah. >> Yeah. >> Cuz you'd be pissed, right? If you're if like um like if I'm a consumer and I'm using like Amazon's endpoint for instance >> and I I use Kimmy and I'm like oh my god like this is bad.

29:37 I'm not going to say Amazon quantized the model in a bad way. I'm going to say oh Kimmy sucks right? So it seems like that that >> they care. They care >> justifiably. Uh this is probably a stupid question but just checking has anything improved from being quantization? Like is quantization always strictly worse? >> Uh well [laughter] technically it's a lossy quantization is a lossy it's a lossy implementation.

29:59 Speed improves. >> It speed improves it. It obviously normally like >> always look for inverse scaling laws. This is something I learned from N Brown where like things that normally act in one direction sometimes they >> well technically when you run a benchmark because these models are non-deterministic. Sometimes your you know MVF4 quant is like you know two basis points higher than your >> Yeah.

30:21 Exactly. Yeah. It's it's within that's why I always say within margin of error and I actually stop saying that because everyone assumes that what I mean is well within some margin of error we're barely inside of that to the worst. So we're saying but yeah sometimes it's just like uh gives you a higher output score but like like Ali said that's noise.

30:40 Um to my knowledge you're not necessarily making the results better. You're just trying to again like keep your fidelity as close to 100% to the original model. There is to your point research that we did on MP I don't know if you are able to pull um a tweet we did one of our research interns Joshua um I think it's a tweet on on how we have 20% better quantized JLM52 than Nvidia um essentially what we found throughout like this two month research is okay quantization is is a lossy it's it's you're compressing the data

31:15 from you know occupying 16 bits to occupying you know four bits for instance and so you're obviously losing some some information and you're trying to minimize that. And so when I say that I'm going to quantize the model, my my job becomes how do I find the layers that I can quantize and how do I find the layers to not? For instance, with image models, I don't quantize modulation layers and I don't quantize out projections because those two were like out projection is what you see as the user.

31:37 Modulation is what the model sees or understands. Right. Exactly. Um and so I guess to his his paper, do you have the I guess doesn't have the the Yeah, it's a long paper. I don't know if I can find >> if there's a part to search or it's probably in the thread. >> It's probably in the thread. But basically the the long of the short is it is very possible that quantizing more of the model makes the results like if I have a model that I quantize layers 1, five and 10 and another model where I only quantize layers one and

32:05 two, it is possible that the model in which I quantize more information is going to perform better because the quantization errors have canceled out. And so what Joshua showed in his mathematical proof where he had like a verifier in is that you can predict which layers are going to have quantization errors that will cancel out with each other and you choose to quantize those layers.

32:24 And so the result of doing this mathematical quantization is you end up with a model that's 20% more quantized than another provider. So you get 20% more throughput of it because there's more layers that running in NBF4 and your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out. like one layer screw to the right, one layer screw to the left, one layer screw to the right, your final logistribution of the model, so you have better fidelity.

32:50 And so the way we proved this was with kale diversion. So instead of just scoring on the benchmarks, we scored the kale diversions between the logic distribution of the quantized model and the logic distribution of the original full precision model. And we show that with this technique, we get if you're if you're, you know, if your probability distribution on the logit switch token it wants to select is more of the same as the original model, you're probably going to end up staying true to the original model.

33:10 So yeah, so it seems like like previously before this it seemed like the industry was well the more you quantize the worse it's going to be because the more loss you introduce. That's not exactly not necessarily true. So yeah doesn't improve it but can cancel it. >> I think it might be this but reminds me a good bit about pruning actually where you can prune off certain layers but very interesting.

33:28 Didn't know this was a whole paper you guys. [laughter] >> It's fun fact. It was originally 72 pages this paper and then we decided we can't uh >> we couldn't really say. So soft 425 [laughter] >> still still 39 page is like very very substantive. We talked about evals and and all these things and uh like what's possible in terms of speed up I guess like it's like probably like the number one thing that people do want to care about and it's something that you wrote about in your post um like official API 70 tokens per

33:56 second and you push it up to 90. Is that like a normal thing? So what's cool about working in inference, the reason that I think inference is going to be a useful place to do engineering for a long time is that if you look at highly optimized domains like say finance, if you're in finance, you measure how much better you got in basis points. It's like, oh, I got five basis points better, like 120th of 1% better.

34:23 That's huge news because everything is is so optimized. when we publish optimizations, it's it's 20%, it's 100%, it's 200%. [laughter] So, there's still probably like a lot further to go, honestly. Like you you'll know that influence is pretty much solved when researchers start publishing about how they got 1% faster at at something which by the way uh because I am from the finance background in the 70s that was the margin at the time when you did quantitative finance research, you would find >> in like the 20%.

34:52 Tens of%. Yeah. And now it's tiny. >> For those for those people interested, look up Andrew Low's paper. Uh he had a really interesting illustration of quant uh stat ARB uh distribution uh narrowing down from like those kinds of 20% differences in the '7s uh down to nothing today. Um which is very cool. >> Exactly. And and we're at the beginning of of the same type of of of thing.

35:14 Now benchmarking is hard. Um I think any anyone will tell you that. and and benchmarking provider speeds is hard because there's so many variables that go into it. What hardware are you using? How much load do you have on the system? What's the exact nature of the prompts and input and output sequence links? All that kind of stuff. But overall, when you start stacking these improvements, you're looking at multiples.

35:40 Um you can look at it the the most common form of course is TPS tokens per second which is bad naming by us in the industry cuz there's actually two tokens per second. There's tokens per second the throughput number and and the latency number >> um to like total tokens per second out of the out of the GPU as a throughput number. Most people only care about tokens per second as the latency number which we should call ITL inter token latency but we don't.

36:06 Um [laughter] anyway, so you can imagine a sort of standard API without many optimizations for one trillion parameter model operating somewhere in the 30 to 50 tokens per second range for a for a reasonable traffic profile. Um, and we generally see the goal of, you know, pushing to 10x that. Um, but, you know, not necessarily day zero, but by stacking enough optimizations, um, if you have say like four optimizations each of which doubles performance, or sorry, three optimizations each of which doubles performance, uh,

36:49 then you you stack that up, that's an 8x gain. That's kind of the the order of magnitude that we're working with in this in this space. We're trying to make things substantially faster, not just go from like 70 to 90. >> Are you saying you you you have done that? >> So, let's say you have as a reasonable baseline 30 or 40 tokens per second. Uh you can you can achieve 10x that.

37:10 So, like on GLM 5.2 too. Um, if you're running it unquantized, uh, perhaps on hoppers even, um, and you're just using an off-the-shelf inference engine with no particular optimizations, no no speculator, um, nothing nothing extra around like KV routing, no disagregation, uh, you you're probably yeah looking at that like 30 to 40. Do you think that's like a reasonable baseline?

37:36 >> Right. >> To get to something like 10x, there's a lot of trade-offs that you're making. If we're running at sort of more like a 300 400 tokens per second range, um obviously you are using the best hardware possible. Uh you have a optimized speculator. You have done all of your quantization work. You are seeing a pretty high cache hit rate. you are running with a reasonably small batch size and a parallelism configuration that is tuned for latency versus throughput but it is possible.

38:11 Um so these spreads that you see if you like go on artificial analysis or you go on open router and you look at you know the the worst provider to the best provider often times can can hit that kind of range. Uh 10x is of course very aggressive. It's often times maybe more of a four to six times improvement, but that's the the kind of performance that makes us really excited is when we can get these huge gains, not just go from 70 to 90 tokens.

38:43 >> It's also like hardware dependent. Like if you if you obviously have a thing where you're serving it on just like a node of H100s and then you throw like you know you shard the model across like four nodes of B200s like like you can definitely increase the speed with just throwing more hardware at it like normalizing for the same exact hardware and the same number of GPUs.

38:59 >> Yeah. Then you're looking at like a 2 to 4x improvement depending on the inference optimizations. So yeah it's some of it's you know what's the car and some of it's who's the driver. If you break down the 2 to 4x say say the example is run GLM52 on >> B200's single node right what's like the cost trade-off for effort to get like the last bit of juice out versus what should people just think of right >> spect quantization >> spect quantization >> that's that's like 95% >> and how far does that get you and how how

39:32 easy is that for the average person to do so say right now I want to throw the weights of GLM 52 on a node of B200s How easy is it to find speculative decoder model or already quantized model? How much work goes into it? >> If you're doing it up front, it's quite a lot of work. If you're doing it today, there's going to be people who have published things that you can just you can just grab some NVFP4 weights.

39:55 You can grab a speculator. Uh yeah, if we're thinking about like what are the 2xs we're stacking going from uh BF-16 to NVFP4, it's not quite a 2x, right? It's like I I think it's about like 30 30 to 40% um from 16 to 8 and then another 30 to 40% multiplied from um 8 to 4. So that doesn't quite get you a 2x but like roughly a 2x speculator roughly a 2x disag on top of that if you're able to get enough hardware and and put enough traffic through it.

40:28 another roughly a 2x and then you add in some you know doubledigit percent increase from having just a a better run time with you know the the latest kernels and stuff behind it. Um and that that's kind of how it stacks up. Um >> so building each of those like building the um quantized weights is for someone who really knows what they're doing hours to days of work.

40:53 um building the speculator again like hours to days of work and the um disag setup hours to days well okay like once once you have yeah getting disag working for the first time I'm saying of course is is very difficult but the the the marginal implementation is >> if you're just grabbing like if you are a person like just a normal consumer who has access to like a node of B200s and you're wondering how can I just host it myself you don't need to quantize the model yourself there's always going to be like an open source

41:22 quantized checkpoint, Nvidia is going to push one out if no one else does. You usually the providers will have their own spec deck that they've trained as well. You don't need to train your own spec deck. You can just use that as well. >> Yeah, like Kimmy uh GLM52 has its own MTP, >> right? >> Minulti token prediction. [laughter] >> I can do it for you in case I get it wrong.

41:44 You know, >> actually you should correct if we're wrong, but their multi token prediction can be used for self speculative decoding. >> I'm actually not sure. I'm semi-confident in but someone can check. Um but you know it's useful to paint the story of okay not just the average person but say a company wants to switch from serverless inference to I want to throw this up on you know I want to rent some GPUs throw it up.

42:06 These are the steps you take to do significantly faster than just put it behind V. >> Right. >> I was waiting for a mention of Dynamo. Uh I feel like uh that's supposed to be the baseline that you measure against. I would think of Dynamo as less of a sort of out-of-the-box system and more of a toolkit for building with. So when we talk about doing KVA aware routing, when we talk about doing KV out offloading, when we talk about doing um PD disagregation, Dynamo fundamentally is by the way Dynamo is an open source

42:41 library from Nvidia. >> We've done that with >> cool. So, so then your your listeners know then that it supports all the different inference frameworks. Um, and it actually is kind of multih hardware which is interesting. >> But it's just a router. It's not like an optimizer layer. >> Yeah. All it does like what Dynamo is good at it is a library for moving information around your cluster around your hardware.

43:03 So if you have, you know, KV cache on one place and you need it to be somewhere else, Dynamo coordinates Nixel for you to to move that around. That doesn't mean that like out of the box you just say, you know, pip install Dynamo and then you get like a a massive performance speed up. Um it's it's more of a a developer toolkit. >> Yeah, I would have said it would it comes with a set of defaults that you can then swap out.

43:31 >> It does. Um if the industry at large, I think was like rolling out all of these deployments um standard, then I think it would be like a a credible baseline. But um [laughter] we've got a we've got a benchmark against like what we're seeing in the wild. >> I did want to talk a little bit more about PD dis this act because that's probably like number three after quantiz and speculative decoding in your book though.

43:54 Uh I was just going to pull out the book like section 522 on Medusa 523 on eagle. Yeah. >> 524 on >> it's 55 um would be um would be disagregation. >> Well no I just wanted to dwell a little bit on the other like so what do you choose to include? What do you choose to not to include? Because there was all these other techniques I guess. >> Yeah. >> Are these still relevant?

44:16 Because I think they came out like a year and a half ago maybe. >> Medusa is quite old. >> Yeah, Medusa is old. >> But is it in the book as a good here's the baseline. You should know this. >> Like I read the paper like it makes so much sense. >> So wi with the book I had a couple goals. uh one was to give people just a working vocabulary for the space as a whole and the other was to give them some intuition about how each of these techniques works.

44:43 As I mentioned in in my AI engineer talk, which is kind of the first public addendum to this, the speculation space has moved much faster than everything else. So yeah, even at the time that I wrote the book, Medusa I very much included as a way for people to understand how the how how the space evolved rather than what the most modern technique is.

45:06 And now of course there's deflash despark um there's there's newer techniques even than than eagle although eagle is still very commonly used. Spec tech. >> Yes. Specular specular decoding. >> What can you >> It's It's bas paper by tree and it's like it's basically doing speculative decoding. >> Uhhuh. >> For the specul. >> It's it's literally just another It's like Yeah.

45:28 It's almost explain it. And it seems like he got non-trivial speed ups there. But it seems that the complexity with training, it's almost like like in our mind at least, it's almost as complex as training gel. It's like a very delicate balance. And often times you it's just but it's literally speculative on speculative >> speculative specul. >> Yeah, we we saw this paper.

45:48 >> It's interesting, right? I wouldn't even expect it to be very particular to train. I don't >> The naive part of me is like, okay, train specular decoder. >> It makes sense. Like the whole idea of spective decoding is you you it's like almost like the iPhone autoproduct version but for a normal model, right? Like you're just you're just, you know, generating three tokens and you're like, "Okay, do prefill on them."

46:06 And so you saved those three turns for your original model. Now your speculative decoder is doing three turns of auto brushing. So why not just have an even smaller model? [snorts] >> The the other question there is what are the size of speculators? So say for GLM, >> right? >> It's like a billion tok a billion parameters >> like for minax it's it's yeah it's like one layer.

46:28 It's like 160th of the original model usually. >> Yeah. >> Actually I think we should do a paper when we get back to the office. speculative speculative speculative decoding. [laughter] >> No, it see it does seem like like how when do you stop? >> But then it also seems like kind of um like if you're able to train spec spec code for instance, right?

46:46 Like if you're able to have a small model that is accurately predicts what the intermediate speculator is going to predict that is able to predict what the original target model is going to predict, then why not just use that smallest model directly, right? >> Yeah. This is adjacent to the routing problem, right? Right. The thing with speculators is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you're running the big model on.

47:11 There is a orchestration and resource competition problem inherent in that and that is one of the sort of constraints on speculation in general is that draft tokens cost resources to create and cost software complexity to manage. And so if you have sort of like infinitely recursive speculators, you're adding quite a bit of that complexity on the actual implementation within the inference engine as well, not just in the training process.

47:40 I was going to say I would wonder if you could do similar like distillation and pruning of you know it's the same thing. It's just a model. Can we not just distill a lot of the weights quantize the speculator but >> out of my domain? Um I guess the the question that also comes up is this is all for big server workloads, right? How much of this applies to say I have this MacBook I want to run Gemma really efficiently.

48:06 Um similar problems not the same >> pretty different. I talked to Sero um about this on on his podcast a couple weeks ago. The difference between inference engineering for the data center and for production workloads versus inference engineering for local AI is that we start with fundamentally like different constraints and different goals. With local AI, it's how do I fit this model onto my hardware and then make it less dumb?

48:34 And with data center inference, it's how do I load this model and then make it less slow. And obviously, you know, we care about less dumb and they care about less slow. But the local AI inference engineering ecosystem, I think, actually has a lot for us to learn from in the data center space. They are experts in various forms of quantization, including dynamic quantization that we just kind of don't touch in the pruning, in the distillation, in the uh, you know, layer removal.

49:05 There's >> removal matters less. Yeah. All those pruning really. >> Yeah. But but >> which is surprising, right? But >> just just to fit something on the laptop. >> So yeah, I mean it's it's a it's an interesting it's an interesting space. Not not necessarily that like their techniques make sense for us to do in the data center. Uh because obviously we have different resources and different goals, but more that the the process as well as the openness of that field is is something to you know admire.

49:36 >> Yeah. Yeah, like to your point like um certain optimizations that would like for instance turbo quantum surf like it made such huge hype on and we we did like a whole deep dive on Twitter what is it how does it work why is it good or not and it took off and it was implemented on local devices because your your memory bandwidth is is so slow on like a MacBook for instance but try putting the same thing on like an Nvidia GPU on a B200 turbo would not be like it would not be used like Nvidia like Nvidia made clear that

50:05 this is not a good optimization and we've seen it firsthand where the overhead of doing deontization quantization of you know in the kernel itself the turbo kernel eone is actually much much slower than the time that you save from doing the bandwidth because on the B200s you have like 3.5 terabytes per second you don't need to >> you know decrease the the storage that much you don't need to do you know FP4 KB cache you don't need to use a recon there's there's there's better optimizations to be made but on on edge

50:31 devices it's extremely important it's extremely useful so you know seems to be like different optimizations there but then they're all uniquely combined with like oh you want to quantize the model you want to do specing like certain common prefixes >> principles >> yeah exactly exactly >> they also do a lot of work on um model parallelism especially over uh you know heterogeneous topology where you have you know some sparks and they are wired together with you know Ethernet DGX sparks >> yeah this is the exoabs >> yeah

51:01 you have you know a number of uh Mac minis stacked >> [laughter] >> there's you know the inter they they one thing that that I think we both have to deal with although they have to deal with a lot more is is the interconnect between machines um which is why like you know one thing that we do a lot is work with tensor parallelism and that's where uh you are using all of the you know all eight GPUs um and and sharding the model across it uh tensor parallelism is not a good fit for local AI because it assumes a very high

51:32 bandwidth interconnect like envy link uh was you know they might be forced to do something like pipeline parallelism which we're never going to do unless we're doing some kind of multiode inference. >> Since you mentioned it I actually wasn't sure if we were going to cover it but let's briefly explain tens of parallelism and expert parallelism. Uh since you have very nice images I just want [laughter] to show off your images.

51:54 >> Yeah. Uh shout out shout out to Luke from Bad's design team for making these these beautiful images. Oh, that's a that's actually before we get into this just one other difference is we we talk a lot about the active parameters of a mixture of experts model and for local inference folks that matters a lot because if you have a batch size of one you're only activating that many parameters when we do >> yes conversation >> yeah we uh when we go through like ae model um and we host it um for an API we assume that all

52:27 parameters are going to be active because your batch you're going to you're going to hit everything. >> Um cool. So >> broadly tensor parallelism you can do with any model. Expert parallelism you can only do with models. Um effectively all models today are models um that are you know at least all models large enough that you would care to parallelize them across multiple GPUs.

52:51 So that that's that that nuance is is less important. Now um with expert parallelism the idea is you put the entire expert on a GPU. Generally you have more experts than GPUs. So you might put like n experts per GPU like eight experts per GPU or whatever. And then you uh you replicate the router which the router is very small across each of the GPUs.

53:12 And then by moving the the generation from expert to expert with each expert being inside a GPU they're not competing for resources. you massively increase the throughput that you're capable of doing and the uh GPU toGPU connection is is not as important because there's not as much communication. Uh tensor parallelism requires that you are able to do this like all gather all reduce.

53:38 So you basically shard the model across the GPUs entirely. Um and then for each step you're combining the results of each of the GPUs which is why the interconnect matters a lot and it is generally of course this is a this is a very high level generalization there's a lot of places where this is not correct but generally uh TP is is helpful for latency and in many cases you will use some combination of these two parallelisms um across the the model rather than just like picking one or the other.

54:12 Do you do you want to you want to add the color though? >> Like in a model it's not they're not mutually explicit. You do tensor parallelism and you'll do expert parallelism. Pipeline parallelism less. So like it seems to me like we never use VPN. >> Yeah. The only reason you would have to do pipeline parallelism um which is where you separate like different layers and you put like half the layers on one hardware and half on another is if you are forced to do multi-node inference um because a model is bigger than you

54:36 have the like let's say let's say you're doing a deployment on H100s for whatever reason and you're putting a a trillion parameter model on there you have to use multiple nodes of H100 and so you the because the interconnect is so flow between the nodes. The only viable way to parallelize there is is pipeline. But then you would do expert and tensor within each node.

55:01 >> And the limiting factor for H100s is HPM. >> Yeah. They just don't have enough. >> How much what's the magic numbers that we need to >> like on a B200 is 180 GB per GPU and then a node of eight you're talking like 180* 8 and the FP4. So each parameter takes half a bite. So that's 800 GB. >> On a H100 it's like 140. >> It's 80. >> It's 80. Yeah. I'm old.

55:24 I've been doing this a long time. I actually remember H100 specs. >> Tell me about the T4. >> Let me tell you what it was like to run a model on a T4 back in the day. [laughter] >> Um, >> well, one one thing I was surprised to see that more people didn't do. Uh, Jamba, I don't know if you guys remember Jamba from AI21. They would actually specifically pick a hardware and then they they they designed the arc dimensions for the hardware and then it would obviously saturate the hardware like it makes sense and like

55:53 somehow all these models don't do that. >> Don't they do this for the training side though? >> Uh I don't know what the >> training for training >> like deciding which GPU which GP >> yeah how to >> yeah yeah they they do and with training it's more of like a math like you can run the math and see the flops and maximize it. with inference it's more of like an autotuning like GPU current auto tuning but like it's it's basically like you you define that oh I have two GPUs I can do TP1 TP2 EP1 EP2 for instance right and

56:21 you so that that gives you like total like two square combinations and then you just like you shadow the same traffic like real world traffic and you just see which configuration gives you the best TPM TPS and you just use that I don't like the fact that it's you cannot reason about which one's going to give you the best performance or that there isn't one specific configuration that's always best but It seems like autotuning is just the way that you find the best one.

56:42 And with kernels and GPU kernels, it's much of the same. After you you design your kernel as you design your configuration, how many threads do you launch, how many, you know, how much shared memory do you use? You just just auto tune. You just sweep the parameter space and decide on this is the best one empirically. >> But yeah, but they are they are combined.

56:57 They're not just like separation. >> There's a few bits of training that are kind of like hardware targeted. If you look at, for example, Nvidia Neotron models, they run very very well on Blackwell. That's that's unsurprising. Um, [laughter] so there's some degree of that, but I think that most open labs are trying to make models that can be run on as wide of hardware as possible rather than targeting just like a single chip.

57:21 >> I see. For usefulness. >> Yeah. >> Uh, okay. One more thing while this chart is still up. All gather all reduce is expensive. One of the things that is a movement in Silicon Valley is mega kernels. Just keep fusing kernels. I don't know. [laughter] Is it that simple? >> Well, I mean like if this colonel can't save you like like here with with with tensor parallelism you're then half the matrix is one GPU and the other half is on another.

57:44 And if I need the entire matrix in order to do like a nonlinear operation in the next step which is for instance like if I'm doing attention I need the softmax or I need to like like exponentiation I need to have the entire row. So I need to know what that partial result was from GPU 2 and what the partial result was from GPU1 in order to be able to do the softax in the next stage.

58:03 So I I like I have to make them communicate with each other. Even if I had a fuse kernel because of the nonlinearities within each one also with with like mega kernels like honestly I'm I'm very bearish on on I'll be honest. >> Please please please. >> No it's just like mega kernels it it was a good research direction and it seems like a very like like intuitively theoretically it's nice like oh my like you have a lot of launch overhead from launching one kernel the data just fuse everything together.

58:27 But yeah, but like like the the kernel complexity itself is is is very difficult to write a very optimized mega kernel. It's it's it's very very difficult to do so. And even the like not to name any companies but like even the companies that haveve worked at people that I've spoken to who work at companies that do fused mega kernels they very very often don't end up running those in production because the TRTL and modular kernels that launch are faster because you can optimize each individual components and you can

58:56 just have them paralyzed with each other >> with the Reuben's I don't know if you guys saw the Reuben's Twitter post yesterday but they're also um he >> Reuben's like no like the GPU. >> Do you have a Twitter account for Reuben's only? >> No, no, no, no. [laughter] >> I was like, what are you talking >> one of one of the tech leads at Nvidia like launched a Twitter post like we're pulling the curtain on Reuben and here's the here's the specs and and the and the third tweet showed like not to get too technical into it

59:22 and I need to read it much more. But the GPU is is designed in such a way that it basically kills me kernels. You don't need to use mernels that much anymore. So it seems like that entire research field goes into like won't be continued. But yeah, >> can I speculate about Reuben for a minute? >> Um, you know, I've been through now we >> and by the way they are covered in the book, but yeah.

59:45 >> Well, I mean they're covered in the book in the sense that like >> I'm aware from the blog post that is going to happen in the future. >> And you even had the the name of the one Fineman. >> Yeah. It's like, hey, this is this is going to be this is very future [laughter] proof this thing. Okay. Uh I don't want to publish a new one until like next year or something.

01:00:02 Uh anyway, so we were discussing the degree to which I am old. Um and you know I've now been through three hardware launch cycles. I've been through the Ampio launch cycle, the Hopper launch cycle, and the um Blackwell launch cycle. Now when I say launch cycle I don't necessarily mean like the uh the actual shipping of the hardware like ampers were racked up well before I got in this industry but there's a lot of time between hardware being racked up and hardware being sort of feasible for inference.

01:00:38 So if you look at like the original VLM and SG lang or VLM especially like that was written targeting Ampure and then had to be updated for Hopper updated for Blackwell with each of these cycles it becomes faster and more urgent but also substantially more complicated. when I look ahead to, you know, what's going to be new with with Reuben, I think that like Dynamo gives me a lot of technical hints around like what kinds of work is going to be very valuable.

01:01:08 Obviously, we're continuing some trends from Blackwell, right? NVFP4 is big. The amount of compute that they have behind NVFP4 tensor cores is is massive. Um, we we're going to talk about video, I think, at some point and and that's the the big barrier there. you've got, you know, much much faster memory bandwidth. Um, but which was the same thing that that made Blackwell so good.

01:01:30 Um, but the the big thing is more systems thinking. You have more emphasis on the CPU to GPU interconnect, more emphasis on the interconnect between GPUs. And when you look at Dynamo, it's a system entirely designed around how do I move the KV cache to where it needs to be when it needs to get there. Um, so I think that themes around like KV cache offloading, KV aware routing and and disagregation are going to be substantially more important in the Reuben era, which means that inference engineering becomes not just a

01:02:04 like CUDA kernel problem, but also like a very traditional hardware infrastructure problem, which is something, you know, we've been building toward for for a long time and something that's like very exciting to me because we're going to see sort of multiple domain s colliding and the ability to reason from the kernel level like up to the hardware level and and back down is is going to be very valuable.

01:02:28 I will take what Philip said one step further actually into that it's I think trending towards becoming exclusively an infrastructure problem where like problems of PD disac training spectacing kernels is not going to be much of a problem because the GPU is moving more towards being an ASAC where it's you're just you're just trying to orchestrate what happens on the GPU but you're not actually controlling it thread by thread level and you see this with like QL QDSL like you're you're just working at levels of like

01:02:54 tiles of data but you're no longer working at controlling what each thread does on the GPU that's being taken care for you. Um, so I guess do you agree that a GPU and future GPUs are trending more and more towards becoming A6 that just need to be launched and then they do the data operation based on your conversations with other people? Oh, I mean, yeah.

01:03:15 No, that that that is a section of the market, right? >> And obviously A6 can do uh a lot more performance for only their their workload, right? >> And the the G and GPU makes makes them continue to be very general. >> Yeah, the the I think that there's like a spectrum >> actually graphics, but I I keep saying this, I have to correct myself in case people come at me for getting the the G wrong.

01:03:39 Yeah, it's and it's it's like a spectrum, right, of of a very very general purpose compute to something like a Talis where you've got the the hardware built for a specific set of model >> weights into the chip. >> Yeah. >> No loading. >> I don't I wouldn't say that like that that we're we're we're going all the way there. It's more like along this spectrum.

01:04:00 It's a step in the direction of more specialization within the hardware. >> Yeah. I'm curious. I feel like he was driving towards something. I guess my point is being bearish on like you say you say like everything else apart from burning the weights into the chip. Burning rates on the chip is like impractical because you want to fine tune, you want to optimize, you want to you want to quantize, you want to release new checkpoints of the model.

01:04:18 If it's burned into the chip, the chip's useless in like a month or two, right? I guess I guess my point is how can you not like seeing Nvidia more and more specialized like take its GPUs from a general programming paradigm where you're just it's a general computer that you can use to program threats and with every new generation you're putting more and more specialized instructions specialized tensor cores specialized you know MMA instructions things that will will allow you to to just control it almost as an ASIC

01:04:44 almost as a collection of AS6. How can you look at this trend and then still be bullish on companies that are coming up with A6 for AI in a sense in a sense? >> Yeah, because they're they're sort of re they're evolving towards that direction. >> They're almost evolving towards an like as in Reuben I guess like compared to Empire or you know T4 Reuben is is is basically an ASC.

01:05:08 It is is basically just the thing that is is used. It's like pre like you can program obviously obviously like like I guess I like it's very controversial to call it as it is it is a GPU it is a it is general it does have threads I can write CUDA to control it and change it operations but but it has the solic arrays and tensor cores and TMAs and and tensor memory and it has these things that are almost exclusively useful for loading model weights it has um you know tensor core instructions that are almost exclusively

01:05:35 shaped around the head dimensions of models that exist in the market today to say that you're going to come up with an async you're going to etch something into it well with the next architectures basically going to be useless. >> Yeah. I don't know. I don't know. I think that the thing to remember is just how long these hardware cycles are. So, if a chip is coming out today, that means the design process for it was kicked off years ago.

01:05:54 And they've at Nvidia they've done a very good job of predicting where the market is going to go and and you know >> I mean they have the most information >> of course but if you if you look at you know there being public open source model architectures that look more or less like early versions of the one today. Reuben's honestly the first chip that was fully built in that world.

01:06:17 And so you can see a lot of the understanding of the shape of the workload that this chip's going to be asked to do in the way it's designed. >> Yeah. Okay. So, so I'm not going to be the best person to to directly answer those questions. I think these are very fair questions that are honestly the first one that's based on Reuben that like I've heard artic articulated so well.

01:06:41 I do think that uh I I will make a case for uh vertically integrated model lab AS6. So like the open AI Broadcom whatever jalapeno >> uh chip which like totally makes sense like uh so I we first had this on the pod with uh Martin Casado where he was like look if you have a a trillion dollar or $500 billion training run then take 50 billion of that and make it ASIC like it's fine like like you won't get more than 10% efficiency from from the ASIC and like that makes sense right so so like a model specific trip yes but

01:07:14 ASIC companies the interesting thing is I feel like you are focused you're hyperfocusing on like like you say like the tiles stuff. >> They are doing a lot more sort of like uh surface area engineering or like like like the actual allocations on memory and hardware and like the uh the communication between between chips that um probably still won't be touched by Rubin but I don't know the details.

01:07:38 >> I see. I see. They typically they often talk talk about things that I would expect to have bigger orders of magnitude than would be programmably accomplished by whatever Ruben does. But who know who knows? [laughter] >> No, I see I see. I see. Yeah, it seems >> Yeah. Like I mean you know like think about what is what are the real blockers to 10x to 1,000x faster inference.

01:07:57 It is not the stuff that can be rearranged uh just within the existing GPU design >> intercommunication. >> Yeah. Like these these guys are aiming for 300,000 tokens per second. They're not [ __ ] around. Like um make six [laughter] >> maybe. I think you know it is interesting to me that you're so beish on so much of this kernel engineering work given how much of it you've been doing recently.

01:08:24 >> Right. Right. Like the more I do it the more it just seems to me that >> it's not mega. I would also add like there's generations of models being out. Right. I think on your guys' end you see a lot of okay one day it's GLM Kimmy DeepSeek uh Miniax throw in the others some are doing completely different stuff right Gemma no encoder the latest thinking machines is all from scratch but when you look at the other side like how long have we been on the GPT5 generation right they've been serving that thing for quite a

01:08:52 while sure there's maybe more pre-training there's there's different checkpoints but like you actually can squeeze quite a bit out and you do a multi-billion dollar train run if you can make it x% more efficient. They they serve it for a while. Same with uh say the cloud 5 uh family, right? >> Like if they release a new model like they release GPT6 now or whatever and they're releasing new model every year and we >> well we don't know but if we assume that they're changing some bits of the architecture and not just

01:09:17 doing like post training like you're going to be spending $50 billion a year every single year coming out with new AS6 for the model and throwing out the AS6 of the previous year away. >> Yeah. Yeah. >> Easy. So, so I think okay, um I I would slightly disagree based on my again my it's all secondhand uh on the longevity of a model. There's still people out there using 40.

01:09:38 >> Yeah. >> Yeah. Llama, not Llama 2, but Llama 3. I still see Llama 3 workloads. >> Yeah. >> Because if if it's done, if it's trusted, uh don't change it >> if it works, [laughter] >> which is one of the promises of open source, right? Like the whole 40 save 40 movement. Like you don't got to have a save llama 3 movement. you just got to have an 800 somewhere.

01:09:57 [laughter] >> I think at some point there's also the question of if a model can do enough and use enough tool calls and be agentic enough uh can it just web search tool search write code do you really need to keep squeezing more? We will because you guys will make it cheap and fast and smaller and I can swap it in. But at some level like you give me 52 today or say whatever 120B model I can run with it for quite a while.

01:10:22 Right? >> This is assuming like you don't need intelligence. >> I think there's a lot of >> you need reliability and predictability like I'm an enterprise like like this is tried and tested. It is signed off by like my 5,000 stakeholders like I'm not touching it. >> It wants a batch job every day and I like the results. The results are predictable. >> Yeah, >> it doesn't make sense to keep using them.

01:10:45 Like stuff gets sparer, cheaper, better, but that doesn't mean that old models GLM50 isn't usable, right? If we hit a stall, say for whatever reason, there's still a lot that can be squeezed out. We're going to run out of time. I I did want to also make sure yes, actually, we happen to have this diagram pull compare this versus any Cerebrus diagram, right?

01:11:04 Um I don't think Edge and uh Maddx have have put out public uh charts yet. Uh but the complete the real estate is very different. The size is very different. Right? This is not wafer scale, right? This is there's probably like I don't know a few hundred of these on on a wafer. Um I don't I don't know how big the comparison is, but like uh it is a it is a very like real estate allocation difference.

01:11:26 >> Few few dozen I would say. >> Few doz. >> Before we move from hardware, I have two quick questions. One, the latest Kimmy which is really big uh 3 trillion doesn't fit on most hardware on single node. Yes. you you you need you need GB300 to visit on a single >> ord [laughter] it's simple math um NVFP4 2.8 trillion parameters uh 1.4 4 terabytes. Um the GB300's have uh 288 GB each.

01:11:52 Um so eight across eight of those. Um you have enough room for the model. And honestly like so the other thing with with GPU VRAM math is you have to leave space for the KB cache and that's going to depend on to some degree on the context length. Um so when a model is both has a very large number of parameters and a very long context length you're kind of like fighting over space um which is why you know the KV cache offloading uh would become like a more salient topic I think with these huge models uh cuz you you just

01:12:31 you're very crunched for space >> with the Reubins. You now have what? NVL72 rack of 20 >> terabytes of >> Now you you still have NVL72 on on uh Blackwell as well, but um you can't necessarily assume you're going to do inference on that. Um there's a whole lot more ADX racks in the world than there are NBL72s. >> Yeah. I guess my last quick question on hardware was do you notice anything with hardware generations for new pre-trained base models?

01:13:01 So one of the things you said for efficiency is you can swap hardware. That's one of the 2x gains. Um, when we see new stuff coming out training wise on Reubins, uh, any changes on locks, does this affect what type of models we will be seeing when these are more available? And can you >> they they get bigger? like people understand the ceiling that you have in terms of how many parameters of a model you can run uh given the sort of latest inference hardware and that kind of forms a ceiling.

01:13:34 And so for example when DeepSec R1 came out it was you know it was 671 billion parameters which at the time was really huge and I think did a lot to push us to really quickly adopt Blackwell and and get good at serving on Blackwell. Um so yeah it's it's it's mostly in my mind about uh model size and then about matching the architecture and the native quantization to the the target hardware like we talked about with like you know all Neotron models of NVFP4 for example.

01:14:07 >> So we talked a lot about LMS. >> Mhm. >> You have a lot more in the book. What about audio video? What's the other side of inference engineering? Ali you're pretty big in video diffusion. Video diffusions I think are like they're just shaped a lot of the stuff that you can think about reason about with LM being autogressive. With video diffusion it's it's it's not the case.

01:14:26 For instance, you don't do you don't do batching. You every request just comes in on one GPU and it serves one GPU. You don't have to shard the models are a lot smaller. Like 1.2 for instance is a 20 billion parameter model. um you don't need to worry about it's like orders of magnitude smaller than the best LM and it's one of those spaces where the the open source models are like with LM we see KK3 is almost comparable to you know methyl like GPT 5.5 the the difference between the best open source LLM and best open

01:14:59 closed source LM is very small like it used to be 6 months I don't think it's 6 months anymore I think it's like basically almost unpared video models are definitely not there's a huge gap If you look at the best video that you can generate today with an open source model like quantu versus something like with cling or video difference is light and there.

01:15:15 So it it creates this disparity where media companies will choose to go most of the time to closed source models for instance and I were to tell you hey I can generate an entire 3-hour movie for you with this model and they'll optimize it so that you only have to pay $10. But if they were to do it on closed source they'd have to pay $1,000 which is 100x like I'm 100x cheaper but it's still $1,000.

01:15:36 they're still going to choose to do all of their cuts with V Uncle. So the it's like a chicken and egg cycle where less demand causes less innovation in the field causes, you know, less open- source checkpoints to be released. And some of the labs that were releasing open source models like one will have closed source their latest models like 1 12.7 is not open source.

01:15:54 We're still in 1.2. The the challenge with with video models especially is the number of of tokens. So so so so models you want to generate a high quality model, a high quality video. So let's say you're doing 16 frames per second. That's like the absolute minimum you'll do. And let's say you'll do like 480p video. So you can think about your like dimensions and I think I have like a good um just like a diagram that shows the number the sheer number of tokens, right?

01:16:19 Let's say you're looking at like just one video of like you know Sparta or Sparta 300 whatever. So let's say we're looking at like four frames, right? those four frames of that video. If you go just if you're doing full attention, if you go a little bit up, like you're looking at uh 480p by 720 by 81 frames in just 5 seconds cuz 16 fps by five, right?

01:16:40 And then you compress it down to latent space, but you're still doing 30 by like 50 by 21 tokens. Yeah. >> Which means that for attention for just 5 seconds, you're running attention on 35,000 tokens, right? So the attention becomes such a huge bottleneck and because it's open squared if you're doing like if you extend that to like 10 seconds well it's just square 20 seconds 30 seconds.

01:17:01 So to generate a good cut scene of like 1 minute it's almost impossible to do within the same compute time and it's just it's becomes unfeasible you can't do it and so you end up with moving towards two direction either you decide to do attention on the entire video at once in which case you are forced to do sports attention. So if you scroll back down to the original the the video image like you can see whereas on the left for instance I would be doing full attention where every single token in that sport at 200 scene

01:17:26 attends to every single other token as you can see the sheer number of like red patches on the right I'm only attending to each token only attends to like the top K top 12.5% that's important to it which can be like spatial so like you know the token that represents the crown attends to like the head the face and then the head on the other frame and the previous frame temporal locality spatial locality that kind of thing.

01:17:44 This results in terrible video quality. And the whole point of the the post or the article here is to show like how you can train and you can do all these things, but you will still suffer your quality a little bit. So you end up with one of two things. Either you bite the bullet, you have huge compute and you do full attention over like a million tokens because you're trying to generate like 2 minutes of video [snorts] or you move towards auto reggressive video.

01:18:06 Auto reggressive video seems to me like that is the bet that the future is going to be making, but there are no good open- source auto reggressive video models out there today. And that seems to be if you want to get like an hour movie, if you want to see video models generating like an like, you know, Hollywood level movies, they have to be autogressive in order to exceed the 5-second frame.

01:18:22 Or there has to be some insane leap that happens in compute that allows us to do full attention over like millions of tokens at the same time in a in an efficient manner. >> Even millions of tokens, it's like you're you're quadratic, so you're going to get there really quick. I think can you explain the pros and cons trade-offs of auto reggressive? So one that comes to mind is you know the consistency across frames you will 10 minutes into generating auto reagressive diffusion you're going to forget but what what are

01:18:51 pros and cons of this? Well, like like auto reggressive LLMs, you can take a lot of your sorry autogressive diffuse models, you can take a lot of your optimizations that we discussed with LM like spectac and stuff like that and you can apply it there and you can if you have a very high quality scaled up model. There is no reason why I can't stream the output as in I can show you the first frame and then I'm like like kind of like GPT back in like 2023 when you're like now it just almost like one shots the text but back

01:19:13 then you could read and it's generating as you read. With video models you can watch and it's generating as you watch. it generates the frames and and so token by token generation will allow us to to scale a lot up and apply the attention mechanisms there. The downsides is every single auto reggressive video model is [ __ ] It's just terrible quality.

01:19:31 If you like it's just if you put if you put the quality of any opens like 122 versus any other autogressive model, you can see like a like a video generated at 1.2 is like you know a cat and dog fighting autogressive model will give you like degraded Tom and Jerry quality. the the solution to generating long output then becomes okay we're not going to use auto reggressive model we're gonna if you look at some of the things that like um like Grock imagine or Grock video does and they do it really really well is they'll

01:19:59 they'll try to stitch these you know 7-second chunks together and so you generate 7 seconds and then you're like okay I'm going to can you extend this video and they'll chunk two videos together open source doesn't seem to have the tricks that they have there and and by definition it's closed source we don't know what they're doing but the closest you can get is taking the last frame of a and feeding into like a text and image to video where it will take the text the prompt and it will take the image of the last frame

01:20:22 and you'll ask it to generate the next 5 seconds and that's kind of like how you you can extend this level of a model to generate like a move where you just you're constantly streaming frame by frame but you get a drift. So you start with like you take the image and then you generate a video and then that next 5-second video is like lower quality and the third chunk is like even lower and the fourth chunk is even lower and like sometimes you'll see things where like the the new video is like just ever so slightly

01:20:46 darker than the first one and the next one is darker than the second one until like 25 seconds and you have black screen. Like it's just it's it's it's a we tried to have a demo that would show this but it was like it was it was extremely embarrassing to show. like we just decided not to because it seemed to like but it is it is um I think models will get there.

01:21:04 They just need to in my mind scale up significantly and move towards being ultraaggressive but the training techniques don't seem to be clear there. >> For those on who interesting Grock imagine we did a pod with Ethan H from from that team right >> uh who dropped a little a few hints but not that not not enough that we can [laughter] fully reconstruct everything >> specifically on this part.

01:21:22 He explains a bit about >> Yeah. So we talked about memory and in uh longer context and all those things >> but as far as I know there it's not autogressive even though like no one in industry is auto aggressive. >> Yeah. >> It seems to be. Yeah. >> The key thing to understand between a auto reagent goes in both directions while auto reggression it only goes forward in the sequence.

01:21:44 So that's why you see this sort of like going off the rails behavior both in if you sort of naively construct a video generation model as simply generating a linear sequence of frames. You can't then go back in that sequence and fix something to make the whole thing consistent. While of course the reason that we need all this latent space for the video model is like you said we keep all the tokens in memory.

01:22:07 iterate over that that full sequence and you can you can adjust the past in order to make the future make sense. So if we think about the architecture that's going to get us there to these longer richer sequences, it's probably like you said going to be a mix of the the auto reggressive and the diffusion um working together to do what each piece is good at.

01:22:32 >> Well, you get like you intuitively get why. like English for instance or just writing in languages like it's just left to right you can stream your tokens you can stream your chain of thought just even as a human you write like you just you write and then you think about what's the next thing you're going to generate and then you write that and then you think about your ideas and you generate forward and sure you can argue that as you write you need to go back and you want to edit some things but you need to do that

01:22:51 you know less often than you think whereas with video there is no sequential you know the the pixel in the top left corner of the video and the pixel in the bottom right corner of the video they both need to attend to each other to understand how the video quality is going to be almost as equally whereas with text you don't need that as much. >> Is there a parallel to audio?

01:23:09 Like I'm not 100% confident on this, but there was a point about a year ago where there was audio LM there's diffusion for audio and auto reggressive and for the points you mentioned mostly on the infant side even though they're shorter clips you know most music is 3 to 5 minutes um we've basically swapped over to auto reggressive for >> yeah I can't speak to music but speech is auto reggressive you uh effectively I mean this this was even back with like the Orpheus architecture a year and a half ago or you just add a

01:23:40 bunch of waveforms to the vocabulary so that the LLM can output tokens that represent those waveforms and then you construct speech and that's how you stream it. >> That's it. Wow. >> That's my AIE talk from 2025. >> Oh, nice nice. >> But it's not with audio it's not the same challenge though because you like audio is sold with an LLM that generates everything like like with audio it's still a transcript that you can generate an LM.

01:24:05 So your your audio model just needs to like transcribe it text speech. >> For music there was a phase of a t a trade-off between diffusion for music and auto reggressive and they were both pretty on par. There's probably more pros and cons to either I just wanted to poke if you had takes. >> Yeah, I don't know about music specifically. You know what with what you said about >> editing your writing, you know, obviously I think my editor would tell me I actually need to do that more often and go back and fix things.

01:24:33 I can imagine music or poetry for example where you have a rhyming scheme and you might want to go back and make a change to make it to to make it easier to set up a rhyme that you want to make later on. Uh there there being some advantage to being able to attend in both directions. uh but yeah to my knowledge you know I I very much bifocate this this influence problem into the autogressive models which have a set of constraints and techniques and the diffusion models which have a set of constraints and techniques and

01:25:04 um I think of text um embedding uh voice in and voice out as being in the autogressive side and then image and video being in the diffusion side. There's some overlap between the two. It's not a a perfect split, but that's the broad categorization I use. >> I should point out, I think it's confirmed, right? Nano banana and uh GPT image are auto reggressive image.

01:25:29 >> It's kind of this blended approach that we're talking about. Um but in the the image space, uh it hasn't like made its way over to the video space, at least in the open source world. >> Yeah. But like I I assume that's not too far away if if that is possible on on the at least the Quen image guys are trying it. >> Yeah. Yeah. With the I'm really excited for Quen image 3.

01:25:50 I hope they open source it. >> And then I I'll also mention on the diffusion for text side. There's been some movement. Not a lot. >> Yeah. We've got Mercury. Um >> you host Mercury. >> Yeah. Nice. >> Yeah. Gemma as well, right? [clears throat] Diffusion Gemma. >> Uh >> diffusion Gemma is open source. Um and then uh >> and we're on the science part.

01:26:09 They just have been releasing uh some uh virtual cell models that use the fusion as well. >> Yeah, they they have built it's definitely still in the sort of cheap fast tokens um world. >> Yeah. >> Uh we're trying to >> it's I think it's the wrong marketing and I've told them this before. I was like look like you're not going to beat the optimizations that you know the other LM are going to do.

01:26:32 Uh but you can you can have different APIs like you should be able to use it differently than chat response chat response >> how so >> because it's diffusion because you can do like what what does like contextf free guidance for diffusion look like >> for for text like give me a give me a poem give me a plot structure that like diffuses into place >> exactly so that's where you know like I mentioned with with poetry for example where you might want to ensure consistency across you I've done a lot of LLM sonnet It used

01:27:04 to be one of kind of my go-to benchmarks. And even today, yeah, they they they don't get the syllables right. And if you can attend across all of the different tokens, uh you can you can get the syllables right. >> Yeah. Uh and uh David Holtz from Mid Journey was investing in text diffusion. I don't think anything came out of it, but like the idea was that you can storyboard a long movie and then you can generate the scenes with video normal video gen.

01:27:30 But the the idea of like coherence across a thing that would just appear where like the the the end should attend to the start and you should not have this auto reggressive path dependency does make sense in principle. Just the API should be different. The marketing should be different. >> None of the most heavily used open source or closed models use diffusion.

01:27:51 But isn't that like like doesn't that point to almost like a >> it's it's chicken and egg because what if you just gave it more scale? [laughter] >> What's the largest diffusion alone? I don't think it's very big. >> Uh I don't know the parameter count on this one, but the under 20. >> The fusion drama is not >> I think it's a 20 something. >> Yeah.

01:28:08 Yeah. You know, like you haven't actually tried. >> You haven't given it a big one and you haven't [laughter] unfair is a 25B and >> and that's what I'm saying. It's like for its size it it does pretty well um in terms of in terms of quality. >> It's almost like the same challenge with video models that have the same size. It's like you're comparing it to models that are much larger in scale.

01:28:29 >> Yeah. Well, unless you do the whole thing where you you have a text uh backbone and then you you know you like glom some kind of decoder thing that uh >> does that like you know we started off the podcast doing this for the uh for the inverse direction from image to text right >> uh and I think like it's it's roughly intuitive that you can do the opposite direction.

01:28:49 >> I agree. I see [laughter] I see what I see. >> Yeah. I mean we're we're speculating on on research in general. Yeah. Um, one part we can end off with this is is the the topic of your talk where uh inference engineering used to just be like let's take an open model make the GPU go burr and then that's it. That's the job of of phase 10. Uh now it looks like people are using inference more and more in post training.

01:29:12 >> Yes. >> Um and training and inference. >> Yeah. So it's training for inference and inference for training both have become big topics. Well, inference for training in the sense that like obviously you just need you need to do you know you need to do rollouts when you're doing like oral training runs and so if if your rollouts are taking a long time if like you know you're using a VLM for instance as opposed to CRTM or if if the model that you're trying to train is not supported in CRTM and you have to fall back to

01:29:39 an older inference engine your rolls are going to be slow and you don't want to do training on rollouts that are too off policy so you have to wait for them so you bottleneck your entire training pipeline and so like obviously the techniques that we we do inference optimizations for um will help them there. The training for inference mostly comes down to like just the spec training eagle head training um and sometimes post training.

01:29:59 For instance, if you want to quantize a model, you'll quantize it down to like NVF4. How do you like sometimes sometimes you get lucky and you can just do PTQ and that works. Sometimes you quantize it down to NVFP4 and the model is is terrible like the quality is too bad and you have to do post training on the model in order to make it understand that it's going to now be in NVF4 and let it still output the same logic.

01:30:22 You can do this with normal SFT PC um you know quantization aware training all of that stuff but more and more so we're seeing techniques like Nvidia released a quantization distillation paper where you you establish a version of the model that's in NVF4 and a version of the model that's in full precision and then you'll do distillation training based on the logits of the two models in order to make the FP4 model understand and so more and more of the team the engineers like of the inference engineers that work on our

01:30:47 team they have to be very familiar with like training techniques and and just being fine writing training pipelines for it. Yeah, it just seems like u they're meshing together in a sense. >> Well, it's coming together. >> Yeah, absolutely. I mean, if you think about the ultimate goal potentially of having a continuous improvement system, you know. Yeah.

01:31:09 I mean, it's it's it's kind of funny, but at the same time, it's also kind of happening. And and I think within a a few months to a couple years, like a lot of leading agent builders are are going to have these loops like really up and running in production where you are doing inference, learning from the inference. We obviously for a long time have been sort of like learning from inference as it's live and dynamically adjusting the system.

01:31:39 uh you know any kind of dynamic adjustment is going to beat a static configuration across you know your um exact config across your speculator across that kind of thing and then the you know you can take the the traces that you're generating from your product continuously post- trainin the model roll those out AB test um get better signal get better model get better product that that loop is is really promising um the technologies and the infrastructure to build it are coming along quickly and so the sort of

01:32:18 unification between training and influence I think is is only going to accelerate. >> I actually was chuckling but I wasn't I didn't think it was funny like it's actually real. One of the big things for AIE World was fair was that you know we have RSI until AGI is the is the rough tagline. uh which like yeah I mean we have uh I saw you pull up parameter golf like we have models training models and uh the next step is obviously models training uh the optimizing their own inference which is kind of funny.

01:32:46 I wonder if uh models will be like on policy better at training themselves than training models that they are unfamiliar with this. These are all like very interesting open areas of research. One big part of my job a couple years ago was for any arbitrary model that came out on hugging face writing a config for it and kind of getting it up and running.

01:33:06 And now the get it up and running config is is oneshotable. Um and so you know I don't have to do that anymore. Yeah. I mean that's not exactly a model optimizing its own influence so much as a model like being able to read the SG lang docs but uh yeah I mean >> well we do see it we do see like with JLM52 for JLM52 is very very good at writing GPU kernels >> and so for like it was very funny internally we had a JLM52 endpoint that we're using to like that we plugged in in our cloud code harness so every engineer on

01:33:38 team uses like our JM52 and it will do a forward pass on the JM52 instance of the you know the node and then it will get the profile trace and it will analyze it and it will find the kernels that are the bottlenecks in Slang and then it will write the new kernels and it will do another profiling trace and when it's done it uploads the image to our thing and then we can pull that image down and repeat the cycle and so for quite a bit of time we had like literally GLM52 GM2 and and like some of the GPU kernels that were

01:34:10 on JLM52 within our inference engine is written by GLM52 >> and the trace and the kernels were guided by J52 as the driver. So it seems like I do see I do see that that circle being there. I think a bit more time is needed. There's definitely a lot of things that they can't do. The models just aren't there yet even though they're like really really smart.

01:34:27 Like they still try to like reward hack their way into like the the cheapest or like they're very like they're not good at like decision- making almost it seems. But yeah, I do like Yeah, like the model optimizing its inference is is already a thing that that happens. Do you think GLM52 was uniquely good at optimizing itself or did it just happen to be the best coding model that we had access to and it would do an equally good job of optimizing a Deep Sea Kimmy or something?

01:34:56 >> Well, to Stress's point, maybe it's going to be off policy when it tries to optimize >> really hurt Deep Sea [laughter] >> to try itself. >> Uh, no. For for what it's worth, I don't believe that. >> But it's just Let's just find out. >> Interesting. Yeah. >> Just just, you know, you have more computer than me. Just go try it. >> Yeah. Any other upcoming trends in inference engineering that we didn't cover?

01:35:16 Like right now, you know, you cuz you guys are so close to it, you can obviously see it that the rest of the world doesn't know about. >> The big ones are obvious. Models get bigger. Hardware gets more powerful. Users get used to a certain level of speed and demand a higher one. I think that some things I'm excited about are systems level. you know, we we still have a lot to think about in terms of composing multiple models together.

01:35:41 If you think about a voice agent, there's three to five models involved in that and the communication between those models. There's a lot of new modalities that are coming out. Uh there's like the cosmos, the new world model. Um there's more research. Speech to speech is still like not entirely a thing. Uh but it's it's getting it's getting closer. Uh there's going to be just a lot of new modalities to build around which is going to be exciting.

01:36:11 And then yeah, I think that the other thing to solve which is something we've been solving for a long time and and are not done with yet is is just going to be continuing to operate at another 10x another 10x another 10x scale as an industry. If you think about the degree of usage that AI has worldwide compared to, you know, some of the more mature technologies both on on consumer and business, it's it's pretty clear that there could be multiple 10xes more of demand.

01:36:44 If you look at the infrastructure work industrywide, obviously it's been stood up very very quickly to meet a unprecedented spike in demand and that is like not stopping. Um, so yeah, there's just just a lot of problems to solve around like longtail reliability and uh figuring out where we're going to get the next like 10x and 100x of tokens from. >> I'm going to say it's going to be a really boring answer, but I think the answer is just faster.

01:37:14 necks like faster network chip communications. It seems to me that like more and more memory is the bottleneck. You want to have larger models. Right now when you're doing serving at large, you have to transfer KV cache from one node to another. But the way that you do that is you find the KV cache, you find where it is, you transfer it to another node, you put it on that node's memory, and then you transfer it from that node's memory into the GPU and like like into the sensor cores of the GPU.

01:37:39 So there's like a two-stage transfer here that makes it such that you're very bottlenecked with with just KV cache transfers outlarge which affects the time of decode and pzag. You have to do this because the HPM is so much it's like extremely fast like 4.5 terabytes per second as opposed to like which is like magnitudes better than than neck communication speed.

01:37:57 If you were to somehow be able to in like this theoretical dreamland, have extremely fast next, you could in theory spare that HBM and you could just transfer KV cache trans like directly from one node to another. This would give you like almost 100x speed up when you're doing this aggregated serving between nodes and nodes. I'm not familiar with the technical challenges of making faster.

01:38:17 I'm certain there's a reason why they're like orders of magnitude smaller like like slower than like HBM, but if someone were to figure that out, it would literally be like a mag like two orders of magnitude faster to do Zcode. That would be my take. >> Cool. >> Uh I don't know if you have a nomination for things that are trends. Uh I I got one. >> Cool.

01:38:41 >> Um so I think inference engineering for continual learning. Um, so what if you just uh like if you if you just had the the idea that you are supposed to learn from everything that you ever process, do you do anything differently or or do you just have the same paradigm of like well stick it in a memory MD >> and then like it somehow gets consumed in KV cache and like this system works, it's not broken or like how do you like reshape inference so that it learns while you inference?

01:39:09 Yeah, I think maybe one relevant topic though is your absolute best friend in the entire world spoke on KB compaction >> correctly. >> Like what changes >> what changes >> if you're trying to continue learn >> there's two things and there's like um like Charlie and I had this Twitter sort of a argument where we like continual learning could take one of two paths.

01:39:29 It could either be that the model learns and so it's continuously pushing its new knowledge into its weights. In that case, you just need to have like your inference just needs to continually fetch new weights or yeah, like you just literally need to do fetch new rights and reads of weights >> or the other path which is you do KVK compaction and >> and there's a Laura layer if you just only update Loras.

01:39:53 >> Yeah, exactly. Exactly. >> Which is that's the engram approach covered. >> The argument against doing weight pushing is that you can only fix one help knowledge as in as in you can only feed it a new feature of like oh what is the best university in the world? The best university of the world is is water. But then a second derivative >> that's not changing.

01:40:09 >> That's not changing. That's not changing. But like a second derivative question of which university should I hire an intern from. So if you know that the best university in the world is water then the answer should be waterloo. But if I wasn't just oneshotting the question and I was to ask it to like use its knowledge to think and then give me a second answer or like should I hire an intern from water or MIT?

01:40:25 It would be like oh yeah both are good. But no like I lit I just edited in your knowledge base that Walter was the best. Why didn't you use that to do reasoning? So that's the fundamental problem with trying to change a fact in an MLP within the way KV cache compaction fixes that with with KV cache or like rather not KV cache compaction but like if you're able to have something like like the still paper which we we came out with which is you're able to sort of make your KV almost infinite and you're able to compact in

01:40:51 such a way that you don't lose any of the knowledge in that case you can actually do continual learning and you can actually solve continual learning and it's as a it's it's a result of um this this argument that Charlie and I that I do concede that his point was correct and I I do see that KV cache is is is the way forward and in that case I don't think inference is going to change that much because we still use KP cache and inference you're just going to update the KB cache but it's going to be like an additional

01:41:15 step but nothing changes in the weight so nothing changes in inference time nothing changes the spec okay surprisingly great answer uh we have it up on the blog it's it's a relatively recent blog so uh we can people can go see it >> uh yeah otherwise this is super enjoyable chat I know we've like already gone 2 hours. >> Oh wow. I didn't realize time flies.

01:41:35 Yeah. >> So much we didn't even cover. >> Yeah. This like we also wanted to talk about the book and all that, but you've covered the book. >> Oh yeah. I mean everyone everyone knows about the book. [laughter] >> Yeah. Uh high highest ROI thing in the history of Ban, right? For the hour. >> Without a doubt. Without a doubt. Absolutely. >> So congrats on that.

01:41:51 I you know and we've covered that in in our meetup which we can publish separately. Uh but no, thank you to you guys for being so generous and sharing. I think it's a fun conversation that um we don't get to have enough. I think in French engineering we never really covered head on and so to have you guys come on uh is is a is a treat was amazing. >> Yeah.

01:42:10 Thanks. Thanks for having us and you know hopefully in a year everything shifts and we can uh come back and say everything we were wrong about. >> Yeah. Yeah. Yeah. I'm excited for this Mega Colonel's comment to get out there. See what people say. >> We got to start stuff. [laughter] >> Should I go into hiding? I know I'm going to get like the Mega Colonel community after me.

01:42:27 >> Yeah. One thing I really respect about you is you are not willing or you are not uh scared to kick the hornets's nest ever. >> It's not I I don't think it's that that controversial. I don't know. We'll see. >> We'll see. [laughter] >> All right. Thanks, guys. Thank you so much. [music] >> [music]