Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM Transcript, AI Summary & Key Points

00:00 How much authority should an AI agent have? At UCON AI New York, learn how senior practitioners set permissions and evaluate agents in production. Use their experience to decide whether your own agents need boundaries and checks before release. Join us December 15th and 16th, 2026 at the Western Jersey City Newport. Explore the program at new york.cucon.ai.

00:29 AI. [music] >> Hello everybody. I'm Olip Pony, [music] curator and I have in front of me today Eduardo Vaki, one of the two Italians that usually talk about boxes inside of boxes. The other is is Andrea Perufu. Eduardo worked for a long period of time in the web assembly ecosystem. He did some spotting around other types of VMs and today he's back at Red Hat and he's trying to make things easier in terms of Kubernetes and inference.

01:05 Is that correct? >> Yeah. Yeah. I'm I mean on my team [laughter] it's a few people working on this and I'm just one of them. But uh fair description, we're working on distributed inference on Kubernetes specifically and that's called LLMD and it's part of the VLM family of projects. So yeah, super exciting to be back at Red Hat and work on, you know, the bleeding edge.

01:33 Yeah, that sounds nice. But before we go there, because that's the gist of your current work, tell us what happens currently in the web assembly space. Uh and I'm asking you because besides working on the Java side of the web assembly, you had exposure to was zero if I remember correctly. That's right. >> So maybe just have like a short refresher. I know that the web assembly space is moving fast.

01:57 So maybe if you can just provide a short summary where it is, what changed and what's to be expected in the upcoming period. >> Right. So for the benefit of our audience I don't know how much all of them are familiar with it but you will know that was born for web browsers but more recently and by recently now it means a few years people were also interested in running it outside the browser and for the backend specifically so running software on was and in that sense it could be seen as a sort of a similar thing to

02:35 the JVM M in in spirit and that's why I guess some of us that were working on the JVM were interested in working on was as well. So was changed fast in some senses. It it evolved quickly in some senses. It evolved just slowly in others. So there's been a lot of discussion around the component model which is this new modular system that's built around web assembly.

03:06 That is probably where a lot of the development happens these days I'd say. But I'd say that the space for web assembly on the back end has changed a lot in the last few years. I think that all of our development landscape has changed. So things have shifted a little bit. I think it's uh very meaningful and relevant today even more possibly than it was when there was much more hype because it has a lot to do with sandboxing and as you know sandboxing is all the rage these days.

03:44 One uh good scoring point for web assembly was earlier this year I was talking to the local first community >> and what was interesting was that they were mentioning that one of the libraries they built the name is eluding me now but what they said is that rather than just thinking about providing implementation in different programming languages they built it in Rust.

04:08 >> Mhm. And that meant that they can just compile it to web assembly and then use it via web assembly to different programming languages. And that was the nice part from their side and they were quite quite happy with the result and I think that's where the power of web assembly comes or I think the previous year I spoke to a couple of guys that used to work on infrastructure and they they spoke about firecracker and they what was nice is that they said that most often than not the hardware that you have in the server

04:38 racks is idle. So what they did on top of those idle space they just had very small web assembly machines well containers I think it's a better wording for that that was doing very small small things that were quite useful for other people. So I I feel that there are all those different scenarios where people are are using web assembly. Yeah, I think uh one interested as interesting aspect that connects to what you were saying right now is uh even migrating workloads somehow like running stuff in the browser but then

05:15 you also have these web assembly run times that can run even on devices or in general outside the browser and you have the same workload that runs on your browser and we have demos of things that are migrated even live migrated from the browser to an enduser device without breakage. That's really fascinating and I think these days when we talk about uh agentic workloads that's a that's an interesting space that's where I think was uh is still relevant.

05:51 I mean was mean is still relevant especially for uh browser related work and many other but these scenarios you know for the word that's that's changing a lot these days for developers those are the use cases where really could shine >> yeah as you mentioned isolation is important and is getting more and more important these days so I think that's that's quite quite useful what is bugging me for at least one week now and uh probably I'll try it the following week let's see what you you're going going to say to that I'm

06:21 just thinking to run a small language model inside the web assembly in a browser is that feasible is not feasible yeah I mean it is feasible there are examples of that the thing is but I I I've been on the receiving end of this question a few times now so it depends is the short answer web assembly is [laughter] web assembly is meant to run on the CPU first and as we So we can do inference on the CPU but uh generally it works much better and it for the way LLMs modern LLMs I mean the modern architectures for LLMs are

07:02 designed they are meant to be run with highly parallel architectures and the CPU is parallel but not as parallel as a as a GPU. Right? So long story short, smaller models which basically make sense to run in a browser for instance do make sense to run in web assembly. Much bigger workloads. Well, those in general we probably want to run on a GB on a GPU anyway.

07:36 However, there are experiments that I'm aware of to translate web assembly into GPU kernes. So yeah, possibly. But the so the short version of it is for all those workloads that are meaningful to run on a CPU. So relatively small language models was can be used and there are many versions of of these workloads that exist. I I think that hugging face for some of the smallest models will run them on your browser.

08:06 I think and there are a few examples on the wasam.io conference there were quite a few talks of people showing this running these workloads in the browser especially to do inference so yeah it does work it depends on your workload yeah well I I'm going to say it anyway it's simple it's more about just trying and seeing how things evolved and fingers crossed that something will happen and not just smoke but in order to work and consolidate the point that I stated about an expert into putting boxes into boxes.

08:47 You had a post uh I think last year or the previous year about tornado VM. I know that you just had a small exposure to it. So maybe just have a sort short summary what's tornado VM and when should we use it? >> Yeah, so tornado VM is uh we could say a flavor of the JVM or an add-on a plug into the JVM. uh depends on how you see it. Tornado VM allows you to run Java code essentially on the GPU.

09:14 This is the bottom line, the the selling point, you know, the the very short version of it. It's a little bit more nuanced than that, but yeah, basically you can uh speaking of boxes inside boxes. Yes, it's called a VM because it allows you to translate on the fly your Java code and compile it on the fly into GPU specific kernels. So, PTX for Nvidia, Spearv, whatever.

09:43 And it handles for you this translation. So, you can basically write Java code and then it will translate for GPU back end. Now the code that you write there it is Java code but you have to take some write the code in a specific way. Uh you might have to use some libraries that tornadovm provides you. You cannot run basically any code any you can you won't be able to run spring on tornadovm.

10:11 That's [laughter] the bottom. I mean yes I mean but it will run in Java mode. You won't run spring on the GPU. That's but yeah. Yeah, you can write GPU kernels using Java and that's the beauty of it. And uh the blog post you were referring to, I think it was last year's. I'm walking through the reader, one of the examples in the code, I think there is an example of writing a simple kernel and running on the GPU.

10:38 That's one thing that uh we show in the in the blog. But then we run GPU llama 3 which is an implementation of the llama 3 model that runs on GPU and this implementation is completely written in Java. So that's fascinating. The blog post walks you through how this works. In the latest DevOps Belgium uh if people wants to catch up want to catch up a little bit they demonstrated I think there was definitely a talk about tornado VM.

11:08 There were talks from the growlvm team of the similar projects to run lama and other language models using growvm that will do the acceleration for you. And then there was also former growv team member uh that spun off his own startup. I think his name is Alonso Alonso. I remember it because his nickname is Alonso squared. and he started his own startup uh uh branching off some work that he did originally in grow VM again to run code uh Java code I think uh on the GPU and uh LLM.

11:50 So yeah, fascinating stuff. And so bottom line, the reason why I started to look into this is fun of boxes inside boxes, fan of languages and compilers. So it felt like a natural evolution of the things that I did in past including compilation to web assembly to learn about compilation to the GPU. >> That sounds quite interesting and it definitely makes sense these days.

12:13 For instance, I saw that a lot of the the big companies are just using Java for inference and that makes perfect sense to just have this uh this closing the loop. Today you are part of the team at Red Hat back again at Red Hat that is working on LLMD and in my understanding what LLMD is taking a step back here VLM is making sure that everything runs perfectly on one box.

12:45 LLMD is trying to make sure that everything is efficient if you have a Kubernetes cluster. Is that a correct affirmation or where did I understand wrong? >> No, no, it is correct uh on a high level. Uh more specifically, so VLM is our inference engine. So you can think of it if you will as the JVM of language models. So it has support for several language models.

13:18 There's an interesting aside here. It it is a platform to run LLMs. The interesting aside here is you would expect that the this is a general sort of VM and the LLMs are sort of software that you load on top of it. But unfortunately every LLM architecture is slightly different. So VLM has to basically include the implementation of every LLM that gets released.

13:42 So when people release LLMs, they do not release the software to run them. They release the the weights. So all of these inference engines such as VLLM what what they do is implementing basically the the software the the kernels the the components that actually load the the the weights and then actually do the inference. So VLM is our platform to run the core and effectively VLM is able to serve it.

14:10 It's a server uh I mean it's an inference engine. It's also a server that exposes endpoints that you can call to do inference and it does so efficiently. It was originally VLM the V I think the V in VLM originally stays for virtual as in virtual memory allocation because it was the first engine to implement page attention. basically a mechanism inspired by good old systems programming to do paging of memory memory pages but for memory that gets allocated on GPU.

14:45 This is very you know in a very high level level way. So VLM is able to do high throughput inference. So you can do several requests at once and BLM will pack them and try to serve them together and do inference in one pass for several requests at one time. This is called continuous batching. But uh this gets you to a point essentially uh at some point you have your GPUs on your machine and your inference engine and that's it.

15:19 you you will not be able to scale further than the resources that you have on that machine. So how do you scale further? Well, you can scale by having more VLM instances, right? So you have several VLM instances and you want to serve uh your inference requests using these uh few VLM these few VLM instances that you have. But now you have another problem.

15:47 How do you route requests across your VLM instances? So if you are on Kubernetes what you will do and and most people are on Kubernetes these days the standard for platforms for uh serving uh services uh you know in a in a you know efficient way. So you can scale your VLM instances and have several VLM instances that serve the possible the same request.

16:11 I mean that gets uh selected by theuler by the vmuler. If if you do it the traditional way and let the Kubernetes normal router to pick across the VLM instances that you have replicated, you won't get necessarily good performance. The the reason is inference does not behave like a traditional service, a traditional HTTP service. Basically, if you send a request to one specific BLM instance, especially if you are doing conversation, if you're doing a chat and you know, if you're doing a gentic coding, it's still a chat.

16:50 Basically, there's a lot of exchanges back and forth between you and your server, uh, between your agent and your server, you want to stick for as much time as possible to the same VLM instance. This must might be counterintuitive in a way because for microservices we know we don't want to hammer the same service on and on because otherwise it will get overloaded right we want to do sort of roundrobing or random robbing across the replicas but the thing is microservices or service in general these are usually in a way

17:23 stateless uh the state is usually stored somewhere else and it's not part of your request I mean the request can be served independently by any instance. Unfortunately for LLMs, in order for LLMs to uh do their inference in an efficient way, you want to stick for as much time as possible to the same instance. But at the same time, you cannot stick forever to the same instance because at some point you will reach some limit.

17:49 You are not the only user of that BLM instance. You want to serve and distribute the load across your uh BLM instances. So there's a balance that you have to strike between staying with the same instance and benefit of what of the caches the KV caches the prefix caches that are in place these systems that allow you to uh do inference in an efficient way and lower costs but at the same time you don't want to overload the same instance too much so you also have to keep this into account this is not the job for a

18:26 traditional Kubernetes schedule and that's where LLM LMD comes into place. That's what LMD does. LMD is not just a router or a scheduler. It's actually a family of projects and they are all meant to give you a set of tools to do distributed inference so that you can leverage all of the GPUs that you have and use it at their maximum as much as possible so you get the best value the best bang for your bucks because these this hardware is expensive whether you loan it whether uh or or you own it.

19:03 So you want to use it 100% as much as possible and that's why we provide scheduleuler. We have uh something that we're going to release. I mean in the open source ecosystem it's already been released. We're going to release also as part of product redhead. This is called batch inference. Batch inference is using your cluster even when it's under load.

19:29 So you can schedule request to do batch processing in the background when your cluster is underused because basically this hardware this hardware is so expensive that you want to keep it at 100% all the time. So even when there is no live inference going on you can schedule botch processing to run when you're sleeping when there's less uh work to be done uh live.

19:53 So that's what LLMD does. It uses your hardware as much as possible so that the money you spend gets you the best value and you do it on your own hardware so you can do inference without leaking data to a third party which is I mean important these days. >> So what you're saying is that we will be able to use our previous Kubernetes clusters. though what we still have from the cloud native era because everything was native these days and just make sure that we can rebuild it to build our privacy also for the AI native

20:32 era. Yeah, Anthony and say Aldi allows you to make cloudnative inference and evolve your infrastructure in such a way that you can do AI on premises in a similar way as you did uh cloud native development before AI. Well, that was always an issue, but now it feels that we are discussing in the cloud native most of the times we spoke about stateless whenever you call something you didn't always ring on the same on the same doorbell because that was one one issue another issue was in the Kubernetes space it was the part

21:15 with IO well most particularly persistent data like database and stuff like that wherever you spoke to infra infrastructure guys they told you well we can have everything in kubernetes except for databases because then those things become more complex but let's take them one by one how do you bring in a stateless ecosystem like kubernetes worked a lot in being something that theoretically should be stateful because if I have a conversation I would like to continue my conversation about I don't know whatever stupid

21:48 thing I speak to Gemini or claude or open GPT or whatever stuff. I don't want to have uh my responses coming for what Eduardo was speaking to his LLM. >> Yeah, it's it's a complex problem for sure. Um as I said one part of the story is that well I mean if people uh people that are hearing the audience know how LLM generally work essentially the stateful nature of the LLM in a way is u is fake [laughter] in that they are really stateless.

22:24 The way they do inference is that we send them the entire history of a conversation every single time. So maybe I I jumped uh at the you know at the conclusion to yeah to the conclusion earlier and I said a lot about prefix caches and KV caches but yeah so essentially the way LLM works is that essentially you you always send the entirety of the conversation to the LLM and what really happens is the LLM completes the conversation.

22:59 I don't know if people recall how open I and GPT uh the GPT lab playground worked a few years ago. Basically, you only wrote it was the completion API. You wrote uh a sentence and then the LLM wrote the continuation of that. And the cool thing was that someone discovered if you wrote that prompt as a conversation as a dialogue with with you know with quotes uh user says da da da computer says the LLM will continue the conversation and that's how chat GPD basically was born like uh people notice this and notice this

23:38 pattern pattern now uh LLMs are being trained with this pattern but that's how basically uh originally worked so if you think of it this way every conversation really is a very long prompt to the continuation of that conversation. The way LM works, you send the history of the conversation and then they continue with the next answer and then you get back the answer and then you continue with your answer and so on so forth so on so forth.

24:07 If you are using LLMs, you will know that pricing for these things, depending if you're using tokenbased spend or uh a subscription. Well, if you're using a subscription, that doesn't really matter. But if you're using tokenbased pricing, you will know that there's a difference between cached uh responses and responses. The reason is if you don't do a cached response then you are sending the entire history of your conversation to the LLM.

24:40 The LLM does not have that conversation in cache. So it will have to do inference on the entire prompt and that's why that will be particularly expensive. But if you've been going back and forth for a while without long pauses and the cash on the server has not been evicted, then you've been reusing stored cache state to continue the conversation. That does not mean necessarily that you haven't sent the entire history to the LLM.

25:09 It just means that even though you are sending the entire history, the history is recognized as part of a prefix of a cache. And so the LLM can compute only the continuation instead of the entire prompt. So we have these mechanism for caching in place. So that basically LLMs if the cache exists can only infer the last few tokens that continue the conversation instead of have to recmp compute the entire state from the prompt.

25:43 LLMD does that for you. So it maintains the this cache both in the form of a KV cache we want to go into the details how that works and of prefix cache and both these caching mechanism allow you to persist this state across the exchanges in your conversation persist as in volatile memory because these are caches usually they're not persistent for a long time and that's the reason also why you want to stick usually to the same VLM instance even though LMD also gives you ways to move around across instances.

26:18 It's it's a lot. We cannot explain how all of that works. But in general, L & D allows you to do this allows you to have an efficient use and even for stateful conversation of your hardware. >> Okay. So, LLMD, it's also donated, if I remember correctly, to the CNC Foundation, right? >> Yeah. It recently reached 1.0 a few days ago. Yeah. >> Okay. Congratulations.

26:47 So, we are right right in time. In what state is it? Is it production ready? Should we wait for it? >> Oh, no. It's definitely production ready. There's a number of companies that are already using it. At Red Hat, we're actually partners with with Nvidia for a bunch of things. We have a lot of success stories already uh people using in production and Yeah.

27:11 Yeah. you should definitely take a look at it. In fact, people are sometimes I go to conferences and people I realize people are already using it or maybe they're using VLM on their infrastructure and they're not aware of LLMD. So, VLM allows you to do inference up to a degree in a very high performance, high quality way and the LLD and VLM projects are sister projects in a way.

27:37 I mean there's a lot of exchanges across the stack so we are part of the same team in a way but yeah LMD gives you that edge for instance BLM does not do automatic flow control even if you are only running an LLM uh an one BLM instance LMD gives you control over quality of service and uh prioritization of HTTP requests. So that is something that BLM at least uh until a few days ago.

28:07 I don't know if the things have changed recently uh could not do. So you can get uh benefit for from LLMD even if you're not necessarily running in a very highly distributed situation. >> So what does that mean? Even if you don't have a cluster, use LLMD. Even if you don't have a lot of GPUs, uh even if you uh don't have to distribute the load across different GPUs, LMD might still be useful.

28:37 And yes, uh you can also run it bare metal. So you might want to evaluate LMD even if you're not uh using Kubernetes. I mean Kubernetes is the first target, but uh it is supported also on bare metal. >> Okay. So it's focusing on um on inference even if you don't have already existing infrastructure. So it just tries to provide the best usage of your resources even if you have or you don't have um an infrastructure already existing.

29:12 It seems that uh at the point when we are recording this it's end of October and it seems that LLMD started at the end of October in 2024. So it's a good it's a good moment to have the first uh major version out in this period. So congratulations. >> Yeah. >> But staying in the in the Kubernetes space. One of the reasons why people prefer to use things that they are already familiar Kubernetes is not that old of a solution comparing to other types of solution that were out there but it's still one of the most mature in

29:51 the cloud native space. uh and one of the things that are very important these days or they always were important but now they are more than ever it's observability what should we expect from that perspective in terms of observability when we're discussing about LMD I mean as you mentioned everything related to LMG is more complex than everything else how can we make that complexity easier to digest through the observability solution already existing >> yeah so I mean it depends it's a it's a it's a very wide question.

30:21 So the way L & D actually works, it leverages a lot of signals that are metrics based. So there's a lot of insights that are being exposed by BLM metrics and data that explains essentially what's going on inside the uh inference engine and LLMD uses this information to actually do what I explained earlier that is this inform the routing across instances.

30:54 What this means is essentially when you use L & D and VLM you don't just have a system to do efficient inference. You also are getting for free if you will a rich system to get insight on what's going on not just on your cluster but in your inference engines. You can see how many tokens are being produced, how many failures you're getting, what's the throughput.

31:17 uh you can get alert on situation where you might want to intervene. For instance, you defin you have control over the saturation of your cluster which means saturation means essentially you have so many requests that you are struggling to serve them and then at that point you can add more machines you can throttle the incoming requests. So you have a lot of knobs that you can work on.

31:42 The observability story for LMG is uh pretty interesting as well. And you have all the tools to get uh insights of what's going on in your cluster from a GPU and inference perspective as well. >> Well, you already mentioned that it's it was complicated to get it right, but which thing was the hardest? >> Huh, [laughter] that's an interesting question.

32:07 Yes. um everything well um one one thing that is interesting in this ecosystem is that uh it is everything is very new. I would say one of the things that are challenging maybe not the hardest is that well Nvidia is probably the biggest most important vendor but there's also AMD there's Intel there's Google's TPUs there and there are more there are also a lot of startups that are building their own hardware and while for the PC we benefit for a lot of time now of a standardized set of hardware components with a very uh

32:51 well- definfined instruction set for these accelerators and uh GPUs the languages are very different. So one of the challenging part for VLM itself and inference in general is that uh and and the reason why we need a framework on top of that is that you can't you you can't write the kernel for a GPU computation once and target every single accelerator there is because everything is different.

33:17 is like having uh Intel but also ARM but also risk five but also power PC and you have to use all of them depending on your vendors and depend and you have to support all of them. I mean in in our case you want to support as many as possible. So that's probably one of the most challenging things that happen in this in this case. The other thing that is also possibly challenging from a hardware perspective is that there's a lot of data that has to be transferred like GPUs.

33:47 LLMs are large language models and we hear about trillions of weights of parameters. This translates in gigabytes, terabytes, pabytes of data and this data have to be moved across machines. You don't just have expensive GPUs but you also have to have expensive connection between the GPUs. So we're talking about RDMA. So you have to connect this this hardware with very expensive type of connection that allow you to have very fast data transfer.

34:22 So I I would say in general from a from an artwork perspective these are these are things that are very challenging and at the same time it makes this space very fascinating and uh interesting. >> Yeah that that's for sure. But one you did mention that the beach of possibilities is quite wide when we're discussing about GPUs >> which were intended for something else but now people like usual they adapted and they used it in a different perspective.

34:52 >> Yeah. But now we can see that there are companies that they started building their own hardware. We are discussing about TPUs, >> right? >> How will that change the the game? >> Yeah, that's an interesting question. In a in a I don't know. [laughter] Hope hopefully it might um make things more affordable. I mean competition is always good like like the the one side is that this makes things challenging for people that are building infrastructure and software on top of these things.

35:22 It makes things challenging for people that want to source this hardware because it's still very expensive. Uh but at the same time if there is more competition hopefully prices eventually will go down. I think one thing that is very important and I think this is also the message that we want to bring across with LLMD VLM. We're getting to a point where first of all we you don't really need extremely expensive hardware to do inference to a certain degree of quality.

35:56 Obviously, frontier models will be always be more capable than older models, but um open models are getting better and better, even relatively small models that can run on consumer hardware. Like I have a relatively beefy Mac myself with 48 GB of RAM. This is a company Mac, so I didn't pay for it. and it was probably bought before the the price rises.

36:24 So, it was a good time. But it's a relatively small amount of uh shared memory because it's shared across CPU and GPU. That's the point. And I can run 27 bill I think 27 billion, which is not huge, but it's pretty capable. And you people have talked recently about quen 3.8 27b. Uh this does not run very well on my laptop but uh it runs on laptops still and it runs on my laptop even though I super it's not super fast.

36:56 So if you get a server that is only slightly bigger than my laptop you will be able to serve a lot of requests in a very efficient way. And this model has been compared to Opus 4.6 and Opus 4.6 is only one year old and we were super excited and impressed by OPUS 4.6 last year. So, let's not get too carried away by all the announcements we read on social media about how all of these models are super capable and the the old models are no longer useful.

37:29 The old model are still useful and they're even more useful if you can run them on your hardware because this is very important. You don't have to share your data with a company that you don't know you can really trust after all. And this is I think the message we want to bring across. You can own your own AI. You can own your own token inference on hardware that now it's relatively relatively is the word here.

37:53 Yeah. Affordable. And and this is especially true for companies. I'm not talking about consumer. This is true for consumer to some extent, but it's very true for companies. They can source the sort of hardware. They can do inference for these models. They're not as big as the frontier, but they're definitely quite something and they can solve a lot of interesting problems already.

38:17 >> Yeah. What I think it's important for people to understand is that do we actually need frontier models for everything? >> Exactly. >> Because in the end, as you mentioned, it's becoming tangible that you can run them on even laptops. >> Yeah. >> I was mentioning it's something quite basic. it's ordering I don't know something and then you don't actually need anything else but details in terms of human understanding and then a very small model it's more than enough but it's very niched >> right >> and then that's

38:51 that's the thing is that yes we do need a perspective where we do that stuff and we need I don't know the argon the opuses and all the other frontier models out there but now a lot of people are doing experimentation and at the point where you're getting into the production space you have to do the refining you have to do the efficiency part and then definitely you get to the right uh on the right part and I saw something similar even on the edge side where they had very tailored models that were just being focused on

39:25 a very thin narrow use case and that's it but now we are just a lot of us are exploring and we are measur measuring stuff that actually don't matter >> for sure I I think Another interesting aspect, this is another project that we are incubating an LLMD. It's called LLMD semantic classifier. It deals with routing not just efficiently but to the right model for some definition of right.

39:52 So right might be cost based or it might be topic based. And so this this uh system there's another sister project that's called the VLM semantic router if you're interested LND semantic classifier is sort of a sibling if you will uh with different targets but the goal is the same. You have a prompt you have a conversation going and you want to route this conversation to the right model which might mean right sized and that's exactly what you were saying.

40:20 So it doesn't really matter sometimes to have a very large model that is able to answer every possible question. It might be useful in many situation of course but you have also to bring to the table the the cost efficiency. So you might have specialized models that are trained or fine-tuned on specific domains. And I think that's where we should go, where the next thing should be because that really brings LLM inference, AI in general, agentic stuff to the people.

40:58 [laughter] It allows you to run this stuff on your hardware with things that are attainable, goals that are attainable, not just super expensive hardware that nobody can afford. I think we are at that let's say teenage years of uh of what >> LLNS and AI will mean for the future because everybody is priding themselves on how much horsepowers the our models have but actually we are stuck in traffic >> hours and we it doesn't actually matter if it's there or not.

41:33 >> Yeah, teenage years is a good metaphor I'd say. Maybe just to close up with what's on your road map. So version one is out. What's in the next period? So upcoming months, upcoming years. What's the target for LLMD? >> Yeah, I mentioned a few already actually. So uh LLMD itself uh is able to run you know on this uh this hardware they use it efficiently.

41:56 It's able to run very large models, mixture of expert models by sharing them across machines. But things that we are experimenting and we are getting more and more refined are this batch inference concept. So you can leverage your hardware more efficiently and use those dead moments in your traffic to serve and uh do inference even scheduled inference uh without the live uh interaction.

42:22 So you can 100% reuse that hardware that you have. And the other is obviously as I just mentioned semantic classification so that you can not just use the right machines but also use the right model at the right time. For instance, let's say you have a very big model that is expensive to run and smaller models that are more efficient to run and you can run on on hardware that is much cheaper.

42:48 Depending on the conversation that you're having on the prompt that you're sending, you might want to use the larger model in some situations or the smaller model. some situation. So these are the things that we are working on currently. There are more and >> [clears throat] >> uh I cannot even list all of them but it's an exciting space and I think you're going to see a lot of interesting stuff in the next coming [music] months. >> Okay. Thank you for your time. >> Thank you. [music] >> [music]