← All transcripts

What Runs Where? Local AI in Production — Ahmad Osman, Osmantic Transcript, AI Summary & Key Points

AI Engineer · yesterday · Science & Technology · 56:09 · EN

Watch on YouTube

Answer

What runs where depends on the workload: most agent tasks are bottlenecked by execution time rather than tokens per second, so small specialized models on modest hardware (even one RTX 3090) with per-chip inference engines and a complete harness — including search — cover most production needs, while continual learning ultimately requires local machines you control.

AI Summary

Local AI in production is a stack-choice problem: the right model, the right hardware, the right inference engine — optimized per model, per chip and per workflow. Open source AI is an optimization problem and local AI is a user experience problem, and both must be solved in parallel. Benchmarks are deeply unreliable — Artificial Analysis Index v5 ranks Haiku 5.5 ahead of DeepSeek V4.1 Flash, while HLE Diamond puts the open-vs-closed gap above 50% and Terminal Bench says open source leads by two points — so metrics must be measured on your own hardware against your own task traces. Since February 2026 agents, not humans, generate most tokens, and agent workloads are bottlenecked by execution time rather than tokens per second, which makes even a single RTX 3090 genuinely useful. Small and specialized models are the future: a 1-billion-parameter model will beat Opus 5.5 within three years, Airbnb reached frontier-quality on a 2-billion-parameter model at 1% of the cost, and Qwen 3.6's 27B dense model beat a 15x-larger MoE a month after its predecessor could not. Quantization has matured (GLM 5.3 at 4 bits retains ~94% accuracy vs 76.1% for DeepSeek 3.1 a year earlier), DeepSeek V4.1 Flash shrank a KV cache 437x, and better scheduling in the Flash Data paper nearly tripled tokens per minute. Below a usage threshold GPUs should be rented, not bought — but rented infrastructure must be verified, because cheap providers can quietly route to a different model. Continual learning is the next big unlock, and it can only happen locally: engrams plus decision models plus weight edits is the recipe, because cloud providers train for their average user, not for you.

Key Points

  • Open source AI is an optimization problem and local AI is a user experience problem, and both need to be solved in parallel — the field is in the '90s of Linux, with mega kernels and new inference engines arriving as more people invest in the ecosystem.
  • Benchmarks: Artificial Analysis Index v5 ranks Haiku 5.5 ahead of DeepSeek V4.1 Flash, which signals a broken eval; HLE Diamond puts the open-closed gap above 50% while Terminal Bench says open source models are ahead by two points.
  • OpenRouter data (October 2026): agents have produced more tokens than humans since February 2026 — 'it's always going to be agents producing tokens for agents' — and open-source models (led by DeepSeek V4.1 Flash) dominate tokens consumed while closed models (Opus 5.5) dominate money spent.
  • Artificial Analysis Intelligence Index v4.1: open-source models jumped from 13 to 51 points since January 2025, with the gap to closed models shrinking.
  • Agent workloads: you spend most of the time waiting on an agent to run a shell command and get results back, not on tokens per second — which is why a single RTX 3090 delivers a lot of value.
  • Specialized models: a March 2025 tweet predicted 'the future is small and specialized models'; within 3 years or less a 1-billion-parameter model will beat Opus 5.5, which is itself a small specialized model compared to Astra or Fable.
  • Agent-made trailer: Claude Opus 5.5 built a talk trailer end to end — quotes, clips, music, text overlays — in 6 hours, using two RTX Pro 6000s for tagging photos and scaling video/audio while a MacBook handled editing and A/B winner picking.
  • Portal harness: Osmantic's local agent harness (codename Cairo, cloud emoji) embodies the argument that 'the cloud is someone else's basement' — every model runs on someone's hardware, so frontier-vs-local is a benchmark question, not a given.

AI in practice

Used for

Agents

  • Portal (Cairo) — Serve as the local agent harness used to build the RTX Spark game and run agent workloads fully locally with search built in. 2 held 12:55
  • Use local GPUs as parallel agent workers for token-heavy agent workloads. 2 held 25:00

Tools & resources

1 item

ONo. 3145
AIAINotes.us AI product

ODS (Osmantic Deployment System)

Open source · Osmantic/ODS

ODS is a self-hosted local AI server stack developed by Osmantic for Linux, Windows with WSL2 and Docker Desktop, and Apple Silicon macOS. Its installers detect the host hardware, select an installable model, generate credentials, and launch a preconfigured stack containing local LLM inference, Open WebUI, a management dashboard, LiteLLM, embeddings, speech-to-text, text-to-speech, agents, n8n workflows, Qdrant-based RAG, search, ComfyUI image generation, and operational and privacy tools. Local mode is the default; cloud and hybrid modes can route through OpenAI, Anthropic, or Together APIs. The stack uses a bootstrap model so chat can begin while the hardware-selected full model downloads in the background. A manifest-based extension system discovers service folders and merges their Docker Compose definitions into the stack, while the `ods` CLI manages services, models, modes, configuration, presets, and lifecycle operations. The repository documents GPU and unified-memory paths for NVIDIA, AMD, Intel Arc, and Apple Silicon hardware, along with model selection, recovery, validation, and operator workflows. ODS is distributed as an Apache-2.0-licensed open-source repository and can be installed with shell or PowerShell installers. The repository identifies version 2.6.0 as its stable release and states that no cloud account or subscription is required for local operation.

Mentioned in
2 videos
Kind
AI

Links mentioned

🔒 Full analysis locked

Unlock more videos and the full analysis

A credit unlocks one video's full analysis for good — the build steps, the tools and how each was used, the methods behind every use case. Pro opens the whole library instead, and raises how many videos you can analyse a day.

Unlock full analysis — free

Transcript

Searchable transcript of What Runs Where? Local AI in Production — Ahmad Osman, Osmantic — AI Engineer (56:09). Search for a phrase, then click its timestamp to jump straight to that moment in the video.

Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.

00:12 Just going to give everyone a second to settle in. So, I have 170 slides for you. I believe that we're not going to be able to go through all of them, but they will be posted online very soon. Um but yeah, this is basically uh what what runs where and uh local AI in production. And the goal is to go through the history of local AI. Uh you know, probably we're going to go back a bit to 2023 and where we're heading in the future.

00:47 Uh and of course talking about where we are today. So uh in June at AIE World's fair uh we hosted the local AAI summit with NVIDIA and back then I said that we are basically in the '9s of Linux. Uh and I uh I think we're starting to finally see more and more people invest in the ecosystem and the infrastructure. Uh since then we have been seeing more me mega kernels or what people call mega kernels.

01:17 We've been seeing new inference engines. uh we're seeing a lot of people trying to optimize basically uh per model, per chip, per workflow and that's the way to go. Uh so yeah, open source AI is an optimization problem. Uh local AI is a user experience problem and both of those things need to be solved in parallel to each other. >> [clears throat and cough] >> This is uh this is from our recent launch uh co- launch with Nvidia.

01:48 We built an optimized inference engine for the RTX Spark. And this game right here, this was built on an RTX Spark on Windows. Um so um you can play this game on ospark.com and you can see all the traces. This was built fully locally. No cloud, no CX, our own portal. uh uh harness. Everything was built um locally basically. [clears throat] And uh this is from the Best Buy launch was at like you know uh official launch uh for the RTX Park.

02:24 Uh we had so many people around us that wanted to play this game. So we're starting to see more of everybody like you know everyday person actually showing up for local AI and interested in what's being done in here. Um, can you tell the difference between these two answers? One of them was a local model and another was GBT 5.5. This is from our uh uh video online with Hugging Face.

02:53 It's Locally AI 2011 and uh we ran some demos back then uh on a bunch of hardware and um one of them was faster than the other. The answers were pretty similar and you can see here that um the model that we ran locally was qu 3.6 6 27 billion parameter and it ran at 51 tokens per second for decoding uh while um GBT 5.5 via OpenAI cloud was running at 34 tokens per second and I don't know if you guys are using Codex CLI at at this point but it's pretty slow uh in comparison to anything local.

03:33 Um this one is an interesting one. Anyone saw the new artificial analysis index v5? Uh they have that uh anthropics haiku 5.5 is ahead of deepseek v4.1 flash. And uh that's a very interesting benchmark in my opinion. If your evals are showing that haiko is beating deepseek v4.1 flash there's a problem there. Now, um this one is an interesting one. It's showing the progress since March 2023 and how long um long horizon tasks uh are becoming longer in terms of what humans could do that machines could solve right now.

04:26 And we're seeing that basically it's doubling every four months. And that rate is not slowing down. We're also seeing something else and more interesting which is this is from open router data uh October 2026 that uh the number of tokens are in trillions and since February agents are the ones producing tokens more than humans. The last time a human was the one generating more tokens out there was February 2026.

05:01 And I guess we're never going to get there again. It's always going to be agents producing tokens for agents. This is artificial intelligence uh artificial analysis intelligence index v4.1. And it's showing that the open-source uh models since um yep January 2025 basically have jumped up from 13 points to 51 on the index and the gap is becoming smaller and smaller.

05:30 Yet again um it it it raises the question why a few slides back we're seeing that high 5.5 is the one beating DC v4.1 flash. So which benchmark to trust is a good question to ask nowadays. Um [clears throat] so three things this is um one of those tweets are from um oh I bought one in the same place twice. Um my but I have a tweet since uh March 2025 saying the future is small and specialized models and um I believe that in 3 years or less we're going to have a 1 billion parameter model that beats obus 5.5.

06:28 Okay, this is an interesting one. This is from Owen, right? Um, amount of money being spent on models right now mostly closed rates. However, the number of tokens being consumed out there are mostly open source models and um you know this is the leather board from opener October 2026 as well and this is showing that um pretty much in terms of tokens everybody's using deepseek v4.1 flash and Jeff for most tasks but in terms of money spent it's obus 5.5 five.

07:11 So what is what are agents doing right now and why do we think that there is a need for new chips to be engineered out there um for like you know or new paradigms for software that run faster more asynchronous uh it's because most of the tasks are being done right now you're mostly waiting on an agent to run a shell command or do something execute something and get the results pack more than you are uh bottlenecked by tokens per second actually.

07:40 And there is a reason that even just a single RTX3090 would give you a lot of good information like a good a lot of good use because you're still going to be bottlenecked more by processing time. So [clears throat] the gap between open and closed models um depends mostly on which benchmark you trust. Um according to HLE diamond the gap is more than 50%.

08:10 Uh terminal bench says that open source models are ahead by two points. So we have an eval problem that's for sure. And u I don't know how it's going to be solved. Honestly there is a reason that sometimes I'm just like okay Google released a new model. Do I care about the benchmarks? Not really. Like when is the last time I used the Gemini? It's Gemini Pro 2.5 the March 25th edition.

08:37 That's the one that I last lied from them. Since then it feels like it's benchmarks basically. Uh this is an interesting research came out in April and it's from a security vendor. um pretty much was able to reduce the same um security gaps being caught by small specialized open models. So this tells you something about the idea of like small models.

09:02 Anyone um anyone realizes right now why like OBS 5.5 uh is cheaper than fable 5.5.2 or u while it's more capable or it's not more capable in every task. It's more capable for the tasks that most people care about. It's RL to, you know, to an extreme extent for coding, but I don't trust it when it comes to problem solving. I trust it to execute and write really good code.

09:32 Uh less slop as you would say, but it's not the smartest model out there right now. So, um you know, the the the the reason I first Okay, how many of you know me? Like how many of you actually follow me on X or like Okay, so only a few of you. Uh, okay. So, my name is Ahmed Osman, by the way. U, let me introduce myself. Sorry, I came in here and I just started talking.

09:59 [laughter] Uh, I'm the CEO and founder of a company called Osmantic. U, my co-founder is parking his car right now, so he's not here, but he would laugh uh when I say that he he says that I'm very humble. He just came in. Um, so I didn't just call this company Osmanthic uh because it's my last name. Um, somebody in the DMs came up and asked me uh if my last name is short for open source man and I decided that okay, I'm going to take that and run with it.

10:26 So, we created um and you know the thesis that I built I'm building this company around is that you don't need frontier intelligence at every step. A lot of what we're doing could be done with small and specialized models. I even say that obus 5.5 in comparison to Astra or Fable is a small and specialized model right now. So again 170 slides or slide uh 22.

10:50 So if you find me skiping some I will share them online. Um so uh anyone saw the trailer that I created for this uh for this talk? It's this one right here. This was created with uh code obus 5.5 end to end the quotes found the clips the music how it's showing the text how it's putting all of that overlay all of this was from an AI model doing this work on its own.

11:27 I pointed it at a bunch of videos of mine and a bunch of articles and it just captured everything. I had to select a bunch of like you know use this don't use that but besides that it orchestrated this entire thing however it wasn't able to build this entire thing without let's go back to this without two RTX pros 6000s that it used for uh pretty much tagging photos uh of scaling videos of scaling audio uh my MacBook used like you know handled the edit the the editing and uh picking [snorts] AB winners, basically AB

12:03 testing for what do I like and what I what do I not like. But pretty much it was straightforward. This took me 6 hours. I literally started working on this on like 900 p.m. and I was done by 3:00 a.m. I was like, "Oh, wow." Like, I could do stuff that you used to need to hire someone to professionally do with just an agent. And I bet you that we're going to have a small and specialized model doing this within three years from now, just at home.

12:34 So um the cloud is pretty much someone's else's basement. Uh I argue that with everyone every model that we use whether it's Astra or Fable 5.2 or Quen or whatever it's someone's basement. So we need to make peace with that. Um and also um you know we started we created this harness called portal and I decided to give it uh the cloud as like an emoji because well you have the cloud at home now so like start using local I more and uh code name for this thing is Cairo.

13:07 I'm from Cairo Egypt so you know I like to throw in stuff in there that are like you know uh trivia. So local is basically the machine uh that you have under your desk uh or someone's AI it doesn't matter someone's data center it's all local it's local to someone um so the idea of like you know you have to use the cloud because it's frontier intelligence frontier based on what based on which benchmark that I just showed you massive gaps in and for which tasks specifically are you calling it frontier some math uh for uh

13:44 people like me that like talk about four bits and stuff like that, but we're going to skip this one. Um yeah, uh long context still we haven't uh we haven't found a way to make that work. Memory is still a massive issue and uh there is an interesting slide coming up. Let's skip this. Oh yeah. Uh this is for my basement. This is uh a 14 RTX 3090s behind me right there.

14:20 Uh this thing was barely able to hold Deep Seek R1 uh which came out in February 2025. I had a live stream 6 hours live stream uh trying to run it on my DDR4 RAMs and I was it was incredible. It was like right after 01 and finally Deepseek R1 came out 671 billion parameter model and I'm like oh it's reasoning. Yeah, it's giving me like eight tokens per second on my RAM and like once it hit like 16,000 tokens it was so slow but I'm seeing some reasoning in this thing locally on my machine.

14:54 So that was like a very good moment like imagine that like just four months after this thing came out um GBT OSS came out. It's like 120 billion parameter model and it was like one of its size and it was doing a lot more. It was way more capable. So like the trend of how fast open source models are catching up, it's incredible. [snorts] Um yeah um actually this is one of those things that like when I tell people that I built this like March or February 2025, people are like really?

15:25 So like even like with GS3 the full brilliant parameter model I was able to like build a screenshot organizer at home running on an RTX uh 1490. Like you models were already so capable but nobody wanted to put a little bit of systems engineering behind it. Um let's let's skip all of the techy techy stuff. Um I'm trying to focus on well let's not skip it completely.

15:55 Uh we know that we're getting a lot of good quantizations. We're seeing a lot more stuff around NVI4. We're seeing more from Nvidia when it comes to um proper implementations. They finally hired a bunch of people that released a new uh inference engine specifically for the RTX Pro 6000. So we're seeing some progress in there. We know that Blackwell RTX Pro 6000 Blackwell is different from the Blackwell of the B300 and B200.

16:20 Um so we're seeing some work being done in terms of optimizing. software for inference at home or like you know bronze proumer not necessarily consumer hardware uh but also there's a lot more to it. There's uh stuff like uh uh quant like um quanti quantization aware training uh which basically is making the model reassess itself as it's training so that you know the most used layers in the model are being kept at higher precision so that the least used are actually the ones being compressed down.

16:56 um this kind of stuff are very important and this is what like people like Daniel and Mike all from ensl basically built the dynamic quantization based on so bunch of benchmarks um the most important thing yep here I mentioned the dynamic quantization and there are two different ones by the way there's ones that happen at runtime and there's ones that you basically run a bunch of benchmarks and assess which layers get used the least and then decide Okay, this one I'm going to keep it at 16.

17:28 This one I'm going to keep it at eight. This one right here, it's useless. I'm going it's like barely being used 0.01%. So, I'm going to b it at one bit. Right? That's how you compress down a model and quantize it. There are so many different ways to do this. Just botting everything at four bit is probably going to nuke the model. So, don't just say, hey, it's all the same.

17:47 Quantizations are not the same. Like, you actually have to be pay attention to which layer is losing what exactly in terms of precision. So yeah, this is an interesting one. Um basically the accuracy in terms of uh being able to like for models again um how quantization have evolved we're seeing that something like GLM 5.3 at forth is able to recall um about 94% of its accuracy while a year ago deepc 3.1 was only able to do 76.1 precision recall right so that shows that the more we optimize these quantization

18:31 algorithms, the better we are in terms of accuracy, the more that we can compress these things and make them work on smaller uh machines or like you know less VRAM basically or less memory and get a lot more accuracy out of them. Oh yeah. So yeah. Um so basically um how many of you here know the difference between like decode and prefill? Okay. So prefill as we know it is compute bound.

19:07 Decode is pretty much memory bandwidth bound. And uh that means that if you have a long prompt and you're starting off with that, the time to first token is going to take more. If you're starting with a shorter prompt, you're probably going to get more tokens faster. Um so these are things that most people do not think about when they are buying hardware.

19:32 So yeah, um you have basically different kind of bottlenecks depending on your use case, right? So uh longer context, right? um means that when you're building the KV cache, you're more compute bound, right? Uh patching, which kind of hardware do you need? And that's that's why I'm I'm I'm bringing this up because what you use in terms of hardware and what you buy depends on the workflow that you're going to be running and the needs uh of your use case.

20:01 Not just I'm going to buy any hardware out there. um [snorts] like the idea of a dense model running on a DGX spark when we know that you need to run the entire like you know when 3.5 27 billion parameter model needs to run every token by the 27 billion parameter parameters that's going to be very slow in comparison to running ane on a DJX park. Yeah.

20:28 And uh how many of you here have like an iPhone? Come on guys. Like if you're not feel free don't raise your hand. you're you must be lying about other stuff as well. So, uh right now an iPhone is basically more capable um than running intelligence and capabilities than a model that used to require a data center 3 years ago, GBT40. So, right now, if you use Quinc 3.5 on your phone, it's more capable than that.

20:54 So, that tells you something about where hardware is going and where model optimizations are going. actually we're able to extract a lot more intelligence from much smaller models and that's again my bet on why I'm saying smaller and specialized models one and two billion parameter models are going to be the norm. We're going to have a bunch of orchestration around them and it's probably going to be another model that's doing the orchestration so that we don't even have to bother with it.

21:23 Yeah, this is an interesting one. Um the M5 uh Ultra is supposed to have something like 1.2 2 terabytes per second bandwidth. Uh, and the DJX Spark has 273 uh, gigabytes per second for decoding. Somehow two DJX sparks still beat the M5. And this is where I say we need a lot of software optimizations. Max Abel does not have any good software optimizations for AI at this point.

21:52 The software is really lacking. Koda Nvidia is way ahead. And there's a reason when somebody's asking me if they should be like what hardware to buy and I'm like how much of your time do you want to like spend trying to get something working and optimizing for it? Because if you have a lot of time, go buy an Intel GPU. But if you want something to actually work and you're able to work with an agent to improve it, it's probably going to be Invidian Koda.

22:25 Yeah. Uh we had this thing on with hug and face. We like uh did like a YouTube video with them talking about local AI and we ran a bunch of different hardware, right? Uh we ran um an RTX Pro 6000 with CR 3.6 27 billion parameter. We ran the same model actually in all of them. Uh M5 uh MacBook Pro with an M5 Max chip. Uh a DJX Spark and Stress Halo. And um you can see the different bandwidth that they have and this is the video from that thing the race where they start generating tokens and you can see the GPU beating

23:07 all of them. So in terms of tokens per second, we seeing a lot more out on RTX Pro 6000 because it's memory bandwidth is a lot higher. But not just that, when it comes to batching and running parallel uh parallel like uh requests, we also see that it's right here. 50 requests per machine at once. We're seeing that RTX Pro 6000 becomes almost 1300 tokens per second while everything else is sequential.

23:51 And this is a software problem by the way. Somebody should build an inference engine for these different machines. And please open a BR into ODS for that specific uh machine for that specific uh platform and we will make it the default for the models that you have optimized for. Um, we need a lot more um people contributing in the open source community with their hardware optimizing for the models that they are using right now.

24:24 That's how we all get better together because otherwise and the idea of everybody contributing to that lama.cbv doesn't work because you're giving away basically the like you you're using a bunch of tricks to just uh uh work for everything, right? you're you support MPUs, you support Raspberry Pies. So, you're taking a lot of shortcuts and you're not able to optimize for the hardware platform.

24:49 That's why like I believe that an inference engine per model per chip is what we need moving forward. So, yeah, again you can see the number over there. So this shows just how much like when I was telling people go buy as you view it's for a reason because if we're using agents and again I was showing that agents since February are what use tokens the most.

25:11 Uh then they are just going to be spinning off other agents and when you're running them in parallel it makes sense to get a GPU rather than a MacBook or an M5 Ultra Studio. Unless of course you want to build an inference engine optimized for it then please again open up BR uh and a so these are just a bunch of numbers you can read them online. Um this is about batching.

25:50 I'm just trying to get through the more important stuff. So again, different engines, different quantizations, different file formats, different optimizations as we quantize these things. Uh ultimately, we're going to have a bunch of these coming up over the next few years. Uh we've created our own ODS weights and we're just like, you know, everybody's going to optimize differently.

26:11 And I don't think that right now it's necessary to just try to like, you know, find the best standard. It's more important to find all the optimizations. most probably everything that's being built for the next year, year and a half is going to be thrown in the trash and that's fine. It just it's more important to actually understand what kind of optimizations should you be building for.

26:34 So if you're building if if you're trying to choose an inference engine, some questions for you. I would say um it's better to just try to optimize one for yourself and like just based on your workflows, but that takes time. I'm assuming most of you here are engineers or people that are actually interested in this space. Um, so start playing with tiny kernels like just get colle.5 like 0.8 billion parameter kernel and see how it works out.

27:06 This is an interesting one. Um this is from open router and it's basically showing the benchmarks of GLM 5.3 and it's showing it on different hardware. This is this is where you can see that an inference engine actually matters. Some of these are clos source some of these are not even available to the public and there are different chips. So we at the bottom we have 10 storing down there, right?

27:31 Uh so this is what I mean when I say it's like per chip or per model optimization because it's different from one platform to the next. The instruction sets are not the same. What you can do with the hardware, what you can extract, unpack, back up and like do calculations on doesn't work the same. Uh so yeah time to first token token per second per user and again per agent I would say moving forward is something you need to optimize for uh request per second and basically task accuracy and all of these things need to

28:10 be measured based on your own hardware. You can't just be saying hey uh I'm just going to trust the numbers on being posted online or I'm going to just follow this recipe and it's going to work great because most probably it's not. Um this is a very interesting uh one. This is actually came out just last months from a company called actual inc. Uh and um they built a ra uh tokenizer that was I don't know I can't remember if it was ra or uh if it was written in assembly but it was um basically like 100 times faster but

28:45 that 100 times faster when it came to the whole request it made it 1.1% faster. And that's still great by the way. Like that doesn't mean that it's like you know what they built was not an excellent piece of software. That just means that everything that we're using in these like you know these LLM pipelines it's system engineering and you need to optimize every part of that pipeline so that you can actually extract more value out of it because the hardware is burning and like you know it's being used without you

29:13 getting the most optimal output from it. So, I'm going to skip this one. I'm trying to get, by the way, to uh continual learning. So, that's why I'm skipping a bunch of things here. Um, we saw a bunch of other levers uh over the summer. We started seeing the flash, a lot of people working with speculative decoding, optimizing for AMD, for Nvidia hardware, for different models.

29:41 Uh, we saw a bunch of like lama.cb forks and a lot of people just started with forking lama.cb, CBB, right? And just adding D flash support or DD3 support um something that just came out yesterday and I'm trying to get to it. This this paper came out November 9th and it was impressive to me. Uh it's basically called flash data and uh I still need to so like what I'm saying right now is just me skimming through it and trying to add it to these slides.

30:15 So, I might be wrong. So, don't hold me accountable on this until I do more in-depth research. And if there's a correction needed, I'm going to post it online. Uh, so I'm saying that as a disclaimer. Uh, but what they are doing is basically uh saving where each turns like ends and instead of just letting VLM guess and it's pretty much better scheduling and this better scheduling is leading to what?

30:43 It's leading to referral tokens per minute being almost tripled. um on different models it's between 3.14 times to 1.5 like two a time and a half that still is good performance gains right um so when I was saying back in June that we need a lot of more people actually optimizing in the space and trying to come up with like you know these uh hacks to make things go faster this this is what I was talking about and we're finally seeing a lot more people do this uh people are not saying okay I'm going to use lama.cv CV or

31:21 lama. Thank god uh they are actually doing more than that. Um and what I'm building ODS with in mind is that we are going to be using the best inference engine out there for the model of your choice. Uh when you decide to use that model, we're not just sticking to lambda.cb or VLM. Whatever is out there that is best, we're going to be using that for that platform.

31:41 That's how we make the user experience better. As I was saying, open source AI is an optimization problem. Local AI is a user experience problem and most people don't want to go read my articles on inference engines. You want to actually just use something that works or otherwise you're going to go to chbt.com. So we're trying to solve that. Um [clears throat] enagram table tables are something that uh deepseek released this year uh I think January uh or February and uh we are so lucky to have Deep Seek by the way uh

32:24 they have released so many great uh like research and like ideas for free for us to use and uh they were breakthroughs that actually like I doubt any uh of the closed models companies uh would share these kind of findings or research with us. Um Strata is doing a lot of good stuff. Um Google it. It just came out now. Yeah, this is an interesting one back in March.

32:59 uh a friend bought a 5090 after Quinn 3.5 uh 27 UB came out and he tried to he hooked it up to close code and told it to like uh change the color of the RGB on the GPU right on the GB fans and it wasn't able to do that and he came back and he's like hey like Obus was able to do this thing but locomotives did not and I was like did you give it a search engine uh and he's like what do you mean I'm like okay so like quote code comes with a search engine All right.

33:30 Uh, so it's able to go look on the internet for the necessary data about the tools that came out after the model was trained to figure out how to change the colors of the fast. So I opened my firewall ports. I gave him an AI. Told him to set this environment variable. Told him exactly how to run the command. Ask it the same thing and was a search engine.

33:50 Surprise, it was able to actually change the RGB colors. It's like wow we're kind of like you know we shouldn't be asking a model of tasks without internet without giving a search access if it was trained on data that didn't have the tool that you're using right now or even was trained on data before this GPU that you're buying and you're trying to work on came out.

34:14 Um so you know this is an infrastructure problem and this is a user experience problem in local AI. So like ODS comes plugged in with a search engine for you. So you don't have to worry about this stuff. This is what I mean when I say like people try local and it's like the first experience is so bad. I'm like okay this thing doesn't work right? They will never try again.

34:34 They will just be paying subscriptions for the rest of their lives. And that's why like I remember like uh a year ago I was like uh telling people online like don't go buy like Apple hardware. And I I have no problem with Apple but I'm just like if you try Apple hardware right now with local models you're never going to use local models again. So like there's an issue with the wrong messaging in the space.

34:58 So again this explains the whole thing. Um so yeah um benchmarks I don't like benchmarks so we skip this. Yeah this is an interesting one about like EXO. It's a paper um it's a paper called EXO. uh they are not related to XLabs or anything like that and it's basically about an engine um about a harness that we wrote itself uh to actually improve uh its own like uh reuse like basically an engineering platform or an engineering harness around the task that it does more of and how that saved it tokens in the long run.

35:40 So this is again like another just systems engineering basic first principles engineering that we need to do more around uh local AI models or models in general to extract more value out of them. Check it out. How are we doing? Oh, so which accounts change their models on open router? Um, we're seeing generous coding does that more about 33%. But like people that are doing translation or summarization just stick to what they're doing.

36:26 Uh so a lot of people move models and uh more people should actually be fine with the idea of switching models as newer models come out. Uh but a lot of people are just going to be like no I'm going to stick to open AI or anthropic office 5.5 is great until the next release. Like just try like give Deep Seek V4.1 Flash a chance. Like again it's I swear to God it's better than Haiko.

36:53 It's better than like the lowest tier anthropic model and somehow artificial analysis is saying that it's better like that haiku is better like what a joke. So Airbnb had actually uh this month had um an article about how at 1% u at 1% of of the cost uh they had the frontier intelligence quality on a two billion parameter model. again small specialized model if you have the right data if you have the right traces if you know how to build some synthetic data pipeline and I'm giving away a lot of information here by the

37:32 way like just go ask f.2 do about this stuff and it will build you something. Uh or come to us atmantic. Uh either way uh if you have some data um traces from your agents with the frontier intelligence models you can start to to build an evaluation pipeline and you can decide on building sens data pipeline and RL something that will be very efficient very very fast and at 1% of the cost of those frontier intelligence labs and it will beat it.

38:05 So again, optimization per chip, per model, per workflow, it's everything. And it will save you 99% of the money. That's not me. That's Airbnb. Let's get this one. Yeah. Um there are labs out there like or companies out there uh like Fireworks that are building specialized models and hosting them. So like this tells you something as well like there's a bad like specialization in models uh distillations teaching smaller models.

38:45 How do we think that obus 5.5 was trained? It's a master student model it's fable 5.1 or fable 5.2 into teaching it and give it giving it a bunch of things based on the normal distribution of anthropic users and finding the most important stuff. But like do I trust uh OBS 5.5 to just go do some frontier research for me and like figure out some um extremely novel stuff?

39:13 Absolutely not. I wouldn't trust it. Out of distribution tasks, it will fail miserably. Astro 6 would be better than it. But then again, it's like speed and efficiency uh kind of situation. Yeah, this is an interesting one. Uh by the way, uh there's a certain threshold for which uh below that you shouldn't buy GPU, you should be renting. Uh and it's funny that somebody like me like buy GPU.

39:42 Uh it's kind of telling everyone about the fact that you don't need to go and buy a GPU. you actually can rent one and use it for your task and overflows and there's a lot of ways to run GBS out there. But what I will say is that if you're just using one of those cheap providers, they could be saying that they are offering you chemicase 3.1 Pro at like 30 cents per million tokens or something and they are basically routing you through another model from a different provider.

40:09 So they could be really just running qu 3.5 uh 35 billion MOE in the back end. Um trusting the infrastructure of the models that you run is very important. So that's why I usually don't tell people to just go use an AI but rather like go get bunch of GPUs, confirm that they work very well. Um make sure that basically they are up to standard and use them from there.

40:40 So we're seeing uh prices drop a lot both for closed models and open source models but that gap between open source and closed models is only closing because open source models have been getting better during that time span. So the effect of open source models that's why like you know um some people want to ban open source AI. I'm not going to name names uh but there's a reason that like you know competition is good for everyone.

41:06 Uh the free market is great and we all win that way. Uh oven research wins, humanity wins. Uh and also we spend less money and get better tokens. Oh yeah. Uh pricing prices per million token doesn't track uh intelligence necessarily. Uh, so this is one of those things where it's like, um, you know, OBS 5.5 max, the highest tier, $8 per million token, and it's beating in webdev.

41:39 U, again, I don't trust benchmarks, but this is just like a a way to just try to put data next to each other and compare. Uh, but it's way cheaper than something like Astra, but again, it's because Astra is probably six to 10 times bigger in terms of size. So it requires a lot more compute. Um but again OBS 5.5 is being like was distilled or taught to just do good web development and software engineering tasks.

42:15 Yeah. Uh GPUs cost the same uh busier idle. Uh so that's that's the one thing. So like when you're renting them uh so there's a certain threshold that after which it makes sense to actually rent even rent. Uh so like figure out if you if what you're doing just requires a few hours then like figure out the best place to rent from and optimize for that.

42:37 There's a lot of engineering that could be done around this. And Mudal like Mudal has an amazing uh CLI that allows you to just rent GPUs on the fly, right? So you can even ask codex if you have a codex or even ask a local model to go rent you the GPU with the CLI as a command and get your workflows kicked like started and once it's done stop. So again a little bit of engineering but like I feel like everybody around AI and open source mod is like I don't want to do any more engineering.

43:05 I just want to offload everything and like if you message somebody what's that code about like what is this BR? It's like they don't get back to you for like an hour and a half, it's because they're probably like sitting down with an agent, making that agent explain to them what they did before and like figuring out the BR themselves. So like it's bad.

43:28 Oh yeah. Uh there's a lot of uh again there's a lot of maintenance that goes into owning hardware. So it's not it's not simple or straightforward. There's maintenance, there's dependencies, there's like silent failures, there's hardware that dies in you. um there is like redundancy and stuff like that. So like there's a reason that that cloud exists and it's so expensive depending on which provider you use.

43:48 Uh but this is also fun if you're like a hardcore engineer. If you like this stuff and you like to thinker, this is fun. Oh yeah, like uh this is one like you know scripts could fail on you and like you know the agents would say it's dumb but then it's like go back and check and it's like oh my god I made a mistake. You're absolutely right. Uh, so like we're still like, you know, models are still like not perfect.

44:11 Like we're not at AGI just yet, guys. Um, yeah. So, three forces of the line. Um, more capable models per parameter. And I don't hear when I say denser here, I don't mean dense models. Just means intelligence density per parameter. Uh so this is one thing that is happening and having happen happening frequently. Then specialist models, models that are trained on special workflows and better engines, right?

44:42 What else? Oh yeah, uh this is an interesting one. Uh when I was saying what else I saw that we're moving to the uh continual learning part, but we're not there yet. um you know the same machine that I had behind me in that picture was like 14 RTX3090s at like 2023 it was able to run like Lama 2 7TB and then somehow today each one of those GPUs is running like quen 3.5 qu 3.8 8 tomorrow probably Quinn 4 and it's way more intelligent.

45:10 So you're kind of like when you're buying hardware right now and when you're like getting into this space you're kind of planning for tomorrow's use case and tomorrow's like intelligence density and optimization rather than just today's. There's a reason that like an RTX3090 right now is like in Canada a friend was telling me they're $2,700. So like God with you, man.

45:32 Like I'm sorry about that. [laughter] Can't believe I bought a bunch of them for $400 a few years ago. Oh yeah. Uh so basically uh in March we had the 397 billion MOE come out from CRM 5 and they came out with the 27 billion parameter dense model 3.5 back then and it was not beating the MOE. However, a month later the 3.6 6 of that quen 3 point of that same model the 27 billion parameter model was beating the 15 times larger dense MOE so you know again kind of intelligence like density of intelligence and it makes

46:17 sense because here you're making the model runs through the entire like 27 27 uh billion parameters for every token however that this model thee had a 17 billion activated parameter so it was only 40% smaller, but it used like it required still 15 times the amount of total memory and it was beating it. Something to keep in mind. Yeah, GBT40 used to be required data center.

46:49 Now it runs on your phone. Um, this is one very interesting thing. V4.1 flash most amazing release this year so far for me by the way. um probably even better than quins 3.5. And I know a lot of people will come out will come after me for saying this, but this one made us [clears throat] go down from [sighs and gasps] 13 GB for 400,000 tokens to 29 megabytes in terms of memory for a,000 tokens.

47:24 pretty much shrunk the KV cache of this deepseek v1 by 437 times. So we've come a long way right since this is one of those optimizations like you know 100 times faster tokenizer only increase your like you know overall performance by 1.1 right uh but we're also seeing that over time this is a 400 437 times performance increase for the same uh for the same area basically for the same like you know engineering task.

48:00 Um, so what else could be optimized for over the next, you know, year, two or three? That's what I'm usually like thinking about. I think the next big thing is probably continual learning. We'll see. And by the way, continual learning will not happen uh from cloud uh platforms like for continual learning, it needs to run locally. That's why I keep saying like this thing ultimately local has to win because every big provider is not going to train a model and host it for you.

48:30 They cannot do that and they're not going to train a specialized for you. They're training for their entire like you know normal distribution of users. And in order for something to learn just about you, it has to be trained for you. Otherwise, we're stuck with markdown files and uh [snorts] memory like uh memory hacks around models that doesn't necessarily work.

48:53 like you know how many times has a model like made a massive uh you know massive mistake because it's a like you know large context and now it's like writing to three different skills while it's already started with its own skill on the same topic right so you end up with like like something like codex or Hermes or whatever you get a bunch of like markdown files and you call that a harness but over time it's becomes like so plotted and useless um the only way to solve that is actually continual learning Oh yeah, this

49:26 is we need optimize for every piece of hardware. This is from the loci summit at AIE back in June. Yeah, again we got like a Windows machine with the RTX Spark running quins 3.8 8 uh flash next um on a machine that has 128 GB of memory at the code speed 72 tokens per second. Uh and it's an incredible model for the fact that it's running on a laptop and like go try to play odsspark.com and just see like what it was able to build on something that you can take on a flight.

50:10 Let's get this. Yeah, mega kernels are part of the optimization. And we have 10 minutes left. So, I'm trying to give some space for questions. Only 32 slides left, guys. >> Yes. Yeah. Yep. Yeah. I know. I know. It's very dense. Thank you. So January I said that uh and people came after my head for saying this. I said that by the end of the year we're going to have um OBS 4.5 running on an RTX Pro 6000.

50:52 By April we had that inqu 3.6 27 day in parameter. We already have surpassed OBS 4.5 locally on an RTX Pro 6000 a long time ago. So do not underestimate the kind of acceleration that open source models are having at this point. the gap is closing and uh specialization even if like some people would argue that big labs uh have a lot of compute so they are able to like you know use the compute use agents research more do RSI right and that open source labs cannot do the same thing well the other alternative is basically

51:30 specialization until we're able to you know deepseek or whoever is able to crack this with some novel research is able to give the open research community something to run with. So, it's going to be a minute, but there's a way around it as well. Yeah, plan for the model you will run next year. So, when you're buying hardware, just think about what's next.

51:59 Um, yeah, 1 billion uh by GPU. Um, shout out to Chris Alex Neimatron from Nvidia. Amazing guy. Uh, and um I'm trying to get to cont learning. Yep. Inagrams. So, enagrams, decision models, and weight edits. These are the last three big things that came out in the space for me. um anagrams allows you to basically have a an equivalent of like a map on your SSD or in your memory for a model to retrieve from.

52:40 So that is solving a big beast. Decision models allows you like you know we always had classifiers but we never thought about how it expands to every surface single task out there. That's another layer that if you bought on top of an enagram plus and weight edits um you're going to get continual learning and like this is the recipe in my opinion and I'm saying this out right now so that in a year I can point to it and say I told you um we're uh we're very excited about this and like I had a tweet uh on November 4th so

53:12 you can see even like at the uh at the footnote right there on November 4th that I said anyagramraphs plus decision models plus n8 programming um equal continue learning uh lookup embeddings basically uh we're we're cooking something. That's all I'm going to say right now about this. We're cooking something and keep an eye on ODS. uh because I genuinely believe that the markdown plot that every harness out there is full of right now has to go away and this is probably the best way to do that at this moment.

53:52 This is not necessarily the the final shape. This is not the final boss but this is one easy way to do it. Oh yeah. Uh loops. Uh so we have loops inside models. We have loops around models. We have loops across experiments. Could we also have loops around enagrams and decision models as part of this entire thing, entire recipe to make container learning work?

54:21 The the same could easily work on CPUs. By the way, anyagrams, just SSDs or RAM work just fine. Yep. Um so learning uh make sure that when you're evaluating something if you're running some bunch of benchmarks that it's not just road hacking. Uh more to come on this. Now we have 5 minutes left. Any questions [gasps] I hate benchmarks. Um really it's uh it comes down to the kind of software that I have built my own traces my own corrections and building around them.

55:02 So like it's kind of like you know the whole idea of like having high taste in data and not necessarily the saying that like you know that that's who I am but like the whole idea of having high taste is very important in data and evaluations. There's a reason that every out there is being hacked right now. Um should you buy a GPU? Um good question. Uh you tell me.

55:25 And uh yeah, um by the way, there's a book coming out that will include a lot of this stuff. Uh locally.com, keep that on your radar. Uh there's a bunch of links in here uh about my articles, videos, etc. And uh I have 9 seconds left. Thank you so much.