Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI Transcript, AI Summary & Key Points

00:01 [music] All right, let's get started. So, hello everyone. My name is Yun Mo. I'm a founding engineer at Friendly AI uh building inference engine with our team to optimize inference performance. Uh so today I'm going to talk about how we built the fast and reliable inference API for frontier openweight models. So oh thank you. Okay so before we dive in let's do a quick show of hands.

00:41 So who here uses closed weight frontier models like GPT or cloud? Okay most of you are already using it. And then who here mainly use openweight models? Yeah. Yeah. Fewer than the closed weight models. So personally I recently using a mix of both these days but I'm gradually migrating more of my workload to open weight models. The reason is that open weight models uh have crossed the threshold these days.

01:17 For example, encoding, reasoning, and agentic benchmarks. Models like GLM 5.2 are competing in the same tier as the leading proprietary models. So, as you can see from this code arena leaderboard, uh GLM 5.2 is ranked even higher than code opus 4.8. So, it is right uh below the cloud fable 5. So, similarly across various coding terminal and tool use benchmarks uh GM 5.2 2 and minmax M3 shows strong results.

01:48 So the question is no longer only whether open weight models are good enough for many workloads now they are. So the next question is how fast and reliably we can run them in production. So benchmark numbers alone uh don't really give a clear sense of how well things actually works. So instead I ran a a simple hands-on experiment to see how these models actually perform.

02:16 So I asked both GLM 5.2 and OPUS 4.8 uh 4.8 to build a tower defense game like this. So as you can see both models produce working implementation with similar structure and quality. So both one has a map and the turrets are shooting the the enemies within the range. So yeah, both looks pretty great considering I made it with a a single prompt. But when we look at a cost, the difference becomes very clear.

02:47 So GM 5.2 was more than 5.6 times cheaper for this task. And in agent SL style workflows where you need to iterate multiple times, the the the gap compounds quickly. So this is not just about motor quality anymore. It is about what you can afford to run at scale. So after the model quality and cost, the next next question is provider quality and provider quality is really about performance.

03:17 So even if the model weights are the same, latency, throughput and reliability can vary significantly depending on where you run it. Uh so this one has a open router snapshot which shows that uh these differences are measurable. So if you're building an AI product, you should not only ask which model to use, but you should also ask which providers gives you the best performance, reliability, and pricing.

03:45 So I will show you another example. So here I use the same model 5.2 and the same coding agent workflow. So the task is to build a simple Candy Crush style mobile game which we call moto crush. So when you see this, the left side is friendly AI. it generates the the the game within two minutes. Another provider needs uh more than three minutes. So the model is the same.

04:10 The agent app is the same but the provider was different. So model developer tools and coding agent this is this exper this is the experience that users actually feels. So from this simple example we saw that even with the same model and the same task the experience can feel very different depending on the provider. The difference comes from the entire inference tech.

04:36 So the model architecture cache system routing policy quantization strategy GPU corners all those things come together should shape what the developer actually experiences. So this is the uh core of friendly AI's value. So we are not just exposing a model API be uh so friendly AI is built as a fullstack inference engine for geni models. This matters uh because fast and reliable inference is not just one optimization.

05:07 It is batching scheduling quantization spec decoding and connors all all everything should work together. So let me briefly explain uh how we built that and how we make uh the inference performance uh to the cutting edge level. So in the modern agent applications inputs are often very long. It easily exceeds uh 10k tokens while outputs are relatively short and more importantly a large portion of the inputs are repeated across requests.

05:43 In many real workloads like 50% to 80 to 90% of the tokens are identical across different requests. So from a system perspective this means uh most of the compute cost is in the prefill stage where the model processes the entire inputs and build a KB cache. So if we recompuse this share prefix every time you're wasting both latency and GPU cycles. So when a new reccast share a a prefix with the previous one, we can reuse those tensor and skip the corresponding prefer computation and directly reduce the TTFT and uh we

06:19 can avoid recmp computing large chunk of the input and but however making this work at scale requires two key ch uh mechanisms. So first one is cache aware routing. So in the distributed system KB cache is not centralized. It lives on individual replicas. So the router must send incoming request to the replica that already holds the relevant prefix in its cache otherwise we re we lose the benefit of reuse.

06:48 The second thing is cache hierarchy. So KB cache is extremely memory intensive for large models and long context. KB cache size can exceed model weights. So and as you know GPU VRAM is the fastest but there's also very limited resources. So uh we treat cache as a hierarchy like hot cache stays in GPU memory warm cachees in uh CPU RAM and quarter cache can be offloaded to MVME or uh the disk spaces.

07:19 The system dynamically offloads less used blocks and prep them when uh when they need it. So by combining cash aware routing and hierarchical memory management we significantly reduce the prefix cost lower TTFT and making long context agent workload much more efficient. So the next thing is interactive speed. So standard auto reggressive decoding it generates one tokens per four pass that makes decode uh inherently a sequenture and during the decode the GPU is often underutilized and speculative decoding changes the

07:57 shape of this problem. So instead of generating one token and stopping it proposes multiple candidates tokens ahead then the target model validates those uh candidates in a single pass and if the tokens are accepted they are free. So this improves time per output token uh without changing models uh output quality and there are different ways to draft candidates.

08:23 So one way is modelbased speculation. It uses a small draft model to predict the future tokens and then and the benefit of it is it can achieve higher acceptance rate but the cost is the uh extra compute and another method is model free speculation. It uses pattern from a prompt or prior output. So it has lower overhead but it works best only when the output resembles the existing context.

08:55 So the important part is the trade-off. So speculation can improve per request latency but validation as overhead to the batch. So the best strategy is not static. And at friendly AI we take a hybrid approach. So the runtime can dynamically select between model based and model speculation uh based on the workload and request uh characteristic. So we also think about speculation as something we can improve over time.

09:24 So with self-evolving speculation uh draft model can adapt based on the real serving traffic. So the speculator continuously improves its prediction of the workload actually running in the in your uh production. So the next thing I want to mention is reliability. So speed alone is not enough. So production AI application uh reliability is part of speed.

09:51 So if an API is fast in average but frequently hits timeout like five 5xx error 4 to9 rate limit or fail or that kind of failure the user experiences uh still breaks. So this becomes even more important for agentic workloads. As you know agent may need multiple model cores to complete one task and if just one call phase in the middle the entire workflow can be invalidated.

10:21 So that's why we do not optimize only for the latency or throughput. We optimize the full API experiences uh including the error rate, uptime, all those things together. And this open AAI snapshot for GLM 5.1 pro uh shows that friendly AI is not only the fast but it's also highly reliable with over 99.9% up time. So what I want to say is many inference provider out here of course in this menu as well they will tell you that they are fast and some point of them they are but on cert on certain benchmark under certain

11:04 condition but for your application the only benchmark that really matters is your own workload. So your prompt your token length your traffic patterns and your latency expectation. So our recommendation is simple. So please bring your workload to us and test a poor experience with us in terms of like speed, reliability, cost and quality. And of course um we build our inference service for exactly these production workloads and we and we believe that we can give you the best experience.

11:37 So we have a boost in this venue. So bring your hardest problem to us and let's solve it together. So to close I want to share a testimonial from one of our customer is a kilo code. So kilo code is one of the largest coding agent platform processing like several trillions of tokens every week. So they have tested many different inference providers over past years and in their split test for GLM models they were especially impressed by friendly AI.

12:08 So compared with many other inference provider, friendly AI was over like seven times faster and had a sign significantly lower error rate. So I believe you can get the same kind of experience with friendly AI. So if you're building your frontier open models, building with your frontier open models, come try them on friendly AI and bring your real workload and let us show you what fast and reliable inference can do. So this is the end of my talk and if you have any question feel free to ask. Thank you. >> [music]