Help organizations replace expensive frontier models with cheaper open models on specific workloads: benchmark open models against evals, split planner/worker/reviewer traffic, and fine-tune small domain models distilled from a frontier model's outputs, keeping quality within ~1% of the baseline while cutting the bill dramatically.
AMP is an agent harness that automates switching between models: it uses a weaker model as the baseline and invokes a more capable model when needed.
Cerebras Systems (branded Cerebras) is a company that designs AI‑optimized compute hardware and software for large‑scale model training and inference. Its products include the Wafer‑Scale Engine (WSE) and CS‑class systems, which provide high‑performance, large‑memory accelerators for deep learning workloads.
An AI software engineering agent developed by Cognition that runs isolated cloud coding sessions, writes and modifies code, and can test applications through a browser. Devin is designed as an end-to-end software-development environment for planning and implementing coding tasks. Vals evaluated it internally and increased its adoption after finding it to be token-efficient.
Featherless AI is an AI inference platform and serverless LLM host for deploying open models through one API key. Its model catalog offers access to more than 40,000 open models, with browsing and testing by categories such as language, reasoning, productivity, and model size. The company also provides dedicated GPU capacity and fine-tuning services, and presents model selection, classifier use, workload measurement, and capacity planning as ways to manage inference costs at scale.
An AI frontier-model API from Google with a dedicated service for generating distillation datasets. The videos present it as an option for producing training data for smaller or task-specific models.
Hugging Face is an AI and machine-learning collaboration platform that hosts and distributes models, datasets, and applications. Its Hub supports public model, dataset, and application repositories across text, image, video, audio, and 3D workloads, while its open-source stack includes Transformers, Diffusers, Safetensors, Tokenizers, TRL, Transformers.js, smolagents, and PEFT. The platform provides a unified API for accessing models from AI providers, GPU-based compute and Inference Endpoints, and paid team and enterprise features such as private datasets, access controls, single sign-on, audit logs, and dedicated support.
Pi is a lightweight coding-agent harness presented as an alternative for people seeking to avoid Claude Code lock-in and reduce cost per token.
Simple Jeff is an open-source project that replicates the Jeff one-shot classifier API using open models for lower-cost classification.
Searchable transcript of The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI — AI Engineer (55:25). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:01 [music] Uh thank you for everyone for coming here. Uh not too sure whether be happy or said that so many people are suffering from their token bill. It's a full workshop. Uh yeah. U we are going to talk about tokconomics. So a little bit of background with kickstarting it right. I'm Eugene from Fellows AI. I'm CEO and founder of it. Uh, and I'm part of the team behind the RWKB open source project, the first AI model under the Linux Foundation and uh, and we work on next generation AI architecture.
00:37 Uh, that's a little bit of my academia background and my Twitter handle is PCO creator. But but more importantly, right, uh, uh, a little bit of background for Father's AI is that our mission is to make AI accessible for everyone, every language, every dialect, every country. And we do this we do this by creating the world's largest platform of open models of over 40,000 open models.
01:00 This includes specialized fine-tune like the Sunflower model which is Uganda's first language model specifically for their language and a lot of other specialized models. So we're just not talking about the biggest open source model but specialized language models or specialized use cases. Some of it which I'll touch as well. And when we where we can go on scale in terms of variety of model Father's AI also we serve customers at scale.
01:22 So some of our largest customers are literally serving two trillion tokens a day and and that's and and they're serving it at around 0.3 per million tokens or 60k a day and they're doing it on around 512 AMD GPUs uh MI325 uh and for this customer in particular it's a private 4800 BOE model and they are serving at around OPUS 4.8 eight class performance.
01:47 So it's lots of concurrent requests, lots of token volume. That's what we mean by scale. And if you look at it, 0.03 per million token at Opus 4.8 class is not something that is not something that you can get off the public cloud market. This is very specialized setups that is meant to run at the scale that we're talking about. So that kind of uh leads in into two major disclaimers that I'm going to start off.
02:14 one we sell inference. So everything I cover here will cover to will touch on any provider. Nothing here is specific to Fatherless but to to for clarity we sell inference. So so this is from our data points from our experience respectively as well. Okay. The other thing is that this workshop is going to cover a lot of optimization at scale and it probably will not feed you if you're like spending less than 10k a month in your organization.
02:43 Uh my word of advice for those who are below this this uh metric mark and this is an arbitrary metric is that if you're spending less than this, you're probably better off focusing on like trying to get strong use cases within your organization rather than trying to optimize the token spending bill because a strong use case can bring your organization way more than 10k a month.
03:04 And and that's and that's should be your main area of focus and that's not my area of focus for this talk. So what we're going to cover in this workshop uh is four moves in in in in four major order measuring right sizing uh how do you buy your capacity and how you check your process so measuring yep you'll be surprised at how essential this is uh like how many of you know your total token spend can I have a raise of hands wow only half the room raise hand okay how many of can measure at the department level.
03:42 Those those who have their hands raise up lift up and so I see about another half went down. Yeah. And it gets lower and lower down to the team level to the use case level. Um we like easily 30 40% of our customers that reach out to us the first thing they had was a single shared key that everyone in the organization used and that was like uh how do we even measure that?
04:06 How do we start from there? So, so this is not an a vendor specific like you need to start splitting up your keys, start splitting it by department, start splitting up by major use cases so that you know where all your token spends are. Um, and this is then from then on it can inform your workload. So yeah, API keys shouldn't be that hard to issue. Um once you know that right you can start right sizing things uh respectively like like which team is uh and workloads spend on it what was the person waiting for like some
04:35 teams deserve the tokens per second some background agent running at midnight probably can don't deserve the tokens per second speed that you're paying for it uh yeah don't pay for ultra on a background agent that's just dumb um and and from there also you can also look at your usage to better advise like uh dedicated capacity that you may purchase be for open models or even closed model.
05:00 Yeah. And also you can also check which team spend the most token and whether they are productive. Uh I'll touch on that as well. One of the things that you should know in AI workloads right uh and you all may already experience right is there are two major tracks with very opposite economics uh respectively. So there is the agentic workload and obviously there's things in between where it tends to be like longunning multi-turn agents.
05:24 Coding agents are a particular example of this. Some some some mayard some devs like run coding agents like literally overnight over a long period of time uh with multiple compaction and and these things maximize the context uh usage. Process automation is the other extreme. Uh think call centers, think email routing, things your meeting notes transcribing and things like that.
05:47 It's all it's much shorter in context in general though there are a lot of examples that defy this statement. uh but it's very processdriven, very task isolated and it's it's a volume game essentially. So what do I mean by more usage and this is probably one of the first major explosion a lot of organization face uh in particular when they jump onto the open claw hype or one of the claw clones.
06:14 Uh so you like like from from from various developer teams that we we support like we for example we sampled one of the team and it was only five devs and they were averaging around continuously eight running agents uh for one of the companies we work with and and so they were previously running Opus 4.8 it uh of the API and they were essentially spending 150k a month continuously running these agents.
06:36 Uh some of it is the agents that they used to code, some of it is the agents that kick in at night. They they were slightly more advanced. They already knew that they they they should kick things off at the night uh when no one's working but essentially they were spending 116 billion tokens a month and we literally helped them switch to a GRM 5.3 instance.
06:54 Uh previously was 5.2 they already upgraded 5.3 and that was like a single node that was less than 10k a month and they were and they didn't see any disruption in your impact. So knowing your team workload allows you to make these decisions respectively and like I highlighted like know processes workload also allows you to split things off. So like for example previously they were running everything throughout the day but they they started scheduling some of the background agents when the team was asleep and it runs
07:21 through night. So this allows to utilize their hardware respectively. Uh the as I I touched on briefly uh the other side of process automation is purely a volume game. Um extremely predefined. Uh I I can give you more specific example of the off the record but you think of it as like millions per day or even tens of millions per day. Decisions in call centers, lines of code scanned, emails and text message filtered.
07:49 Like one of the tos we work with like they easily hit pick 10 million messages in a single minute and this shouldn't come as a surprise. Happy New Year is the worst. Yeah. So, so, so this and they and they help and the reason why they filter messages is because they provide a feature about like providing safe phone for kids. So, they needed an AI to scan like basically is it profanity, sexual content, etc., etc.
08:13 So, so this this is really not a long context but a volume game. Other things like other things is like maker checker processes, approval processes, document processing, these are all volume game for multiple organization. So I suspect for the financial institutes here since we are in New York a lot of OCRs, a lot of documents, a lot of approval, a lot of transaction risk the is is under this category.
08:39 Okay. So yeah, if you're not yet tracking your token token spend and token view split, right, we are open sourcing uh uh tokconomics tracking tool which is just a very fancy way of like splitting up your API key and tracking which API key spends some how much uh uh which is separate from this talk. Yeah. So feel if you want to again update feel free to to scan this QR code and respectively also if you want the slides for this talk and when the recording is released from AI engineering to be notified about this talk you
09:06 can also get uh fill up the form there. Okay. Okay. Okay. So yeah that was about measuring. Um yeah if you are not measuring you can pretty much ignore everything else I say and start with measuring first because all your biggest ROI is just by measuring. But once you have things measured right the question is how do you right size it and this is where things gets more complicated.
09:31 Uh in in general right one of the things that we start to observe as as AI gets deployed at scale within organization is that there tends to be a pattern of like because AI is the shiny new hammer. We try to apply it to everything especially within the innovation labs. we just try to apply to every damn thing and we tend and and and I have seen organizations literally apply fable to really the silliest of things.
09:59 So So you you there's a lot of models and a lot of models are really expensive and and and and figuring out to do and where it fits is it's it's important it's an important step of the process. I know open models I mean a lot of you probably heard about open models and that's our line of business. Uh yeah uh and and so a when a lot of companies come to us for example to like find open models that fit their use cases uh and drop in replacements this is uh we generally guide them along their use cases.
10:33 So these are this is a general rule of thumb that is probably going to be outdated next month because open models release every few months and same thing with closour models things change rapidly but but there but the things what we find very exciting actually within this year is things have started to change like in the previous year the best close source model is without better than the best open source almost but as this year approach right we are starting to see the gap close we're starting to situations where
11:03 where some of the open models are even on par or even surpassing. So like for example before the Opus 5.5 update came out Kim K3 was beating Opus in UI design in some of the UI benchmarks and and this is like very public benchmarks within customer benchmarks right we were already starting to see open source models beat very specialized benchmarks in specialized use cases.
11:28 So like a particular one to highlight is like for example step fund five model they it was able to achieve near us and estra quality despite being like a 10x model model in financial benchmarks uh financial use cases and this is it the best coding model no but in that use case it's actually extremely strong and competent. So, so, so the landscape is slowly changing where where where you can have drop-in replacement for a lot of use cases that are sometimes even just 10x cheaper respectively for if if you know your
12:00 workload and you can drop it in respectively. Special sh is also deepse v41 flash for some reason and this model is extremely popular across all workloads respectively because it's easy to prompt steer uh for a lot of companies. So well I well the rest of the models I have listed there that they they're like default for good in their categories deep for1 flash seems to like be quite popular regardless the case.
12:27 Yeah. Uh though benchmark score doesn't seem to justify why is it so popular. Yeah. Yeah. Uh another thing to highlight right is that recently like and this just like for a few weeks ago. Uh the a new class of uh AI models have been introduced called uh classifiers uh oneshot classifiers most popularly known as Jeff. Open source projects have already started uh taking off like simple Jeff and these are models that are entirely focused on just making decisions and classification on on task.
12:58 Uh the main advantage is that because on the technical level they are preview only they are commonly 10x cheaper than existing models or even faster at the same time. Uh the downside is it cannot give a chat response. So so so these are things that are h changing and happening that you may want to start looking into on like segments of how you want to improve your your token economy.
13:19 Yeah. But but back to the the the the token economy part there is two major tracks and it is important to separate out and track this uh track them respectively. On the agentic side, we're talking about long running long context as I said earlier and to and you have to understand this workload tends to be extremely cash heavy extremely uh input preview heavy and cash heavy and there's only a small percentage of output.
13:49 We see across like agentic use use cases and deployment only 1% of the request is to the output tokens and and and this is this is very logical because it takes multiple turns rinse and repeat but this also decides your token cost because like if you seen token cost right uh deployment cost right output is may have the biggest number but even your input well is much smaller or the cache is much smaller it it gets repeated heavily in in this use Please.
14:21 Yeah, this is not a coding agent workshop. Uh, but one of the most common patterns that we we see in a lot of organizations, right? Is that is that typically uh typically in terms of like trying to scale up their AI agent deployments with within within the coding teams, right? They tend to have the best agent do the planning and a faster uh a faster cheaper agent do do the implementation and then maybe the best agent do the review.
14:47 Again uh there's a good reason why this split is very common across all the agent frameworks uh is is that you get the best of both world because like the implementation phase is tends to be tends to be very token heavy in process. a lot of processes repeated back and forth like keep modifying the file testing checking for the results rinse and repeating and and you and most and most of these agents especially this uh especially like JM 5.3 flash is more than capable of doing a task at the same quality level as a much
15:17 more stronger agent like let's say Astra or Opus uh uh respectively assuming those agents do the planning we have worked with some of the organizations where literally the planning and review agent is the s frontier close source model and then the workers that implement it is the GRM 5.3 flesh as an example or deep 4.1 flash yeah one of the observations that we made uh from from this from from this kind of split pattern right is that unfortunately as much as we wish everyone does so nobody really switches their model
15:51 to save the company money uh this people are lazy to switch the models uh in general you you want to give an incentive to switch or or alternatively if your agent harness is good enough. So like for example some harness like AMP uh they automate the process like like it uses the weaker model as a baseline and then use a big the smarter model to automate switch in when it needs a smarter uh consultation.
16:18 Um this this helps reduce the friction respectively and helps improve your your token economics like you use at scale like a ma a major rule of thumb that we we we now give our clients after going through a few times is that you want the smarter model uh and since it's already so expensive as a baseline but you uh but you want the the faster model to the cheaper model to be faster and that naturally sets an incentive.
16:45 people want to get their work done fast. If the model is smart enough to get it done and it's cheaper, they will just use it because it's faster, not because it's cheaper. Uh we had teams where where where where both models were the smarter one and the cheaper one was maintained at the same tokens per second speed and and as long as they had the token budget in excess, right?
17:05 No one switched. And it and on the flip side, we have seen teams which is the other extreme where they set an extremely low budget and so that if they use a smarter model, right, they just burn it in one day. Yeah. Then everyone just switch to the to the D into the faster model D and faster model because that's their budget. It's like almost as if the smarter model didn't exist.
17:26 So like be aware about your team dynamics on how they switch. The the best trade-off is still speed because people are naturally incentivized for it. So faster, cheaper, they'll just switch to it respectively. Yeah. Once again, reminder measure and you measure through separate token keys. Uh but this also goes beyond just measuring. For example, like CI/CD is a very common use case right now of things running in the background.
17:56 uh we we have helped set up for teams like faster tokens per second uh that that is running for the the users like the humans but much slower tokens per second um speed but much cheaper at scale for the CI/CD pipeline and and and this is just the nature of like what's your priority respectively respectively across the task. Yeah. And you can control that at a key level.
18:25 So another major finding as uh that we had experience with uh deploying for for teams right uh so like for example for some companies um how many of you had the fortunate or unfortunate experience of the CEO saying everyone must implement AI now and everyone must have a second brain now. Yeah, multiple organizations have started this uh in some cases like even in the UAE it became literally mandated by by by by the law of the country.
18:54 So so like a lot of departments are not prepared to know what is the trade-offs respectively. So a good rule of thumb right well developers tend to know okay all these smart models what's the trade-off uh between deepse flash and G53 for example most uh your your finance department have absolutely no idea your your support department has absolutely no idea they they'll just pick whatever is the first model and probably just go with it so so our general rule time is that give department models labels but not the model
19:31 names and this also gives you a lot more control as you switch it out uh uh like like I advised earlier fast versus smart but in some cases also like finance agent or Excel spreadsheet agents um these are hyper specialized agents that will fine tune to those tasks respectively and and this also gives you a lot more control it allows you to swap out the model it allows you to split split the traffic and test in between with without without telling hey users please try this downside is also users may sometime complain
20:02 when you do that and you screw that up But having that control is a good thing. Uh another pattern that we see even though you do this is maybe someone will add a dash preview version. So that's a variant of it. But it gives you control without without taxing the communication burden across the organization. And this is extremely important because like like some of the teams that we work with, right?
20:26 We literally saw members running Llama 2 models when when we are talking when we're talking about like in this year because no one switched the option. Yeah. So you really really need to need to uh need to set this in place. Yeah. Okay. Process track. Process track a lot of things that I mentioned earlier don't really apply as much to process track because process track tends to be a lot integrated into the applications into the workflows.
20:55 uh respectively and and that's just by the sheer volume of the traffic. Um yeah and and a for a lot of teams in the process track right uh they tend to be a lot more focused on focused on the right sizing already by default because because if you're running like trillions of tokens you most departments will not approve it like $10 per token million token it's just not possible and and so so so they start they start early with the evals the benchmark and then subsequently finding the model that right size fits it Yeah.
21:31 So, so the thing is like the common question is how do I know the smaller model is good enough. You don't need to that's the wrong question to say you just need to make sure you set up your test cases, your evals and your examples and then subsequently just do the search process from there. Uh companies like us will be more than happy to help you because we have over 40,000 models.
21:51 But then but then if you it's also not that hard to just download 10 models from hugging face and just try uh so uh so from that you just do the search find the cheapest model that kind of like pass most of it and then you just see how you iterate from there fine tune from there uh we also have a more specific process that I don't think is unique to us that we'll cover as well as so for example like in for one of the major tos that we work with right uh we actually help their team lower their inference costs 200 150K
22:20 to 7.6K a month switching from existing clos model just by going through this process of like okay you're running the best model first let's see what what workloads we can split off and then subsequently what's the models that we can fit in and then what can we find and this was this was like to be the customer support use cases and and so on so forth.
22:38 Yeah. And to reh highlight back Jeff uh and and simple Jeff which is a open source project that we did uh that replicates it. You'll be surprised at how many percentage of your AI traffic is classification traffic. uh uh at the end of the day right like yeah sure answering the customer support is text but even then right even then right a lot of teams right already have predefined text output that they put into the context for the model to output for most cases and and you could use a classifier that and there's a lot
23:16 this is a new new new new shift within AI worlds and that this is one of the big levers for these kind of processes in lowering your your token Yeah. So one one one example of uh such a model when it comes to fine-tuning. So a lot of models in processes right uh do benefit at this scale where you start to fine-tune your own model. Um on the flip side for coding models well there are organizations that will fine tune models for you like Devon for example.
23:48 Uh a lot of developers generally just want the best model. So it's kind of unless you are in this AI space, it's kind of hard to keep pace in fine-tuning the best coding model. But on the flip side, for your domain, it might be worth it to fine-tune a domain model for your use case, especially if the model is much smaller. So for AT&T, for example, when they finetune your models, they they they went from being OPOC class to like 27B to even 7B models and they have published this.
24:22 So, so we all want their launch partners and and you can actually try their hotel line of models uh respectively and and and this is basically what they use behind the scene and and it's all about volume here. I I think I think most of you understand the scale of AT&T and the call centers and things like that. Uh yeah, but it's a good example of how you fine-tune models to to specialize use cases and once you have reached ever other organizations that are major example like Shopify has started doing this and and things
24:50 and and it doesn't necessarily need to be a large scale finetune. So for example, in the Shopify use case, they did it because Shopify web store had a proprietary programming language for how to configure the web store and the existing models just weren't that great at it. So they went to fine-tune it. Uh but and AT&T also they had examples of that uh in particular like how to arrange the telco towers and things like that.
25:18 Yeah. Uh very telco specific task. I might not be the right expert to answer that question. Uh but they they had those use cases. So, so they fine tune respectively. You two may have those use cases. And how do you bootstrap such a model, right? Is that the good news is that uh you can easily bootstrap things these days because if let's say you're willing uh if the best model that is too expensive to run at scale can do the task, you can already get started.
25:46 So, so, so an easy way to do this, right? uh that we see very commonly is like either you generate your own data or prepare your own data which is hard. No one likes to do that for some reason. But the but another common path is that they start with an expensive model and then they deploy in a small scale and then they generate they data set. I understand that some labs it's against their terms of service to do that.
26:14 So you may want to check with your lawyers but there are labs that openly embrace it. So for example, Google Gemini um their their frontier models, they literally have a dedicated API service for this. Uh so so so this allows you to build data sets respectively on your use case on a strong model which you can then which which you can then reduce dramatically to let's say a 7B scale.
26:38 Uh alternatively, open source models are more than happy for you to to to run and distill on. So you can run the best uh open source models respectively as well. Once you collect them, you can then fine-tune the data and then you can fine-tune and respectively like different sizes to see what fits best. So once again like in the 808 example, they have the 27B and 7B for different levels of the call center.
27:01 Um like your rooting your rooting cost may come in at a wait I don't think I can say which size goes where but your your rooting cost can come in for the much smaller model and then it escalates due to some classifier triggers as an example. Then then so so so that's how you get the traffic to flow get the traffic split and then you get it at an efficient level.
27:23 Uh one of the things that we also tend to work with teams on on when doing this is that even even when they go through the stages and they completely split over their traffic to a finetune model because the best the best model still progress uh they they uh we typically advise them to cut over uh cut a part of their traffic as a shadow traffic to the best model to keep building the data set and and and this allows you to ride the wave and progress of the best open models respectively.
27:54 Uh and the unit economics is actually very important here because like one thing that I would like to remind people right is people severely underestimate at the the the speed that AI is going especially for models that you can run much cheaper at scale like you take this laptop as an example I like to measure right what is the best model that I can buy in the most expensive MacBook Pro period and today's best models be models at 31B, 32B, 27B size is already much better than GPT4.
28:31 So essentially, this model that can run on a laptop is already better than the best model that you have last year. Give and take. People like to argue on the specifics, but let's just say broadly. And when you fine-tune it to your task, that means if you can run that model on a laptop respectively, you can run it on on on your GPU servers at scale much more efficiently.
28:56 Yeah. So, yeah. So, that brings in to the next topic, buying GPUs, which uh which which is apparently increasingly a challenge. So, yeah, that's where companies like us come in. One of the things that we we realized through a lot of customer conversations, people are rather surprised that things don't scale linearly. Uh what I mean by that is like there is a huge trade-off between tokens per second on an individual request to to the cost to the cost of the request itself.
29:28 So for example, if you do not care about about the cost, you could run the best the model, let's say a fixed model on the best hardware on all eight GPUs, batch size one, and you could get let's say 100 tokens per second, 200 tokens per second. If you like if you really want and and and this is not our specialization like you really want like thousands of tokens per second, that's where companies like Cerebras is more than happy to offer that as well and it costs even more.
29:54 So there are definitely use cases where this makes sense but it's but a lot of people fall into the trap of thinking hey so if I have the speed maybe it's half the cost it's not even close like like a rule of thumb that I sometimes say right for every 2x of speed is 3x the cost and it compounds uh in this specific example for JM 5.3 uh it's like one agent on is like you can see the same server 112 tokens per second for running individual requests but if you run 16 requests in parallel you get about half the speed for
30:26 all the requests but you literally 10x the output total output you and and that that gives you like literally around 9 to 10x lower in cost in just like changing changing your target speed profile and so when a lot of customers ask us like how do we deploy things at scale sometimes the first question is what's the model quality you want and what's the tokens per second speed that you want and then we start recommending the amount of GPUs the amount of deployment And this this target tokens per second is an important
30:57 tradeoff essentially um in more extreme cases right if you really really can run things at 10 tokens per second rarely anyone does but there are some weird use cases that allows it you can get even more incredible token total throughput volumes respectively. Typically we see customers choosing between 20 30 which is kind of like uh your chat GPT speed without without too much thinking all the way until like 60 which is like the fastest speed and if they want 120 to 100 like it tends to be this this range most stick
31:29 between 20 30 to 60 depending on how how long the chain of task is. Yeah. So yeah uh I kind of mentioned you generally only want to pay for the tokens per second speed uh when when it's in front of someone uh whe whether is it in front of your developers or whether is it in front of your customers those are the trade-offs that you want to make and even within them right we have seen cases where even just that slight tweak down right can easily save two times the cost but has near minimal impact for the c uh customer
32:04 respectively. Yeah. And once again, background drops, please don't run it at 120 because it makes no sense when you do that. Yeah. So once you have previously I said measure, right? You're able to see your your your traffic pattern. So a lot of traffic patterns tend to be bursty in some nature with some base load because someone's always running something at some time.
32:28 uh especially background agents and and this is where you can you can also plan your deployments respectively. Uh I know since we are dealing financial institute a lot of financial institute may not have this as an option but because they are only dedicated instance deployment but if for some cases where people allow mix where they have dedicated instances with some burst to the public cloud this is where you have some flexibility respectively.
32:53 So like the biggest cost savings you have right now in the market is deploying a dedicated instance if you know your base workload. So as long as you fully use it you kind of have a savings there but it's it's a bit more complicated depending on your traffic pattern and a lot of these things can be tested and validated uh respectively and this is like essentially what any inference company will work with you.
33:13 It's not unique to us. We'll be like hey this uh do you know your traffic workload can you share some data and then subsequently we'll be like okay we propose this much you can try switching to this and then we we generally we can generally show you your savings respectively and in some extreme cases where you have a lot of spikes you may want to have limit your dedicated capacity to a certain amount and let the spikes peel over to the public cloud respectively.
33:38 Different use cases have uh have different right sizing but this is something that by measuring your token workload once again at the beginning you can start exploring these options and the savings can be huge like once again like for the customer that we serve two two trillion token and that was like uh 4.8 OPC class model they are paying less than 0.03 per million token like you can't get that price anywhere.
34:04 Yeah. So yeah, simple disclaimer is the spikes really matter. Uh it's very rare that someone has a short spike at the extreme end where they like run a workload every month at that scale and then nothing else matter. So so usually some base workload exists. It's just how big of a base workload. Yeah. Yeah. So yeah. Then the last one is check your processes.
34:33 Uh this has nothing to do with inference optimization from here onwards. Um some slides may sound that very critical of AI but I'm generally an optimist. Uh just just I just want to lay that out because things can be coded out of context. So one of the uh Pandora's box question or problematic question that we are starting to see is that a lot of organization are starting to ask hey I'm 10x or 100xing my token spend but I'm not seeing results at that same scale um resp like like we are starting to see reports from from
35:11 various organizations like uh where AI has limited productivity only has 6% productivity or 7% which is not the scale that a lot of us were promised or we envisioned and and unfortunately this is the case because we are at a stage where everyone is just taking a giant hammer and hammering everything. Um it's not that AI is not useful, it's just that we tend to just spam it everywhere.
35:37 And yeah, if there's a there are times where I just want to like can you please not do this as another co-pilot because it's just too much of it. you need to think of how you use it respectively. Uh um yeah and and another major issue that we start to realize is that while a while AIS are are being deployed at a larger scale and more frequently and people get faster in creating stuff the organization processes may not go as fast.
36:11 The first bottleneck for for coding coding environments will be the CI/CDs and you may heard that CPU becomes a bottleneck respectively. Uh another pro another thing is like is just the processes in place uh respectively. How do you trust? How do you review? I think the previous speaker spoke at length and trust. Yeah, the lack of trust is what slows things down.
36:32 And this could be rightfully so. Maybe the agent the the AI agent was badly designed and shouldn't be trusted. But it also flip inverse. What if hypothetically the that agent was doing everything perfectly but people weren't trusting it because it didn't have the test in place, it didn't have the validation and it didn't build that trust. That slows things down and these things ma these things have a much bigger impact.
36:59 So once again be very wary about how the hammer is being used because you need to make your adjustments accordingly. So and this goes back into saying like tokens are not work done. You have uh um and and one major example that I will highlight right is is this happens more often than than I would like is that sometimes spamming more agents and more tokens sometimes means you get the same task done multiple times.
37:26 So while the antipatterns that we started seeing right uh yeah well well in overall the engineering organizations improve productivity by clearing more issues quickly automating clearing of issues for example like when someone like when a QA and and things like QA teams reporting an issue and agent automatically fixing it without even an engineer those are great but one of the is one of the common issues that we start facing is that people get ambitious really fast and they instead of doing small changes they start
37:58 doing huge changes like changing like half the code basis or one quarter of the code basis. Sometimes when this is planned with proper testing is great but a lot of times right the change is happening at such a large scale right the bottleneck becomes the on the reviewer to merge the code and because because everything is done on such a large scale where a human cannot process it one of the common antiatterns that we start seeing is that okay someone has implemented the the there's issues being done and because there's
38:27 so much code being generated issues that do not have code generated do not gets review did not gets picked up by the reviewer. So someone needs to generate the code using an agent and then when the agent generate that giant pile of code which should be a lot less if they were to like properly review prompt it and stuff the reviewer look at it and it's like huh this is too many code.
38:47 Can I ask my agent to highlight what changes are relevant and then reimplement it and and and in the end they started rebuilding hand by hand with with their agent. So they literally discard the first agent's job. There's multiple good reasons why this happen. For example, like reviewers who want to maintain control and quality of the main codebase, but this is a very common example of a bottleneck where you where where if you didn't think the process through, you're just throwing more and more things respectively and
39:18 the speed is not increasing. So review your processes. A good way around this is naturally to to to instill better testing practices and so that it can automate that and filter out also reminding teams to like hey split up your your your merge or or make it more modular to make it easier for the review process like things like that. These are human processes.
39:43 It has nothing to do with the AI and yeah we tend to get too excited about AI sometimes. So this is not just a code problem. uh we are starting to see like because uh AI agents have been deployed to make decks. This deck was entirely design-wise was generated by AI. I went through multiple iterations. It's not a zero shot. I I went through every page like at least 30 times to to change the text, but the design was done by agents and and sometimes within your organization, you're starting to notice people that just one
40:17 shot prompt the entire slides, things like that. and and they start passing it around. They start doing meetings and then it's on everyone else to filter. What's the inconsistency? What's uh what's the misalignment? Why is these two data points completely different? This is also another example of like same as the previous example, people generating code or people generating slides or people generating report and the burden shifts onto some human down the road to verify things.
40:45 So be very aware about this organization change. um why am I still an optimist? I don't think this is necessarily a bad thing in itself because the problem is everyone needs to learn and this is where we just start adapting our processes accordingly. Uh and the other major thing is that well there are a lot of common examples on how to use AI. You all know your subject domain for your organization better.
41:10 And there will be use cases where an AI could be used in place to automate a process. For example, that could save your organization or millions of dollars or increase your productivity at a grand scale that that only you will know and not me as an AI expert will come in. I like to point to the most uh famous example that Walmart had where was it Walmart?
41:36 I believe it's Walmart where they literally started putting a stool a stepping stool inside every single truck and that was like the simplest change but it lowered their medical bill by the millions because it lowered back injuries across for heavy goods respectively. It it's some something specific like that with the use of AI and it may not even necessarily be the best AI model that actually can have the biggest impact.
42:02 Yeah. Yeah. So uh doubling down on my previous point uh one of the term that that was coined by uh that that has been trending is called the slop grenade. So generating lots of things AI and then leaving it to someone else. Do not let slop grenades happen. Uh and this is a culture thing and a processing. Yep. So the some guideline rules that I would set and this is more subjective people people disagree.
42:29 um to prevent the stop grenade from happening enforce that people should own what they hand over or generate. So for example these slides I own every text that is generated. If there's an issue that's on me it's not my AI agent who who who has an issue. I do not get to hand it off to an agent. I must own the responsibility. Same thing for code. Same thing for anything that is generated.
42:53 And you also have to measure outcomes not usage. uh and so this gets more complicated because I I don't want to expend too too long into this but like there there's like this simple formula that's that I think people take as a rule of thumb tokens times price divided by outcomes someone actually use and and and this help guides a lot of things smaller more efficient models faster model bigger model yada yada but what I don't like about this is that another analogy that I'll use about AI is uh think of it more as like
43:29 supplements or steroids that enhances the team. Think of it as uh like if let's say I use construction work as an example. Let's say I can use it to increase the muscle mass for the workers example. Not that it's ethically to do so. I I know that's not the case. But let's just say as an example the train of thought some teams will have it and they improve productivity.
43:48 Some teams don't. But even without that it's quite clear some teams are much better than others. and and so the AI being a supplement here right doesn't necessarily change everything so you need to be aware that outcomes outcomes are hard to measure respectively but it's something that you can start measuring and observing most importantly back to the first step measure you know the token spend per department per team and you can start seeing which team is performing better and you start seeing how much tokens they use
44:22 you can start seeing what works yeah as I said this flaw measurement. So yeah uh that's uh that's the high level overview uh respectively and if if you want uh to to like kickstart your process internally within your organization you can you can either talk to me and my team or like you have six questions for you to start asking like who team spend what what's the what what must you test to use a cheaper model what's your base load what's the default model for each team and which workload has a person waiting that you
44:52 need something faster. Yeah. and how much of what we generate is useful. These are like guideline questions for you to start asking to refine your tokconomics. Yep. So Q&A. >> Yes. >> Yeah. So when you mention models, is that >> test um >> like SFT sort of fine-tuning or is there like other sort of uh fine-tuning methods that you have depending on what specific use case there is or what data there is?
45:23 >> Uh okay. So in general it tends to be SFT but I don't think uh uh um well we do provide fine tuning services and it tends to be SFT I there is a lot of methods that are uh happening right now uh currently I know RL is very sexy right now especially in the coding environment because it's easy to generate the data in the loop but what we realize for a lot of organizations especially like in New York for example financial institute things like that there's not really an RL environment for financial task for example and
45:58 and so it tends to be more SFT based uh um and and there are also other organizations like adapt adaption AI um uh where where they are starting to also automate the the the refinement of your data sets for you to train the model be SFTbased or thinking thinker machine based focusing on this might be the wrong question. It to me generally what people should focus on is the evals respectively.
46:31 So here's a good example uh for Jeff that came when the Jeff model came out and then uh we did the open source project called simple Jeff uh which replicates the Jeff API for existing open source model a lot of people quickly realized that the quen 27B was really good at this task and and there were multiple teams that went to fine-tune the quen 27B I on on our side because we were trying to take the approach like our company like we want to accept any model and then we just build the API on top of it we started taking
47:01 approach of like hey we see the benchmark we see the existing model why don't we prompt tune it instead it's not even fine tuned and then and then two weeks later as hugging face went to do all the benchmarks at scale like over 10,000 questions and things like that somehow my prompt tune qu 27B beat most of the 27B fine tuned so yeah this this is not a flex because I'm quite sure now my model is beat by the recent cloudflare fine model that was done on a more proper basis but it's more of like the tuning method might
47:32 uh the tuning method might not be the right question. It's more like what's your evals and what's the results that lead to it. >> And a follow for that. Do you need do you need to collect golden traces for the um set of data you need if you're doing SFT or whatever it is or can you do let's just say like your your for your task the frontier models get like 80% accurate right so maybe 20% of the time it makes some sort of mistake if you fine-tune on that won't the fine-tune model also get like 20% of it wrong are you
48:02 supposed to filter that out somehow and get only the golden traces for training or uh can you just throw all the traces in um and train So this is more a personal opinion technique because uh so um the generally I try to split the data set because people are busy filter um so I try to at least get a golden trace data set and the golden trace data set may be much smaller uh respectively and we just uh and we then we fine-tune that with a much heavier weightage uh as well but there have been multiple papers uh especially
48:38 uh uh um especially um with thinking tokens that even the non uh the failed task right the as much smaller model when you fine-tune tends to benefit from the training because it actually improve the thinking process even if you got the wrong answer. Um this is something this is really a good case of like try and then measure and it works 70% of the time.
49:04 Sometimes you you need to increase the golden trace rat ratio but it's a general rule of thumb. I think for a lot of customers when they're trying to to do SFT down a smaller model, the general rule of thumb they're trying to keep is can I keep to the near same level of performance and maybe the eval is down 1%. Or even in some cases it improved by 10%.
49:23 Because the base model wasn't specialized by this task and more importantly does it 100x reduce my bill. So that's usually the motivation uh respectively. Um yeah, golden samples are critical if you want to do beyond the baseline and sometimes that requires humans to actually look at the data which as much as I love my customers to look at the data more people don't like to do that hi um how would you deal with uh code spend that would mostly come from cloud code and all the integrations there because then your model
50:03 is fully locked into the cloud an ecosystem I suppose. Do you have any advice on how to deal deal with those type of situations? >> Uh yeah, unfortunately clock code is the example of fully locked in. [laughter] So um sometimes sometimes the first movement is actually getting um developers to switch the harness. Um harness it seems to be a lot more flexible thing right now.
50:29 Like we have seen organizations that are like fully locked in like I'm only going to use devin only or m only. We have seen organizations that um that are more flexible like literally they are running their they creating their own agent harness and my response is like why why are you creating your own agent harness from scratch? But like a common one that we see people switching to is for example pi.
50:51 And this is also because there's a lot of excitement around it. people have shown that even with the same model, it gets a much cheaper cost per token because of the way it's simple and efficient. Um, and from there the I think the beauty for the PI coding agent harness is that it's easy to adapt because it's so so lightweight. We've seen a lot of teams that just start prompting their agents to make modifications and once once they have that then then they start getting the cycle of experimenting and changing.
51:19 Uh a common way to get them to be very incentivized to change is to actually introduce a a good enough model that is really fast like a 100 tokens per second and then suddenly they start giving that a try and some people start like some of the first shift will be they still use their clock code as their planner and then they instruct the clock use this PI agent with this model to do the implementation hence the planner implementation split.
51:46 It's more Yeah. And that's where the transition start to happen. It's not easy to change people's agents, but yeah. >> Um Yeah. So an example of this could be perhaps to expose this this smaller agent as an MCP in inside cloud or or having these tools where you can offload. I I suppose is that something that you would you you would say could work? Uh MCP is more tricky because MCP agents unless done right don't have access to your file system.
52:18 So typically it's just something way simpler like PI is installed and they run the bash command pi this. Yeah, that that's usually the case. >> Thanks. Yeah. Can you share if possible any case studies about uh caching solutions being very thoughtful about caching and the outcomes that have been generated for specific customers even if it's anonymized?
52:52 >> What's the specific challenge you're talking about? You do want caching you okay is your question more like you want anonymize you do wait you do not want customer private information to save in the cache. No. So, so like with customers that you've worked with, have you got any case studies you can share about the outcomes that they've been able to generate because of being thoughtful about how they cache and different sort of caching methodologies.
53:20 >> Uh I think this one we can take offline but in general for our larger scale customers we deploy them with dedicated cache. So it kind of solves a lot of PI issues because it's the the cache is only for that customer. Um, yeah, and obviously there's prompt formatting, but I may need to dig in more specifically like what you're trying to ask here. The short answer is yes, but there's a lot of way to interpret this.
53:42 Yeah, >> cool. >> Sorry about that. >> Uh, thoughts on turn or taskbased auto routing? Oh um I'm a I'm generally more towards task based um mostly because it's much easier to control expectation at a department organization level. So this this is back to like hey a department may have a label model and I kind of auto routed that model for them. >> Uh I know there's a lot of uh hype right now for turnbased auto routing.
54:17 One of the issues with turn-based auto routing is that you need to lock down your your turnbased auto routing to let's say at most two to three models in not your cache will be extremely inefficient to be used. So um so for example we have seen turnbased for JRM 513 and JRM 513 flash makes sense but if you put in 30 models and it keeps changing a model every turn your your low cache heat rate will start to wreck your your optimization.
54:46 >> Yeah that's what I've seen before. I I started like request level and it just didn't make sense and then >> yeah but two models tends to be okay. >> Uh task and organization makes the most sense. >> Yeah. >> Yeah. Yeah. I agree. All right. Okay. Thank you very much. And uh if anyone wants to find me specifically for more questions, feel free to just come up front and ask me or find me by the side once the next speaker is here. Yeah. Thank you very much.