Why No One Agrees on What Their AI Tokens Cost - Pratyusha Duvvuri, Cisco Transcript, AI Summary & Key Points

00:01 [music] >> For 7 seconds, cuz longer than that you'll sleep, um close your eyes and I want you to think about what tokenomics means to you. Humor me for a second. Close your eyes and think about that. Okay, that's about 7 seconds. Now here's the thing. I'm willing to bet a million dollars that no two of you in this room have the same answer. Whether it's hi there or Christy there, like nobody has the same answer here and I can tell you this with confidence because we surveiled at least 100 different customers/companies.

00:49 We talked to engineers, we talked to PMs, we talked to engineering managers, CFOs, like the whole spectrum, right? And the variety of answers we got from yes, just about observability, right? Like how much what is my token costing to know we want a full financial breakdown of the ROI on every single team. It was a pretty crazy wide spectrum. This is probably a good time to tell you that I don't have an answer to what tokenomics really is, but I can humbly try to give you my version of tokenomics.

01:24 It's probably also a good time to say hi. I'm Pratyusha. Um we well I was at Galileo, which was acquired by Splunk, which itself was acquired by Cisco. So we have a Russian doll situation. Um and I was a tech lead for agent observability, distributed tracing, most recently this crazy topic, tokenomics. And while I'd love to just jump in and show you a product, I want all of us to sort of sail together on this crazy journey.

01:48 So let's come to the same page on three trends in AI. And if you dispute me, like please do. Like I'd love to hear from you after the session. The first one is we all know tokenomics was definitely like a status symbol of some sort, right? Or token maxing, sorry. Um and what I mean by that is our friend here Jensen literally said, this was in May if I remember correctly, if a 500k employee is not spending more than or at least around 250k in tokens or token spend, I am terrified.

02:20 Now though, like just yesterday or well I woke up this morning and saw Twitter, there was a post from Eric, I forgot his last name, the CEO of Ramp, uh one of them, and he said, uh our our CFO doesn't want me to post about this, but I'm going to do it anyway. We spent 1.5 million dollars in one week and he freaked out. So there there's this trend towards let's talk about efficiency, right?

02:46 And that's also probably a good time to segue into the second point here, which is before let's just use the soda model for whatever task, right? I don't care what, I'm going to use Astra Opus 5.5, but now it's more about okay, how can I get the most efficiency out of my tool calls, out of my model usage? And there's a nuance here that I want all of us to think about.

03:08 For example, at Galileo, well now Cisco, um we had this internal use case where we using a cheaper model, like a Luna model, which gave us 96% in Evals. It was an internal use case, right? Um just for fun, we ran Evals on like Astra instead, like GPT-6 Astra, and our Evals jumped from 96% to 99%. Did we change the model to Astra? No, because I don't care, I want to save money, right?

03:33 When I'm inside the company. But if I'm trying to do anything external facing, if I'm trying to talk to my customers, use an agentic like harness to get the best utility for my customers, I'm willing to take that 3% hit for that extra, you know, like boost of customer satisfaction that I will get. The third trend here, and this is probably the most that the biggest one that people don't think about, which is that before OpenAI, Anthropic, Cursor, you name it, whatever model provider, they would tell you, okay, here's

04:04 our cost per credit, here are the models, this is our license plan. Thank you very much. We don't do any customization. Today, as we talk to all of the spectrum of like telcos, fintech companies, I mean, you name it, everybody has a different like agreement with these providers. For some people, it's 4 cents per credit. For some people, it's 3 cents per credit.

04:24 For some people, it's, oh, up until $30 we won't charge you anything, but the moment you cross that, it's 10 cents per credit. Everyone has their custom way of defining like costs for tokens, which is especially why tokenomics starts to become important. Um before I jump into my version of that definition once again, we, as I mentioned before, surveyed a 100 different companies and different customers in each of these different companies, right?

04:50 And I will tell you, it is extremely hard to bucket all of your ICPs into just three categories, but we had to try. I mean, there's only one way you can build a product, right? You got to have some version of ICPs. And what we came up with are these three. Number one, probably the most that some of us could relate to, the engineer, right? Number two, the engineering manager.

05:14 And then the third one, completely something that most tech folks don't think about, I will admit it, the finance folks. So the engineer is worried about, okay, well, uh what models should I use? How much should I like I have a $10,000 budget for this month? Am I am I blowing past it? Like where am I on that spectrum? What skills am I using? Should I use like Sonnet instead of Opus?

05:34 You know, whatever it is. The engineering manager has a lot more concerns. That person is like, okay, I have let's say an observability team and I have the product team. The observability team is getting me results, but they're also blowing past their costs. The product team doesn't seem like they're getting me, you know, the KPIs that we promised. However, the costs are lower.

05:55 What exactly is happening? What's the trend? That person is worried about like broader usage and adoption. Um and then the one I you was completely orthogonal, finance folks. These guys are worried about budgets, about alerting. They need to justify to their bosses or the CEO or the board or whoever is there, what exactly is the ROI on this spend, right?

06:17 And the problem with tokenomics, I think I read a really interesting article like again this morning um on X. It was uh some guy was saying that token usage is sort of like you're driving a taxi with a meter and you have no clue like what the taxi is going to show you at the end. It's sort of like we landed on 2 million. Oops. Um very different from prepaid prepaid plans, right?

06:41 So we're worried about all three of these folks. And now is a good time to talk about okay, well I I hear you but we already have dashboards. Like I log into my OpenAI portal, I can see a dashboard, I can see skills used, I can see my cost, I can see maybe a projection. Sure. But that's just OpenAI. Then you have Anthropic, then you have GitHub Copilot, then you have like four other providers and then how are you having a one-stop shop to see all of that information?

07:08 Plus if you really think about it, I'm a power user of those dashboards, right? Before well my my own token tokenomics product. So um before that I had to a shuffle between all of these and I'm an engineer. I'm not like a finance person or an engineering manager. So you're not really servicing all your ICPs with those products even though they're good.

07:27 Plus something I'm going to touch on in like a couple of um a couple of slides is that there's this crazy process called reconciliation where a lot of things that OpenAI might have said oh you know we have 100k uh that we're charging you for this month might actually not be true. They might come back to you and say oops it was 120. It's just our recon process caught it.

07:46 So we need a way to catch all of that and like give you the one true like honest place for you to see all of your usage across all of your providers. Um oh also that last bullet point, finance needs spend mapped to ownership. Yeah that's that's a little different. See in in most companies you have your management structure, right? You have the workday structure.

08:04 So it's me, my boss, that person's boss all the way up to, in my case, Chuck Robbins. Um, that's not the way finance sees things. Sure, it's the same guy. You have like one CEO, right? But the way they map out the finance hierarchy of a company based on OPEX, COGS, whatever they're thinking about is very different. And their their notion of ownership is very different of like from Workday or my manager, like that notion of ownership.

08:29 If that was a mouthful, don't worry. We'll come back to that in just a second. Okay. This is a good time to talk about my definition of tokenomics. Tokenomics is the observability and interpretability of token usage and cost. I kind of just like shied away from uh uh more complex definition there and tried to hide behind these crazy words. But if you really think about it, what is interpretability?

08:54 It's ROI. It's uh trying to figure out like which what is the set of models I'm using. It's trying to figure out the proper usage and categorization depending on your use case. The point is that these words are very like vague, but when you really jump into like your ICPs and think about what they want, that's when you can think of what the best product to build is.

09:17 So um before I actually show you the demo itself, I also want to talk to you about what we need from you. So let's say I'm going to show you my product like you know, you're going to use tokenomics tomorrow. It's going to be great. Before that, the first thing we're going to tell you is hey, we need a Workday hierarchy from you or ADP or whatever like tool you use to store your like company structure or tree, we need that information.

09:42 The second thing we need from you is we need your API keys and org IDs. So for OpenAI, Anthropic, whoever you're using for your provider, we need that information. Why? Cuz we're going to talk to them every single day to get that data and then service you like the latest data on like a cadence of one day freshness. And the third one, the the really gnarly one is the finance hierarchy which differs from company to company.

10:07 And it's it's honestly crazy how finance people deal with that. Um, okay, let's jump to the product. By the way, fun fact about workforce hierarchy, people are very sensitive about what data they share, which makes sense, right? Like they're not going to share their entire like workday history. So what we've noticed when we talk to customers was we just want to share with you like the names, the IDs, the job functions, but I don't want to tell you anything else like tenure, duration, etc.

10:33 So based on your use case, we have made our workflow so simple that we literally accept a CSV file. We're like, bro, just send us a CSV file with like these six things. If you want to give us more information, we'll slice and dice and show you more stuff. But at the very least, we need these six things from you. Anyway, back to the product. Oh, I I had to do this thing where I blurred the information because I didn't want like to get fired by Galileo or Cisco and they did not give me permission to show you the actual

11:05 numbers. So sorry about that. But apart from the actual numbers, you have everything else in terms of the product showcased here. So as you log on and if I am running out of time, feel free to like pause me for questions towards the end. Um, you log in to the Tokenomics dashboard and the first thing you see here is different ranges to filter, right?

11:25 Like you can filter month to date, you can filter by like weeks. There's a fiscal calendar for finance folks. Notice that on the top, this is right, yeah, top right, you see the leader view. This is not for finance folks, although they can access this view. This is for the engineers and the engineering managers. Um, as you see here, there is the cost dashboard and the adoption dashboard.

11:50 Cost is, well, as the name suggests, how much money am I spending, the tokens, etc. Adoption is very different. It's hey, is is my team adopting these AI tools? Are they is the Codex usage increasing, decreasing? Like you see these like red numbers here, trends. That's the trend over like whatever time range you've chosen for adoption and cost increase and decrease.

12:12 And then this one is actually one of my favorites. We got this after talking to customers where a lot of folks mention the biggest stresses for both engineering managers and finance folks is forecasting. People don't know like are they going to blow through their budget in like a week or a month or like what that looks like? And so this ghost-like graph that you see here, this weird like opening graph um yeah, that's the projected usage.

12:32 And if you think there's a lot of variance, I agree with you, but that's also because that just is how the AI world is today with like new things coming out, new models coming out and cost. Um so if we look down here, this is the probably the most colorful part of the product where you actually see the cost trends over each day, right? But then here we have Chuck Robbins and everyone under Chuck Robbins uh as like direct reports, we see all of their usage roll up.

13:02 So I think I'm under this guy G 2. Let me see if there's a slightly clearer view. Okay. So G 2 is our CPO, Chief Product Officer. I report to him in some way and my usage is under G 2 Patel's usage, right? But and G 2 Patel obviously reports to the CEO, so we have that. And if you like scroll a little past, let me see if I can tease my way here. Yep, you see that G 2 has people that report to him and and it goes on and on and on and then you can very fast because that's very important for user experience see each

13:32 person's usage in their team level and also like an individual usage as you keep going down this this path. Um another really cool thing and this is just me continually going down until I reach me and people who report to me. But what's also cool here is you see the consumption by job family. Think of my manager's manager's manager, right? That guy is responsible for a product team, a sales team, and uh the engineering team.

14:00 That person wants to know okay, well uh is my sales team using more tokens or have a higher cost than my engineering team? That's crazy, right? Funny enough, that was actually one of our customer stories where like sales was using more than engineering and that was a little little worrisome. Um but that's exactly the kind slicing and dicing that we provide to you, right?

14:17 With this sort of like one-stop shop. Um rest of it is more intuitive, I would say. Like yeah, you want to know your costs per model, per thinking pattern, high, low, medium. Uh you want to be able to filter by the person, their job function, their total costs, etc. I want to I want to spend more time on this, but I know we're running out of time, so I'm going to move towards the finance view, which again, as someone coming from a pure engineering background was a little crazy.

14:44 Um but once you really understand how finance works and like the cogs and OPEX reporting structure, it's it's not that bad. It is also a hierarchy of some sort. So we have the same thing here for finance. We have um their finance hierarchy costs map, uh a tree with different nodes. And then all the way at the end, you hit departments. And when you hit departments, all the way from someone like me to Chuck Robbins, the CEO, everybody is part of a department.

15:13 And so everybody's cost rolls up into some node in this tree. And then you have a financial controller and an analyst, and I can keep going, but that's going to like make you guys get a little lost in my explanation. But the point is you have strict RBAK. You have strict viewership rules. Like if I'm a manager of team A, I should not be able to see team B's usage unless I'm an admin.

15:33 We take care of all of that in our product where you only see what you're supposed to see. Um though what was left in this demo was honestly just me talking through the settings, but for the purposes of interest and time, we will skip that part. Um this is a very very evolving product. I know I spoke through a lot, but you guys will be alarmed if I say we went from zero, like talking to customers, right?

15:57 That that zero to building a full-fledged product and shipping it to GA in 10 weeks. That was an insane sprint, off the record, a lot of weekends burned. But it is what it is, and we're getting paid, so. Um I will say that with our customer like outreach, we're getting a a of like requests. As I mentioned, forecasting, that's something that 's not public yet, so please don't go talk about it, but it's going to come out in like the next couple of weeks.

16:21 Um, we have ROI inference. So what how do you tie business outcomes, whether that's customers landed, migration completed, a new product that's attracting some eyeballs, how do you tie that to token usage? And I can keep going on like all the feature requests that are coming, but I want to spend at least a little bit of time on the challenges we faced.

16:45 So the way I would describe economics, right? At this point, I think you'll understand me when I say this. You're riding or sailing, I don't know what the right word is, a ship where you don't even know the destination, and it's a big ship. So it's very hard to like steer your product that you're building for scale when you don't even know the destination because it's an evolving subject.

17:05 And the engineers in the room will be able to empathize with me when I say imagine you have like 90,000 like people that you're servicing, right? You have this like data model built. You you're like so ready. And then you get a crazy new request that completely changes that. And now you have to account for like 10,000 users accessing your platform while also accounting for the new product users.

17:24 It's just very hard to do that when you are like a nascent product, but we managed to do it. So I'm pretty proud of the team. Um, the second challenge I will touch on here is the three different audiences. I know I made it sound hopefully a little easy when I was talking to you about engineers, engineering managers, and the finance folks, but if you really think about it, the true challenge of a product, especially in the age of AI, is not just building something and like shipping it, right?

17:48 The true challenge is how do you make it so simple and dumb and easy to use that everybody just gets what they want from it. That's very hard to do when you have crazy different ICPs. And then the third of many challenges is the thing I mentioned before, reconciliation. It's like this data is not constant. It changes over time, especially the finance folks in the room can empathize with me here.

18:10 And how do you like fetch that data, the new data that was so different from the old data that you promised you would not change, but that is just how providers are. Um, not just that, your people come and go, right? I might quit tomorrow. You might like have a new manager tomorrow. Like, things can change. And how do you find a nice way to loop that in to your visualization of like attribution?

18:31 It's It's a pretty hard problem. With that being said, I know we're towards the end, so I want to leave at least, I will, 20 seconds for questions. And I also want to end with please keep thinking about this definition of tokenomics. Like, what does it now mean to you? Thank you. >> [applause]