Tokens Are the Cheap Part — Steve Jamison, Jellyfish Transcript, AI Summary & Key Points

00:02 Hi, I'm Steve Jameson, AI research and engineering leader at Jellyfish. I'm here to show you why tokens are the cheap part. The research that I'll present is drawn from the Jellyfish platform. The industry's largest set of engineering workflow data joined AI telemetry data. It's no secret that AI costs are climbing. They're up 60x from this time last year.

00:27 The use cases are wide and varied. The models and harnesses are getting much better. The implications are exciting. But discussions around budget and spend are not as much fun. These conversations involve your peers and senior leadership and the board. Not always the easiest audiences, especially when you don't have the right data. And it's real money these days.

00:51 Output is up, too. Merged PRs per engineer are up 43% over the same one-year horizon. This is phenomenal. We've not seen productivity inputs like this, especially on this time scale. There's plenty of research and anecdotal data to know that AI and AI-driven coding are driving these throughput gains. Heck, even our charts show the correlation. Notice where the inflection points are.

01:18 AI spend increased in early 2026 when throughput gains start a steady increase of their own. There's a tension with these two numbers, though. How do you reconcile them? Spend is up 60x. Throughput is also up 43%. Is that good? Bad? 43% is a big number. But so is 60x. How do you know if you're spending enough? Is there more productivity gain locked behind a checkbook?

01:49 Should we increase spend 50%? 2x more? Or how do you know if you spent too much? I'm sure many of you have felt the tension here. The CFO or board are asking for the receipts of the spend. Can you show them? Well, merge PRs are up 43% satisfy them? Also, what happens if you cap use and spend? Are you stopping from setting tokens on fire? Or are you stopping further efficiency gains?

02:21 You can ask peers at other companies what they're paying. For many, it's still a moving target though. But, how do you equate that to your org? Surely, just comparing their bill to yours is inadequate. We've heard that reports of We've heard the reports of companies spending 100k, 500k, even millions on AI. Setting then resetting their AI budgets as reality sets in.

02:48 Ultimately, these two numbers can't tell the story. And even together, they miss the biggest line on the cost side. These two numbers leave out the biggest cost of all, developer salary. AI is making developers more productive on a scale we haven't seen before. It isn't just another tool, it's intelligence. This intelligence dramatically changes how much each dev can produce.

03:15 That's why we need to think about its cost differently. Not off to the side as overhead, but it's part of what it takes to produce each PR. We're going to call this intelligence costs of a PR. What you pay your engineers plus what you pay for AI. Put them together in one cost bucket. AI is about 2 and 1/2% of the cost to produce another unit of merged code.

03:45 Salary is the other 97 and 1/2%. This is in line with other reported figures, and this data is from August 2026. Here, tokens are the cheap part. But, that's current relative costs, not dollars. Let's look at what happened to the cost of a merged PR over the last year. >> [clears throat] >> The dollar cost of a merged PR fell 26% over the last year.

04:21 Not PRs created, but PRs merged. Review time is factored in, and some level of quality gates has been cleared. What That decline in cost per merged PR includes a 60x increase in AI costs over the same time frame. The change is nearly universal, too. 83% of the companies we studied showed a drop in cost per PR. What does this tell us? The additional throughput gains are being spread or are [clears throat] spreading salary thinner across PRs, and this is happening faster than the AI costs are increasing.

05:00 What's the range in token spend? We ranked companies by token usage, then grouped them into quintiles or fifths. The range here is quite large. The lightest group, about 0.3% of the cost of a PR is is AI. This represents about 60 million tokens per dev per month. The top 20% of AI spend per dev, we're going to call the heaviest spend group. There, 8 and 1/2% of the PR costs is AI.

05:36 This represents about 2.5 billion tokens per dev per month. About two orders of magnitude more than the lightest group. We'll use these groups on the next slides to see how token spend influences other metrics. First, PR throughput measured as merged PRs per dev. Split by the same AI spend quintiles on the previous slide. The bottom 4/5 of spenders saw PR throughput gains of around 40% give or take.

06:12 But the heavy spend group improved more at a 67% increase. More AI spend led to more throughput gains. Now let's look at the cost per PR. Positive results are seen again across the board. Cost per PR is dropping for all AI usage groups. However, again, the heaviest fifth so the the companies that spent the most improved the most. They saw a cost per PR decrease of 32% over the last year.

06:50 The other four groups saw about a 25% decrease in cost per PR. Investing in spending more on AI fueled more of an improvement in cost per PR compared to peers who leaned in less. So we saw that the pop- same same as we saw in the population average, cost per PR changes are driven largely by PR throughput, not by AI spend. The heavy spenders experienced PR throughput gains of 67% compared to about 40-ish percent for their peers.

07:28 Spreading the same salary across more PRs in the month has more of an effect than the rising AI costs because tokens are the cheap part. This is not to say that org should light tokens on fire and spend with abandon. Here we're looking at tokens used on the horizontal axis and cost per PR on the vertical axis. There's a steep drop at the beginning where relative small AI spend results in large gains initially.

08:01 As token usage increases, the cost per PR improvement decreases. This is diminishing returns. I want to stress though, diminishing returns is not the same as no returns. We're seeing that the top 10% are spending 8x the tokens compared to the median and achieving a 27 25% lower cost per PR. There will be a time when tokens aren't the cheap part, but by and large for most companies out in the wild we're not close to that threshold.

08:42 More tokens leads to more PRs, which leads to a lower cost of delivery. So far we've talked about companies. Next I'd like to take a tangent and talk about a study we did with developers. Here we looked at developer weeks with heavy token use compared to the same developer weeks with lower token use. >> [cough] [clears throat] >> Here we saw the same phenomenon just at the individual level.

09:19 Compare each developers heavy agent skill weeks with their own weeks without skills. They spend about twice as much on tokens, merged 27% more PRs, and reach PR and each PR costs about 18% less once salary and tokens are counted together. There are two fair curiosities I get when I talk about this work. First, does faster work result in worse work? And second, are the more PRs the throughput gains that we're seeing, are they being driven by smaller chunks of work?

10:12 First curiosity is cheaper, faster work, worse work? If AI makes PRs faster and cheaper, are we just shipping more mistakes? One proxy for this is river reverts. Merged PRs that undo an earlier PR. The heaviest fifth started lowest before most of the token growth, and revert rates drop slightly for every group. Though these deltas are just slightly below the noise in the data.

10:42 It's not evidence of a improvement here. However, more importantly, we're not seeing any worrying signs of rampant or severe quality issues with higher token spends and faster PR throughput. My second curiosity is throughput driven by smaller PRs? And this is demonstrably no. Not smaller PRs. In fact, AI is driving an increase in line level lines of code generated per PR.

11:21 This is no surprise to the devs in the room. Raise your hand if you've rolled your eyes at verbose, unnecessary inline AI comments, or a file full of mostly irrelevant tests. Gen AI Gen AI likes to generate code. Also, the bar to create is low for devs, and there's a pull to do more per PR instead of spinning up another ticket. It's just one, maybe two prompts, and I'll handle this edge case, or I can combine this and that element.

11:49 Fewer PRs to wrangle. This doesn't mesh well with the PR review stress folks are feeling these days, though. There's >> [clears throat] >> Also, AI is just becoming more capable, so we're getting more ambitious. It is beyond this talk to dig into what's actually driving those level of lines of code changes. However, the increase in PR throughput is not coming from smaller PRs.

12:21 This is intuitive for most, but it's still good to verify. Back to the opening question. Is 60% more spend worth 43% more output? You can't tell in isolation. You should track both. They are important numbers, but they're supporting metrics. AI spend per PR is the intelligence that you're buying as a resource to help make your devs go faster. PRs per engineer is what you're getting with that spend.

12:56 However, looking at those hides the trade-off. >> [clears throat] >> At but put salary into the equation and take the ratio of cost to throughput, and there you get your answer. Cost per PR is down 26% over that same time period. It's clear AI is driving cost efficiencies despite the increasing bills. And the AI the AI spend needed to get those efficiencies is small, relatively speaking.

13:34 AI is still a sliver of what a PR costs. Two and a half percent on average. Under 10% for the heavy spender group. Tokens are the cheap part. >> [clears throat] >> In the range that we see, more spend kept paying. Heavier token users saw 67% PR throughput gain compared to 40% for their lower spend peers. This drove cost per PR down 32% compared to about 25% for the lower spend peers.

14:09 Looking at the PR cost to token ratio curve, the top 10% of company months by token usage resulted in 10 8x the tokens, but 25% lower cost per PR compared to the median. AI is making developers more productive on a scale we haven't seen before. It isn't just another tool, it is intelligence, which changes how much each dev can produce. That's why we need to think about its cost differently, not off to the side as overhead, but part of what it takes to produce each PR.

14:50 You need to add the AI spend and the human spend the human salary spend together to calculate your real costs per PR. This isn't to say spend frivolously. Sure, you should watch for runaway jobs. You should track your AI spend. I'm not saying don't look at how much AI is going how much AI spend is going in each PR. I'm saying don't let that be your driving metric.

15:22 A new framing might steer how and when you think about other investments, like integrating model routers, post-training your own model, implementing token limits, etc. Will you get more efficiencies out of trying to control AI costs or by continuing to invest in things that increase throughput, like more AI enablement? Either way, track the total intelligence costs, human plus artificial, per PR every month.

15:54 Next to that AI bill and next to the PR throughput. How are you trending? Is throughput continuing to improve? What about those cost efficiencies, the cost to deliver? Watch reverts and quality metrics beside all of this as a counterweight. We don't need AI slop mucking up the gears. So, heading into budget season, bring cost per PR to the table. Not just your token bill, not just the throughput bump.

16:25 The bill tells you what you spent. Cost per PR tells you whether it worked. And right now, for most companies, it says keep investing. [clears throat] Tokens are the cheap part to get efficiency gains. Well done. You made it to the end. Here's how you can stay up to date with our latest research, which again draws from the largest engineering workflow and AI telemetry data set in the industry.

16:54 You can follow me on LinkedIn, my colleagues, or Jellyfish. Subscribe to our Substack, or you can check out our Jellyfish blog on the website. If you want help tracking your org's engineering workflow and AI effectiveness, including cost per merge PR, go to our website and book a demo, or reach out to me and I'll get you connected to the right folks. Again, I'm Steve Jameson, AI research and engineering leader at Jellyfish. Thanks for watching. Cheers.