Building the Cloud for AI Agents | AWS CEO Matt Garman Transcript, AI Summary & Key Points

00:00 Agentic workflows tend to perform better on AWS than anywhere else. Compute sandboxes, gateways, agent permissions versus people. A lot of those are things that we have built and are building and thinking actively about. >> The top frontier labs gobble up every available GPUs. At the same time, you want to promote newer companies that are going to become the enterprises of tomorrow.

00:18 How are you thinking about balancing? >> From the very beginning of when we launched AWS, startups have been the lifeblood of the core of what we do. These are the innovators that are at the edge of technology understanding what's possible. We're very intentional about keeping capacity available for the startups. We recently announced we're going to be buying 2 million NVIDIA GPUs over the next couple of years.

00:37 >> Your capex is what 200 or >> $220 billion uh for 26 and we don't anticipate slowing down anytime soon because the demand is just massive. >> With all the debate around AI extension risk and the hugging face attack, what are CEOs asking you about all these things? Welcome to the podmat. What a time we're living in. So I have a lot of topics to talk to you about.

01:01 >> Awesome. Thanks for having me. I'm excited. >> Yeah. Yeah. Absolutely. So let's start actually right from the beginning, right? So you are the first GM for EC2. >> Mhm. >> And that was 2006, right? And today you guys are what 160 170 billion in revenue. >> Yeah. About 169 170 billion. Yep. >> Yeah. growing 30 >> 37 37%. >> That's 37% at 169. >> Yeah, it's that's crazy.

01:30 >> There's a ton of o and it's Yeah, it's it's it's interesting to think about it from, you know, we're there in day day day one when we had like, you know, the first dollar of revenue, but um >> yeah, and it's, you know, the interesting thing is we're still at the early stages of what the business can be and what the opportunity is for customers. um most workloads still, you know, there's a huge amount of workloads still that live on prem today and the amount of compute um that people are doing every single day is is

01:54 more than it was the day before. And so you see the tailwind from AI, you see the tailwind from migration into the cloud and um you know the the business has grown really fast and it's it's been a super fun part to be a um thing to be a part of. >> Yeah, it is. It is. I mean they'll be writing uh history books about this and business books about this for a long time to come.

02:15 I want to touch on the on prem that sort of boggles my mind. I obviously did my best to keep them there for a long time. >> You built a lot of stuff on prem back in the day of trying to now get all of that to move into AWS. >> Yeah. So we can talk about that later. But uh so if you think about EC2 in the early days and you got your start with obviously selling to startups and and today the AWS cloud service I mean as you reflect on the evolution what's what stands out.

02:45 >> Yeah. >> Part one and then part two we'll talk about how this nature of how you serve startups has changed. >> Sure. Well like you said actually it's funny. So I actually interned for um AWS in 2005 when we were first kind of was an internal project. It was my business school internship and my project was actually to come up with uh an analysis of who we thought AWS would be most interesting to and uh the the answer was startups like probably not surprisingly and uh and so you know from the very beginning of when we

03:15 launched AWS startups have been the lifeblood of of the core of what we do for for a number of reasons. One is the value proposition is just so attractive what AWS provides to startups. >> Um and and we we spend a lot of time and effort making sure that we're great partners to the startups helping them not just provide infrastructure but also advice and how to get your company up and running and how to think about your architecture so it'll scale eventually and and a whole bunch of things that we do for the startups.

03:38 But we also think that for us it's just good business cuz >> the startups today are the enterprises of tomorrow and and you know it's it's an imprecise number but >> today we estimate that you know maybe 30 40% of of AWS revenue comes from companies that well there's a one-time you know startups in in AWS's lifetime. So and it's uh you know and so it's fun to see for us to see the companies grow over time and and to get bigger and become enterprises effectively.

04:04 Um, and so that's why we invest so much in startups and why we pay so much attention to the the brand new, you know, two people in a garage type startups. And um, and and not um, and and not just because of the business outcomes, but also because that's who we learn from. These are the these are the innovators that are at the edge of technology, understanding what's possible, uh, pushing our services to say what they'd want more of, what could help them go faster, what could help them achieve their uh, outcomes.

04:30 And a lot of times they're pushing more than the the banks or the healthcare companies or the governments. The startups are the ones that are pushing that envelope. And it really helps us to be better and make sure that we're ahead of that wave where the banks are going to want some tech some capability 5 years down the road that startups want today.

04:48 >> Yeah. Yeah. And uh compared from then to now what have the how have the startups changed and what they want from you? Obviously >> they all want a lot of GPUs and we can talk about that but besides that >> well I will say uh there's a couple of things that have changed. One is um you know startups started at a much smaller size than than when when when we first started right they they might have gotten $10 million of funding and they had a app idea that they were kind of slowly iterating on.

05:18 Now it's like you know from day one they're valued at a billion dollars they have $200 million of funding. It's like it's a, you know, it's a a a team and an idea and all of a sudden they're worth a billion dollars which um so it's you know I think the >> they just go from our offices to your offices. >> That's right. That's right. Um uh and you know like the the size and scale and ambition of the ideas I think requires more capital.

05:41 They're they're bigger to start with. Um they're obviously more expensive too, right? To you know to um uh to go after kind of a training a model or doing something that that a lot of the folks are doing today. Um, so that's number one, which is just the size that they're that that they start from is is really big. Um, but I would say number two is uh some things haven't changed, right?

06:01 They they actually are still thinking about, okay, once I scale, how do I think about an architecture? Um, how do I think about um security? How do I think about performance? How do I think about having all the capabilities that I need? How do I think about setting up my IM setup so that when I have more than three employees, this thing is going to work and scale, which is a lot of times why they like AWS as opposed to just, you know, going to a Neocloud or something like that.

06:28 They actually need the all of those other security and and um and capabilities that that come around from um from training. Um and so I think that's that's you know that's something that hasn't changed and and is exactly the same. I think the the scale at which the ramp up um is definitely different today. I also think increasingly we're seeing um teams that want a cloud that is great to work with agents and not just with people.

06:53 Um and I think that's where we've spent a lot of time how do you think about you know exactly how do you think about broad scale? How do you think about performance? How do you think about um you know an interface that is you know a well- definfined API interface that agents can actually easily traverse and work across. Um, and so that's also something that we both I think we were naturally set up to do well, but that we've actually doubled down on to ensure things like, you know, how can you start a database in 3

07:19 seconds and and and and really get to um some capabilities that agents are excited about? >> Um, I should have looked, but I haven't lately. Have you introduced any specific new services that are explicitly targeted at agents or people building agents? I I'd say what we've done is we've optimized some existing services that that so that they can both work for people and agents and it it because you know sometimes um and and so the answer is there we definitely have some services designed for building agents.

07:49 So we have things like agent core and um and bedrock and pieces like that. But when we look at just taking an underlying component like S3, right, which is where the vast majority of companies store their data and have their data links. Um it turns out like a lot of the use cases for people and agents are similar, but you want to have um and one of the things that we have in preview right now or in in beta.

08:09 It's called AWS context that allows you to build a context layer so that agents can actually more easily find all of the data they want across your various data links. Um and so whether you have your data stored in um Aurora or you have it in S3 or you have it in somewhere else in AWS can kind of build this context layer where when people aren't necessarily going to access data in that way but agents actually are happy to go across lots of those different things.

08:34 And so there are some services that we're building like that but for the most part you also care about what does latency look like, what does throughput look like, how do you make sure that the underlying engine is actually fast, scalable, um that they actually care a lot about tail latencies which is interesting. um where people are not they don't always care about the the P999 S3 latency like agents do care and get blocked by that and so >> um that's the thing that we've cared about for a long time and um from a

08:58 performance perspective it's one of the things that um that really popped that that's why um aentic workflows tend to perform better on AWS than anywhere else. Yeah, obviously we see um a lot of companies starting out and the very common refrain of course is that uh agents are writing all the code for them, right? >> And agents are selecting the databases and >> the email servers and everything that you can name it, right?

09:23 So has there been a lot of uh thinking that and obviously they read the documentation uh has there been a lot of thinking on the AWS like your feels funny to say quote unquote legacy services that have been around for a decade on how do you rearchitect them. >> I think there so yes I think there are some things that we've thought about actually the core underlying building blocks I think are in in really good shape.

09:48 I think there's a usability layer um that we're thinking about of how we make um easier to use for we'll call it like the very simplest the the the use case you call out where somebody's just coding up an app really quickly and they ask it to deploy. We we find the vast majority of our customers will go and tell their coding agent whether it's Kira, whether it's Claude, whether it's uh Codeex, they'll they'll say like, I want to build on AWS, here's my credentials, here's my stuff.

10:11 This is how I want to deploy it. And then the agents are great and they go do it. There are use cases though where you're just you're already not on a cloud, you're just trying something. You say deploy. And frankly, a lot of times that'll go to some of our partners that have a an easier to use layer on top. >> Um which we which we love, by the way, too.

10:27 and we love um our partners on on those fronts too. Um but I do think that there um there are some things that we're doing where if you're brand new, you don't already have an AWS, you already haven't set up your IM. Um today or we'll say a couple of months ago, it was much harder to start up an account, right? You had to it's been like this for 20 years, but you have to define your VPCs, you have to define your IM roles, all these kind of things, which are actually super important things once you get to be large.

10:53 Um, and what we hear from customers is like it was a hard trade-off because they know that they're going to want those n months or years down the line, but for now they just kind of want to use the services, not worry about that. >> Um, and so what we've actually launched is if you go create a new AWS account today, >> and we're slowly rolling this out.

11:09 I don't know if it's actually full all the way out yet now, >> but um, you don't have to do those things. You don't have to give it a credit card. You can sign up with your Gmail account. um you get to just all of those things are handled in default behind the scenes and within less than 30 seconds you're up and running and can be and can be operating in AW full AWS account which is you know for those type of systems much more what the agents want to be able to do because they don't want to have to go through all that

11:32 setting up your VPCs and your >> um uh and and pieces like that. So there are some things where we're adding kind of some ease of use to that which we're we're quite excited about and I've seen some really positive feedback from customers on and and we'll keep doing more things like that over time. >> And I and just one more thing that which is the the the great part about that though is it's not like a a simplistic account and then you have to migrate.

11:52 If you basically say okay now I actually do want to scale into the or I want to develop an organization. I want to go and and kind of fine-tune some of the things. you can easily come just you're you're already in a real AWS account and you can actually then go and do all of those things later when you need them and so there's no migration or move later and so that's part of the hard work that we really think about which is how do we make it not a choice for the customer but an easy on-ramp into the depth of features

12:17 that we know customers um and startups are going to want when they start scaling. >> Yeah. Yeah. Got it. And um what has been some of the hardest things to accommodate as agents have taken over? I mean you made a name. >> This is the way you guys became the phenomenon that you became was serving developers, right? >> Um and then uh infra teams u now developers are substituted by agents and pretty soon infra teams are being substituted by agents.

12:50 >> Yeah. Are there what has been some of the hardest things for you as the largest service provider in the world >> to handle that transition? >> Well, I you know look I think um like I said much of our infrastructure was is >> yeah pretty well set up to handle the scale um which is good. I I think there's been a couple of things which are interesting which are I think interesting paradigm shifts where um you could argue that some of our systems are um I wouldn't say overengineered but um as an example many agents want

13:23 to create a database uh do a little bit of work and then have the database go away. You really need that database to have 59s of durability. Exactly. Right. And so there's some things that we rethink there where when you're creating an Aurora database like as your production database like you do want five ends of data, you know, you want you you need durability, you want availability, you want all of those things.

13:44 And for the agent use case, that's that's arguably or maybe not arguably like overengineered for for what we need. And so, you know, it's hard. We don't um and we kind of have this this uh um a belief that we don't we don't really want to have like a non-durable option that's that's going to cause problems either cuz you never really quite know if the database that's created is wants to stay around for a long time or a short amount of time.

14:07 And so we're we're um trying to do the hard work to think about how do you accomplish both of those things where you can create it quickly and throw it away and you don't really waste a lot of resources. But um but um but if you do want it and it can be durable and stay for a long time, it actually grow into a big production database. So those are some trade-offs that we think about actively as we think about how does the the the more traditional kind of this is going to be my production system kind of capabilities um

14:31 match with uh with with some of the more transient nature of the infrastructure that agents uh want to use. Um, so that that's one I think, but there's there's there's a bunch. I think the um scale and speed and latency of creation of of durable resources is another one that's interesting. Um, you know, I think the other one that we actively think about too is just are there new building blocks that agents are going to want that we just didn't really need before.

14:57 Exactly. And so compute sandboxes, um, uh, gateways, um, agent permissions versus people or or or service role permissions, um, a lot of those are things that we have built and are building and thinking actively about cuz they are just brand new building blocks. It's not using the existing building blocks differently, but it's brand new ones where pretty pretty clearly you want a different permissions for agents.

15:20 You don't just want to give it Ragu's permissions and let it go do whatever you can do. you actually want um you know very time box short-term permissions to just go do a task. You actually may not want to give permissions to a whole tool at all. You may want to get very fine grain commissions what an agent's able to do >> from a sandbox, right? You actually want a sandbox that um uh you want to >> virtualization is having its act too.

15:43 >> That's right. Exactly. That's and and they're they're different. It's similar. It's the same idea, but it's, you know, how do you have that lightweight and and think about um unfortunately we've done a lot of work with um firecracker or our kind of microVMs. >> In fact, a lot of the sandbox company startups use fire. >> They all use fire which which we invented 10 years ago maybe or something like that.

16:02 And it's it's it's really purpose-built for I mean it's not wasn't purpose-built for agents, but it's actually quite good for agents because you can spin them up really rapidly. they have a great security boundary um and you don't have a lot of that virtualization overhead that you have from a traditional large VM. Um, so you know, I think those are some some new building blocks that we're thinking about and um and more emerge every single day, you know.

16:22 Yeah. And >> yeah, >> so I've rattled enough into agents. I'll come back to it later, but it's a fascinating topic. >> Super cool. >> But uh let me switch gears a little bit and ask about something that uh every one of our startups face and we get asked most frequently >> which is how can we get GPUs, right? >> Yeah. I mean obviously you guys have running massive GPU farms and increasing it every day but uh the top frontier labs gobble up every available GPUs more power to them.

16:57 >> How do you deal with the internal at the same time you want to promote newer companies that are going to become the enterprises of tomorrow like you said >> um how are you thinking about balancing >> yeah it's capacity needs and these small companies that don't have a lot of credit and whatnot. So, well, yeah, it's a great question. Um, and and a couple of these things make it more challenging to one is um the the capex expense needed to go deploy kind of all of the compute that everyone needs right now um is

17:25 massive, right? And so we are uh I think >> your capex the same as what 200 or something >> $220 billion uh for 26. It's uh I I by by at one point I saw that it's you know it's it's a pretty I mean that is a larger expense than we've ever had maybe any company has ever had in a in a single year. Um and um and we don't anticipate slowing down anytime soon because the demand is just massive.

17:46 And so at some point you're limited by how fast can you build data centers, how fast can you deploy capital, how how fast can you get memory and chips and all of those kind of things. Um and so all of those are at different points constraints that we're you know whether it's power data centers capital memory chips they're all kind of constraints at various times construction people to build buildings like you know construction people are are at a premium today.

18:10 >> Um and so all of those we you know we we work really hard to make sure happen. Um and then with the capacity that we are able to deploy, which is still a huge amount and and and not enough, we know, um we really think intentionally about what is that allocation strategy. And so, um we're great partners with the the large frontier labs, the Anthropics and OpenAIs and and Meta and and other large customers.

18:31 And so those folks are are really good customers of ours. >> Yep. >> And uh and we want to make sure that we invest in them. we have large enterprise customers whether it's uh sales forces or or JPMC's or you know other large companies that that have demand for fewer numbers of GPUs but or or accelerators sometimes they want tranium sometimes they want um uh Nvidia GPUs um but we want to make sure that we can support them as well and we're very intentional about how we make sure that we have capacity for startups and

18:56 and so what we do is we actually do allocate and we basically say okay we're going to keep we could you're right we could sell every single um uh you know GPU or or AI accelerator we had to probably just the lab the big frontier labs and call it a day. >> Um we choose not to do that because we actually want to keep growing the the full ecosystem. They get a large number but but um we want to keep supporting >> um a broader set of customers cuz we actually think that both the whole ecosystem be more healthy for us.

19:24 There's some diversification but it's also just we know these are going to be big companies over time. Um and so we try to support them. Um I saw recently that we say you know yes in some way shape or form to something like 60% of the the requests we eventually get. Sometimes it's a little bit later sometimes it's in a different region. Sometimes it's a slightly different configuration than the customers are looking for.

19:43 Um but we really try to lean in and and and try to allocate as much as we can and every single startup that you have wants more and so you know so we're working hard um to try to make sure that we have all of those. Um, but you know, it's it's hard. Like that's that's it's you know, like this is the as a lot of people say, it's a good problem to have.

20:01 Uh, but it's it's a problem nonetheless. And um and we're you know, we continue to look at at um you know, we we recently announced we're going to be buying um you know, 2 million Nvidia GPUs over the next uh couple of years. like we're we're landing a massive amount of capacity >> and 2 million >> 2 million um so you know it's a lot um and it's over the next couple years but um and uh and who knows if that's enough but that's at some point again we're limited by other other components as well um but it is we're we're

20:31 very intentional about keeping capacity available for for the startups and I know it's it's painful to not have enough but we we keep pushing >> and when did uh your capex cross I mean pure AWS. I mean Amazon was a bigger company. >> Yeah. >> Cross like let's call it even 1 billion or 10 billion a year. >> Oh, I don't know. I'd have to go back and look.

20:58 I'm not sure about that. But it it's definitely scaled up over the last couple of years. Um uh in a pretty meaningful way. >> The the yeah the AI buildout has has definitely ramped our our capex spending. And so I'd say over the last, you know, call it 3 or 4 years, our our capex has definitely accelerated um pretty pretty meaningfully. Um I don't know when we crossed a billion, but um given we're 220 now is probably I mean we we've been spending capex for a while and the and the company has been really good about

21:22 funding that um and obviously the AWS business is is a good one that we like to invest in. >> Yeah. Yeah, of course. And um uh and you know I think we were for the longest time actually um we you know we we were investing ahead of where the demand was and >> um you know I think one of the one of the most painful things is that with the real ramp of of GPUs like a lot of the elasticity has unfortunately you know kind of gone away.

21:46 Um and so hopefully we'll get back to in our core compute and and storage and things that elasticity is still there and that but um but you know that um but but so we've been spending for for quite a bit of time and and we feel really good about the spend that we're making now. um which is you know I think I get lots of questions sometimes about how you feel about that spend and like are you nervous about bubble and other things like that and I will say you know >> we um because of the position we have like one we do

22:12 take this diversified approach and so not all of our capacity is bundled up in one customer and I think you go to a some of these whether they're neoclouds or some of the the other providers out there and >> you know you'll see sometimes concentrations of 30 40 50 60% with one or two customers >> we're nowhere near that obviously we're you know singledigit percentages at the highest and and and usually it's less than that for one I think we have a lot less risk on on one particular customer but also because we have

22:38 that rich set of services like AWS is where people really are coming to launch their production workloads and so the majority of our usage today actually is either core compute and storage and inference which is part of that application and so those are the workloads that I think just aren't going to go away just cuz we see enterprises getting positive ROI you go talk to the customers and you say at the capability today and the cost today are you seeing positive returns to your business and almost to a person they'll

23:06 say like oh yeah >> and so you're like well that's not going to go away there's no bubble in which they they stop spending on that you know and so >> you know and you know this like the VC model like is every billion dollar startup going to make it no they won't but you know that's that's kind of the game and it's been that's been true for 50 years they haven't always been billion dollar >> start numbers have changed but the principles are the principles are the same right you bet on 10 and one makes it and pays for

23:29 the other 10 or whatever the whatever the percentage is. Hopefully it's higher than one. Yeah. Right. And so that that there you know you saw that with the internet where there was a bubble and a bunch of of internet companies didn't make it and the internet's still a thing >> and and a lot of the companies that had durable businesses >> the Google, Amazon's that others like that they did pretty well >> and so for us we think we'll feel really good about that investment and um and and and the continued investment going.

23:52 >> You guys have a view of the demand >> that's unparalleled, right? because you're seeing across the globe, you're seeing across every segment enterprise and the big labs and the AI native companies and so on and so forth. So, >> if anybody should call it, you should be able to call it first. >> I would hope so. >> Yeah, that's our that's our plan and and honestly, we spend a lot of time thinking about it.

24:15 We're very intentional >> um about how we spend um you know, our shareholder capital and and we think that we're making great investments. We have a lot of good protections of how we intentionally invest that money. But um but yeah, we're we're very bullish on and you know, I think Andy's been public about saying this, we the potential for AWS is is really really large and over the next decade, the potential is there and anytime you have an opportunity that's that big like you kind of want to make you want to you want

24:39 to invest to go after it >> and as the scale of these numbers go larger and larger, right? >> Yeah. Has your planning and process dramatically changed in terms of I mean you're not writing >> Yeah. >> billion dollar checks, you're writing like $50 billion checks or $20 million checks or whatever it is. >> Yeah. Um Yeah. Yes and no. I mean I think look a lot of times we're the the we're still very bullish about the investments and and lean forward but there there the process.

25:12 So there's a lot of things that have completely changed. >> Like how do you even estimate 2028 demand? And there's things that we have to think about now that we just never had to think about. Um and so >> if you go back 15 years, >> um if we needed more power, we asked the power company to give us another, >> you know, couple of megawatts or whatever it was, right?

25:31 Tens of megawatts like and they would just give it to us cuz 10 megawatts wasn't that much. And uh and that was plenty for us to keep growing. Um now we have to bring our own we bring our own power. And so we we pay for power projects. We pay for renewable projects. We're one of the biggest uh renewable power purchasers uh each year for the last 10 years.

25:49 Um and so it's you know we're regularly bringing on new solar projects, new nuclear projects, new >> So you're talking about behind the meter or are you talking about working with the operator? >> We'll do both. Um and so both of those things and so often times with these power projects that we bring on um we'll pay the the capital and and pay for the project and then it'll go into the grid and then we'll get credit for that.

26:08 So we'll bring those on. Um sometimes we'll do behind the meter too. like it's it's a mix at the at the scale that we're doing. You have to think about all of those things. Um but um but you know that's that's planning where you're thinking you know 20 years out of how you're going to think about um power, how you're going to think about transmission, how you're think going to think about that um capital and and that's stuff we'd never had to think about before.

26:31 That's planning we just never had to do. Um but you know we always had to it's you know the the other things is when we used to think about server demand that we needed we you know we'd have you know multiple quarter demand things and we'd talk with our our suppliers and things. Now we we work multiple years out um just because the the size is so much that we have to think about kind of what do we need for 26 we need for 27 what do we need for 28.

26:53 Um, and so that's but you know, but it's also one of the value that we bring to customers, right? That's a thing that legitimately customers can't do themselves. Like they're not going to do power. They're not going to plan their memory footprint in 2028. Like they're just that they can't do that. And so that's one of the real values that we bring to to our >> broad set of customers is just that's just a whole set of things that you don't have to worry about and that that we spend a huge amount of time thinking about.

27:17 >> Yeah. Yeah. You guys are generally, I think for the record, the largest buyers of practically every component of a server. Correct. >> I I don't know that. I don't >> I am sure that we're one of the the the bigger um uh purchasers of of components out there for sure. Um and uh who knows about everyone but uh and it kind of depends on how you think about them and how you measure but um >> so where I was leading to where do you see the the constraints being most severe let's call it 2728 and where do you see the

27:54 constraints easing up? Yeah, it's funny. It's um so uh I'll answer this in a roundabout way, but um you know uh I remember that um in undergrad we actually this was this long time ago and I never thought this would be like a useful book that I read but we read we read the goal and have read the goal right and so it turns out there's never one constraint there's always just the latest constraint and so you have to think about all of them and as soon as you hit one there's another one right and so you know what do you

28:20 like what is the constraint that's going to happen in 20 and 28 like I actually don't I think there will be one. I think it's like every month for us it's do you have enough power and then as soon as power is no longer the constraint you know it might be memory it might be TSMC capacity it might be you know HBM it might be um uh networking components it could be you know there's a a blip somewhere in the supply chain and like you know connectors or whatever it is right at some point you have to think about all of those

28:48 pieces um and it's um you know it's it's not also the um uh kind of where they happen matters too, you know. So it's it may be like, hey, we have a ton of power in Indonesia, but we don't have enough in Germany, right? And so you think about kind of where in the world you want that capacity, too, cuz it's turns out that not everything is totally funible.

29:11 Some is and some's not. Um but might be disc drives, it might be SSDs, it might like we we think about every single component and we have you know tens hundreds of thousands of components all that we track and think about and um and some we rely on our you know our um suppliers to to manage some we direct much of it we directly manage and uh um and yeah that we have a whole team that does that and they're fantastic they're I think they're industryleading and and we um we kind of we we saw this problem coming um

29:41 probably a decade ago and really started not just thinking about okay how many servers do we need to track but just thinking all the way through the supply chain you know four tiers five tiers down what is the component that that could cause an issue for us and making sure that we had kind of guaranteed supply on that >> um if you think if you remember >> I don't remember when this was over a decade ago when there was like the floods in Thailand and and no one had disc drives anymore um uh and so that was >> there was

30:07 disc drive crisises and then there was memory crisis >> exactly and so you know it's like I think we think through all of those things and so we also think about where's their diversification in manufacturing like all of those kind of pieces we try to work through and um and we're we're never going to be perfect on it but there's always a different supply constraint.

30:22 >> Yeah. So um obviously there's a lot of wide ranging debate about data centers. >> Mhm. >> Right. And uh it's clear that folks like us where we stand. But do you think as an industry we have not done a good job >> of explaining why data centers are good for America and >> generally the world? But uh >> and how do you what's the internal talk >> amongst Andy's team >> on how do we deal with this?

30:54 >> Yeah. Well look I I think um and I think you'll hear more from us over this and I agree. I think we need to be more vocal and be more upfront. Um cuz we actually do a ton that's that's really beneficial both for the communities we operate in um for the you know we we think a ton about how do we bring renewable energy to these data centers. How do we think about being water positive?

31:13 >> Um actually our our our data centers use a really really small amount of water. We mostly use free air cooling. How do we think about being great participants in the communities where we are and um and how do we bring highpaying jobs to the communities we operate in? Um and not all data center operators do that. And I think there are some well there are some well chronicled examples of others out there that are not great um at that and they just >> don't really pay attention to regulations.

31:35 They think that the rules don't apply or they just you know launch really quickly without thinking about those. And I think that it causes problem for the whole industry because everybody kind of gets lumped into that. And so >> um so look I think we'll be we we I think you're right. We um you know vocally is self-critical. We need to be more vocal about the benefits that we do bring and think about additional ways that we can help communities understand the benefits that that we bring to them both for the the services

32:00 they use, right? If if you usually go to a community and say, "Well, do you not want to use Netflix?" And they'll be like, "No, no, like I I still want Netflix." Like that's it. Um and um you know like it's important for us to think about and highlight the benefits that we bring where I I recently saw a report where one of the communities that we operate in >> the everybody in that county pays $5,000 a year less in taxes because of the taxes that we bring to that.

32:25 And we don't we don't tell them. They don't even know it, right? It's just invisible to them. And so I think we just need to be more um um clear about those benefits that we bring cuz I think if you told the communities, by the way, your tax bill is $5,000 less than it would otherwise be if we weren't here, >> they might have a little bit of different thought about the building that's over there.

32:43 Um so and and but but not everyone does that. Not everyone kind of um >> presumably there's more transparency that's needed. >> And look, I think much of the data center um community, not just us, is actually pretty good actors. And there's just a few that aren't that um that >> that kind of I think have have caused some of the angst recently. And I think we just need to do a better job of highlighting, you know, who's who's being good citizens and who's not.

33:07 >> Yeah. And uh before we leave the hardware topic, uh um I want to touch upon training. >> Mhm. >> And your whole history with building your own uh chips, right? >> Um we were one of your first partners using Nitro a long time ago. Mhm. >> Uh and since then uh Graviton and you made tremendous progress. So what was the thinking that led to saying look we're going to do our own thing and then how has that progress been and where are you >> it's actually a fascinating story and I think it's a great example of where um

33:42 Amazon AWS will innovate and we'll iterate over time and continue to think bigger about what we can do but but um but kind of prove our way there as opposed to you know and so takes as an example um this probably now I 10 it's probably about 13 14 years ago we were seeing that there was a pretty significant um virtualization tax on the overall number of resources and um and we were kind of thinking about how do we our customers were telling us I want bare metal performance and they you know they're comparing to having

34:15 all the resources of a of a server >> and so the first thing that we did is that um we took a a network offload card and virtualized all of our network virtualization and pulled it off onto an offload card so that network virtualization got closer to to bare metal performance and back then it was not quite bare metal but it was close it was closer >> and then we got really excited about that and we said okay what if we could move storage virtualization off as well right and and um and none of the network offload cards

34:42 could do that and then we we found this one company who had some ARM cores on an offload card and they were doing it for other reasons I can't remember their original purpose but we're like could you use those to do storage virtualization and some of these other functions and they're like maybe and so we really iterated with them this was the product Yeah, this is the team >> and um and just loved that team like really innovative, really missiondriven, really wanting to solve problems.

35:05 >> Um and so we acquired them and we said, look, could you build us a slightly bigger card that actually could take all the network virtualization off and basically give us a bare metal server that has no virtualization on it, no VM virtualization, everything is through APIs on the card. And because we had this view that one performance would be much better, resource resource utilization would be better.

35:24 Ruby be we security isolation >> and security isolation would be much much better and we tell people you know could then legitimately tell people we have no access to any of your VMs that are running there >> and and um and this has been a huge benefit for us for the last decade where frankly like we've been leading and and others have been kind of slow to do this because they this is not a generalized thing that people can do >> but um so we got to that and we basically said look we're we're making a lot of progress

35:51 here um what if we take you know And there's a bunch of ARM cores that were on this offload card. And we said, "What if we turn that into a server?" >> Uh, and we did that first with Graviton. It was a very underpowered um very small server that we launched. >> Um, and and customers were excited. They're like, "I'd love to have an ARM server. This is super interesting."

36:08 And so we went down the path and um and Graviton um you know and part of what we did is we looked and saw that there's the the slope of you know ARM cores were getting faster and where the you saw the power utilization and and the graph where they and you knew the intercept was going to happen for where this architecture was going to be really good for for parts and they just needed somebody to drive the ecosystem and and get some of the pieces in place.

36:30 So we did that with Graviton. Um and Graviton's been a a runaway hit at this point. Um, >> have you been public about what percent of your fleet is? >> We land more graviton ships uh uh every year than um uh than than any other um type. So it's it's it's very popular. Um and it's um you know look we they're 20% cheaper at a 20% better performance and have been like that for the last kind of five six years.

36:58 So that's a it's an easy value proposition that um and I think the vast majority something like 90 plus% of our top 100 customers all use Graviton in some way shape or form across their fleets. Um and so that's been a huge win for us and for customers it's been it's the single big single easiest way that customers lower their bill is to move to Graviton.

37:17 >> Um they can often we've we've have examples where people have moved their whole fleets and cut the number of servers they had in half. >> The performance is so much better. >> It's amazing. And so half a number of servers, each server costs less. Like it's it's a it's a big win. And so then about five, six years ago, we said, you know, we saw the rise of of AI um compute happening.

37:35 Um not nearly expecting what it was today, but still saw it was going to be a big material mover. And so we went in, built our first chip in and Tranium, and uh we're now um you know, in market with the third generation in Tranium 3. And um have seen fantastic results. So, it's, you know, we're we're sold out um uh for capacity through probably maybe towards the end of next year or something like that.

37:59 And and and we we're trying to again we're trying to figure out how we can save some capacity and get startups to be able to use some of the capacity cuz it's uh we see great results. So, most of you know the the the majority of of traffic on Bedrock all runs on Tranium. Um and we have um great deals with both Anthropic and OpenAI to build on top of Tranium.

38:16 um as well as a a a set of um of smaller startups and you know I think we have half dozen to a dozen startups that are building on top of tranium now too >> and uh >> so now the name suggests it's a training chip >> yeah we're back >> but everybody's using it for inference too >> so where is the lean architecturally and where is it going >> it's a good point um look the vocally self-critical we're terrible at naming and so it's not it's not our strength originally we had a chip called inferentia for inference and

38:45 training for training for training And then as the models got bigger and bigger and it turns out you actually want to run the inference on these really large systems, it turns out it turns out that tranium is actually maybe the best inference chip on the market right now. >> Um from a a per absolute performance and cost performance point of view. >> Um and so um >> is it better memory memory bandwidth?

39:07 Where is the >> it's just it has it has um it's just architecture is a little bit different than than um uh than others and it's much cheaper. Um, and so from a cost performance perspective and absolute performance perspective, tranium is great. Um, uh, and so we're, you know, we we use it a ton for, um, for inference and, and, and that's going to say, Bedrock, um, it drives much of the Bedrock um, uh, inference today.

39:30 And, um, but it's also a good training chip, too. And it's, you know, I think we're we we, you know, a lot of our broad set of customers usage is not in training models, but is in using it. And so that's where a lot of people get to use it under the covers, and that's where we're excited about it. But um but a lot of the big customers are interested in it for training clusters as well.

39:47 And particularly as you get to tranium 3 and four um uh which we've announced we haven't we haven't launched trainium 4 yet but announced it. Um folks have kind of looked at that architecture and said like yeah that's the future of where I want my training clusters to be as well. So we're we're quite excited about the future where that goes for these really broadscale training clusters also.

40:04 >> But um but yeah it's built. >> Yeah. Now uh let's get back to uh talking about agents but from a perspective of uh large enterprises or large medium enterprises. >> Yeah. >> Where are they in their adoption? Um >> and uh have they what sort of benefits are you seeing them >> reap already and what is the roadmap for them as far as you can tell from your vantage point?

40:30 >> Yeah, it's it's a really good question. And I think it's one that um that we've spent a lot of time thinking about. And when I talk to customers all out there um today, they view, you know, they're getting a lot of value out of what they've done today. And I would say the agents that most enterprises have built are are relatively simple and straightforward.

40:50 Um and they're starting to think about um and they're um mostly >> non-aututonomous, right? They're still kind of people in the loop, if you will. And so I think we're and and by the way, there's customers are still getting lots of value out of that today. And so they're really thinking about how do I have these be autonomous but in a safe way. And I think there's two things that that I think hold customers back today from just continuing to to scale.

41:13 And it's it's already a pretty big business today, but I think it's it has a massive opportunity to really change every single customer out there and every single workflow and and really thinking about it. And so number one is just how to think about it. I think what we originally saw was that enterprises had a workflow and they're saying great I would have the enterprise I would have an agent go do the same workflow.

41:35 >> Yeah. >> And what we encourage them to really do is think not just replicate you know Bob does step 1 2 3 4 5 so agent is going to do one step 1 2 3 4 5 and then Bob's going to check it at the end. That's not really the model you want. You want to actually step back and say if I want to accomplish something how can an agent do it differently? It can do it in a massively parallelized way.

41:53 can try 50 different things and get to that and and how do you help it get to that right outcome and rethink how a computer would solve a problem versus a human solving a problem. And so one of the things is us just helping customers understand how to think about that and really kind of have that blank slate because that's where you really get value is not just replicating what you're doing today but but thinking from a green field approach about how you go solve a problem completely differently and I'm sure that's how

42:19 many of your startups are thinking about this too which is how do you help customers green field solve a problem not replicate the thing that happens today. >> So that that is number one >> people running fleets of agents swarms whatever you want to call them. Yeah, >> very common these days >> and and and you just want to think about it, you know, and and so enterprises are not as as again this is one where you learn from the startups and you try to how do you apply that to an enterprise world where they're, you know,

42:41 an insurance company is not necessarily as forwardleaning, but they would love to figure out how they can have a better um you know, approval workflow or something like that. Um so that's number one, but then the second one is this this how do you turn those into fully autonomous workflows and how do you actually trust the agents? And so we're spending a lot of time thinking about how do we build services to help enterprises feel like their systems are secure and that they can trust an agent to make a decision that

43:08 they can have the right guard rails that it can have the right permissions on their data that it's not going to delete production systems that it's not going to make tragic mistakes. And right now I think that nervousness is probably holding people back maybe appropriately by the way is holding enterprises back from just saying okay go nuts. It's like you don't actually want an agent to just go crazy and accidentally delete a production database.

43:27 That's going to be pretty bad. >> Um and so you that's how we're we're kind of actively working through this with customers on how do we both help them architect and frankly invent new technologies and capabilities that are going to help them solve that problem. And so that that is one of the areas where I think we'll continue to innovate and um and and um and we'll get there.

43:45 I think we have some really good ideas um and and some good technologies brewing that I think can really help. So are enterprises learning how to do eval systems and so on and so forth to keep the agents they need help honestly like you like both like evals how do you have a constant loop of testing? How do you how do you think about um you know goal seeking in a reasonable way?

44:08 How do you have your data labeled in such a way that it actually even makes sense that the eval can actually kind of um approximate what you're going to be doing in production? How do you measure in production and and back test it so you're not seeing drift? All of those things are problems that that um enterprises don't know how to solve today. I shouldn't know if anyone really is great at solving these today.

44:26 Um it's why you've seen so many FTE teams kind of spin up and and AWS is um and our partners are really leaning into the the FDE motion to go and help and this is the single biggest area where customers need help and and when we think about how you you you and my view is we want to train our customers to be able to go and do this themselves, right? This is not the traditional notion where I want to have a people-driven business that goes on forever where you just keep paying consultants over and over and over again.

44:58 our our view on how FD should work and and and is um we want to go into a customer who's ready to accept the you know to really kind of accept owning this when we're done and in 45 days do work where we can teach them how to make an eval teach them how to get their data in a labeled way do work alongside with them and then at the end of 40 to 5 days you leave and that customer is is good and ready to go and trained up and that's what our customers tell us they want they don't want to be beholden to a external workforce

45:25 for the next five Yes. >> And um but they need help today. >> Yeah. You made a massive investment in >> FTEES. >> Um so taking that even one step further, right? Some of your industry peers have said, look, you can't have all of your data going into a big frontier model. What enterprises should really do is to take an open source model and then post train on your own data and workflows and phrases and whatnot.

45:55 >> Where do you stand on that? Are you seeing customers actually trying to do that or do you guys >> how do you how do you think about that? >> It's it's a great the first point I I wholeheartedly agree on that first point like the customers and like enterprise data is their most valuable asset. >> And so from the very beginning it's why we built Bedrock like we did.

46:10 Um we have a guarantee that your data never leaves your VPC. And so your if you're running inside of Bedrock um your data doesn't go back to the model provider. They never see your prompts. um that stays inside of your own trusted environment. Um and so that that is why um enterprises kind of run they they prefer to run on top of bedrock and it's why you see that business growing massively like hundreds and hundreds I mean it's every um every month we see that just business just every week we see that business

46:40 exploding and and it's why um you see open AI workloads migrating to bedrock. It's why you see Anthropic really growing really rapidly. And so whether you're using open models or um uh or or um closed frontier models. Um I think bedrock is a great solution that our our customers told us by the way like you know we if you remember 3 years ago I got a lot of heat.

47:02 >> Speaking of bad names, that's a good name though. >> Yeah, Bedrock is a good name. That's good. But we we got a lot of heat actually for being slow to the AI world because we actually built the foundations of this where we said look we're not just going to rush out of service. We really want to think about how do we make sure that we protect our customers data and build a service that we think is going to be durable for the use cases that we knew about.

47:20 And if you remember we got a lot of heat and we said look we're going to go build the right thing and now as people move from proof of concepts to production vast majority of them are landing in AWS on bedrock for for much one of the reasons is because of this. It's also because of the the set of services that we have. We also offer open models. We offer proprietary models.

47:38 Um we offer um a whole set of capabilities around those agent core. We build these building blocks so it's easier to build agents with any of the models that you want. >> Um whether they're in bedrock or out of bedrock for that matter. You can use Gemini or other things for it. But um but I think that's a it's a it's a differentiating piece for us and it's a super important thing to think about because >> having that data go back into the the model provider I think is a is a dangerous thing.

48:02 You talk about open weights models though. I do think that there's a and it's an area that I'm excited about. um uh we're we're really ramping up our support of open weights models and and trying to build a good um environment and and frankly this is where today I think and I think it's true a lot of customers believe that they have meaningful proprietary data that if they could mix in you know do some post- training do some fine-tuning um um to an open weights model that they could distill down they could actually get

48:32 a better performing model at a lower price. Um there's a bunch of pieces in here that have to work out well. They actually have a good eval to actually prove that that's true. Most people are doing that in SageMaker today. I think there's more that we can do to make that easier. But actually like if you go look at where are people doing that, they actually do it in SageMaker on AWS today and they actually host the inference via SageMaker.

48:51 >> SageMaker is getting a new lease of life as a >> uh I mean it is it's it was always kind of a model building platform, right? And now you know there's if if you think about what enterprises are doing in this world that's what they're doing is they're basically effectively building their own custom models. And so that is SageMaker is is a great place of doing that.

49:07 Um and I think there's some things that we need to keep building on to make that easier and easier to do >> um and to test across different open weights models and things like that. But that's a um it's a it's an evolving space. I think it's a super interesting one and it's one we want to make sure that we have all the right things for um for customers to be able to do if they if they have the the right data and expertise to actually go down that path >> and with all the debate around AI extension risk and this that

49:31 and the other and the security um uh vulnerabilities and so on and so forth and the hugging face attack. How are enterprise what do what are CEOs asking you about all these things? Yeah, that there's a bunch and and they look they they mostly want to say like and it goes back to this like when I launch agents, how can I trust that they're going to do what I want them to do?

49:53 And so we're in increasingly >> we have been for a while like we're basically we're heavily investing in building capabilities that allow people to deploy agents safely into their environment and think about those controls. And some of those are how do you make sure the agents have the right permissions? How do they have the right sandboxing? How do you make sure that you have the right set of guardrails?

50:12 How do you how do you really intentionally think about what you want the agents to do and not do? Is there a human in the loop or not? And so we we spend a lot of time with our customers thinking about how do you think about safe agent deployment and get better over time and um and what other capabilities do we need to go build to help people deploy agents safely into their environment.

50:30 Um so there there's a lot there. Um the other a the other angle on that which um a lot of people are worried about which is just um are some of these really powerful models going to be um attack services and and and kind of mythosphere um and so I I kind of have a view on there they're yes that is a real risk I think to to customer environments but it's also um a real opportunity and so we we recently launched a service called continuum that uses these powerful models to help customers go secure their environment and

51:02 So, we'll look across our environment, uh, look for vulnerabilities with them. We'll use some of these powerful models and and, um, um, and help customers find vulnerabilities they haven't found before. And most importantly, by the way, prioritize which ones cuz we know context about their environment, how it's set up, where their permissions are, where they may have compensating controls that that make it harder or easier to do.

51:23 Um, and so Continuum is incredibly popular with customers. Actually, we're um um um really bullish about what's possible from AI to help with AI powered security cuz look at at some point customers are going to need security at machine speed, not at at human speed, not at like an alarm someone goes in look at it. >> And so that you know we're running fast to go help build that for customers to help them protect their environments and and I'm very excited about the the continuum team is building on that front too.

51:51 So within AWS itself >> Mhm. >> like how what's the state of adop usage of agents in the broad and >> well continuum is basically us trying to expose what we do internally and so we we use uh AI extensively for our own security. We use AI extensively for our own software development. Uh we use uh agents actually one of the things that's really cool is we use agents across our entire business.

52:18 And so we we rolled out um Amazon Quick to every single Amazon employee. And now I see HR teams building agents to to help drive what what used to take teams of people weeks to do that a single person can now do in a couple of hours to think about kind of team planning and and and resource management. Um I have finance teams that are building um agents to to go think about how do they go pull tax rules from everywhere and ensure that we have compliance on a bunch of different pieces.

52:46 And um super cool to see that things that used to be blocked by software developers >> actually the the the line of business folks are able to go and and unblock themselves and and innovate more quickly. Um and so Quick has been an enormous blow and and that has has grown like wildfire. see um customers like small startups using it all the way to the the largest enterprises in the world rolling it out to their entire customer base to get the benefit of kind of being able to access all of your enterprise data and easily

53:14 apply agents and and capabilities to to help you accelerate your jobs. And so I said we use it across uh everything from software development to security to um to to driving HR policies. I mean you guys are notorious for measuring every everything about your operation. Where have you seen the biggest gains? Yeah, it's I mean obviously you know the answer software development like that's the real answer >> the derivatives of that >> but yeah I mean the software I mean the speed of software development has been uh and

53:41 and and really product development uh as a whole not just coding >> actual >> absolutely the the case at which we're deploying new products is is um is massively different than it has been in the past. I think you can see this where you know AWS has always been known for rolling out features really quickly and uh and it's you know we've we've seen a turbo boost on that in the last year or so as as we've get we call them frontier teams as they think about agentic development as opposed to kind of traditional development

54:09 and it's you know it's not code completion it really is agent first the agents write all of the code you're just managing a team of agents and and driving that um and it's it's been fun to see the the the pace at which um innovations for customers has has been massive and um and it has to be cuz that's the that's the our customers out there have a a almost insatable appetite for new capabilities and and that's what we got to do.

54:36 >> Yeah. So organizationally are you like do you have any insights on how organizations should change and how >> I don't uh I'll say that >> agents manage people manage agents >> uh there's like there's going to be there's going to be lots of people for a long period of time. I do think organizations will change. I don't know the magic answer yet. Um but but we're actively thinking >> inside you're just trying various experiments or what?

55:01 >> We're trying experiments. We're thinking about pods. Um as you think about you know here's one example is in a product organization you used to have a team that would own a particular kind of capability for a long time and you might have 10 people working on that thing. Well today you can innovate so rapidly that one doesn't have to be 10 people.

55:18 It can be three to four people >> and they build something so fast, you actually want to move them to different projects and and problems. And so thinking about how do you both operate and and maintain the things that you built while being agile and flexible to move around on an organization that's as big as AWS is as active things that we're experimenting with and playing with.

55:36 But it's um it's fun and it's enabling for our employees. They actually love it because they can build faster and do more. But um but there's real work there. >> Yeah, it's a fascinating time and uh so thank you very much for your time. We could be talking about this for hours together, but uh thanks for all your insights. >> Yeah, thank you for having me and uh and thank you.

55:53 We we we love having uh all your uh your companies as customers and uh we love learning from them and uh I appreciate having me here. >> Yeah, we'll keep sending them your way. >> Excellent. Thank you. >> Thanks.