Why Building an AI Agent Is Easier Than Deploying One Transcript, AI Summary & Key Points

00:00 If you want to build an aircraft, you need to procure thousands of suppliers. Someone sends a confirmation of like, hey, sorry, like this part is going to arrive 2 weeks later. And if they missed this email, hundreds of millions of damage. >> Procurement historically may have been more in a box and now it's touching legal, it's touching finance. It's touching a bunch of different software systems and people.

00:16 The opportunity for the AI native startup is to say, we're going to own that entire endto-end arc. >> No company and no enterprise starts with fully autonomous negotiation agents from day one. Why? because they don't trust us and they don't trust the technology from day one. And by having this unit in the loop approach, we have feeding our agent with all the feedback and all the learnings and then they suddenly trust us for like 10k negotiations, 20k negotiations, 100k negotiations.

00:41 >> When you think about what a durable vertical AI company looks like, what are the qualities that you look for? It's really really hard to forecast your moat going forward. If you look back at all the best businesses at the early stages, they were Welcome back to the A16Z podcast. I'm Elena Burgerer. And today I'm joined by Sema Amble, a partner at A16Z, and Vlad Kyle, a co-founder and CEO of LEO, which builds AI agents for enterprise procurement.

01:12 Sema, you recently wrote a piece called the incumbents are coming and it asked a question facing almost every AI application company. If an established software vendor already has the customer and the data and the model can work across its tools, where does a startup have an advantage? And um we have Vlad here and Vlad can really help us understand where a startup has an advantage.

01:37 And so I think a a good place to start is the cases for and against the incumbents. So if a company can connect a capable AI agent to the software it already uses, where does another application actually uh have a use case and an advantage there? >> Yeah, let me back up and frame it up a little bit. So historically we thought about there's the incumbent and then there's the startup and they're they're fight they're you know it's a it's a fight between distribution and innovation which you know to to co to take my

02:11 partner Alex Rampel's phrase. Um however now there's like this third piece which you're you're you're pointing at which is um the incumbent can layer on um one of the models on top or and and then there's a much more formidable competitor in the market. And so why do you need an AI native startup if you've got cloud force which is taking cloud plus Salesforce and putting the two together and then you know you've already got all your data and your your customers uh sorry your employees are all used to using the product

02:44 so why another product um I absolutely still think there's obviously still a case for uh the AI native startup and it really centers around the fact that um the legacy incumbent is limited to their system of record and that record that they have and they're not completing the endto-end job. So let me put that like more concretely in an example. Um so say you're um a customer and they customer calls and says they got charged after they got cancelled.

03:14 that um resolving that cancellation history isn't just the you know customer going into the chat and saying hey um I got uh I got overcharged and the response there it has to hit billing it has to look at all the chat history it has to look at the contract that's not one system of record that's the knowledge around that customer and everything it touched and that's something that one system of record wouldn't touch however the opport opportunity for the AI native startup is to say we're going to own that entire um end

03:49 to end arc that endto-end arc. So that could be in legal. So owning everything from uh brief all the way through trial. Um well I can talk more about procurement but it's it's really this concept of owning the end to end end to end work. >> Yeah. Uh Vlad, do you want to talk about just kind of where where that does show up in procurement? Like where a existing incumbent plus a model just is insufficient and what have you seen just with the companies that you work with?

04:19 >> Sure. So like when we think about procurement, I like I would assume that most people think about like prices, right? So like what is the end price that we negotiated on? And surprisingly the the record looks always very very simple and very easy. it's just 8K that's the result for example and um I mean it's the same for sales right so um even if um if I come to Sema and tell her like hey we we now partnered up with another enterprise um look at the signature here for like this contract it looks like very easy but

04:48 like Sema doesn't see like all the work behind it right so like there probably like 30 stakeholder meetings happened 500 emails 20 excel sheets and um that's the same for the counterpart procurement right so you see in ERP system, 8K for aluminum. Um, but you don't see that maybe the um the supplier did like a push back and asked for like um 10K. Um you don't see that like a cost engineer had run like 3 weeks of Excel sheets and 3D modeling to find out the prices of the um of the part and everything else.

05:23 Um so that's what we see. like most of the work um in procurement actually happens outside of this um ERP or any system of record. Yeah. >> Yeah. Um, and I'm sure just in in the workflow that you described, an agent can do, you know, a huge number of things and different kinds of agents can do a large number of things, too. And I actually I think that's a good bridge into the next question, which is like, you know, a year ago, I think a lot of the incumbents were releasing chat bots and that was kind of the extent of

05:55 what you would see. But Sema, in this piece that you wrote, you you lay out four different kinds of agents. retrieval agents, which are kind of the the chat bots that we're familiar with, process agents, policy agents, and principal agents. So, can you just walk us through all of them and explain kind of what changes in the kind of judgment that's necessary across all of them?

06:18 >> Yeah. So, a year ago, um, I I I made this meme, which was the slap on a chatbot strategy, which is essentially like all the incumbents effectively had a a chatbot that sat on top of the system record, which you could chat with to retrieve information, maybe do some light analytics. And that really um was in that first bucket of what the retrieval agent is.

06:37 Um, and maybe let me walk you through an example of what each of like the retrieval, the process, the policy, and the principle. Back to the like customer um the the customer support example just because it's very it's easier to understand. So imagine if you're a customer and there's a service outage and you're calling in uh to say, "Hey, I want to get compensated for this service outage."

07:00 the retrieval assistant, which the incumbent may have, is going to be able to pull up, yes, you know, there this was what the contract term said and um yes, there was an outage and just verify that information. It's pulling up information about the customer. It's in the database and it's just like sharing it back and and maybe synthesizing. The second agent or the second step in the agent um sequence is the process agent.

07:25 So that process agent may be able to pull through an approval of credit and say okay um based on the our policy handbook um it was out from these dates. Therefore we are say you are entitled to um money and it can just apply the bill and just do process. There's no judgment involved. Then as you keep going to the policy agent, so the policy agent isn't just going to um apply the process, but it's going to say, "Okay, uh in this situation, it was out for 20 minutes.

07:59 That's enough to be considered a significant outage." And they're applying that judgment because there isn't really like a strict definition around it. And then the last stage, which is the principal agent, you're actually weighing um you know, okay, should we offer more compensation because that was a pretty terrible outage and we want to reserve the relationship and it's worth doing more beyond even what a process or policies.

08:25 But the importance of these four things is that most incumbents started out in if at all in bucket one. Now, at least they're marketing that they are moving towards process and policy, meaning they'll be able to apply more judgment. And you know, if you look at what they've they've launched, um these are workflow agents that will help you get a document signed or um input information from a transcription or things like that.

08:51 They're very much still limited to, I would say, retrieval and a little bit of process. They've not gotten into more judgment. And I can get into why like there are all sorts of incentives uh that are preventing them from that. But the incumbents are trying and I think what they're not able to do they're like some of them are think okay let me partner with open AI anthropic one of the labs and try to build out um take that model capability and complement what they have um and superpower it supercharge it I should say.

09:22 >> Yeah. Well do you want to say why sort of some of the incumbents are holding back? >> Yeah. Okay. So there I would say they're holding back but they are held back. Okay. >> Um yeah I I'm sure they they want to be full force but there's probably two pieces. One they have this advantage of distribution right they have the customer trust which enables them to then sell more products with the customer.

09:44 So take Salesforce when Agent Force launched uh it was very easy for customers to say yeah I'm going to sign up for the Salesforce agent especially if it was offered at almost no extra cost. Um, and so they have this trust, they have the distribution, it's often pretty seamless to turn on the product. The flip side is there's all these internal incentive issues, right?

10:04 Which is if you start getting into more complicated agents, it's um it's a it's an internal conflict with an existing product that's offering workflow versus you're resolving the work. And those are two products that are um like if you're solving a customer support issue end to end versus providing uh um a workflow for a support agent, a human agent, those are different buyers.

10:29 How do you um you know those two teams are in conflict and then on top of that I think from a sales perspective like what are you selling to the customer? And then oftentimes in the classic like incumbent issue, right, is like there's two uh two VPs, right? and they're different orgs and they're selling different products and like they will never be able to uh figure out what the the right set of incentives and the right person to sell to.

10:54 But anyway, so there's all these sort of classic incumbent issues I think that the incumbent also runs into. >> Makes sense. Um Vlad, where across this this kind of spectrum of retrieval agent, process agent, policy agent, principal agent, where where does LEO sit? >> Yeah. So like we span across like I would say like all of those categories and it really depends on the complexity and the risk um our agents take right.

11:19 So there >> sometimes we um can like we already have use cases where we run fully autonomously um sometimes you have the human in the loop. It really depends on like the budget approval how like how how complex and how how risky this is. Um but maybe like coming back to what Sema said about trust for for for the incumbents like you talked about like internal trust.

11:39 I think there's like also an external trust thing, right? So like how do you convince someone to go through from hey just like an agent that retrieves some information and maybe runs processes to do to do like something completely autonomously is like obviously there's like a product component to it but mainly is a people component. So they need to trust you.

11:57 Um and like we as a startup scale up we have to earn the trust. Incumbents already have the trust but this also means they like they can destroy the trust if they ship something like too early and the product maybe doesn't work or it's bad or like works um decisions um in in a bad way. So that's um that's that's what we we can do. Um and then interestingly or what was like really surprising for us like those process agents um actually became kind of like a side quest um for us because it like when we talk when we look

12:29 at the invoice process um or like invoice agents as a process. So you we we retrieve some informations um you match this across like other documents and then you push this back to um SAP or Oracle like a very clear um clear process um and then we figured out okay that's like 100% of the software market right so that's that's invoice software um that's how you build it today but it's actually only like 20% of the work of the job to be done of the problem because 80% of the problem is like um what if like what does the

13:02 invoice is fraudulent. What if there's like a mismatch? Like what did what is like if we don't take the happy path? Um and so we like um very fast shifted to to the next step of agents like doing those doing those exception um um exception handlings. Um and we convinced the customers by um I think we got like lucky being like always slightly ahead of the curve.

13:24 So we were able to to pitch the next generation of agents. So for example like when we started out 3 years ago it was like just retrieving a document like not impressive at all today but like three years ago this was in like crazy impressive. Um so we pitched this to customers we find out okay that's a real problem that's a use real use case they would pay for and then we were able to ship this some weeks later.

13:47 Um and in the same way we are doing this now for the next step for process agents and for like fully autonomous agents and for longrunning agents. um we are pitching this to them, finding out it's a problem and then we're able to ship this very fast. >> I think an interesting point on trust is this internal versus external trust. The other lens on that is yeah so you need your customer to buy the procurement software and trust to use it for their internal processes.

14:12 But one of the really interesting things when we first met Vlad was that they're doing they're also doing the negotiation. So you have to trust that the LEO agent is going to then interface with a third party and uh and there is that piece of trust. Um and of course I think a lot of people feel burned by the incumbents and like you know they're pretty limited and haven't been able to do what they've marketed they've had in the past.

14:36 So but putting that aside like I don't know maybe Vlad I'd love to hear a little bit how you convinced the customers to to trust an AI agent to now take on negotiations. >> Yeah. And can when you do that, can you also just like paint the picture of like what's involved in procurement and who who are your customers? Um and and what kind of sort of legacy systems are they used to?

15:02 >> Yeah. So like when we um when you think about procurement, you like maybe just think about like purchasing or like you when you look at like B2C world like just you buy something. Um but actually it's like a very intense process process which like runs the economy, right? And it includes multiple stakeholders and a lot of stakeholders and a lot of um um a lot of departments um legal, cost engineering, um obviously procurement, finance um all of them have to work have to work on this um um on this decisions and

15:35 essentially like when we um like I think I think the reason is like how how we convince them is like what I um already said said earlier on. So like we we're like always a little bit ahead of the curve. Okay. So we knew the technology is coming. So we like even before CHBT came out um like we we just started a few weeks before like this CHBT breakthrough.

15:57 Um, so we already heard all of the all of the problems. Then we hear the the hype in the let's say like tech bubble and we were able to pitch this enterprises and we quickly figured out okay like this is a this could be like an interesting use case like just chatbot applications or retrieval agents um document processing um and then we we are able to find out the problem and then ship this quickly to them.

16:19 And then obviously like this is like the people factor of like trusting. So we we telling them something about and then we're able to really ship something in production. Um but then there's also like this product perspective where we have um a lot of like evils in place, right? So like we um the like no company and no enterprise starts with fully autonomous negotiation agents from day one.

16:41 No one does that. Um why? Because they don't trust us and they don't trust the technology from day one. Um so you have like a very easy approach of like um having a like human in the loop and this is like extremely helpful for us because we see like we have like let's say like the perfect negotiation agent which is like overall the perfect procurement negotiator but we don't know exactly how a fortune 10 enterprise like the specific fortune 10 enterprise operates and by having like this human in the loop approach we

17:11 are feeding our agent with all the feedback and all the learnings um and then they suddenly trust us for like 10k negotiations, 20k negotiations, 100k negotiations. Um, and then you also have like other um other agents that are like more longunning. So when we talk about like multi-million dollar negotiations where you analyze complex 3D models and technical drawings um there we on purpose have always experts in the loop, right?

17:36 So there's an agent running for multiple hours and then we ask for feedback of the cost like cost engineer and then it does the next um the next work um and so on. Maybe just to double click on that, how do you what where do you put the human in the loop on the like negotiation side? Uh you mentioned the cost engineer, but like if I were you were going back and forth on a deal, is it mostly around the like uh you know data for you know something like cost engineering or is there anything else where you have humans in

18:04 the loop there? >> So again depends on the level of negotiation right. So like we have to distinguish between like um negotiations where you just like a negotiation where in u like where enterprises they never did those because they didn't have the capacity but by by by deploying agents they can just um capture savings that they were unaware of right so they they like before agents or before Leo they just didn't care about everything which happened below 50k right so you can just like this is maybe a hack for like

18:33 other startups you can just send um an enterprise an invoice for 40k they will probably not negotiate because they don't have the capacity to do so. Um um except they have leo agents then we are going to negotiate um against you. Um but other than that they are just like just um just paying that. And obviously there the risk of like like what is the risk of like you don't you didn't ne negotiate at all.

18:55 So what is the risk now of having a bad negotiation agent? Nearly zero, right? So um maybe we miss out on some negotiations but like it's better than nothing. Um but still um in like in we're talking about business relationships and business relationships are not always about the cost and the money, right? So maybe you're not spending a lot on the vendor.

19:16 Um but maybe you you're like you need this business relationship, right? So good example might be like podcast or marketing services. Okay, that's like probably like a a friction of a um of of the spend, but you don't want to like you have a clear business relationship with with someone like setting up the studio. Um and you don't want like like a random some random people doing that because they already know how how A6 operates and how you want to record all the stuff.

19:44 Um so there we have human and loop um approaches um where like procurement people care about the relationship. So they care about the voice of tone and and how it works um but mostly autonomously um and then we have the other set of agents where we always have um a human in the loop approach and it's like um there like multiple steps in like negotiating.

20:06 It's also like it's not only the price, right? It's also like um how is the contract design? So we talking about legal, how is like um collection design? So we we talk about finance obviously like cost structure. So we're talking about really like cost engineering. Then we talk about commercials. Um that's procurement. Um and those are not back office people.

20:29 Those are like highly trained um people where they have very specific knowledge of a very specific process of a very specific company and very specific industry. um and they feed are those longrunning agents of LEO with with those insights. >> That's another example of how procurement historically may have been more in a box and now it's like okay it's touching legal, it's touching finance and it's touching a bunch of different software systems and people and both specialists and more generalists.

21:00 >> Yeah. Um, can we can we map this on to a specific uh c like not a specific customer but a specific vertical like I'm a drone manufacturer. I manufacture humanoid robotics or something like that. Like how many parts do I have to you know order and procure? How many factories am I touching? How many suppliers am I touching? Just all of all of those things.

21:24 if you want to pick maybe Vlad a vertical that um you know is is just managing all of this complexity with Leo and and just kind of take us through what their experience is. I think that would really help just illustrate exactly um just everything that you touch >> again like when we like when we as Leo when we talk about procurement of purchases like we don't talk about like laptops and pencils okay like we think like that's solved also by Leo agents but that's easy we solved this like three years ago um we talk about

21:54 like when we like you want to build an aircraft or robots or drones or even like we now like doing a podcast about um again like AI hype but even AI needs to be built right so you data centers. Um, and like building means procuring like someone needs to like if you build an aircraft, you need to procure thousands of suppliers. You need to build a factory to build this airplane.

22:19 Um, and like really small frictions can have like a crazy impact, right? So like there's a like if you're running a very large project of like building a data center, building an aircraft, if there's like one specific part which arrives 2 weeks later, this can have like a damage of like hundreds of millions of dollars um and postpone uh and postpone the project.

22:43 Um so like all of all of those that's why like all of those decisions have to be co coordinated. Um and one part is like you you need to figure out like what you need with what suppliers you work. um what are like what is like the best supplier to um to like to to get this part. And then once you decided all of the stuff, there's like all this operational um back office um stuff behind it which might sound like unnecessary and boring but again like operational means someone sends a confirmation of like hey sorry like

23:11 this part is going to arrive 2 weeks later and this is like one of 500 emails um in the Outlook or Gmail um of a procurement manager and if they missed this email hundreds of millions of damage done and this h like this happens regularly because the only thing they store in their um system of record is then just the date, right? So like it will like not like not this Wednesday, next Wednesday.

23:38 That's what you see in the system, but you don't see like like maybe that's okay, but maybe that's a $100 million damage and someone has to decide that. And that's that's also what our agents are doing, right? They are not only retrieving the information, they are then making the decisions um or like does this have an impact? What kind of impact and how can we resolve this?

23:58 >> Yeah. um when when you are sort of so deeply embedded in the physical world um what what kinds of you know physical world problems can you intervene with like some things I would think are just like unsolvable you know like let's say you have a shipment coming in and a bunch of stuff like falls off the ship or the straight is closed or whatever like they're there they're they're kind they're things that like you can't do and obvious Obviously there are things that you can do.

24:29 So, so where where can you intervene and where does that really make a difference? >> But but that's like actually it's about like probability, right? So obviously like you can't like you can't change if um if like some like if there's like a damage on the ship like every every example that you manage like you um you can't um change that but you can if you have like all of the context you can predict that.

24:55 >> Mhm. um because you can predict like how um how reliable is a supplier. Okay, so they're like there are ways on like protect the goods that you're shipping. Um and if you have like all the context, you have like one supplier where like 20% of the goods are missing and then 1% of the good is missing. Um and maybe like this one with 20% is like 10x cheaper.

25:18 But for this use case, it's fine for you to pay 10x the amount um because you have like a higher probability that this thing actually arrives. Um so and this is the powerful thing because you have like context not only within one enterprise. Um so we like we talked about like multiple stakeholders but there's also context on the outside world, right?

25:37 So just the agent should like have context of all the news out there. um maybe even having like context of like some bets like on poly market I was like okay those disruptions are going to happen um then like um information about like on the supplier side on the seller side on the demand side and by combining all of those contexts um I wouldn't say that there is a limitation um in the long in the long run obviously um like today we have like different sets of like probability but we can we can help throughout the

26:10 process and this is what we are building um building at Leo right so it's much bigger than just procurement inter company it's more like intra company and like how um how are like businesses like how enterprises are doing business with each other so like buyer and supplier side >> you describe LEO as a as a multi- aent system so can you describe what the different agents are doing >> one level is that we um that Leo agents spend across like all those four categories that that Sema mentioned um in her in her article and

26:42 um it again like depends on the um on the risk and the complexity. Um so we use like all of them. So like multiple agents um but the other thing is that like in in order to do a job end to end those agents need to share information um with each other. They need to do this in like a very specific order. Um and when we talk about like a multi- aent system this is essentially what we are We are solving like the task end to end and because the also like human level task involves eight people, eight stakeholders um and

27:20 maybe like three departments and five different um software tools. Um we need to cover like all of those to to do like the job end to end and those agents need to then communicate with each other and only with a multi- aent system you can do a job end to end. Um when we started out we um like started off with like an retrieval like more like a co-pilot obviously like 3 years ago but then the next step was like a single agent but then we very quickly discovered okay like that's like you can't solve an um you can't solve

27:49 in like negotiation even um without having a contract um agent without um maybe like having having an agent looking at the news and everything that I described before. Um so that's what yeah that's what we define as a multi- aent system. So if you have like a bolt like an airline company needs to procure a bolt for say you know Boeing needs to procure a bolt what exactly is that process for procuring the bolt and like where does LEO step in on that process?

28:19 >> Yeah so like this is like one of one of the um one of the purchases where that we can like run fully autonomously and we can do this because of this multi- aent system. So like first of all someone has a demand right? So they need to um somehow communicate it and even like this part is extremely complicated. Um so you need to call someone maybe you like you like you open up your laptop because you're like a construction worker.

28:44 You open up your laptop only every second week. Um and now you are required to like work with SAP um or any other like um EIP system. So you can't even like issue the demand. So this is like how we make it very easy. So you like you you you take a photo like you you you upload a quote an Excel sheet um and that's actually everything that you that you should know about procurement like no one cares outside of the procurement department about like categories GL accounts framework contracts no one cares um we in the

29:14 procurement will care but no one outside that cares and then our agents take off and they're like they check the inventory they find out okay they they ask another plant okay can we like can we source those um those bolts internally no okay then I'm calling the um am talking to the sourcing agent finding out do we have internal suppliers do we have external suppliers then some like another agent has to draft the RFQ send out the RFQ over um over email then bunch of emails arrive some of them are like completely

29:43 nonsense some of them are like just in the email some of them are PDFs some of them are Excel sheets we retrieve those informations um then we do like the next step oh maybe like based on our price benchmarking there's an opportunity to negotiate Um and then then we have like agents that essentially decide on the next step. So negotiation could mean strategic negotiation with a human in the loop.

30:04 This could mean autonomous negotiation. This could mean auctions and e auctions calling then the specific agent doing the negotiation. Um and then like doing this like end to end think about like confirming the order, shipment tracking, um invoices. Um and we we are able to run this like fully autonomously capture like all the context. Um and then obviously the next powerful thing is do this for like more complex um parts where we talk about like direct procurement um where we um where we also operate.

30:37 Yeah. >> What what's uh direct procurement? Yeah. So like um essentially like everything I just described is um the main goal is here automation, right? So you can like run this process fully autonomously and then throughout the process you can like generate even more savings, right? So it's not like so we look at this like okay what is like this like end to like job to be done?

30:59 How does it look like? So like what are they doing like thousand times a day but actually they want to do it like zero times a day. We've like run like fully autonomous agents, but there's like also opportunities of like what are they doing zero times a day, but if a business would do this thousand times a day, that was that would have a crazy P&L impact.

31:16 Autonomous negotiations on um spend they never negotiated before. Um so, so this is um and this is like in the indirect procurement part like think about um MRO parts, building a factory, the bold example that we did. Um but also laptops and pencils, marketing services, someone who needs to build up this podcast studio. Those are like all indirect. And then we have like direct parts.

31:40 Um this is like when you build an airplane. Those are like all the suppliers that actually um that you actually need to build the airplane um or to build the drone or to build the robot. Um and then we don't talk about 50,000 suppliers. We talk about 100 suppliers or 2,000 suppliers maximum. Um, and those are like extremely strategically important. Um, and you have maybe on one supplier like 1 billion of spend.

32:07 So you don't want to run an autonomous negotiation. You want to run a negotiation which which takes 3 months and where you're like crazy prepared and where you have engineers on your team analyzing, okay, what's the indicy for aluminium? What's the indicy for oil? Um, how did the price change? Um, so you like really take over like all of those drawings.

32:28 you check the quality um of this part and this is what like where it gets like really exciting deploying deploying agents. >> Yeah. And and for something like that presumably like you'd have the expert engineers and the other procurement people kind of more as the front of house and like the agent is more back of house. Is that the idea or is the agent like actually it's like you you sit across the table and you're shaking hands and it's like the robot instead of the human who's like negotiating like um is so yeah is

32:58 it is it more back of house or is it like still front of house? It it's it's obviously more back of house, right? Because you like because you like >> need like these complex multi-million dollar negotiations and and that's that's again like a beautiful example. 90% of the work is preparation. >> Like the end result that you see in your system of record is like oh instead of like 1 billion I paid $900 million.

33:21 Okay. But like there's like three months of preparation and like 10 people working full-time on that. Um and obviously this is like happening in the um in the back. Actually we have some use cases where um it's also like h like helping in real time. So think about um let's assume we would now have a negotiation and I have like a pro perfect preparation the same as like with like those um those those notes um those notes that we are having here um imagine like while we are negotiating um I would have like real time

33:52 insights on my screen popping up um where you tell me the indicy for oil changed 10%. So like it's increased by 10%. So that's why we need to increase the prices by 10%. And I would have like an initial like an an immediate popup with like that's true like oil increased by 10% but the product um has only 30% um of oil contained. So like you you can you shouldn't increase the price by 10% but maybe only by 4%.

34:19 Um so yeah there are like exciting use cases also like in in the real life um part. Yeah, we we know that you know companies like Harvey and Dakagon are really fine-tuning models now. Um what what kind of underlying models do you use and and how do you approach things like fine-tuning or or harnessing? >> Yeah, so like we believe like you can so like we use um multiple models from like all um all providers and we really see this as a like obviously like as a commodity, right?

34:50 So they like have um really good like general business purpose. So like reading creating a PDF and like creating creating Excel sheet like all all of this stuff. Um but we also believe that like for some use cases you um you you can get extremely far with like combining the foundation model with a with a hardness and you can maybe reach like 100% of like the job to be done.

35:15 Um but there are also some use cases where you can have like the best foundation model, the best harness, um whatever that means, but like um and the best harness, but you still can get only to 80%. Um and like I would say like good examples for that is like for example what um when we talk about negotiations what like cost engineers doing, right? So they're like analyzing drawings and then they're like defining okay what should this um this part actually cost like and that's why it's called like should cost modeling.

35:49 Um and there's um there is definitely like an opportunity where we like thinking and already started um um fine-tuning the model to get then to um to get them to to 100%. And this part another example is like price benchmarking um where think about like the the like you would have a quote and in a perfect world you would just drag drag and drop the quote somewhere and you would get the perfect price but it's ex like and all those like all those informations they're like not publicly available right so there's all like

36:26 those are like all proprietary data based on like one um one enterprise or like across multiple enterprises so like general purpose models can't train their models on that. Um, so what we are like what we are thinking about is like maybe like not training just an LLM, but I think like what we see now with models like Jeff um or so popping up where you have like an um you train it on like text data but the out like the the output is actually like an outcome um or just like the perfect price.

36:55 Um, and you can't do this with harness because like if you would give me like an um a quote from BCG and a quote from McKenzie, they they could do like the exact same work, but this could be like a 10x different price and I would have like no idea like what is better. But if you give this to a procurement manager, he would like initially have a gut feeling or like okay like this quote makes sense.

37:19 Um or like I think I think like good examples again like content creation like always like I don't know like how much I should pay someone for creating a video. Um but there's like a gut feeling behind it. If I ask like another video creator of how to um how to do that but if you ask them to write down the rules they can't do this because it's like just like gut feeling and instinct.

37:40 So, and that's where we see a lot of opportunity actually um training an agent but not maybe like a classic LLM but more exact like again like what what companies like um or like models like like Jev are now doing um on the outcome based so we have like price benchmarking short cost modeling. Yeah. >> Um Sema uh we've talked a little bit about how labs are really moving into industry specific work or working with incumbents to do this.

38:06 um when you think about what a durable vertical AI company looks like, what are the qualities that you look for? One is around, you know, owning the endto-end work that we're talking about, building up this data asset and being able to do um something that uh hasn't been done before. In many cases, this is all said, I think when you talk a lot about moes, it's really really hard to forecast your moat going forward.

38:36 If you look back at all the best businesses at the at the early stages, they were they were just thinking about, okay, I'm winning customer trust. I'm selling more to them and there's a lot of opportunity versus, okay, I'm going to do these six steps and then get to the seventh step and then we'll have a moat. And so I think we we talk a lot about defensibility and durability.

38:55 And I think part of that is you're locking in the customer. They there's more dependencies, they find it valuable, and you're doing more of the work. And here it's it's truly like okay, you know, old CRM company was uh log for all the deals. New sales AI agent is actually owning a lot of the sales prep process and the outbound process fielding inbound and doing a bunch of the work.

39:19 um the comp the overall customer is dependent on that product and that's like a really important signal of getting to the moat. And everything we talk about in terms of um in terms of stickiness and network effects and all that is sort of downstream of um of of that initial like customer use and the value of the product. >> Yeah. Um Vlad, have have you had conversations with customers or potential customers who ask you, you know, why should I buy your product?

39:50 Why can't I just, you know, plug into a model and do this myself or like use whatever existing system of record I have plus a model? Um like like what what do you tell them and and how do you how do you convince them to to use Leo? >> Yeah, 100%. And and that's a very fair question, right? And like even if you look like internally at LEO, so like the the first use case 3 years ago which like kind of like went viral in the procurement world was like just like having a quote and then getting this information into SAP.

40:23 So like very like again like technically like but like like tremendous business value. Um and so you um you have like this retrieval agent um getting like all of the information putting it into SAP. We like this was our like first product um and we had an engineering team like obviously small just like the three of us or maybe like four people building it um and then selling it but this is nowadays a case study um if you if you're applying to work at LEO so we give this to people like to build this and they have like 8

40:55 hours to do so. So what I want to say by that is like a product that we like one of our first use cases um can now be somehow built by engineers within 8 hours. Um so because it's very easy to build stuff nowadays. Um so obviously there's a question okay well so someone can build this within eight hours. Okay cool but then couldn't like just procurement department partners also just build everything in 2 months.

41:20 Um and the answer is like yes you can build this in 8 hours and you can build this but you will only reach 70% let's say like the of the performance. Um and the problem is 70% of performance or accuracy or however you measure it. It depends really on the task doesn't mean 70% um automation, right? So it's it's this can be mean that you're like have 70% of the performance but you still need to do 100% 100% of the work um because 70% is not not that much.

41:52 So again like all of the people have to um have to check the data. Um, so maybe you even created like more work um more work than um than you did before. >> Two things to to layer on to what Vlad just said. One um overall it's good if there's more adoption of the base models or just like GPT products because it means that people are also willing to trust um vertical specific products as well.

42:22 So I think that increased uh familiarity, comfort, excitement about AI tools is generally just good for the market. Um the second thing is I was chatting with the management team of a Fortune 500 company two or three weeks ago and um one of the things they mentioned was they had tried to build out their own cash collection uh product. This is a big enterprise business and they like after I I don't know three or four months of work at a minimum they had found that there wasn't enough context.

42:53 There was poor quality context. They had a lot of recordings, a lot of screen grabs. Um they tried to pull it all into one system, but they there was no um there they wasn't good enough. Uh and then there was this giant question around okay like you know we've got now two different ERPs and we were about to acquire another one. Um who's going to update all the mappings test the you know test out okay does it work?

43:19 Um and then like we keep talking about exception handling. You now need to map that onto a totally different system a different way of doing things. And I think they quickly realized that the internal build didn't make sense. And so we keep hearing stories of this where people are like okay I'm going to do the internal build. and they're like, "Wait a second, it's not different from what DIY's ever been in the past, which corporates have always tried."

43:37 But, um, I think enterprise companies generally realize that like there there's their core competency and then there's, um, building internal tools and, um, they should focus on the the first camp. Yeah. Yeah. Makes sense. Um, >> yeah. So, that's exactly what I what I've meant with like obviously like they can they can get to 80%, but those last 20% really matter.

43:58 um and you can only get like they matter to get into production. So that's why you need like this harness, right? So you need all those like integrations memory workflows and sometimes you need vertical data um um to do that and like like like the parto princip so like those 20% can make like 80% of the of the effort um or like they are making 80% of the effort um so like to all like those fortune 500 150 companies so you can do this um but then let's say like procurement and workforce of AI procurement agents should

44:35 then become like one of your core competencies. Um, and you should evaluate whether this makes sense or not for you um to to have this in as like your core skill. >> Um, Vlad, I'm curious if you're seeing suppliers start to use agents or AI at all and kind of like what happens when both the buyers and the suppliers are are fully AI enabled. >> Yeah.

45:03 So like we 100% believe that like in the future there will be like agents on both sides. Um and um this this makes so much sense. Um but like surprisingly what we what we see is like so like when we look at the supplier side this also kind of like equals the seller side right. So what we see is like that the the sales side was like always ahead of the procurement side.

45:23 Um but what we're now seeing with those suppliers for this Fortune 500 companies um this actually is not true. So they are like maybe advanced in like let's say um video recordings and like using tools like granola and all all on all the stuff but like not like really um having agents deployed they're automating the work um and the cool thing is is like procurement is like is unsexy right so sales is sexy procurement is unsexy but it's like one process and procurement is the the counterpart but now the good thing is in

45:56 those industrial companies for 5000 companies Procurement has the bigger power to the supplier because you as an like typical use case like automotive supplier you dictate to your suppliers what they should use what the quality has to be um how they have to how they have to answer to an specific RFQ. Um so the opportunity is now if we serve the procurement department and then can and they can dictate what a supplier should use.

46:27 um why are we not like not like why aren't we like also pushing them to like LEO agents um that are also helping them automate the work um owning then both sides of um of the transaction >> and Elena I know we were talking earlier today about you know how can you have two parties on the same platform is how does that work um I think it could even extended to like legal work right and and these are two very adversarial parties right but like if you have two law firms with clients with different interests, but both

46:59 benefit from knowing, okay, here's the latest draft. Here's where we are with the open issues. Here are things that have been agreed upon. And just even tracking that that doesn't doesn't really exist right now, right? That's all being human that's being created by um humans. And so being that coordination effort an agent could be doing. >> Yum. Yum.

47:18 It's well, it's super cool to think about how like, you know, both sides are kind of maybe evolving in tandem. one side might be going a little bit faster as you're talking about Vlad. Um, but over time potentially people are just like on the same platform and actually it's like it's way better coordinated for for everyone >> 100%. Because like like also like what we mentioned in the beginning, right?

47:41 So like like the obvious like the obvious question is like okay but like we we also talked a lot about like negotiation agents. What if like both parties have negotiation agents and price is 100% a point where it's like a zero something or like they have like different interests. Um but we what we also discuss is that like price is like the outcome of like 5,000 different like other task that happened and on those 5,000 other tasks they have the same incentive.

48:11 Um sales wants to have as less like little friction as possible. Um, buyers want to have a really fast time to market, right? Again, like coming back to building aircrafts, building data centers. You want this data center to be built as fast as possible. You don't want to be build it like 6 months later just because it takes so much time um to analyze all of the um responses from suppliers.

48:33 Um, and you want to make the sale also fast. So like all of those five 5,000 other tasks, the incentive is ex exactly the same. And that's why you can deploy or Leo can deploy agents also on both sides um doing like automating the work um of all those other tasks. Um this is the beautiful thing. >> I think that there used to be this logic of like you shouldn't customize your software too much to to one end user or one end customer.

49:03 But I think something about LLMs and AI in general is like it might increasingly be possible to customize without slowing yourself down as a business too much. Um, so I'm curious if if that is something that you're seeing Vlad or Sema um and and kind of what what does that mean for for end buyers of software? I think the overall principle there's a lot of forward deployed work happening right now and part of that is because the state of the customer data and understanding customer N um is a lot harder than

49:43 understanding N plus1 and so we're sort of in the early stages of deployment overall and that's why there's still a lot of humans as part of this product um and by the way that is something that is harder for the incumbents to do because they're also like not set up in in a way to have even their the way their product feedback cycle works where they have a they have implementation teams but that's very much an afterthought versus something that feeds into the product side.

50:06 The beauty of AI is a it learns over time. So that's when we're talking about learning loops and you have the right eval process, you can do more and more complicated jobs over time and part of that is automating the deployment itself. And so you can and and I'd be curious to hear how Vlad is doing it, but a lot of our companies even are doing that at you know at a rapid clip where more of the customization a is being handled in an automated way and b the customer is able to turn the knobs and levers around

50:39 customization via software um versus okay I needed to bring in a the original it was like you know you brought in Accenture to do your SAP customization and now it's like Okay, I've got a forward deployed team that's going to build some, you know, help build and spec it out. And ultimately, it's going to be completely software driven. >> Yeah. Um I mean that's the reason why like when you look at the or structure of LEO, like 85% of the people are um engineers.

51:05 Um and even if you look at like the people where you like they don't have an engineering title, they have like mostly like an engineering background. And reason for that is um we obviously like we don't want to be a consulting company right so we make sure that we have like overall the best um agents in indirect and direct and finance um and those part but then like as you mentioned like there is a lot of like forward deployed work um to do if you go to enterprises because they have like different nuances in their um

51:34 in their processes um but how we work is we as you mentioned like building a product in a way where it's like um where we reducing this customization but also where's a lot of like self-service. So the job of our like FTDES and forward deployed engineers um is on the one hand making it like self-service but like in internally is like automating their own job right so like their KPI is like you are seeing this happening like multiple times like you like your literally your job is like to automate yourself um and then if

52:08 you automate yourself you go to the next task um I think it's like also like an um approach that Google or so is doing um so like but 100% agree with you Um, and that's exactly what we are um what we are building um and what how we're doing it. >> There was recently a a very big system of record event and and we're not we're not talking about Dreamforce, we're talking about the the bots and buyers summit that that Leo hosted in in New York.

52:35 Um and and just wanted to to kind of hear, you know, stories from the ground and and what what you're seeing among buyers. What are people excited about? what are people looking ahead toward? What are people you know asking you for? Um can you just kind of tell us some stories from from that day and that event? >> So I mean it was the third time that we are doing that we are doing this event.

53:00 Now we did it in um in New York just a few blocks from our um from our office here and over 100 procurement leaders arrived right and what we like we made sure that we like when we do when we do such events that we only invite like high caliber people right so like seale co vice um vice president um and there are like two things very like very different um of of of how we do this or like why why they're so amazed so the first thing is um when we look at like how procurement used to work in the last 26 27 years um a lot

53:30 of tools emerge right so you can like they're like all those technology landscapes and you can find them on LinkedIn you will see like there are like 500 procurement tools but like if you like and this is also the reason like but when you talk to procurement people like very painful and like I challenge someone like to find someone who like loves to work with procurement like no one does like you can really state people hate working with procurement and we're talking about the requesters and we're talking about the

53:59 suppliers and even people in procurement hate working on procurement. Um so like what's going on if there are like 1,000 tools? Um and the reason for that is like the all of the tools. Um um they just made the the process more efficient. That's that's all. But they never changed how though how those people actually work. Um, and it's like crazy to see that they work in emails and in Microsoft Teams there's and in Excel sheets and powerpoints.

54:29 like their main channel where they work on and it's like zero like zero AI enabled um obviously um um in in in this part and the the other thing is like we um we give them like a very cross department perspective on procurement right so we are not talking about like look at this crazy invoice feature that we developed but we more looking like like someone needs something um and in the end you have it on your table and this can be like a cross indirect direct, logistics, finance and you can see how we are doing it here

55:04 and we also putting it into like more into like a physical world because like AI agents that's very abstract um so what we are doing is we building up booths and we we even have this in our offices um in um also in New York where you can walk through the booths and experience like um all of those agents like really hands-on um and that's what the what's what what the people love and it's like um I mean the next event um will be with around 700 people.

55:30 Um so you can imagine how how crazy this this grows. >> Procurement people gone wild. Yeah. Yeah. It's going to be that one is in um Munich. >> Um yeah. So we doing them like in um in Europe Munich and in New York um all the time. Yeah. >> Cool. If you're listening and in procurement, you know where to go. Maybe I guess one question uh one question for me.

55:58 What do you think it takes to get uh to to get people excited about procurement? Is it is it the agents? Is it the people? Is it the time saved? Like what or like something else like you mentioned it like procurement is one of these things that I I I remember you know people aren't they don't like it like it's like a universally kind of disliked low NPS area.

56:18 I remember talking to a guy who was out of procurement like I don't know seven or eight years ago as I was looking at this category and he was like h I hate talking about this product like I I use you know this legacy system of record. I'm on Koopa and I don't I don't want to buy anything else. I don't want to talk about it. It's fine. It was like the most disgruntled customer call I've ever done out of like millions of them.

56:37 Um but I'm I'm curious uh yeah what it is that you think you know really gets people excited about this category. >> Yeah. So I mean it's like it's like and this is also why why why I like procurement is like it's on one hand like so like the reason like why we started in procurement like it's not essentially like what happened but like how people react to it right so if you talk to the to the people they're like really frustrated so this means it's an like highly emotional topic but it's like like let's be honest like

57:08 um B2B Zars okay but it's like highly emotional um so that's a good thing and then if you combine this with um like something which is boring and niche. Um this is also an advantage because um again like the it's um it's like also easy or easy for us for to amaze those people, right? Because like the the really last revolution they have seen is like 20 years ago um and then maybe a nicer user interface 10 years ago um but nothing else happened.

57:38 Um, and so you have like boring, highly emotional and then plus crazy business impact, right? So like it feels like it's unnecessary, but like I I told you like some examples. So like it has like obviously crazy P&L impact, but it has impact on like the whole economy, right? So we're like we are talking about like how data centers are built, how aircrafts are built, how cars are built, um how drones are built.

58:02 So it's extremely important that um you have a fixed procurement process um not only to like make it happen and build something um but then also when you talk about like when we when you look at like the competitive landscape so to get like 1% margin increase um you need to make 10% more revenue 10% more sales um so like if you just manage to get like 1% savings it's like equals like 10% of sales um that you have to um to get the same outcome in your P&L.

58:37 Um so like it's tremendously important and like you combine all of those three things and then you have like um a trillion dollar business opportunity. That's that's my opinion like for procurement, but there are probably also like other things that like emotional, boring, and have a crazy business impact. >> Well, Vlad, thank you so much for joining us. Uh this was a ton of fun.