A service that points Claude Code at two merging companies' contract portfolios to answer vendor-spend questions — renegotiation leverage, sized savings, and renewal deadlines — with every answer traceable to a specific contract page or invoice line. The core insight is that raw contract data silently breaks naive LLM extraction (long documents that never ingest, orphaned amendments, restated totals, tables split across pages), so the engagement is won by building a robust structuring pipeline first and only then letting the agent work.
AlixPartners is a global management consulting firm specializing in complex business challenges, including bankruptcy and restructuring, business and technology advisory, and risk and compliance. Its senior-led teams work across strategy, organization, technology, and governance, and the firm also advises organizations on applying AI to business operations. In the project described by the videos, its AI engineering and AI/data teams used Claude Code for a post-merger procurement analysis involving hundreds of vendors and more than 40,000 contracts, developing a five-stage data-processing pipeline and custom tools with business rules to make findings traceable to source data.
DSPy is a framework for defining language-model program contracts—inputs, output schemas, and scoring functions—so prompt and instruction optimization becomes a search problem. In the described workflow, it searches for shorter instructions while preserving the application's interface and evaluation contract, with candidate changes checked through repeated model runs, shadow evaluation, and human review of disagreements.
Searchable transcript of Claude Code Looked Right but Missed $800K - Natalia Connolly & Kevin Madura, AlixPartners — AI Engineer (18:32). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:01 [music] >> Hey everybody. Nice to see you here. My name is Kevin Madeira. My colleague Natalia and I are from Alex Partners, which is a global management consulting firm. We focus primarily on bankruptcy and restructuring, business and technology advisory and risk and compliance. And I'm part of the AI engineering team there. >> Awesome. Thanks, Kevin.
00:32 Natalia Conley, part of the AI and data team at Alex Partners. And today we'd like to share our experience with putting cloud code on a real procurement analysis with you. Our talk will have three parts. So we'll talk about the business context and then our approach and then some lessons learned. All right, let's get to it. Okay, so some business context first.
00:55 Let's consider a very large healthcare company and it just acquired a smaller sort of niche player that's extremely good in one particular area. And the parent company has some experience in that area, but just not that much. So with that deal handed, the procurement team is hundreds of vendors across the two companies, over 40,000 distinct contract documents and multiple databases that really were not linked to one another.
01:21 So there was no holistic view really of each vendor and each contract relationship. Okay. From here on now, let's deal with with a synthetic replica of the situation. So here it is, right? So Harbor Health is our big parent company and it acquired Summit Health Plan. That's the small regional player that's extremely good in this one area. And both of them have a relationship with a particular vendor.
01:45 Let's call it Acme Health Services. Okay. So after the merger, what the procurement team at Harbor Health really wants is answers such as, well, can we maybe get better terms from Acme now that we've merged? And if so, uh can you maybe size up the savings? And how quickly do I have to act in order to realize those savings? Right? So that's sort of the bottom line.
02:06 And our job is to make sure that that procurement lead from Harbor Health, who'll be sitting across from Acme during negotiations, is well equipped to answer any questions that they might raise, any objections, anything, right? With a specific citations, uh citation with a specific contract page or a specific invoice line. So everything needs to be traced back to the source.
02:30 Okay. So originally when we put Cloud Code to it, this is what it gave us, right? It came back with these answers. They look complete. They're certainly well formatted. They look plausible enough. You know, and uh so then we went back to the warehouse and we checked every single answer. Here's what we found out. The total annual spend, the split between the two companies and the contract counts were all correct, so that was fine.
02:53 However, the total contract value was wrong by approximately $800,000. Right? So that's one. The second one was the the date uh by which we had to act in order to prevent the soonest expiring contract from auto-renewing was also wrong. The agent said December 1st. The actual date was November 1st, right? And so if we just kind of believe the agent and waited until December 1st, the contract would have auto-renewed and uh you know, whether we wanted it or not, right?
03:26 And so that's the problem. So what happened? How come Cloud Code did those things and got two answers wrong? Well, the first thing to check of course would be the data. So let's talk about the data just for a second. And when you think about it, it just seems so straightforward to extract data from contracts, right? In reality though, contract data are a lot messier than than appear than they appear.
03:48 So just some things that can potentially go wrong with contract data. For instance, you can have very long contracts and they just get kind of stuck in preprocessing and they just hang there. And maybe you catch the exception um with the timeout and and you kind of go on and you marry away, you never even know that the contract never made it into the database.
04:06 That's one. Another one is orphan amendments. So for instance, there's an amendment to a parent contract, but for whatever reason it's just not linked properly to its parent. So it's just kind of floating out there as an orphan. And this is actually exactly what happened in our example where Cloud Code missed the $800,000. That's because it couldn't find that orphan amendment.
04:27 So that's that's another potential issue. Another one, and this is pretty common, is where you have multiple amendments restating the original contracts terms. So here you have one MSA, Master Service Agreement, and two amendments that restated the total contract value. And if you just extract the structured outputs, the dollar amounts, right? Then your Cloud Code will very happily go and just sum up those dollar amounts and go, oh, the total contract value is about $17 million.
04:56 But if you read the actual text, not just look at the numbers, the dollar numbers, right? The text says that every amendment restates the total contract value. So the correct answer is seven, which is the most recent amendment's value. That's another sort of gotcha to watch out for. Finally, you can have situations like that where you have very long tables.
05:14 Let's say these are the tables of rates for various services, right? And they go across multiple pages. And so if you look at page two, it just starts out with, you know, some some numbers and there's no header. And Cloud Code might not necessarily know the context. I mean, it will just do its best to kind of guess, but it might not know what each value really means.
05:33 These are just some examples. And none of this is exotic, but it just shows that real contract portfolios are full of issues that may or may not be apparent when you first started out on this on this journey. And that's why Kevin and I spent quite some time really building a super robust data processing pipeline in the beginning. We spent about two weeks on it just making sure that every single little thing like that that you just saw and more, all of that stuff was accounted for.
06:00 And so [laughter] it was really great. So we spent those two weeks just building that robust pipeline and then 40,000 plus contracts were processed overnight once it was done. Um I imagine if we did it by hand, it would have required probably a much larger team than just us two and probably a little bit more time than that. But just a summary of the pipeline, it was a five stage pipeline.
06:21 The first one was mostly preprocessing, so document intelligence, OCR, that kind of stuff. The second step was classification. So what is each contract? Is it a master service agreement? Is it an amendment? Is it an NDA? Something like that. Then there was the terms rates and spend extraction. That's the next step. And this is where we would account for things like very long tables that could break across page boundaries.
06:45 And then we would map all the services to spend categories. Only then would Claude code go over the extracted data and and sort of do its thing. It was very important to have those structured tables, not only because it helped out Claude code, but also because it made it that much easier for for us later on to go and check every answer programmatically in code.
07:06 All right, so Kevin. >> Sure, thanks Natalia. And so obviously contract data is very text heavy, but not always. And so there's sometimes there's multimodal elements. There might be charts or graphs or other things that if you just run it through a naive OCR process, you might get what you see on the right in terms of the the axes, but you actually lose the information that's contained within the chart.
07:29 And so a lot of what we're talking about now, if you're bitter lesson pill, this will likely not be relevant maybe like six to 12 months, we'll see. But what we did here to solve this specific problem was take the contract pages as images and feed them through using DS Pi to arrive at a highly structured output, which is what you see on the bottom right there.
07:49 So you go from this large unstructured data set to basically a database of elements that you know are going to be relevant for the balance of your work. And so it's very tempting to just let Claude rip, you know, on on the unstructured data and kind of have it figure it out, which is which is a valid approach, but there's certain engagements like this one where we know upfront we want rate tiers, we know we want rebates, we we know we want certain data elements.
08:19 So you structure some of that upfront and then let Claude fill in that framework behind it. And so once you have everything in this highly structured format, you can then use Claude code to dynamically build tools for itself. And so you have common ones, of course, like execute SQL or list tables or things like that. But we found that it was it was really helpful for later implementations or later agents when they're coming in fresh to to the data to have some of these tools available to them.
08:50 So invest a lot of time upfront to structure the data, as I mentioned previously, where you you essentially create these derivative databases off of contract data or other spend data or invoices. You normalize all of that together, and then you have Claude build these custom tools so that it has you know, kind of a shortcut, if you will, to get certain information that are specific, in this case, to this deal.
09:12 Um and so you have things like get rate card page or get specific vendor spend. And that's where you can embed some of the business logic where you might know of some gotchas or some context that isn't immediately apparent or obvious in the contract data. You can start to embed some of that in the tool calls themselves, the way that it queries data or pulls things back um is very helpful to enforce some of those, you know, more custom pieces of logic.
09:41 Um and Claude code was perfect for something like this because it's used to navigating autonomously around kind of messy code bases. This is a a similar concept. It's just contract data and spend data. So it's all largely in tabular format. Um but it can dynamically explore and pull in the the different elements that I mentioned previously. Um and then once we found something that was useful, that would work, we would package that up either into something specific in the prompt, but I think we we lean more on these
10:09 custom tools and custom views within the database to enforce some of that business logic in code versus just relying on on a prompt. Um and once you have all that package together, you can start to develop custom MCP servers and and configurations that live on. And it's it's a it's a bunch of different elements uh that really get at how do you perpetuate and and accrue knowledge across the the life cycle, in this case for particular engagement.
10:37 But if you're working on something internally, you right, you want to accrue the the knowledge and accrue certain business rules um that you want to enforce later down the line. So that this is a nice way to do it. And then of course, you have the ability to better govern access to the underlying data so you can block certain calls. There was a great talk right before this um regarding the security of how agents interact with the rest of the environment.
11:03 It's actually pretty simple, I think, at the end of the day. You just have a defined set of operations that you're comfortable with and you block everything else and you can enforce that up front. So we'll walk through just a a couple of lessons here uh very quickly. Natalia mentioned previously about the agent using the wrong date. And so it's a good example of again, some external business context that is helpful and useful to embed into the agent up front because in this case, yes, it had pulled out the date from
11:34 the termination clause, but actually the question we were asking was supposed to use the date from calculated from the renewal clause. And so it it seems like a minor thing, but it's it's uh it's a lot of these nuanced issues which compound your error rates. And so the more you can build these sorts of rules in up front, the better uh the result ultimately of course will be.
11:59 Uh Natalia mentioned also the kind of the dangling references for different amendments. Um and so this is something that we run across a lot. So many organizations have a contract lifecycle management system, a CLM. It's intended to be uh a highly structured database of your contract data, right, the vendor name and ex uh extraction of termination dates and whatnot.
12:21 They're almost never maintained properly or well. And so we often find we have to of course use the data that's available to us, but take somewhat of a bottoms-up approach to actually have Claude read through the contents of the contracts to find um instances of where it might not be properly captured in the CLM. And so you you have to go through and have Claude kind of build out this hierarchy of relationships.
12:45 So you have an MSA that is tied to SOWs and then there are multiple amendments to those uh to those SOWs, and sometimes they're just not organized properly. And so again, structuring everything up front makes it really easy to tack on these kind of loose documents later on, which has a meaningful impact if you're looking for savings opportunities. In this case it was a an M&A, you know, consolidation.
13:11 So you don't want to double count something, but you also want to capture the latest kind of state of the relationship with the vendor, which this kind of process allows you to do. So Claude was very good at at doing that. Uh I made a point before about governed access to the data. This can come back to bite you as well because there were two separate entities that were being uh put together here.
13:34 The way that we accessed data from entity A versus entity B was different. And so we ran into a few instances where the returned value would have been something that was malformed or zero or there was some type of access permission. And Claude when making the tool call didn't really get the the right type of feedback, and so it kind of just went forward thinking that maybe an invoice was zero or or, you know, basically an incorrect value.
14:03 And so a lesson learned there for us was anytime there's some sort of um you know, governance failure or access denied error or something like that, giving Claude more feedback about what went wrong allows it fail gracefully or otherwise recover, maybe flag something to a human or say, okay, I know this information is actually in this other authoritative database I can go to instead.
14:26 Um because when you're relying on it for some sort of savings analysis or something similar, uh you want Claude to have as much of that context of course as possible. Another thing we did um was uh dispatch agents for different savings levers. So our colleagues who have been doing this for 20 years or whatever it is, they have a whole library of ways to identify savings.
14:49 Some are more obvious than others, but okay, you're putting two companies together. Maybe they share vendors, maybe different vendors but same services. So you have a whole library of different activities and analyses that you could do. So we dispatched agents to do each of those in parallel um and they came back uh with actually pretty good results, but there's of course this step afterwards where you need to deduplicate the findings that come back.
15:16 So um you can't ultimately double count in this case a potential savings if um you know if it's coming from the same source or the same vendor as an example. Um and so the way that we did this was um essentially build out an additional process and a and a very specific and precise uh set of workflows where you would kind of let Claude do its thing, do its analysis, but then the reconciliation step is very uh it's built around kind of DSPI ultimately where you're very prescriptive about which data points you're
15:52 evaluating, how you do the reconciliation, and what's an acceptable next step in the process. There was another pretty good talk this morning about uh using DAGs for for policy execution. It's the same idea. You have a highly structured workflow for how you do this this sort of work. Uh and then the most common one in retrospect and the most obvious I suppose is just making sure that the agent has general context about what you're trying to achieve, but more importantly, like what's actually going on in the business at
16:22 the time. So, in this case, it's an active M&A deal. You know, there were things that management had already started actioning. And so, in our pursuit of potential savings opportunities, it will come up with these great recommendations, but it would be shot down immediately because, hey, we already started that 3 months ago. So, seems obvious on its face, but it's actually a pretty tricky problem to try and get the right context and the right elements into Claude at the right point in the process such that you don't
16:50 hamstring the analysis up front, but you don't come out with recommendations that are irrelevant or invalid at the end of the day. Um and so, it goes back to the point about knowledge accrual of just starting with an analysis, accruing things that you've kind of tagged along the way as, hey, this isn't, you know, a project that's in um in motion or something something else that's relevant to the ultimate objective of what you're trying to do.
17:21 And having Claude have some sort of, whether it's a memory system or you embed it in your tools or something similar where you don't have Claude kind of spending tokens to rediscover the same problems you've already you've already flagged. It It's already just embedded in the way that it does its work. And so, I'll leave you or we will leave you with two main takeaways.
17:39 The first is it's worthwhile to invest some tokens and effort and money upfront to structure your data first precisely so that Claude doesn't have to spin its wheels and spend additional tokens rediscovering the environment. So, highly structured data is very helpful. And then ultimately, of course, context is still king. We don't have 100 million token context windows.
18:03 Yes, we have RLMs. I know Raymond in the back is a big fan of those. Um but that doesn't help you with the business context. So, putting in the right business context is incredibly important. Um and that's uh that's what we found during this engagement. So, happy to take any questions afterwards, but thanks so much for your time.