Right Answer, Wrong Reasons: AI Forecasts Costco - Hassan El Mghari & Omar El Mghari, Together AI Transcript, AI Summary & Key Points

00:01 [music] >> All right. Hello everybody. Thank you so much for coming. My name is Hassan. This is my dad Omar and we're very excited to be here today to talk. I think the only father-son talk at this conference. So thank you. [cheering] Thank you. >> [applause] >> So I'm here to talk about you know can LLMs actually make financial forecast of a company.

00:36 And so we used Costco. We gave an open model kind of everything that was public from Costco's filings up until a certain point and we had it try to predict Costco's profits over the next eight quarters. So we'll start off with some intros. My name is Hassan. I lead developer experience at Together AI. The main thing to know about me is I really like building AI apps.

00:54 Everything I do is open source on GitHub and this is another one of the apps that I built. But this one was in collaboration with my dad. And I work over at Together AI. We used it for the open source models in in this talk. We're a platform for running and deploying and training open source models. >> Hey guys. My name is Omar I am Gari. So usually parents they teach children but Hassan he was teaching me a lot during the last six years when I I got to coding after 25 years of experience with the big four firms,

01:25 auditing firms, couple of positions as a CFO and audit head of internal audit of multi-billion dollar company. So today I'm the human in the loop for Hassan's engineering work. Okay. So my coding experience didn't start just last six years. I had a lot of passion for coding at at age of 16. I was coding in assembly and the reason I did that is start with basic a tennis game, small tennis game on an Epson with four lines on the screen and just this four um, you know, tennis uh court, but the ball was too slow.

02:01 It took like almost a minute to get to the other side. And then I learned assembly because of my brother's uh IT background, and then I rewrote the the same game, and it took the ball I can't see it anymore. It was too fast to So, I had to slow it down to really see things. So, just wanted to share that with you. >> All right. So, spoiler alert, um the LLM was accurate up until about 4.4%.

02:24 So, the our our forecast of Costco's actual membership was off by 4.4%. And so, I I thought this was pretty good, right? We had this LLM build a financial model, we eval'd it, um ended up giving us this result. Um and then I showed it to my dad, and it turned out it got the right answer or got close to the right answer, uh but for the wrong reasons.

02:44 And so, the rest of this talk is going to be going over, you know, how we built this financial model, how we eval'd it, and actually what it ended up getting wrong. And and kind of lessons learned along the way, including kind of the overarching lesson, which is having like an expert human in the loop is is very very important when you're building, you know, AI agents in in these different spaces.

03:02 So, really quick background about Costco. Costco sells a lot of stuff. They sell $300 billion worth of stuff every year at close to cost. So, they make very little money from that $300 billion. Um actually more than half of their revenue is from their membership fees, right? So, they charge you to to come at the door. Um their total operating profit was about $12 billion, and and a little bit more than half of that was from membership fees.

03:26 So, membership fees are are, you know, extremely important when we're trying to forecast uh Costco's uh revenue and profits. And thankfully, Costco gave us the perfect test on July 2024 cuz they raised their membership prices for the first time since 2017. So, Costco has two tiers. They have a standard tier that they raised from $60 to $65, and they have an executive tier that they raised from $120 to $130.

03:50 So, what we did is we came up with this cutoff, which was a couple weeks before the announcement, and we took an open-source model, and we gave it access to all of these documents from Costco uh up until that point, and asked it, like, "Hey, if Costco, you know, increases its membership fee, um what are the the the revenue and the profit that you would uh forecast?"

04:10 So, we actually started doing this with a couple of recent open-source models, like GLM-5.3 and Chimera-3, but it turns out they already knew that Costco actually increased their prices um cuz it was already in their training data, right? So, we had to go and find, you know, older open models who were trained, you know, up until before July 2024. So, we found GPT-LSS-120B and Llama-3.3 uh 70B, uh and we This is an example of a question we asked it, "Hey, did Costco change its member membership fees in 2024?"

04:39 The newer models uh knew the answer, the older models didn't. And so, we wanted to give We wanted to use these older models cuz, you know, you want a completely unbiased uh uh point of view here. So, that's kind of the first lesson. If you're trying to do any sort of forecasting, you need to try to find a model whose training data ends before the event that you're actually trying to forecast.

04:56 Um and uh for you know, open-weight models are actually really good for for this kind of use case. Um so, this is kind of how everything works. We went and scraped about 214 uh public filings from Costco, uh you know, quarterly and yearly earning reports, sales reports, um all of that. We uh ended up making it into a list of about 3,000 facts that were all timestamped.

05:19 And and then we filtered these facts to only the facts available before our cutoff date. We gave it to a model, and we actually spun up um 16 sessions of of GPT-LSS-120B. We had the model actually write a bunch of formulas and write this financial model. We ran some checks, and then we picked uh the From those 16 runs, we picked the ones that actually passed uh our checks.

05:42 So, in this case, like, what does the model actually do, right? In this previous slide where we we gave it all these facts, what does the model output here? So, we actually had the model output formulas here. Um and And so, an example of this is all the way to the left, we say, like, we account for membership fee revenue on a deferred basis. This is straight from Costco's quarterly report.

06:00 So, we give that to the model, and it constructs this uh uh structured JSON where it says, you know, the operand is ratable, the term is 52 weeks. And so what this means is basically, you know, even if a user comes in and pays a $65 membership fee, it's actually spread across 52 weeks. Um, and so this is really, really important when you're doing, uh, uh, uh, forecasting to understand how to to to forecast.

06:22 And so we have the model construct a list of all of these formulas. And so the main takeaway here is like, have models write structured data instead of, um, having them, you know, do the math and do and do everything all at once. Um, the model also came up with a bunch of assumptions. Uh, and so it had to guess on a few things. So, for example, what percentage of people are in the standard tier versus the executive tier.

06:41 Um, how many members were in Canada, uh, US and Canada and impacted by this change. Uh, so it made a few assumptions. My dad's going to go over some of these assumptions later and why some of them may or may not be wrong. Um, and and here's kind of the full picture over here, for the full financial model. On the left, you have the assumptions, which we've gone over some on the last slide, um, that the model came up with.

07:01 And then we have 14 formulas that the model also came up with. Um, and and then we have a few numbers that are kind of directly from, uh, Costco's filings. And this is actually an Excel spreadsheet that it that it created. I'm kind of just trying to visualize it in a way, uh, uh, where where it's, uh, it's a little bit nicer to to look at. Um, and so after we had the model create this financial model, we, uh, actually had it go through a bunch of these checks.

07:24 Uh, and so they're just basic checks. For example, a formula can't depend on itself. For example, raising the membership fee needs to raise revenue. You know, just making sure that like some a few basic things were were correct. And whenever a model, whenever one of those runs, cuz remember we had these 16 runs, whenever one of those runs got one of these things wrong, we ended up passing the error back to the model, uh, up to two different times, uh, so that it kind of tried to self-correct itself.

07:48 And it was interesting here, and so these are actually the the 16 different runs that we have. They're all running. They're all creating their financial model. They're writing the the formulas. Uh, we're passing the errors back. And what ended up happening is something interesting where we had a bunch of them fail anyway, um, cuz they just failed more than And two times we we we gave them back their their error.

08:07 Um And so we ended up uh using these five that ended up passing uh for for our uh financial forecast in general. And that's kind of another lesson. We ended up starting with just like one prompt that we kept tuning and tuning and tuning. We're like, "Oh, like surely like we're going to give this to GPT-OSS-120B and it's going to pass, you know, all of the checks."

08:24 Um but that ended up like just taking so much time that we were like, "What if we just spun up, you know, 20 uh uh 16 different runs?" Uh in this case it works really well cuz the model is really cheap. So it only cost us 19 cents to spin up these like 16 different runs. And we had more of them that that passed this way. So this is the million-dollar question that we asked, right?

08:43 If Costco raises its fee uh this September, what happens to its actual revenue? So we had JLM 5.3 translated into just like some structured output that we can feed into the financial model. And then the model gave us this number of like it predicted that uh profit would increase by $560 million over 2 years. Uh and so this is it uh just in the app. We send this over.

09:06 Uh JLM 5.3 interprets it. It sends it as structured data to the model. And remember the model is just a financial model. It takes 10 milliseconds to run uh cuz the model cuz the uh our GPT-OSS model already made it. And here's that $560 million uh number right there. So um this is a just like visualized on a scale. The blue line is the predicted uh uh uh profit and orange is actually the real profit extracted from Costco's uh earning reports.

09:33 And so you can see we were accurate to about uh 4.4%. Uh which is the average of the five builds. The builds uh gave us between 0.7 and 5.2%. So again, I thought this was pretty good, right? We had the model build. Uh we had our AI model build this financial model. We ran it. It was accurate within 4.4%. Uh and then I showed it to my dad. >> Yeah, definitely it's amazing to see that AI can can get to financial models which is the, you know, the area of accountants and and financial people.

10:01 But with with very small guidance. But the first thing you look at when you try to look at a financial model it to audit it is to look at assumptions. So these are some of the assumptions that were used in Hassan's model and uh I just want to draw your attention to two things. The executive share, like Hassan mentioned before, uh it was um you know, big swings there.

10:19 Um US Canada, the last um you know, um assumption did not move um sales and um uh and and fees with much, with zero. So, it basically that's a lever that is not working and it should be and something was wrong. Something went wrong there and it should be fixed. Um so, these are why the executive share, for example, the executive share, the 15%, the model took 15% um and the problem is that the model had just one citation, increased penetration of executive membership, but there was no number to back it up.

10:53 So, that's what the auditors looking at at how are all the facts backed up with, you know, with with real numbers. So, the citation is real, but the problem and the the the number was was made up by AI and not only that, AI also drew the boundaries of that assumption, which is uh 0 to um 0 to uh 5 to 30%, but we will see that it's not correct, actually.

11:16 Um the That's what the AI did, 5 to 30%, but the problem is the real number that was already disclosed in the in in the filings um was 45%. So, in this case, AI failed to um you know, uh identify that number from the from the filing. So, that's one uh one problem. And um the surprise, why did it land correct, like Hassan mentioned, it's 4.4%, which is an okay uh uh a performance because of two errors that canceled out, two assumptions that canceled out.

11:51 And the first one was an overstatement of the base, how many people. Instead of applying the um increase on just the US Canada portion of the members, the model applied it to international, to to the whole population, actually. And the second thing that was understated, it was too low, it's the percentage uh instead of 45% which is the real number, um AI made up this 15% and that was too low.

12:18 It landed at $69, which is close to 66, but um again, the model behind it was wrong. And this is um this is what Hassan was mentioning before um that um like that led to uh like the 16 the 16 models, they failed and they failed um all with the same thing with big ranges from half billion to six billion dollars in in terms of fee increase, while the actual number that we recalculated was just 450 million dollars.

12:50 So, that that was one of the other issues here. And then um just a trick here, the way to know whether a model or um to to something is wrong, if you have two things canceling out like an auditing trick, you just fix one of the two and the model will look worse. So, before the performance was 1.7 because things were washing out, canceling out. And then, once we correct just one, we just we put like the 45% instead of 15% that was made up by AI, suddenly the problem, the error was 28%.

13:22 So, that's what we say, luck is not a control in the auditing world. Usually in auditing, we use three lines of defense. We have management controls, which is the engineering checks that Hassan um coded. And then we have the uh internal audit with some some internal um accounting checks that should be performed by internal um uh audit function. And then we have a third layer, which is the independent check, that's external auditors.

13:48 So, these layers, they will benefit like the model will benefit from adding layer two and three, and that's the role I was playing, like this CPA in the loop or what I call uh actually the judgment, professional judgment in the loop. Um to because we can we can hardcode most of those checks, but still we need uh we need a human uh looking at that. >> Great.

14:12 So, you know, as my dad said, we we had the model here, we passed it through engineering checks. It did pretty well. You know, we got a little bit lucky with the result. What was really missing is these like accounting checks or this like kind of human in the loop activity. So, in a lot of these accounting checks that my dad came up with, you know, we were we were able to to codify.

14:31 So, one of it was the the executive 45% like my dad said. So, it's really important to have these like domain specific checks when you're building these these AI agents. You know, I don't come from the from the accounting world. So, when I built this and then I saw 4%, I was like, "Oh, amazing." But but really it's it's Yeah, this is really the number one thing to remember from this talk.

14:49 If you take one thing away, having a human in the loop or ideally a subject matter expert in the loop is really really really important. They can take a look at your agent or your model or or whatever you've actually built. They can check the assumptions, check the inputs, check the outputs, make the right corrections. You can codify their checks in the model and it just makes it into something that's a lot more autonomous.

15:11 And uh just a lot a lot better overall. And that is it. Thank you all so much for coming. >> Thank you.