How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver Transcript, AI Summary & Key Points

00:01 [music] >> My name is Nayan. I'm the head of AI engineering at Optiver, and today I'm going to talk to you a little bit about how we scale AI with P&L on the line. Like, what does that mean though, right? So, first I think to contextualize, uh let me tell you a little bit about what we do at Optiver. So, we're a market maker. Uh what that means is that we facilitate trading.

00:30 Uh we're a trading firm. We trade with our own money. Um and we don't have clients. Uh basically, you know, we in in a tech sense, we take a lot of data, you know, we build models, and then we make decisions with them. And these decisions need to work in a very high-performance manner, you know, we make 10 million plus trades a day, pricing million plus instruments, and petabytes of data processed daily.

00:52 And also, we're a pretty global firm. We have 10 plus locations across the world. Um and that comes with, you know, certain implications that we'll get into for a second. >> [clears throat] >> So, typically, if you look at a traditional tech company, you can think about how they think about, you know, how they measure how much money they make. Uh it's something along the lines of you implement some change, you roll it out to users, and then users use it for days and weeks, etc., and you collect some engagement metrics.

01:16 Uh and then over, you know, months and quarters, maybe years, you look at like revenue, long-time value, etc. I used to be at Apple, um so I know how this works. Um but um at Optiver, we implement something. Uh we roll out that change to our traders and our our quants, and literally within hours you can see the P&L impact. In fact, you know, at the top of every day, we basically zero out the balance sheet, and then look by the end of the day how much money we've made.

01:40 So, this has certain implications around how, you know, we make decisions around how we want to scale AI and how do we want to build things. >> [clears throat] >> First of all, the feedback loop is much more immediate, right? Like, when you make a decision, by the end of the day you'll start seeing the impact of that decision. Of course, like, over long term, there are instruments that we trade that are, you know, longer horizon and there's more value to discover over long-term, but immediately you'll start seeing the

02:02 impact of something. But, the other side of that coin is that if you miss an opportunity, then that means that opportunity loss is potentially pretty costly. Especially if, you know, there's like a special [clears throat] thing going on in the market, let's say that there's a SpaceX AI IPO or something like that and you miss out on that, then that's probably, you know, pretty costly.

02:20 Which means that we need to make decisions pretty quickly. Uh we need to be able to measure them really effectively and really efficiently. And we got to have a bias for action. Um that means like the ceiling of the value and importance of a decision decision you can make is really, really high. Um so, in the in the case of like a service, like a user-facing service business, like Spotify or whatever, like fundamentally, they're competing for a finite amount of attention of the customers.

02:46 Versus with your trading, the more you trade, the more money you make, right? So, you know, the economics are a little bit more different. So, to accomplish this and to be able to scale and operate at this way, we basically needed to build our own AI platform around a few core principles. Move fast. I mean, we talked about that already. But, of course, part of moving fast is that being able to be modular.

03:09 Because, you know, in the AI world, every day there's a new model, there's a new product, there's a new paradigm. We're doing loops, now we're doing swarms, now we're doing graphs, etc. And we need to be able to account for that. And if that paradigm is valuable, like measure it. And if it is, bring it in. Because the moment you bring it in, you're going to start, you know, getting more value.

03:27 And of course, to be able to understand that, we need to build to measure at every layer. You know, of course, like trading means lots of backtesting, etc. But also, you know, we need to when a new model comes out, yes, there's of course external benchmarks as to how this model is performing, but we also need to be able to understand how in our internal benchmarks these things perform.

03:43 Because the data distribution of the benchmarks that we have is different from what they have available externally. Now, the thing is, traders are pretty smart people. They're pretty high agency people. So, we want our platform to be very self-service. But, of course, anytime you provide something purely self-service, it can be really chaotic. So, we also need to be able to provide good defaults.

04:03 Uh and one way we kind of have this um way of working with them is also being able to forward deploy. So, we have a fairly decently sized AI engineering team and we take our people, we have them sit down with the traders, work with them, understand how they trade, teach them how AI can add more value to them, learn and bring it back to the platform.

04:24 Now, of course, you've all heard the stories of like opening eye hugging face, etc. So, of course, as we're doing more and more with AI, we also need to make sure that we're doing in a safe manner. As you can imagine, there's potentially pretty negative impacts if the agents kind of go haywire and do stuff with our stack. Um but, I do want to underscore that everybody in the company is using AI.

04:42 Like, everybody's using it a little bit differently, but fundamentally we're very very um AI native, I guess is the term that we're using now. Um And, of course, because we're a highly regulated industry and we value our IP deeply, everything that we do is on prem. So, we have our own on prem GPUs, our own prem infrastructure, etc. The only thing that we phone home outside of our service really is for like third-party model inference.

05:03 And, of course, in those cases, we have like things like zero data retention, etc. agreements with the companies. So, here's kind of like a very, you know, shock and awe, like a lot of information on the slide type of view of our AI platform. I don't expect you to, you know, digest all of this, but, you know, if you want to take a photo and talk about me afterwards, be my guest.

05:25 I'll flash this slide again at the end. But, broadly, the things that I want to highlight is that um we all have to be accountable for every layer of the stack. At the bottom, you know, we have models from third-party providers. We also self-host our own models and the open-source models. We have an inference gateway that provides things like routing between these different models.

05:42 Um we can uh, measure how much we're spending on each of these models. We can assign budgets, etc. I'll go to that in in a second. And of course this gives us resilience. Like if one provider goes down, we can always swap out to a different provider, etc. Because again, any time that we're down, you know, that's basically we're not making money. Um, and then we have our own agent platform where uh, agents have uh, registered uh, capabilities like skills, plugins, tools.

06:08 Each agent in each team with the using an agent can bring in their knowledge and memory systems. We have an agent gateway which is basically like a single ledger of all of the agents in the company being able to come in together, communicate, orchestrate humans and agents with similar identity management, uh, etc. So this kind of goes into things like the agent swarm, etc.

06:26 concepts that you also may have heard about. And finally, what's really important to me, I'm a product guy as well, is that um, we have the right surfaces for our users to engage with the agents because you can build a great platform, but if nobody wants to, likes to, or knows how to use it, then it doesn't really matter. So we've built our own kind of um, web interface for people to access like their sessions, etc.

06:46 Uh, we have this thing called our agent workbench which is basically persona management. Teams can kind of build their own profile of how they use the agents and share. Uh, we have a personal assistant that everybody in the company uh, has access to. Um, yeah. So again, I'll get back to this later, uh, but if you have questions we can talk about it afterwards.

07:06 Right. So how do we move on these principles? Uh, one principle that we have is that, you know, if a new frontier model is made available publicly, uh, we want to be able to leverage it in production flows the same day because there's immense value in that if it is valuable. But is it valuable, right? We want to be able to benchmark it on golden sets.

07:23 So that means that we need to have a mature evaluation framework. Um, and then we partner with key customer trading teams where we collect their benchmarks. We onboard onto them and whenever we get a new model, we run the model on these new benchmarks, immediately know the metrics, and can share with the folks in the company to see, "Hey, is this really valuable for our use cases?"

07:42 But additionally, you know, anytime there's a different model, usually that means that we have to make the model available in our gateway, we have the right APIs, interfaces, we have the cost management, etc. set up correctly. And often times new models mean like the harnesses need to be updated, etc. and we also manage the update and delivery of that as well.

07:59 And then of course, yes, we launch it, but then that also comes with the expectations. We're a global company, so we need to have round-the-clock support. So our team, you know, has in different regions support rotations to make sure that the uptime is high. I think some people here in the audience actually are in support today, but anyway. Um and then of course, like we run these benchmarks, we also need to be able to have um the people in the company understand like, okay, we looked into this model, we benchmarked

08:22 it, here's how well it works, here's how you use it, here's our recommendation, and kind of keep that communication line open. Um another thing is that because we have optionality between different models, uh if any model's adding value, we want to integrate it. Uh and we need to be able to insulate against model availability in like different regions, uh or different, you know, cloud providers, etc.

08:44 Uh and also we have a requirement that some of our workloads run in a completely DMZ on-prem zone. Uh so we also there's cost reasons for that, there is um reasons of isolation for that. So for those reasons, we want to be able to be agnostic to the model and harness, so which means that we have the third-party providers, uh we have on-prem models, and all of it kind of goes through our inference gateway.

09:06 We log everything, uh every interaction going with the models is logged, and we also automatically move them to our evaluation system. Uh we have a bunch of different harnesses that we support, and on top of that we have a meta harness layer that basically lets teams configure their experience on top of that and have some portability between these different harnesses.

09:25 Um >> [clears throat] >> so teams can define their own harness configs in this meta harness layer, and that plugs into the team's knowledge base. Automatically knowledge is harvested over time, we look at the traces and then derive ways to uh get additional knowledge as well. Uh and then through this meta harness, you can either deploy manage the agents on our platform or you get access to them through the web portal IDE UI that I described earlier.

09:50 I see some people taking photos, so I'll give it a second. All right. >> [clears throat] >> So, um this type of a chart may be familiar to a bunch of you. Uh this is kind of a roughly illustrated version of how much we're spending on AI, no numbers on it. Um but we are doing more with it. And this is, you know, in December you see like the Claude code moment and things go up.

10:07 By the way, this inflection point here is like when I joined the company, so I don't know if I have anything to do with that. Um and then um then we have here, you know, like the big frontier model releases of the summer. As you can see, like spend really shot up there, but then we kind of got it back under control because we built some optimizations into the process that helped us really get this thing uh in check.

10:31 So, um the thing is the more we spend, the more value we can get. We are pretty confident that we can, you know, keep getting returns. We want to make sure that we're spending wisely, right? So, one thing that we discovered was that um we have a bunch of MCPs, we have a bunch of knowledge bases and information, etc. Um and on a fresh session, we were seeing that, you know, sometimes people have like 200k tokens worth of context like by default minimum, sometimes even more.

10:57 That's That's ridiculous, right? Because everybody was kind of dumping a bunch of stuff in there. This is the kind of problem with like doing things completely self-service, right? We need to have skin in the game. So, we implemented a bunch of different changes. Uh we brought in MCP tool search. Uh we gave the agents the option to um rather than calling MCPs, have their own CLI that's wrapping around a bunch of the MCPs, which is like much more efficient.

11:19 Um we looked at the caching characteristics of like when caches are getting invalidated, how long do we need to set these parameters? Do we need to set them differently? Do we set them dynamically? Um and then we work with the individual teams to kind of come up with further optimizations to their harnesses. You know, we do we put some things into knowledge base rather than this like massive system prompt etc.

11:41 And the good thing about these improvements is that um these things all scale with token cost, right? So basically if the tokens are more expensive, you're saving more on this. But the point here is that um yes, you can dump models out there, make them available to people. And I'm not even going into the work that we're doing on things like, you know, model routing or sub-agent optimizations etc.

12:02 Sheer single session stuff, there's almost always value left on the table. And we're, you know, ruthlessly optimizing towards that. Yeah, there we go. And we shipped optimizations at this point in time, but of course there's always like roll out etc. Our tech stack is a little bit interesting when it comes to rolling out across the different global regions.

12:26 But then also other things that really helped us spend less was that um turns out that if you tell people like how much they're spending and like give them a little bit more visibility, a little bit more alerting, and give them the agency to self-manage, they actually do a pretty good job, you know. We built a system to basically have teams kind of set their own alerting and limits.

12:41 And we've seen like many teams, you know, basically set reasonable limits for themselves. Another option is, you know, exploring open-source models. Uh you know, there are a bunch of open-source models that are on token cost pretty competitive with the frontier models out there. It's not necessarily right for every use case, but we found opportunities where they're a good value replacement.

13:02 And we've been basically making that available internally. Uh and we've seen more and more people leverage those. Then of course, like there's things like model routing etc. There's we've been exploring different approaches to dynamically send the prompt to like a cheaper model etc. Or sometimes have a way where we have the like plan generated by like a more frontier model and the implementation be broken down by smaller models, etc.

13:29 Um and finally, you know, I think knowledge bases are underrated, uh which yeah, I mean they're very important, but they also help get spend in control if you do your knowledge base effectively because then you're retrieving less context fundamentally if you just retrieve the right amount of context. Um so, long story short is that we're still investing I mean using a lot of tokens, but this is just about getting more ROI out of the same tokens.

13:54 And finally, I kind of want to talk about uh meeting the users where they're at. Um so, a few months ago we were in this place where like a bunch of people are using different harnesses. They have all these weird bespoke setups to kind of team up into their dev boxes and some people were building, you know, hacky web UIs, etc. etc. Whatever. So, basically what we said was like, "Hey, you know, there's a bunch of people doing these types of things.

14:15 Why don't we just build a nice version of this and then make it available to everyone in the company?" So, that's exactly what we did. In our web portal, we built this basically system where you can manage your sessions. You can have chat windows. You can have like artifacts in there. Um there are similar products out there offered by, you know, you know, some of the different uh harness companies, but we feel that this worked very well with our infrastructure um and we can kind of iterate it at the rate that we would

14:41 like to. Um so, basically this was actually uh also at a at a place where different traders in the company are now kind of contributing code. They find something that's valuable to their workflow and then they did they just send up a PR and we just update it and then now we have things like, you know, collaborative document editing and things like that.

15:01 And in fact, we're kind of getting this to a place where uh we're doing this, you know, agent multiplayer thing where now I have my sessions in there. We can create rooms with like my agents, your agents, like other deployed agents and we can kind of have all of that in one place and this really helps uh people be more productive and uh you know, we have now a Slack client, etc.

15:19 So, you can also do it on the go and whatnot. So, basically, um our users are very high agency. Uh they will go and build something if we don't build it for them. So, we just built a better version of it. Yeah. So, uh basically, this is um the overall overarching philosophy of how we did this. Uh we have a platform, very modular, uh and um we continue to iterate on it.

15:46 But, so far, we're having a pretty good time. Um that's the end of the talk. Uh but, if you have questions about anything on this or any other area, we can talk. And by the way, we have a networking event uh this evening. Uh please come to our booth and register. It's going to be fun. We have cool prizes. >> [applause] [applause] >> Questions? Yeah.

16:19 So, we've um definitely been exploring a bunch of different options. Um personally, I'm not happy with any of the options out there. There's tradeoffs, but I mean, if you ask like some of the Frontier Labs what you do for knowledge base, they're like just huge markdown files, hierarchical markdown, etc. Um there's places where that works, but um there is um vector search has its own benefits and disadvantages.

16:43 We have a hybrid approach that kind of uh allows us to explore these different options. Personally, I want us to spend more time around like doing things like perhaps some graph databases, etc. that are going to be uh more valuable. I mean, every day on Twitter, I see something like new way of like handling memory and knowledge. Um that's a problem that we want to spend more time on.

17:04 I'm not happy with our solution. Go ahead. >> There's hand built or >> Yeah, good question. Um, value gained from using hand built solutions versus Codex. So, there's a few things. Um, I mean, one kind of boring answer is that because everything is kind of in our own infrastructure, a lot of these tools don't work in the way that you would want them to.

17:37 Um, un- until recently, Codex didn't really have the option to kind of run managed agents, etc. or some of these things on your own infrastructure. Now, they've started offering them and we've been working with them on it, actually. But, um that was one reason why we started this way. Secondly, I think the um most Many of the places that offer these types of solutions also try to kind of lock you into either their model or some rest of their solution.

18:00 But, the moment we kind of go down that path, then we end up in a place where if our infrastructure is overly invested in one particular provider, we move at the speed that they let us move. Uh, we've actually encountered with um that was one of the uh inference gateway providers that we used to use. Um, and basically, we would encounter scenarios where something comes out, we need it, and they're like, "Okay, uh let's talk to our GTM team.

18:27 Let's put you on the road map." Well, it's like and then by that point, it's like, "Okay, no, we need to actually move faster." Um, so that's also one of the reasons. Uh, and finally, I think the in some of these areas, we've been um consistently ahead of the curve. Like, uh talking to some of the frontier labs, um the the way that we've been doing managed agent capabilities, etc.

18:49 like, they're asking us to tell them how we did it. Not to be bragging too much, but I think the um we feel pretty confident in our engineering and kind of our road map. So, that's also uh been well. I don't think this approach works well for everybody, but I think we have a unique um coalition of circumstances that allow for us to do that. Good. >> Can you talk a bit about your Um yeah.

19:24 So, just to make sure I understand your question correctly. What are some of the best practices of how we integrate the rest of this into eval and memory? Right. Um so, I think the we started at a place where, you know, we log everything, like I said, every tool call, input, output, etc. Just hotel logging. And that goes into our evaluation system. Um the challenge that we found with that, and we it's kind of an open problem, I would say, is that um when you're kind of trying to evaluate like a tool call, I mean, we

20:02 have like, for example, a support bot agent that we run in Slack. And we want to evaluate that agent. Let's say, what if we ran this agent on a different model? Well, the support bot is leveraging a bunch of tools that depend on the state of the world that were different at the time when you ran it. And that's the pretty like typical problem. Um we've been working on basically things like, can we snapshot world states with these tools?

20:24 And depending on the system that it ingresses with, it's, you know, there's different answers to that question. If you're looking at some like trading back test database, easy. Like that you can always point in time. That's what we do. But if you're looking at some other system, like if you're looking at Slack, for example, what was the state of Slack when this was originally evaluated?

20:41 Maybe there's some ways to kind of replay Slack from a particular point in time, but other systems, not really. So, we kind of try to assign the confidence value as to how well we can point in time this particular tool invocation. And we still log it as such. And when we're rerunning the eval, we will run the tool with kind of saying, okay, I want to kind of the snapshot.

21:00 We have like a proxy around the MCPs. But, you know, it's imperfect. And I don't know if there's a perfect solution other than just record, you know, everything. Yeah, I maybe I talk time for one more? Good. Yeah. That's a very classic problem that we've actually run into several times. You know, we had this one team that built an agent that did like coding and like analysis with them.

21:46 It was kind of scoped to their repo and it was like, well, what if we also had worked with this team and have their thing also come to us and immediately kind of explodes in complexity. Now it's checking out multiple repos that have different off, etc. like typical problem. So, initially our solution was explore, okay, what if we like literally just did that?

22:02 What if we just made the new agent that just had a union of the functionality, but often times that ends up, you know, a lot of tech debt, works very poorly, etc. I think our current answer to that type of paradigm is obviously every agent is version, etc. So, you can if it's small increments, you can push a new version to the agent, make make new capabilities, but going forward we would more encourage people to basically leverage our agent gateway where rather than one agent having like a 10 different things that it

22:30 does, scope your agents to like a task that they're good at. You can easily measure them at, you can easily manage the life cycle of it and then there'll be another agent that it can talk to easily as if it was, you know, kind of operating on itself because we kind of have this construct of agents being able to go through one ledger, spin up each other, etc.

22:47 And that will help us scale these types of functionality more effectively rather than just bloating one agent to do everything. It becomes kind of like a multi-agent orchestration problem, but you know, that's I think keeping things more measurable is really important because if you really bloat the agent then you can end up in scenarios where like, you know, some of the functionality gets better, some gets worse, and you can't really, you know, tweak it easily.