Agents That Self Improve While You Sleep - Fuad Ali, Arize AI Transcript, AI Summary & Key Points

00:01 [music] >> Well, thank you guys for coming to the workshop. Really appreciate you. My name is Fouad Ali. I am a product manager at Arise. Arise is helps you guys build the self-improvement loop for your agents. Today today I'm going to be talking through what that loop is, how it gets built, and I'm going to be walking through a little bit of an example agent as well to kind of show how that agent works.

00:36 Now we have an hour for this workshop. With that in mind, I know that workshops can be hectic, Wi-Fi can go out. So I have everything ready on my screen. I'll be walking through it. It's optional for you guys to follow along. Highly recommend you do. I personally learn a lot better when I have hands to the keyboard. But if you are having issues, don't worry.

00:53 And we also have Lori in the house who is more than happy to kind of help troubleshoot along as well. Sound good? Cool. So I'm Fouad, I work in product at Arise, and we work with a lot of agents. And I'm assuming you guys are all very familiar with what agents are and how they fail and how they are non-deterministic. And you know, this is sort of like a classic engineering problem that has reared its ugly head, you know, 10 times worse with agents because of the indeterminism, right?

01:20 So when you have agents that can do a lot of things and have a lot of different ways of doing a lot of things, understanding their failure modes becomes trickier. And then add on to that the fact that there are thousands of these agents. Maybe I have internal agents. Maybe I have coding agents. You have marketing agents. And you have obviously customer-facing agents as well.

01:44 And that all adds a ton of noise to the problem. And so how do you actually start to build to a loop where these agents use the feedback that you provide as a human being, customer feedback, and obviously actual evidence, so things like traces, um to inform how they can improve. Uh and that's what I'll be talking about today. Uh Signals are out-of-the-box version of this, which I'll be showing heavily today.

02:10 We have a managed agents platform that lets you kind of define on your own terms what these do. Um obviously data sets, evals, experiments, uh all that will be part of this as well. Uh but this is what we're trying to solve, and uh apologies for the meme. Claude's uh sense of humor is not that good, but um I think it illustrates the use case pretty well.

02:30 Um in the previous world of software engineering, um your SRE would probably be in the first picture, right? So, classic situation, PagerDuty uh notification goes out 4:00 a.m. Your agent broke something in production. Um in this case, we're we're doing like a little, you know, uh shopping assistant, and shopping assistant is saying, "We sell zero board games to all your customers on production."

02:57 Not a situation you want to be in. Uh there have been worse use cases of agents as well, where uh I don't know if you guys are familiar with what happened in Canada with a car dealership, where someone was able to get um a uh I think it was like a a Prius or something for like $5 uh through a chatbot. They obviously didn't honor the sale, but that's legal's problem now.

03:15 Um and so, these agents can fail in a number of different ways for a variety of reasons, and it becomes really important to try to move to a world where we're in the after stage, where the agents can learn how to fix themselves and build that feedback loop into it. And I'll talk a little bit about what where humans still add value in in that sort of world as well.

03:38 So, this is what you'll need to follow along. Again, like I said, this will be optional. I'll be walking through the flow for you guys. Um we do have an hour. Um we can spend the rest of the time with questions. You can spend some of the time trying out the loop. Um I would recommend if you have some time I'll I'll put up here for you know 30 seconds ish to a minute.

03:56 Um if you can sign up for a completely free Arise X account uh you'll need a space ID and an API key which I can show you how to get. Um you'll need some sort of coding um API or sorry um LLM API provider key OpenAI or Anthropic both work. Uh you'll need Node and Python on your machine if you don't have that don't worry we'll we'll still talk you through it.

04:20 Um Cloud Coder Chris will help kind of set this up for you as well. Um I assume none of you guys are strangers to those tools either. Uh and it's very simple. So this is kind of the app we have. So it's a chat base toy buying experience. It's called Wonder Toys, right? Very simple. You can ask in the chat about different products that are available.

04:41 You can chat to purchase. Um this is all running locally on my machine. So this is the agent that we're going to be dealing with today. Uh we're going to be walking through how we instrument it, how we set it up so that we can build that feedback loop. Um so I'll pause here real quick. I'll give you guys 30 seconds to try and do some of this again optional.

05:04 I don't want to you know block the entire workshop on API keys for sure. Cool. So really three things getting the repo running the actual app making sure you have an API key that actually powers the agent. Um so not uh a provider API key like OpenAI or Anthropic. Um then this this part here is about installing the skills that helps your coding agent work with our tools uh in a more kind of opinionated manner.

05:32 Um Node and Python just to run the repo and that's about it. Cool. I'm going to continue in the interest of time. So open question for you guys of these you know six Did I count that right? Five boxes. Five boxes. Which of these boxes would you classify your team is in? And, you know, the usually it starts from, "Okay, we are tracing what our agents are doing.

06:00 So, we know when they fail, we know what tools they call. We can see the entire trace tree for the interaction points of the agent. Uh and that's box one. Show of hands, does anyone have tracing for their agents right now? Cool. Okay, we got a bunch, actually. It's good. Then the next stage of that is, "Okay, well, I know what what the agent is doing, but is it doing the right thing?"

06:27 Um and how do I actually score and evaluate that? So, we call those evals in the industry, and so I would say, you know, some people probably are in that box. Are Are any of you guys evaluating your agents? You have evals running on traces? Okay, we got a couple. Okay, a lot. Yep. Then there's someone reading the traces, presumably. Uh and that's sort of what we're trying to unlock here, right, with the agents.

06:50 Um but realistically, if you have coding agents internally, there's no way someone's actually reading all these traces, right? Or, you know, a chatbot serving millions of customers, no one single human being is going to be able to read those traces, right? Then you actually want to be able to save those failures as tests, right? So, every time something fails, you capture that, and you're able to use that as a regression test moving forward, so you don't, you know, mess the same thing up twice.

07:19 Um and then finally, you want to prove that you want to make the fix, and you want to prove that you actually fixed the thing, right? So, in those last three boxes, have you got anyone kind of in in there, in that stage where they're actually Awesome. Cool. We got a couple. Cool. So, how do you actually thread all these things together, and how do you make this workflow as easy as possible, and, you know, have humans in that workflow in their highest value leverage points, but not necessarily reading tons of traces,

07:49 right? Um the flow generally goes trace signal, which is our out-of-the-box kind of SRE agent, which I'll show you guys in the platform. Signal then runs reads the traces, creates a data set off the failures, proposes an eval for you, automatically fixes what's going wrong, and then runs an experiment that actually proves that that fix is um something that's valid.

08:17 And then you review that fix, you choose to merge that fix, and you go back into the loop. And every single one of these steps builds additional context as your agent continues to improve, continues to, you know, move into new use cases. Let's say I want to add, you know, one-click checkout on this on this agent or something like that. Um that then feed not just your product roadmap, but your engineering reliability organization as well.

08:43 So, uh this is kind of what we'll talk through. I'll walk through the loop. Uh I'll show you a bit of the agent and its traces, signal reviewing the agents, the failure data set, the eval that we build off of that failure data set, fixing it, proving it with an experiment, what the agents did themselves, and what still needs a person, um and where you guys can add the most value, and then we'll have some time for questions as well.

09:07 So, Wonder Toys of the Shopping is assistant agent that we're doing today. Uh it's got a bug in it um that we planted for the purpose of this demo. Although, I hope you guys are not planting bugs in your own agents. Um it's built with uh like I said, the repo has a bunch of different agent frameworks out there. Um Mostly Fast API, Next.js, Chroma DB, uh and then agent SDKs as well.

09:28 Uh and it sends traces to Arize AX. It has a few tools available to it. Um and one of those tools, specifically the one with the bug, um, misfires a filter. And so, I'll kind of show you at a base level what the tracing experience looks like. Um, we made it very, very easy for you to set up traces, so you're not instrumenting things manually to get, you know, the information out of your agents.

09:56 Uh, we have skills set up that wrap our CLI and our MCPs uh, that help you get started uh, with tracing. We have provider um, all the major providers uh, instrumented already with auto instrumenters uh, and all the major agent frameworks as well. It's all open source, so very easy to get started with tracing. Sounds like a lot of you guys are familiar with tracing, and so you'll be very familiar with what the trace tree looks like.

10:19 Um, a user asks, "Show me toys for 9-year-olds." Um, and that goes through an entire request path, right? So, the agent's called, the agent calls a model, the agent has an end uh, LLM call. It has some tools available to it, whether it's searching products, etc. Um, and you can kind of see all the information available um, to ask it what the agent's doing under the hood.

10:47 Uh, so all these attributes kind of show, you know, start time, end time, latency, what the input was, what metadata was passed, what stage in the in the graph it is, um, results that it fetches from the DB, uh, and what at every single one of these stages, what information is available to the agent. Uh, so this is valuable, but again, you probably don't want to read this at 4:00 a.m.

11:07 Um, I think it would be kind of tough to read through thousands of these at 4:00 a.m. just to find where an error lies. Um, so what we want to do is we want to enable agents to make use of this information, and then see how we can actually close that self-improvement loop. So, I kind of walked through the trace. If you want to instrument, the AX CLI is what we use.

11:34 Um it has a guided setup. It's very easy to get started with it. All you need is an API key from Rise and it will take care of translating all those nasty little attributes into the beautiful trace tree that you saw there. Uh you'll get a span per tool call with arguments and results uh to the tool call. You'll get model calls, prompt token counts, all all the basics that you need to actually monitor what's going on.

11:56 Um as well as grouping. So, you can see sessions uh where there's multiple back and forth with a customer instead of just individual traces that are single back and forth with a customer. So, enter Signal which is sort of the second step in this platform, right? So, once you have these traces, what Signal does is Signal is an always-on agent that is able to read your traces on a schedule.

12:24 Uh and you can configure this in the platform to be kind of like whatever schedule you want. Um by default, it's uh set to every 6 hours. Uh you point it to a project that's, you know, has live traces coming into it. Um you enable it and it reviews those traces and can then do a number of different actions via its skills, right? So, [clears throat] the base action that we're going to show today is Signal connected to a repo on GitHub.

12:54 So, this agent Signal is reading another agent's traces. It's reading the source code of that agent based on sort of what's actually happening in production traces from that agent and the coding definition, it's able to actually propose fixes to what you're seeing. Uh so, I'll show you guys what that looks like here real quick. So, we have this Wonder Toys agent.

13:17 It's publishing traces in. We can see all the user requests. You can see I did a lot right before uh this presentation at 1:00 p.m. We can see, you know, kind of base facts like latency, token count, cost uh of running this project. And you'll see this signal tab. So, what Signal is doing is it's clustering issues it finds with the traces and actually proposing and identifying um a fix for for these issues.

13:43 Um so, for the purpose of this agent, the bug we planted uh is causing redundant search retries. And so, we're see what Signal is seeing is that based on the traces that are coming into this project, we are uh causing three sequential LLM calls and high latency because of something that's happening with the search products call, right? And specifically, what that's happening what's happening is that um the casing on some of these queries for the filtering is not working, right?

14:17 So, what Signal does is instead of you getting paged at 4:00 a.m. uh if you remember the meme in the beginning of this presentation, um Signal's finding this issue, analyzing, you know, clusters of these issues, writing an overview and analysis, writing a fix for you, and then what you can do is you can create evals to catch this issue moving forward automatically and trigger off those uh evals.

14:42 So, you know, if we run into this issue at any point in the next, you know, 30 days, we automatically fire an alert and that gets triaged by our team immediately. You can add failure cases to the data set here as well. So, that you have a snapshot uh for future regression tests on exactly this failure mode. You'll see a GitHub icon here as well to connect and open a PR or open issues well, so that you can actually get Signal to write a PR.

15:09 I've actually already done that, and so you can see the PR attempts uh that Signal has done to fix this, right? So, I'm going to go click into the session. And the agent worked for 6 minutes, found the root cause within the coding repo as well, based on the traces, and found that product product categories are stored as lowercase hyphenated slugs, but the element is passing human-readable names with title case and spaces.

15:40 Uh so, when someone is searching for car games, for example, without that normalization, we actually never get back any results. The agent is then going and searching every single product, right? So, you can imagine why that causes the latency spike up to 14 17 seconds. So, everyone following along so far? We have an agent, we have an issue. Signal is reading the traces for you, proposing a fix in connection with GitHub.

16:09 Uh so, actually reading the the definition of this agent, uh and then doing some analysis for you, and kind of making sure that that's available to you. So, this is one of many skills you can connect to Signal. Um you'll see here on the manage agents platform, you have the ability to use one of our out-of-the-box templates, so Signal is reviewing your project traces on a schedule, clustering failures, suggesting fixes.

16:36 We have a cost agent that can flag expensive models doing cheap work, recommend cheaper routing, and actually quantify your savings. Uh but, the interesting part is you can actually start from scratch. Um And so, what the agent studio allows you to do is you choose an agent config, you point it to a project where traces are coming in, you give it a task, and you can connect skills to it as well.

16:59 So, like I mentioned, GitHub is one of those sources. Maybe you have um you know, a third-party system that holds your R back information that and you want to check permissions on this agent. You want to make sure what the agent is calling is actually within its permission boundaries. You can You can a skill to do that. You can connect a skill to uh another APM system like DataDog if there's more logs you want to pull in.

17:21 Um there's really no limit to what skills you can define as long as you have an API key and it's callable from the agent harness itself. Um and we're actually under the hood running these as a Rise managed sandboxes uh within the platform. Uh you can bring your own harness, bring your own sandbox provider as well uh to kind of you know whether you're using Modal, Daytona, E2B, whatever it is or whatever harness you're using as well we support Codex, Cursor, Cloud Code, and Open Code with more harness support rolling

17:50 out. So, does everyone kind of understand, you know, we have a managed agent doing this in the platform. You can set up your own automations to you know either be triggered on a time schedule or off of an alert. So, anytime you see latency spike above 10 seconds, we automatically kick off this agent to do an investigation. Um and investigation agents are part of it.

18:12 Kind of solves the SRE use case. There's a cost lens to this. There's a product classification lens to this. Um there are many ways in which you can kind of use this agent platform to operate on your traces. Cool. Uh I will continue through these. So, Signals are reading your traces, filing what it finds. It keeps one ticket per problem uh the way a reviewer would.

18:37 So, you're not getting hundreds and hundreds of issues on the same uh piece of evidence. Uh it opens those. You fix it. You can either resolve it if you think it's resolved, you can ignore it um and it will dedupe those issues moving forward so you don't, you know, kind of get inundated with noise. Um the add to data set field or uh flow I think is one of the most important parts here because how do you build the actual continuous improvement?

19:04 You have to have a record of what's failed in the past and guard against those as regression checks moving forward, right? So, adding these two data sets give you that sort of living fixture quality. So, you know, you define test cases in code before you ship code. Everyone should be doing that, and hopefully we are all doing that with the help of some agents now.

19:25 Um but this idea of living fixtures that change over time as you add new use cases, as you see new failure modes, is a pretty novel one and that kind of helps with the self-improvement uh loop where, you know, you're not beholden to one definition at one time of what has failed and what our failure modes, but you're organically growing that definition of failure mode as your agent is failing in production, right?

19:50 So, um think of it like, you know, every time you go to try and ride a bike, you notice something about how you're riding, and then over time, those become so ingrained in your memory that riding a bike feels like nothing. You don't even think about the failure modes of um I actually fell off a bike and broke my foot 2 months ago, which is why I'm mentioning this this little anecdote, but typically you don't think about, you know, putting your left foot on left pedal anymore because it's been so ingrained into your

20:16 learning process. And that's what we're trying to kind of accomplish with these agents. Um so, on the data set side, once you get these living fixtures, you actually want to codify the behavior on these fixtures, right? And that's where the eval comes in. Um so, I saw a few show of hands of that people are running evals. Uh can I get a quick gauge of the room?

20:40 Who here is running LLM as a judge evals or you're running code evaluators? Cool. So, if you were in any of the earlier workshops of the Rise, we went into evaluators in great depth. Um there are many, many different forms of evaluators. Um so, just quick quick overview. There are code checks that are, you know, really, really cheap to run that are for deterministic checks, right?

21:02 Is this field present? Is this valid JSON? Exact rules. Was this filter applied? Was this tool called with the right arguments? >> [clears throat] >> Very, very easy for that, right? >> [snorts] >> LLM as a judge, evals also help with this use case where you need to have more of a qualitative or nebulous kind of analysis of the trace, right? And so, this is using another LLM model to score the results of one model's trace.

21:32 Um obviously costs will be higher here because you're making another model call. Um per record, obviously you can, you know, kind of set your own sampling, filtering. You don't have to run these on 100% of the traces that you're running at. Um there's some trade-offs in terms of eval design that we can discuss, but that's sort of what an LLM as a judge is for.

21:51 Then you have for the even more open-ended use cases, you have this concept of agent as a judge. So, where you're not actually sure what you're evaluating, you have a goal in mind, um and you leave kind of the the determination of whether that goal has been achieved to the agent. So, the agent can actually pick its own rails, it can actually pick its own scores, it can go through that process itself.

22:16 And that's obviously the highest cost method because there's multiple LLM calls happening under the hood. Um [snorts and clears throat] decision models have been around for, you know, what? A month-ish now, and with Lower actually at Arize has published a ton of material on sort of benchmarking them and how they operate, and what trade-offs are present with decision models.

22:38 I will say they are a very, very cheap way to run some of these checks on your traces versus um using a classic LLM. Um and they give you, you know, type type choices, yes, no, choice, score. So, once you have sort of this eval signal, right? So, you you understand, you know, qualitatively is this in the right direction? Is it not in the right direction?

23:07 Is it passing my eval checks? And you have the trace data. That's all really rich context for a coding agent to go um implement a fix, right? So, signal in this case is is running on on an Auri managed sandbox with a coding harness, implement those fixes for you, but if you're in your development phase, you can actually pull these traces live into Auri through the CLI as well, right?

23:31 So, it's instant context to feed other agents to actually work on those PRs for you. Um and this is where we have kind of the human check and balance in in the room. Um so, I would say the the places a human adds the most value here are one, obviously validating the fix, right? Reading the PR. There are, you know, maybe some of us in the camp of we don't want to validate every human PR.

23:58 I assume if you're in a high-stakes use case like finance, you're very much in the camp that humans read the PR before the merge, right? So, that's one very high-leverage point for a human to be here. In the data set curation phase as well, so adding that to a data set is part of why we left it manual is are these good enough findings to add to my fixture, right?

24:16 So, that's a manual step in the process, uh and that relies on kind of human judgment as well. Um the eval design, so we have agents that kind of help you write these evals, um but the eval design portion is something where humans can definitely be in the mix because you're imparting your taste of what is good onto the agent, whether that's in the form of an LLM as a judge or a more deterministic check.

24:39 Um and then folding all this information into the coding agent gives it really rich context for it to work from. Um so, just as an example of kind of like how you would create an eval off this, I'm going to go to my traces tab. I'm working off of this Wonder Toys project. I have signal running and signals found two issues so far. One is the orphan spans issue, um, which was actually just my mistake instrumenting this.

25:05 Uh, but it did catch it. Um, and two is the redundant search retries. That's the bug we planted. Um, so I'm able to create a new evaluator with one click off of this. And that actually, uh, looks like I need to use a different model. That actually opens up Alex, uh, which is our in-house agent that I'm doing a talk about tomorrow, um, that is able to go and create that eval off of that evidence for me, right?

25:37 So, based on the traces that are supported in the evidence here, we can kind of codify in a prompt template what we should be expecting moving forward and how we how we make sure that the eval fires and fails on any mistakes that are made. >> [snorts] >> So, this is the eval I came up with. You're a helpful AI bot that checks for redundant search product retries, inspect the complete trajectory including tool calls, see if there's any unnecessary tool call, return a redundant retry when search product is called more

26:07 than once, um, and put the score and classification choice on the trace itself. And then you can run this eval on incoming traces that are coming into your project, so you make sure you catch that, you can monitor that, you can graph that, you can create an alert off of that, uh, and you can track spend associated with that eval as well. So, over time, you know, maybe your development flow has codified this as a regression check and you no longer need to deploy this eval on production data.

26:35 You don't want to spend the model tokens on running this on every trace. You can actually view the cost over time and retire that eval as well. Cool. So, the next part of this is okay, we have a fix. Signal found something. It analyzed the repo. We created an eval to catch it for the next time. We added the evidence to a data set. So, now we have a living fixture that's growing with time.

27:06 How do we know that the fix is right? How do we actually know that, you know, it wasn't just slop code that the agent produced and tried to find an issue that didn't actually exist? Um and this is where the world of agent experimentation comes into play. And this is a very, very rich and deep world. Um so, I won't spend, you know, time talking about all the different frameworks and providers that exist out there.

27:27 Phoenix team did a good job earlier today talking a little bit about the harbor integration they have in Phoenix. Uh but really what we're concerned about is prove that the fix works and prove we didn't break anything else, right? Um and there are many cases where, you know, in this non-deterministic world, if you make a small change to a system prompt, it can then have ramifications on other tools that you didn't think were going to break.

27:51 Uh and, you know, kind of um redundancy issues there. Um and so, making sure that we can prove that this fix actually worked and that nothing else broke is really, really important. And that's kind of the experimentation stage. Um so, I will talk a little bit about this through the flow as well. So, in the Rise, we support a bunch of different types of experiments.

28:13 Um this is the category code enum fix experiment that I just ran. And so, you'll see here that we have actual evals running on this experiment. So, what we're essentially doing is we have this living fixture of failure cases. We have this proposed fix. We actually want to rerun the agent on these failure cases and score the results. And if they meet a certain um threshold, that's kind of up to you to define.

28:49 In my case, 88.9% I thought was pretty good. So, I was like, ship it. Um if they meet that threshold, you actually have results to prove that fix from the agent, right? And the agent can actually run these tests for you through the AX CLI as well. So, as Signal is going and finding these failure cases, and it has access to the agent code, it can actually pipe those failure cases through and replay them through the agent with the version two fix, and then score those results themselves, right?

29:19 So, you come back in the morning, you weren't working up at 4:00 a.m. You not only have a PR that has a proposed fix, you have some validation results that say, "Hey, I proved this fix because I actually took the test cases that we failed on, and I re-measured latency or tool call retries, and I saw that we were no longer retrying tools, and latency was lower.

29:41 And hey, by the way, you had three other evals that you don't want to fail on, either. So, I made sure to check that those evals are not regressing, right? And so, as that living fixture grows over time, you have new cases, these experiments become more and more robust, right? Um and in the platform, obviously, you can go view those experiment results, see what the output is, see what the evals are.

30:01 If you disagree with the agent, you can annotate that output, and that becomes another piece of information to the context for the next run, right? So, everyone kind of clear on this experiment flow at a high level, like how you can run some of these experiments to take a test case, take an eval that you've created as a benchmark, and actually validate your fix before it goes out, before it goes to prod.

30:26 Um and you can shove this into the CI/CD process as well. Um up to you on how you want to configure it. Obviously, running the the agent may be expensive. Um so, your discretion on how you want to run that. Evals as well, right? If you're using an LM as a judge eval, you want to be careful about which model you're picking, when you're running those evals, what sampling rate you're running those evals at, to make sure you're not just ballooning costs and you know, just sending a ton of spend on validating these fixes on

30:55 every push. Cool. So, kind of continuing on, we approve the fix, we look at the task completion rates. Um we validate those results and while you kind of step away from this process, what keeps running is that the eval that you created runs on every new trace and becomes additional context to the next signal run. The task system that we have in arise can also automatically add failed eval results to new databases or data sets so that that fixture grows over time.

31:33 And signal keeps rerunning every 6 hours or off of an alert or off of a triggered monitor and make sure that new failures are surfaced, new evals are created and that data set grows with those new examples over time. We also have, you know, weekly digests where you can kind of consume these in a little bit more of an easy-to-read result where you just have a summary of what's going on with the agent.

32:00 What is waiting for you is confirming that these issues are real, right? You can't just trust every single issue that comes out of this agent. I would say use it as a starting point, validate these fixes and use your judgment on when those issues are actually real. Writing down what the right answer looks like, so this kind of goes in the annotation flow, right?

32:19 You run the experiment, you get a new output, maybe you don't think this is the perfect answer or the perfect trajectory for the agent and you can actually use the annotations product to kind of codify that taste. Merging the fix, this is up to you. You can auto merge these PRs. I will leave that to your discretion. Uh and then actually deploying and promoting the new version, right?

32:40 So, I have V2, I validated it, I think the issue is real, I created the eval, I looked at the score, I validated the fix, I like the code, I reviewed the code. Now it's time for me to actually upgrade the agent. Um I think the the other thing that's really interesting about this is like you can codify the confidence scores, especially with these decision one models that actually return an explicit confidence score, and have those auto route to human beings uh in annotation queues, right?

33:12 So, um not only are you, you know, responsible for reviewing the outputs, but you actually get a ready-made queue for you that is based on all the low confidence results, and um you know, in some of our benchmarking uh we've seen these confidence scores to be pretty good. It depends on your use case and uh specifics in the industry, for sure. Uh so, I wouldn't say that an 80% confidence score is the right threshold for everyone, uh but setting those thresholds is important, and then once you have those thresholds

33:41 defined, you can kind of auto route those low confidence cases to a person to go annotate. Um So, I mean, there's, you know, in the spam test that we saw, 4.6 % of cases scored between 0.3 and 0.7 confidence to people, left the rest at 99.5% accuracy, right? So, that is one example of an industry where, you know, specifically for spam detection, where those confidence thresholds might be more defined.

34:09 It It depends on the use case, for sure. Um So, I kind of talked about some of the stuff we've built so far with signal manage agents and traces, right? So, the ability to actually read those traces at scale, the ability to add things to data sets, uh the ability to run those experiments with the help of a coding agent, um, and actually review those results in line.

34:35 You can also interrupt sessions. You can look through the model's reasoning. You can kind of continue a session, kick it off to a local agent, uh, so that you can continue development on your own side if you don't want this all managed by a rise. But, I think our vision for the future is this running this full end-to-end loop with almost zero human involvement until you need to go test something, right?

34:56 Until you need to go click around in a UI and validate something. Um, and so there are a lot of providers out there for that experimentation setup, right? There's the harbors of the world. There's many, many different ways that you can test the agent. Um, I think most of it, and and this is where a lot of the discussion is happening, is how many dependencies do you have to run to run this agent, right?

35:17 Um, running an agent in the experimental phase, does that mean you connect it to a mock DB? Do you stop some tools? Um, do you actually run the entire thing in a docker image? Can you even do that safely? Um, those are all sort of questions that I would encourage you guys to think about as you start thinking about this experimentation flow. With this agent, you know, it's very simple.

35:38 We just rerun the agent in its entirety. There are no dependencies. It spins up its own DB, and then we run requests through that. So, that might be a very simple use case. But, when you have external dependencies that really matter, your use case become can become bit more convoluted. Um, so our goal for this is really enabling the one click from issue to experiment.

36:02 Hey, I have this proposed fix. Go test it. Spin up all the infrastructure necessary to go test it. Um, in addition to this, and we've we've worked on the CLI around this, is connecting signal where you work as well. So, whether that's in Slack, so tagging uh, from Slack to communicate with signal, whether that's your local agents, which we have through the API, CLI, and skills, where you can actually ask your local agent to go do a digest of what signal is finding and and check those.

36:30 Um making sure that we have the fix drafted straight from the issue and its traces and that's part of what we've done already and actually having the evidence in line. So, here are the actual five traces that failed in the PR description so you can actually go validate those. Um we have signal now proposing evals as well for you so it'll actually check your evals see which ones are healthy as in, you know, if this eval is passing 99.999% of the time maybe we don't want to spend the tokens on this eval and so we can

37:01 retire this eval going forward. Uh we feel pretty confident on it. Or, you know, vice versa if any eval is in poor health, making that eval more robust. And we have a whole process on eval alignment which is how do I make sure the judge is actually aligned to what I think and isn't just outputting scores that I don't believe in. Um so there's a whole process for doing that.

37:25 And then, again, the experimentation flow, running your agent against running these data sets against your agent wherever it lives. Uh whether it's your laptop, in my case for this very simple agent, or whether it's on more complicated infrastructure. Cool. So, I think it would be really great to get started with some of these. Uh so I have the workshop repo URL.

37:48 Um we have docs pages as well for agent experimentation. And um we have a few other talks throughout this week as well. Um So, if you has one tomorrow at 3:20 about um agent fixes and their run-off um implications. So, you know, you fix something somewhere and that eventually has impact downstream. Um and I have a talk tomorrow at 3:45 talking about Alex which is our in-house agent.

38:16 Um and we use Alex to build Alex so it's all kind of the perfect dog fooding tool for us in the agent space. So, I'll talk a little bit about how we designed that agent, you know, how we design context, memory, and some of the decisions and lessons we learned along the way. Really, I would encourage you guys to try to get some traces from the agent.

38:37 We have this built out for you, but if you have an your own example and you haven't traced your agent yet, it's very, very simple to install our skills and CLI, have it instrument your agent for you, and send those traces to Arize. Um once you get those traces in, it's one click to enable signal, and then you can start to see the clustering of issues and patterns across your traces as well.

39:00 Um so, yeah. I would say the things I want to leave you guys with are the hard work of reading a ton of data should definitely be just be done by an agent. Understanding where to add the highest leverage human points is kind of the theme of this presentation. Just because you can automate something doesn't mean it should be automated, but you should use that automation to really inform the reviewer.

39:27 So, things like auto routing low confidence threshold scores to that reviewer queue are really important. Um and making sure that all the context is fed back to the same system, and obviously we're here with Arize, and you know, we're telling you how great Arize is, but really I you want to make sure that this context is recorded somewhere, right? And that's the only way you build the self-improvement loop is when the human judgment, the eval scores, and the traces, and the the trace evidence are all held together in

39:58 the same place feeding that context for the next run, so you're catching new issues instead of just fighting to stay afloat. Um and that's what I have for you guys. Thank you so much. Um I'm happy to walk around and kind of help you guys get started on some of these. Um you can check out the repo, clone down the agent. If not, uh we you can install the skills, instrument your own agent, and yeah, I can take questions as well if you would like.

40:23 I would say there's like two lenses to take how we're actually evaluating. There's how do we evaluate signal as a product line, and that one of the key factors there would be, okay, how often is this right about our PRs, right? That's sort of like an internal evaluation. Do you mean more of that how we evaluate our own product or how you can evaluate an an agent use case, generally speaking?

40:43 Uh we will do a whole talk about evals for Alex tomorrow, but to give you the long and short of it, um eval design is you know, as an as an AIPM, I think that's probably my primary job is like how do I actually design good evals that actually measure the right things? The shorter answer to that question is I think codifying qualitative decisions is very hard.

41:09 Codifying things like latency, retry count, like these are more classical issues that, for example, the issue signal found, I think, is relatively simple, like you found an inconsistency in casing, you saw evidence in retries, make the fix, measure the latency. That's really easy to do, right? Now, qualitative outcomes, I think, are a little bit harder, and I would say there's there's a couple things to index on there.

41:32 One is the the actual like definition of what is good is only emergent from as much signal as you can give it, and I use signal as like the word signal. Um so what I mean by that is, okay, I can look at one example and say it's right or wrong. I can't get an agent to learn from one example because then it's just going to overfit to that one example, or that example might not generalize properly.

41:59 Um and so part of what I was talking about with the living fixtures and the annotations is a lot of what we do. So I'll give you an example, Alex generating um filter queries, right? So we have Alex in in platform. Uh we have a flow where, okay, if you want to search agents or search traces, you can obviously type in a filter, or you can ask Alex to filter with AI.

42:22 How do we actually measure the performance of that? There's no right or wrong answer, cuz it could return a filter that succeeds, and it doesn't, you know, parse this JSON, and it will give you some results, but how do you know that's actually a good result? So, there's a few things. One, we look at the entire session of users, right? So, we're not just saying, "Okay, for this turn, was this a good response?"

42:42 But, did that user ask again? Did it Was it happy with the results? How many additional filters did they create, right? So, that can be one thing to as a signal for you to evaluate on. The second is, I mean, annotations like reading through, okay, what was the query? Does this look like a right filter? Did it return the right eval results? And actually annotating that by hand as a PM who knows what to expect.

43:05 Um, and then you combine both of those with new use cases, and that rich context becomes the eval template, right? So, you can actually use another coding agent to feed that context in and write a really good prompt template, and then it becomes like a a prompt optimization essentially, right? So, like, how good can I get this eval template on the lowest cost model so I can run this at scale?

43:28 Um, so, that's kind of how we think about it. We have some more specifics to talk tomorrow where I'll talk through a little bit of which models we choose, and um, how we think about which use cases are worth running an eval on or not. Um, but yeah, that's the short answer. So, it is kind of like a, you know, post hoc algo propter hoc, like, where can we make sure like what we define as healthy is actually healthy, and where should we define where it breaks?

43:54 So, So, what I mean by that is, okay, it's right 99.9% of the time. If that 0.1% is catastrophic, and we need to catch all of those, it's a very different discussion, right? And then evals, when I when I say like Evals there's kind of two lenses. There's the Evals that you can run in the development flow as CICD checks, as regression checks before you deploy code.

44:15 There's Evals that you're running on live traces, which that becomes a cost-benefit ratio question, right? If I'm running this on 100% of live traces is the cost of running this model on all my traces worth it compared to the cost of this 0.1% failure, right? So I think a lot of it boils down to ROI. I think general guidelines are whatever you can catch in CICD you should, and I understand it's a challenge to run some of these.

44:44 So like and and convince teams to run some of these in CICD I would probably make some of these non-optional checks in the CICD process, right? So you have a data set of fixtures that shows where this Eval has actually failed on the 0.1% of cases and you run you rerun those with an Eval score every single time you push or before a PR is merged. So you're at least you at least have some confidence that that set of cases is not going to fail.

45:11 And then you run an Eval that has a sampling rate on your prod traces, right? Um so instead of saying, "Okay, we're going to ignore the development process, evaluate everything that comes in production." That can be a very high cost thing to do versus "Okay, we ignore prod completely and only test in development and try to capture every case, which you probably can't do um in a data set with coverage, and then just let it happen in prod."

45:35 I would say both of those are extremes, and the answer is somewhere in the middle. And it depends a bit on the ROI of the Eval, right? Um and what model you decide to use has an implication on cost, etc., etc. It's a very good question, and it gets very meta, and uh I I hope I hope you haven't noticed how meta this conversation has gotten so far where I'm using signal in multiple different ways.

46:00 Um the short answer is you can do that entire loop again, right? So, we have a project for eval traces. This is a Rise created. These are all the evals, and those evals are traced, right? And you you can have an agent reading the eval traces and making sure the evals are good. And then, each of those agents actually produces their own traces, right?

46:21 So, when you're running Signal, these are Signal's traces. So, you'll see here you are Signal, an automated reviewer for Rise. These are these are steps you have to do. These are the instructions. This is what Signal actually called under the hood. So, made a few bash calls. It called our MC internal MCP to go do this investigation and then mits it, writes it as an issue, right?

46:41 And then, you can run Signal on top of Signal as well to make sure that And that's kind of what how we evaluate internally, right? Uh, it's kind of what I was alluding to as the the ultimate dog fooding tool. Um, now I I assume your question is more of like a when does this break down? When is this useful to do? Um, I think the harness changes are useful up to a certain point, and I'll talk a little bit about this in in my talk tomorrow on like how we evaluate Alex.

47:08 So, like there are certain use cases where you just a- after a certain point, you don't need to continue evaluating, uh, and there's just diminishing returns on cost from there. Um, where we've seen it is, you know, we monitor Signal and we have, you know, traces coming in from Signal. We have some evaluation score criteria, but there is a certain point at which we actually don't know what is good for our customer beyond what the customer knows themselves.

47:36 Um, so, I mean, you can turn on Signal and it can propose the PR, and we can use some proxy metrics like merge rate on the PR, or the session transcript or whether session was handed off or whether data set was created or eval was created as feedback mechanisms, but beyond that and beyond that and the merge rate of the PR, we can't really tell that much more.

47:59 So, it becomes, you know, diminishing returns to continue to evaluate from there. Um cuz then you're just kind of relying on an agent transcript and you have no ground truth, right? You're just vibing with the agent's output and pretending you're increasing shareholder value, right? So, um I don't think that's useful. Uh but I think we have, to answer your question, we have used signal to evaluate signal as well.

48:21 I think in execution time checks, because such low latencies like probably the most exciting, like where you can actually route a request during execution time. So, these these evals that you're seeing are are, you know, post talk where the the agent has already run and we're evaluating it from that. So, due to the low latency of decision wall models, I think that's something I'm actually genuinely excited for uh in the execution loop.

48:45 Um Like selfishly, or maybe not even selfishly, but like for my personal enjoyment, I like that people are much more likely to use evals now because they're cheaper. Um so, that's like just, you know, purely good for the ecosystem health. Uh like more people thinking about evals, more people thinking about actually classifying their traces. I think it'll unlock that to a large degree just because it's cheaper.

49:10 Um And then, maybe not related to to this talk, but like just my my personal opinion is I think like um because these models are so small that there's some really interesting stuff you can do with on-device stuff um that I'm really, really excited for. Um we're also like kind of pioneering it for some multimodal evals as well. Um because I think multimodal evals can be really complicated because of latency.

49:35 Um and they can also be super, super qualitative. So, in the multimodal eval space, like trusting an agent's taste on an image that was generated, for example, or how it interpreted a click box can be very hard, but getting confidence scores allows you to at least route this to human a lot faster, so I think that's also a space where decision models will help a lot.

49:55 Yes, 100%. So, I kind of briefly alluded to this when I talked about skills, but you can actually create your own agents. Um And we have the out-of-the-box signal agent, obviously. Um Let me do this one. So, you'll see here we have a DataDog skill as well where you can pull up metrics, logs, and APM traces around the alert. Um we have a bunch of providers available for you to do that as well.

50:29 You can also write your own skills, so the the agent is doesn't just have access to the agent traces, but downstream system application metrics as well. Um so, you can connect it to basically any data source. Um Obviously, there's like an element of taste to the data source like connect the ones that you need to connect, make sure it has the right access, make sure, you know, the scope key, make sure that you're not over flooding context because more isn't always better to actually triage the alert.

50:56 Um but yeah, we have the ability to do all that as well. The the shitty answer that everyone always gives is it depends, but it it does kind of depend. So, for example, like what harness are you using? Um what underlying providers are you using? How often are you running this? And we expose those as configurations. We allow you to kind of monitor the cost, too, so you'll see here that this is the signal traces.

51:18 This is how much it's Oh, I guess this is still flowing in right now, but the same way that you can view kind of your cost for like Wonder Toys, you can view the cost of signal. Um so, we make it easy for you to understand the ROI like, okay, if this shopping assistant agent cost me $8, and it cost me 20 bucks to improve with the self-improvement loop, is that really worth doing or not?

51:41 Um Now, like I don't have like a hard number as like 5% of your agent budget should be allocated to the self-improvement loop or something like that. I think it really just depends. Um but I will say like that is exposed to you and available to you and there many ways in which you can kind of pull the levers to make that a little bit cheaper or more expensive depending on how you're seeing results.

52:05 Um so one is the amount of context you're providing, right? So which skills are you including up to a certain point without like just spending more money to give it more information that doesn't actually help. Um and then the other is like the eval kind of design question, how often you're running these evals, sampling rate, etc. Um And then the other thing is like you I mean you can use this part of the platform, but all these are consumable via API as well.

52:30 So if you want to shunt this to coding agents, which you get a subsidized cost for or something like that, you can just export that to a coding agent and have it do many many of the similar things. Thank you for the wonderful questions. I'll be around over the next 3 days. We have the Rise booth as well. Um feel free to sign up. Try this loop out for yourself. Completely free to sign up. Completely free to use. Um and yeah, happy debugging. >> [applause]