Scheduled Agents, Dreaming and Q&A: Claude Managed Agents (Part 3 of 4) — Anthropic Transcript, AI Summary & Key Points

00:01 [music] Okay. So, yeah, let's let's see what this looks like. Uh, analyze uh Google and Microsoft. Um, so ideally what would happen and I hope this works live for y'all um is that uh this main agent is going to um evaluate if it has information and then it's going to give us a custom tool where we're going to then spin up individual sessions for each of those financial artifacts.

00:36 So you'll see this dispatch analysis thing. Um it's going to call it for two tickers and then it's going to kick runs for each of those. Um so this is how we can orchestrate our multiseession. um harness and each of those sessions has multi- aent uh that are doing collaborative work and each of those uh also have outcome definition so that there's self-supervision happening using memory stores to learn from one another um etc etc.

01:03 Um so let's move on to the finale which is um what if you want to do recurring work? What if you want to do uh you want to have a list of companies you care about and you want every day at 9:00 am or first of the month to do the analysis over all of these companies uh on a schedule. So that's what deployments are. Deployments is everything is API based.

01:25 So we you can obviously interact with our platform via the console. Um but their main way that we intend to you for you to interact with it is via the API and the SDK. So you can build uh custom UIs and custom workflows. Um all of these are just building blocks uh that you can compose together to build your application into something bigger. Um so you can see here what cloud did is it created a uh deployment um and that is right here.

01:52 A deployment is takes in a agent definition. Uh in this case it's this research uh desk weekly memo using this specific agent. um it takes in a schedule uh of when it's going to run and it gives it an initial set of messages that it's going to run with. You can also pass in uh resources like uh uh memory stores or GitHub repos or files that it's going to use on each of those.

02:17 Uh and so what this means is that every I think this is first of the month at 1 p.m. or something like that. It's going to run um this uh specific session. I could run one now. So I could trigger it via the console. it's going to do that. Um, but I could also build a UI for it and that's what we've done here. Um, so I can, you know, I have all the crowd.

02:40 I can list and and search uh for my deployments and then I can also kick one off um at any given point. And I don't know why they're failing, but I'll ask Cloud about it later. Um, and this has come up a few times, which is what about dreaming? What about self-improvement? Um, so before I get to your questions, which I'm sure there are many, uh, I'm going to show you a preview that we've, uh, have out for for dreaming.

03:05 So what dreaming is, it lets you select, um, so, okay, before I show you a previous one, it lets you select a memory store. In this case, it we'll we'll use the memory store that we've been using. You can select a model. Um, you can give it custom instructions if you want to, but we'll use a default prompt. Um, and then we're going to let it uh run uh the the last 30 days of sessions through this streaming analysis.

03:31 So when I click start streaming, what it's going to do is it's gonna spin up a session using manage agents actually um spin up many sub agents to analyze the transcripts of each of the other sessions that we told it to evaluate at all of them which use the memory store. Um and it's going to analyze, hey, did the memory store encode all the information that was required?

03:51 did a session uh run into any issue that uh that caused problems? Can we learn anything from it? Did this happen multiple times? Can we update the instructions in any way? You can see here what it actually did is that it found that the skill that we created actually had a few gaps um on the way that it was written and it updated the memory store to tell it even though there is this skill, you should extend this knowledge with the fact that we've learned from it.

04:16 Uh and you can imagine that we will continue to extend this to not only improve memory stores but improve skills directly uh agent prompts, sub agent prompts, etc., etc. Um that's that's it. That's the demo. I sorry about the Wi-Fi. I hope you follow uh later on when when you get home. Um and let's uh keep going with some questions. >> Yeah, we can all take questions.

04:40 >> We have a question over here. >> Yep. >> Go ahead. >> Uh thank you for the demo. Uh we didn't see any uh telemetry uh instrumentation in the agent. Is that a default native feature? >> What do you mean by telemetry? Because you have the full observability of all of the events uh that happened. Right? If you go into the console, you can see it live, but you can also obviously get it from the API where this transcript view is like a nice view, but if you go to debug, it's every event that occurred in that session.

05:07 So you can see exactly, for example, when did we start calling the model? uh when did we finish calling the model? So you can get latency metrics in that. Is there something else that you mean by telemetry? >> Yeah. Uh the thing is that uh usually in software development cycle you kind of define what you want to observe. Uh how did we kind of pick what did we want to observe.

05:28 >> Uh right now you get all the events and you just filter them on your end. If there are things that you think are missing, uh hit us up and we'll talk about adding them. >> Okay. And can I bring my own observability stack in here? Uh you mean like export to OTEL? >> Yeah. >> Uh you you could because it's an API you could build that connector fairly easily.

05:48 Uh we just everybody has different requirements and we don't want to be opinionated but everything is just API calls. >> Mhm. Thank you. >> We have a question. >> Uh I I had a question. So how long can these agents run? I I I know you mentioned this dreaming thing but can you run them indefinitely or do you see that persistent store keep increasing in size and at some point you need to recompact it offline or something?

06:15 >> We we are running compaction for you so that you can run these for days. We've seen 10-hour sessions in production uh from some of our users. >> So you could run these agents like for a year or indefinitely in theory. >> Yeah. Are you talking about running the agent for a year or you talking about like one specific session that keeps going for >> No, basically the agent if I had to if I had a task that I need to do every day, can I just keep it running for 6 months?

06:39 Will it not blow its uh persistence? >> That's right. >> Yeah. >> And is there any orchestration piece like where I could have three agents, agent A does something and delegates to B and C. That's some capability that you have or >> for for now you would spin you would do sub agents and you would tell claude what you want. We find that uh there's a desire as humans to constrain what claude does and the way it does it.

07:03 As cloud gets smarter we have continuously remove those guard rails and let claude just let it cook uh let it do what it thinks it's best. So uh right now you think maybe three sub aents is the best topology for it. But if we just let Claude figure that out, it actually ends up outperforming. >> But any topological or like graph based workflow thing in the UI here like where I could have >> Not yet.

07:30 Not yet. Not yet. But but maybe soon. >> Uh >> you can probably just like speak up. Yeah. Sorry. Sorry. We'll come back to you. >> Sorry about that. >> Hey over here on your right. >> On my right. Okay. Yeah. So question. Um, looks great. A lot of integration points. What I'm curious about, uh, for, you know, an organ, a a complex organization that potentially manages multiple agents, um, you know, I I I think they want some sort of separation, arbback, whatever else.

08:03 Um, how is that what is what is I guess the recommended approach for uh, a complex organization that has those different agents in here to effectively manage? We we have workspaces that's our primitive today where you can put different customers in different workspaces and then uh segment them that way and give access to specific users into those workspaces.

08:25 Uh but we know that there's still a gap in what we offer and um that's that's something >> our back is on our road map. So we we do want to provide more like granular controls but right now workspace is like a clean way of like separating your agents where you have like a group of people access to workspace A and they can access all agents within that.

08:46 You can have workspace B with like different set of users that can have different levels of access. >> So hi excellent presentation really excited. Uh two questions. One is deployment. My understanding was that a deployment is like at a snapshot a moment in time of all the agents. >> No >> deployment is essentially a way for you to run sessions for a specific agent on a schedule.

09:11 So you can say like oh research desk agent go research NVIDIA once every quarter and it'll just create a session every quarter to go research and produce you that report. And then memory which is a state is persisted across all sessions. >> That's right. >> You attach your memory store to the session. So you decide which which session you want to attach which memory store too.

09:33 So the memory store is like a independent entity that you just attach whenever you need to >> right and the final question is harnesses right one what is harnesses for you because everyone have their own definition >> and second is the confident right. So when I change something here, how do I know that it's not >> it's working as expected? Right? My own confident as a developer.

09:58 >> Um for you're right, everybody has a different definition of a harness. Uh we when we talk about a harness in CMA, we mainly mean the tool loop execution. So managing the context and managing the session state and making sure that the agent can do the work that it wants to do. When most builders talk about harnesses, they they really mean, you know, they really mean um the ability to coordinate multiple agents and sessions together as a unit of work to achieve some sort of bigger task in a topology of sorts.

10:29 Um we're giving you the abstraction of a single session as a unit of work for you to go out and build your harness uh with it as the session being one of the primitives. >> Um >> and then your second question about how do how do I know how my agent is performing? Our best recommendation would be to run evals on your agent with like different types of tasks that you want the agent to perform and see how the agent does on those tasks.

10:52 Um outcomes is a really good way where we steer the agent for you to get you the specific outcome where you can define an exact rubric for like hey this is the exact outcome I want and you can define max iterations and the agent will keep going into it until it gets you that outcome. >> Yeah, the the the verifier agent that that Gabe talked about. >> Hi So >> after.

11:17 >> Cool. Hey, thanks for the demo. I enjoyed it. Um my my question at the core is a bit about how to choose between skills and agents and I wanted to ask how would you all say what's like the fundamental difference? Is there an advantage to the harnessing that surrounds an agent that makes it produce better results than a scale? Um, and one of the reasons I'm asking is because I'm afraid of always investing in new kind of like I guess harnessing and tooling around models and it's it can be kind of tiring always trying

11:47 to do a new thing. Um, so yeah, I just want to hear what you think is like is there a big difference and is it going to change any time? >> I yes, no. Um, the qu thick hardness, thin harness, thick skills um is is a question that's constantly evolving. I think what we find is that as Claude gets smarter, the boundaries and and and help that we want to give Claude start to melt away and disappear.

12:12 And so what you need today might not be what you need tomorrow. The boring answer is just run evals and and evaluate these things. That sounds exhausting obviously like yeah >> every time a new model comes out I need to evaluate all these things. Um what what we find is skills is a good abstraction today. uh it gives the model enough >> information to do what it needs to do.

12:35 But I presume that it will start to shift and melt and become thinner and thinner where the model will just be able to do things on its own without any special requests and instructions uh down the line. So uh multi-agent seems to be more useful uh in my opinion, but your your mileage may vary on your task. >> Hi. >> Hey. Hi. So my question ties up to last two questions and especially on top of evals.

13:03 One thing I'm excited about is dreaming but how will dreaming actually affect the final outcome >> something that's where eval comes into the picture but you can have multiple iterations of dreaming sel that can go in different directions. So is there a plan or do you already support some automated kind of dreaming with run with evals and see that hey how to different direction go into which one should we pick based upon what's going on.

13:30 >> Yeah. So so dreaming what it does is it looks over you you can choose the duration for which you want the agent to dream over. So you can say I want to dream over this agent session over the last 30 days and then look at all the transcripts and analyze what task you're trying to achieve and then come up with like a new memory store that's more geared towards like what you were trying to do and we typically will run evals on our end to make sure that dreaming is providing you with better outcomes and we've seen

13:56 customers were able to get be get a significant improvement on their evals by running Dreaming and then using the new memory server but again everything kind of ties back to eval. So like that that's how you would know whether something is performing >> as you expect it. >> Yeah, I think it'd be good to have your own set of evals. Um >> the other thing is dreaming is in research preview.

14:25 So these are early early. It's like a big experimental thing. We've seen uplift uh on it generally. This is we we hear you right like uh how do I gain the confidence that this is >> yeah it's like eval connected to dreaming and then we can find out multiple pathways that that has been taken and decide oh this makes more sense because even though nine sessions may have been saying that hey this is wrong >> one says that hey this is good so if you just rely on that session dreaming thing I feel like we may not be

14:55 grabbing all insights without evalu >> if I tell you just to trust me is that no [laughter] Thank you. >> Yeah. >> Hi. Uh thank you very much. Um this is very impressive. Uh those of us who have been working with cloud code for a year or more um have been doing things that are similar to this but this was nice to see this in a really mature form. Um, one question I have is how much of this was just built on top of cloud code?

15:27 That is if somebody was, you know, ambitious, could they could they build something similar to this? And um, what I wanted to ask you was uh, encourage you to keep things open. uh some of us are already harvesting what's in the JSON session files uh and starting to experiment with things like uh hepo and that kind of thing for for managing the um the memories and um and creating a memory pool.

15:52 So uh congratulations. Thank you. Please keep it open and um can we is there are you doing anything here that's not just built on site on top of cloud code? Thank you. We we took the cloud code harness as an inspiration. We found that it was very well attuned for coding tasks. Um and different it's different levels of flexibility for other tasks. We're building an API where we want you to bring your context, your information, your capabilities and build on top of it where cloud code is trying to give you a a product

16:26 out of the box. So we have slightly different uh you know value uh props um and so it's not just clocked out in a box uh though you know ideally it performs as well if not better and all sorts of tasks. Um so not not exactly but it's definitely inspired and motivated by by some of the uplifts that we have seen from cloud code >> and and cloud code is a terminal tool and we obviously want to be an API product that and infrastructure uh that is going to scale that's going to be reliable.

17:01 So there there there were like challenges that I talked to you about um in the beginning of the presentation that we really worked in to make sure we're providing a really solid um product for you all to use. Yeah. Um, so for a bedrock, is there a console like clawed console like you have here? That's quite straightforward question, but yeah, I just want to manage and see everything.

17:24 >> This is not available on bedrock, but it is available via cloud platform for AWS. So there is a AWS version of console that you could go use this with. >> Yep. >> Yeah. Uh first uh uh is there any uh how you say is there any restriction on the model to be used like uh non-anthropic models >> uh >> just like cloud code I can run on nonanthropic models >> for for now we we you >> yeah it's cloud models only for now >> okay so so the second question is what about end user use uh do they have to have a console accounts

17:59 in order to use it or just a free uh I can create a customized UI for this and they can use it >> so cloud Cloud Console helps you with creating your API key. >> So that's how you get access to our APIs. So you cloud you're not paying separately for cloud console. It's free to sign up. You create an API key and then you're paying based on like token usage.

18:20 So for for for the case like uh SAS like uh maybe we have multiple tenants right. So in order to provide this uh agent what kind of a uh how you say model you have already uh depict on so that we can follow for example what kind of a each company are supposed tenant are supposed to have a uh a different space or or even there is a a sub module for this uh each tenant.

18:45 So are are we going to how do we actually going to organize that? In terms of this >> we we we have seen that the best model for multi-tenency is not bring your own key because you lose observability because it runs on your end users API key rather than than yours. So we encourage you to model things with workspaces as a tenency primitive. Um and we're improving that so that it's easier and easier to build uh on top of this for multi-tenant applications like >> yeah so I think currently you could either have put all

19:17 your customers data into like a single workspace and then you maintain a mapping on your end on like which agent maps to which of your customers. Alternatively like Gabe said you could create a workspace per customer of yours but that can get more complicated because then you have to like recreate your agent. So, so is any you see the examples that uh people did the additional security layer or RBAC layer over the this workspace because to me it may be a little bit you know >> yeah that that would probably be like a

19:47 good good way to do it right now because we don't have any arvback >> uh controls uh on our level other than like workspace because everything within your workspace is accessible to everyone in the workspace but we are working on making our back better and then making the experience better if you're working with cloud manage agents but building on behalf of your customers and how do we make that easy something >> so so is there any way you can show how to scale this uh if there's a multiple uh how you say tenant

20:14 working on that >> yes >> any scaling parameters like uh what is the how you say the the minimal uh pod or whatever these parameters that's was yeah >> uh I mean you're you're defining on a you can create as many sessions as you want yeah like there's no limitation I mean we have some rate limits obviously but um but there's no scaling parameter for you to say now I need to run 10,000 pods >> so so currently no no such thing as like uh like HPA in the Kubernetes like you can do these kind of setting scaling uh

20:47 parameters right like timeout or whatever >> you you don't need that >> you don't need to because we're managing it all for you you're just making API calls that's all >> okay okay sure uh thanks for your uh answer the last uh by the way I the offering link does not work for me I'm not sure if it's the only one. >> Yeah. Thank you for the sharing. >> Thank you.

21:06 Thank you. >> Yeah. Go ahead. >> Hi. First, thank you very much. I've been having a ball with everything that you've been releasing and um I'm really interested in how you're using UI. Um you're spinning up a server, you're distributing this, but how might we deploy that either internally, but I'm interested in to sort of a customer base. If I want to make for example an astrologer >> um that person has to enter sensitive data.

21:36 So it's actually really nice for me to be able to like have that in a >> have a layer. Yeah. >> So there are some procedural things you get like a star chart example. I'm not building this but it's a good example like I can have a sub agent do an interpretation that could last for like 90 seconds. It could create some assets etc. When the outcome is met it could shut down.

21:56 So this gives me a way to actually just make a really clean implementation where all I need is off >> and then everything can spin up from there. >> Yeah, exactly. >> So there are a couple of things like MCP channels that kind of point to this with like fake chat for example, a way for you to kind of just use claude and have a channel to communicate with it through UI.

22:18 But I'm seeing a lot where you're just spinning up a server inside a project that's easy to distribute internally. But how can you deploy that at scale? >> Um I mean we've this is uh being deployed today at scale in production. For example, notion external agents are built on top of this. So this is something that exists today that you can go and use and uh basically leverage cloud managed agents through notion in Atlassian for Jira integrations for example.

22:46 Uh there's definitely ways to do it. I think if you have specific questions we could talk uh >> yeah we we can talk after this but typically you would probably build your product your UI that you're distributing to your customers and then calling into us for like your actual agent >> the phrase >> management everything is through the API >> everything that's right >> so you will manage all the API calls and build your product and UI around that yeah >> hi >> hi hi first thanks for a great workshop and demo and uh my

23:15 question is Not really to like the agent itself. It's just like I saw the Android just released the sunet five. Yes. >> So I just was wondering if any anything you guys could share with us. Yeah. >> It's a great model. >> Yeah. [laughter] >> Try it. >> It is supported in cloud manage agents. Go give it a try. >> You can use it today and see. >> Hey there.

23:43 Um hi if we've got uh business partners that are using like cloud co-work today some of them more techsavvy cloud code to build some like spot solutions for specific business cases. >> What do you see as sort of the level of sophistication up level to start using man like managed agents instead because this seems like more powerful >> but the you know cloud platform could be intimidating to folks that aren't engineers let's say.

24:05 Yeah, I think the console quick start is probably like the part of my demo where you're literally just like telling Claude the agent that you're trying to build should make that pretty easy for even non-technical users to come in and build their first agent. But obviously this is an API product and is naturally going to be a little bit more intimidating.

24:24 But if they're comfortable with cloud code, we do have a manage agent skill with cloud code that basically just like in natural language will walk you through building your first agent. And that would probably be my advice uh to try it out. I think the biggest difference is co-work building building personal agent. With this one, you're trying to build probably an agent that's like available to like your team, your internal organization, or you're building a product that you're shipping to your end customers.

24:48 But if you're building personal agents, co-work is still a really great tool to continue doing that. >> Are there any skills to kind of like take something that exists in co-work? there's a skill to interact with our API. So that cloud is probably good enough. So if you say here's what I did in co and here's the skill to call the API make the translation I feel pretty confident.

25:13 Yeah five we'll we'll do it for you. >> Um my question is about enterprise. uh there are c lot of enterprises who'd like to run these things inhouse because they have sensitive data sets that they won't want going out also they may have already set up their own secure proxies to anthropics models um GIS and also the other question was about scaling so within the enterprise now you can actually allocate compute to scale it our own way right uh so an on-prem kind of solution have you all thought about it >> uh definitely

25:47 been a customer ask from a bunch of for enterprise customers and and we're seriously thinking considering it at the moment, but we we don't already have something that we can we can share with you, but we're we're we're seriously considering it. >> Excuse me. Could you show the last slide that show the QR code of the workshop? >> Uh are you looking for the repo or the credits? >> The repo. Okay. Yeah.