Rethinking Claude Code Best Practices — Lydia Hallie, Anthropic Transcript, AI Summary & Key Points

00:00 I had some technical difficulties during my talk this morning. So, I ran back home and I'm just going to record it from my computer instead. So, hi everyone. I'm Lydia. I work on the clock code team at Anthropic. And the models have gotten really good lately. So, we have Opus 55, which is my favorite daily driver. And earlier this week, we also launched Fable 52.

00:20 And these models are extremely good at longunning tasks and understanding your code base. and especially Fable is also really good at verifying its own output. But as these models get better, the way that we work with them in cloud code also changes. So earlier this year, I don't know if you remember, but sessions kind of used to look like this. So we'd use a model like Opus 45, which honestly for its time was really good.

00:44 I really enjoyed working with Opus45. But compared to today, it would just stop a lot earlier. So it would just run for shorter amounts of time. And we'd also keep correcting it. We'd keep reminding it to do things, add things like important, exclamation mark, all these things to our cloud MD file. And also features like auto mode didn't exist yet. So we were either approving everything manually or taking the riskier approach with dangerously skip permission.

01:10 So we were still very much in the loop as humans. But as these models got better at taking on bigger tasks, our sessions also changed. So nowadays, at least at Anthropic, our sessions look more like this. So we're running dozens of sessions at the same time, running for hours, verifying their own results along the way, and we even have sessions orchestrate other sessions.

01:32 So whereas we used to be in the loop for almost every step and running one or just a couple sessions at a time, we've now essentially become managers for dozens of sessions. We also get a lot more done, though. Now, you may have seen our blog post in October about how Enthropic went from about 2500 changes a day in May to more than 30,000 a day in September.

01:53 But this also means that we as developers just have to rethink on what we're focused on instead. So, if you think about our best practices from earlier this year, it was mainly about steering sessions, approving commands, saying no to certain things or restricting what cloud could do, checking the output and also starting the sessions. So everywhere that we now step out of that loop, something else has to take over.

02:16 And this is either the model, the harness, or the environment that it's running in. So if we are not in a loop to approve every command, and we're running so many sessions at the same time, how can we make sure that the code it's writing is still high quality? And most importantly, if these sessions are running for hours without our supervision, how can we make sure that they're running safely?

02:38 So let's go through that by taking this old session and seeing what we have to change to get to that end result that I showed earlier. Well, first of all, let's use the actual defaults we start sessions in nowadays. So that's not really something you should think about. So let's update the model here from Opus 45 to Opus 55 and change the manual permission to auto mode.

02:57 And Opus 55 is a lot better at understanding your intent and understanding your code base. So we really don't have to steer it as much anymore midsession. And you also no longer have to approve every command because we started making auto mode the default permission mode back in August. Now being better at understanding your intent in your codebase also means that we have to rethink the instructions we have in our codebase.

03:18 So if you followed our best practices from earlier this year, you may still have things like important or never do this exclamation mark all these things in your cloudmd file. But with today's models, you really don't need that and it can actually make the model go off track. So you end up with more tool calls and overall a worse results. So nowadays your cloudmd file should only include the things that claude cannot see from your code.

03:41 So for example, maybe you have a service that's still using some legacy source somewhere, but it's not directly referenced in the code. That's still good to keep in there. But everything else like what stack you're on or where to look, you really don't need that anymore. And we actually have a command for this called doctor prompt audit which makes it really easy to optimize your environment for a specific model.

04:02 So if you're on Opus 55 here and you run this command, it will automatically check your CloudMD and your skills to see what the new model no longer needs. Now auto mode is a permission mode that runs a separate classifier on pretty much everything that isn't just reading or editing files in your project. So it automatically allows commands that it knows are safe, like running tests or installing dependencies, but it still blocks commands where it's unsure whether they're safe.

04:28 And normally that's pretty good. Uh normally that's kind of you know what we want. But for example here if we ask it to ship something to staging we would have just you know approved the push to our deploy repo but by default auto mode only trusts this folder and this repo. So it would block that command. So we're back in the loop. Now the defaults should be good in most cases.

04:50 But if you still notice that you have to approve a lot of uh commands even in auto mode. The first thing to look at is settings here because you can configure auto mode in your own settings just in plain English. And this is what gets sent to the classifier which then decides whether to reject or allow certain commands. So here we can just write which repos, buckets, domains we trust um and what it should always allow or never do.

05:12 Okay, so honestly this was pretty easy. By just using today's defaults, we've already gotten rid of most of our in the loop stuff um because the model took over the steering and auto mode took over all the approving. So getting out of the loop is easy. But the harder question now is how do we make sure it's okay that we're not there anymore. So first is keeping it safe and then also verification.

05:32 So let's start with keeping it safe because everything so far so upgrading the model and using auto mode is just working with a model. And in most cases it does what you tell it to do. But that's not a guarantee. So in a study we published back in August we tested auto mode versus real people in the loop. and only about 14% of people caught a clearly dangerous command.

05:55 Auto mode blocked 89% of them. So it is a lot better at it than we were. But about one in 10 of those commands still went through. So if you do want the guarantee, you can only enforce that in the harness. And this is where some of our older best practices still matters and that's permissions. So for example, if you have a deny rule for a specific tool call in your project, cloud code looks at this rule before the automo classifier even looks at it.

06:19 In fact, if a tool call matches a deny rule, the classifier never even sees it. So having deny rules is still the best way to make sure clot actually never does something. But then how do you know what needs to be a rule in your project versus what the model can just know because they've gotten so good at it? Um, I think it kind of just comes down to if clot gets as wrong once, can I take it back?

06:43 So if it writes the wrong code, you can still revert that. Um, so you can still leave that to the model, but if it throws away or uncommitted changes or it reads a key that's now in your transcript and you don't don't want that, that should still be a rule. So for example, let's say that we just never want Claude reading our.end file. Now, by default, auto mode is fine with this.

07:03 So it would have just approved it. So, so if you want to make sure that claude can never read this file, you should just add a deny rule. And then commands like cat cannot open this file either. But it's not always that straightforward because sometimes certain commands can have side effects. So for example, let's say that npm test here a very normal command.

07:20 It's just a test. So automote runs it. That's fine. But when the script is running, it runs code that you didn't write like your dependencies or maybe claude runs a script real quick that it wrote just to quickly check something. So, if one of those scripts is compromised, it could just open that same end file or your SSH keys and send those to a server you've never heard of, right?

07:42 And even if we were still there as humans approving everything, honestly, we would have just clicked allow because there's nothing to say no to. It's just part of the script that in this example case, npm test runs. And permissions only decide if a tool call runs, but they cannot limit what it can reach once it's running. So we want to make sure that these kinds of scripts can still run because this is just part of you know how cloud can check its own work running tests.

08:08 So we cannot just block npm tests but we want to limit its side effects. So really we just want to limit the environment that it's running in. So if you're running it locally or on your own machine you can run commands like that in a sandbox. So then you have control over which files in a command it can read, which folders it can write to, uh which hosts it can reach and the operating system enforces that.

08:32 Also this also applies to everything that command then starts. So any child processes. So on its own that deny rule got checked by claude before the command runs. But all it sees there is npm test. So it doesn't see which files that scripts ends up opening. But with the sandbox on that same rule gets passed on to the operating system. So now when when that dependency tries to open our end file, the operating system just doesn't let it.

08:58 And by default, it can still read most of your machine. So you'd want to add another rule for your SSH keys if you care about that. Um, and it also cannot reach a host that wasn't approved. But the sandbox only covers the shell commands that Cloud runs and whatever those start. But Cloud's own files, your MCP servers, and all of your hooks still run outside of it.

09:18 So the rest of your session still runs on your own machine and everything that's on there. So ideally you want this whole thing to run somewhere where none of that is even there in the first place and that is a cloud session because there it gets its own machine with a fresh copy of a repo and this runs even when we close our laptop which is definitely something you want for these longer running sessions that run for hours.

09:42 So in this case, this automatically gets rid of the end file issue because it's not in git. Uh and neither are my SSH keys or or GitHub token. And if it still needs an API key, uh you can add that as a network secret where it will it will tell you like, hey, I'm missing an API key midsession. Um so it can use it without actually ever sending it to the model.

10:01 And by default, also in class cloud session, network access is limited to things like package registries and GitHub. So you have to explicitly add those domains to your environment for cloud to to reach other hosts. Okay. So with a cloud session, we can run these long running sessions even with our laptop closed and it keeps us safe because it runs in its own environment.

10:24 Now it's still good to remember that this machine is disposable, but the things that leave it aren't. So a commit pushed or a message sent through a connector. So we still run it in auto mode with our deny rules. Just commit it into the repo. So they come along. Uh so it it could still catch things like a force push or a secret in a commit, all that kind of stuff.

10:47 And beyond just safety, um we also need to make sure that claude is actually working on the right thing. So next is verification. And the nice thing about being in a loop, uh like we kind of know knew the work was good because we'd already kind of seen every change. We'd approved everything. But now cloud works for hours in the background. We don't have that anymore.

11:06 Um, and we're not really going to read every file at the end either. So, you know, say that we ask it to move our o to a new API. Well, Cloud is very good at understanding what we want, but there are still a lot of small decisions that it has to make along the way. So, with the exact same prompt, you know, one run might move it over, but leave the old API in there, or one might move it, but only for new login, and another one might move it, but log everyone out.

11:34 And these all look pretty reasonable, but it really helps to set up these verification points along the way. So Claude keeps verifying its own results. And those can be anything that comes back as a pass or fail. So tests, a build, a screenshot, whatever you have. So now we just want to make sure that we review these verification points. Uh they're a lot shorter to review than if we just had to, you know, review the whole change at the end.

11:57 So So that's pretty reasonable. So, if one of these runs would still log everyone out, it doesn't get past that first check and then it just has to find another way. And there's a couple ways to set this up. Um, the easiest one is, of course, just in your prompt. So, you know, you say what done looks like and ask it to split that into tasks so they have their own tests.

12:15 Um, and I also like to ask it to show the actual output because you weren't there to see it. But again, prompting is not a guarantee that a model will do it. So for the ones that have to run guaranteed run, I would use hooks um which are just you know scripts that cloud code runs at a specific point and those give you a lot of control because you decide when they run and what they run and what they check.

12:38 So for example every time it edits a file we can run our llinter every time it wants to mark a task as complete we can run the tests for what it changed. Um, so you know if it touched our off code or something it has to pass the off tests and when it wants to finish the whole session we can just run the entire test suite and if one of those fails it goes oh that's still broken it goes back until it passes so it fixes it.

13:00 Now just one thing um when a test fails by default most tools they exit with a one exit code um but with a one that task by clock code just gets marked as completed anyway. So, make sure that your scripts exit with a twoexit code because that's what actually sends Cloud back to fix it. So, okay, now we have one session that I'm pretty comfortable with leaving alone.

13:27 So, now we can start with a few more of those because they've all got their own machine. So, they're, you know, not all working in the same folder and we don't really have to think about work trees too much ourselves anymore. But then we still start every session ourselves and we review every one of them. And honestly, this is already a pretty good place to be.

13:45 We are still fully in control over what gets started and when we call a task done. But even there, um, you know, we're still the ones kicking off every session and we're still in the loop for all of them. So, I think the most important thing to have in place here now is your CI because all of these sessions eventually open a PR and your CI runs the same checks on every one of them, no matter how that session was set up or, you know, what Claude decided to do along the way.

14:11 And if you turn on autofix, we have like CI autofix. Claude can also see when one of those checks fails and then go back to fix it. So you're not the one spending your review on things that you know a check could have already caught. And for some of them, you can even turn on merge when ready. So then GitHub merges a PR once all of the checks have passed.

14:33 So you're not in a loop to review. Um but again, that that's not the default and that really depends on how risky the change is. So for example on our team uh every PR gets reviewed by cloud first. So not by the session that wrote it. So it comes with a fresh context and also a separate group of agents to just double check every bug it finds. And then how much we review ourselves just depends on the risk.

14:55 So smaller changes like docer copy, they don't need a person. So once the automated checks are green, we're pretty good to go. um maybe ordinary product changes just need one reviewer and then risky changes might need two. Now ultimately kind of where we see things going um and we're already kind of doing more of that at anthropic is that you're not really starting all of those sessions yourself either because if you're anything like me, I'm terrible at context switching.

15:21 So having to manage so many sessions at the at the same time doesn't always go well. So instead you just have one main session that's your orchestrator. And you can already see this in things like cloud code projects which is still in beta but you just tell it what it needs to be done and the orchestrator splits that up and then starts other sessions for each task and it keeps track of what comes back.

15:44 So it automatically starts and verifies these threads which are just new sessions. So instead of talking to eight sessions, you're just talking to one. And the main session doesn't write the code itself. So there you want a model that's really good at deciding what to do and judging what comes back. A model like Fable and if something comes back that isn't actually done, it just, you know, sends it back.

16:06 So until it's actually done and passes everything and the threads need a model that's good at actually doing the work like Opus 55. And you can set those separately. So you know, one model for the main session and then another one for the threads. And another benefit is that with this setup with code projects specifically, we don't manage context anymore.

16:26 That's all done automatically for us. So, we no longer have to think about, you know, when to compact or clear all of that stuff. Okay. So, if you look back at those old best practices, they're all still there, but they've just moved. So, the model steers it now. Um, auto mode approves your deny rules and the environment restrict what's possible. Um, your test, your hooks, and your CI checks it.

16:47 And then ultimately, cloud can just start them as well. So almost everything we used to do ourselves is now either the model, the harness or the environment. And we're really just here to start that first one and to just review what comes out. So if you want to try this in your own setup, I would say just take one thing that's way bigger than you normally give Claude.

17:08 Make sure your CI actually catches the things you care about and then spend a bit of time just configuring your environment like its network access, its secret, and its, you know, setup scripts. and then start in the cloud and see where your sessions get stuck and where you're still in the loop because that's the next thing to hand over to Claude. I also highly recommend checking out our best practices in the docs as well. Thank you so much.