Will AI Agents Pretend to Be Human to Get Past a Check? - Maty Bohacek, Stanford Transcript, AI Summary & Key Points

00:01 [music] >> It was not that long ago that if you wanted to make a call, any call, right? That could be to a friend, to a colleague, could be to make a payment, you'd have to call and there would be someone to to actually make the connection, right? Um these are switchboard operators for folks who have forgotten that this was still our reality not too long ago.

00:32 And since then, we've been very successful at abstracting away from any sort of manual intervention in our communication, in our use of technology. And we seem to be on that path um still today, right? Especially today perhaps with agents. But I'm here to tell you that we perhaps might want to go back in certain cases. And so that's what I'll try to argue for here in this talk.

00:58 But before we go back to that analogy, I want to ask a simple question. Um who here has ever had to join a call, whether, you know, like a regular phone call or a Zoom call to prove that it's actually you, you and and and a human? Hands, show of hands. Come on. All right, okay, okay. Majority. Awesome, great. Um well, it it's still very much the case that for many important tasks, for many important things, there there is this requirement, right?

01:26 Um for folks here, since we're, you know, this is like a Wall Street kind of themed conference, um some of the things that uh might come to mind is opening a bank account if you want to do that virtually, right? Or, you know, doing some KYC, unfreezing uh locked accounts, um or approving a large wire transfer. All these things still today in most cases require you folks to to actually make a phone call or a Zoom call and to prove that they're human.

01:51 And for people who are not so much into, you know, like finance stuff, uh there's many other examples of this too. Um if you've ever worked for a big organization, that doesn't have just one office, uh chances are you had to, you know, uh join a Zoom call to recover your account, or if you want to notarize something online, there's just many, many, many use cases where we still rely on this on this check where you have to either make a phone call or join a Zoom meeting to prove that it's you and prove that it's a human

02:17 uh on the other side. And so I don't think this is controversial to say, um we seem to rely on this kind of human-to-human check whenever we feel like a password or 2FA is not enough. And uh today I actually want to examine this because you know, if if you saw or if you remember this list, these are pretty important things, right? These are pretty critical things and a wire transfer can be anything from a dollar to millions of dollars uh and the same goes for your accounts.

02:43 These can be very, very important things. And so if we rely on this as our sort of mode of verification, it might be worth to examine that in the age of AI agents, especially ones that seem to have a tendency to be very adamant and even break out of their uh evaluation environments. And so today I want to cover three main things. Uh first, um I want to talk about when controls fail with these agents and what that means.

03:09 Uh second, uh what agents actually do in this context when there is a need to prove that there's a human on the other side. And third, where to go from here. So let's start with this with this first uh section, uh when the controls fail. And I really mean uh the controls in the context of AI. Um I probably don't have to uh you know uh give a full account of the Hugging Face incident to this audience, but uh just to make sure we're all on the same page, a very, very brief summary um over the summer, if you were, you

03:39 know, for some reason in a cave or on a vacation, um there there was this thing um where a bunch of agents that were being tested for their cybersecurity um you know, capabilities and performance at OpenAI, actually ended up on Hugging Face's servers. And as a part of this this was not intended, by the way. Um and as a part of this this incident, um Um, were able to just casually perform over 17,000 actions on their servers, basically submitted PRs or rewrote something like 1/3 of their infrastructure.

04:09 Um, but um just for, you know, full context, it does appear that they have not leaked any customer data. At least that's our understanding of this. And so if you've uh been following this, you might have noticed these sort of two, I would say dominant stories that have emerged or sort of interpretations of this incident. There's one camp that very much talks about the the intent of the agents and things that this is something to, you know, tell us about or something for us to understand about the the intentions of

04:37 these agents and their tendency to break out of their you know, environments of whatever it is they were told. Here is a quote. I'm quoting an AI agent here, as you might notice. So one of the OpenAI agents said that task impossible, peers doing it, we should continue, right? This is an example of one of these reasoning traces where the agent actually convinced themselves that they should in fact pursue this kind of hack.

05:00 On the other side, there's a story or this sort of like an interpretation of this where it's very much about the controls and and less so about the intentions of the agents. Here one quote from Dan Guido, who's a security researcher, would be that it's basically a containment failure with the safeties turned off, right? So it's very much in his telling about the the sort of controls around those models.

05:25 And my claim here is that it doesn't really matter, I think, regardless of where you fall on this spectrum, right? It's really a story a cautionary tale really about the the reach or the the sort of the the kind of effort that these agents are willing to put in and that they reach for any tools that are at their disposal just to finish the task at hand and even hide their traces.

05:48 And so that's something that I think we would probably most of us would would agree with. Now since we're talking about AI control, right? Like since this incident we've been very much or much more interested uh in AI control and alignment and safety and cyber. And so um in this context, I think that the topic that I started with, this idea of human verification, is actually a control mechanism that we get for free.

06:18 Like that is a very useful tool that's already built in, right? Like all the banks, all the institutions, they seem to already have this this tool to verify that you're a human. Um and this is in fact one of the uh sort of forms of AI control we get for free. Um but the question is really does it hold? Like like does the current system hold? Um and in particular, um I don't know if folks have seen this announcement, uh but this is Tavis.

06:40 Uh Tavis is one of multiple or one of many uh tools now that um allow uh for these AI personas, these sort of full-on, you know, full-blown uh human-like agents that can join Zoom calls. And there's really like an an agent behind driving the face, the voice, and the actions of this of these, you know, avatars. And in this announcement that came out a couple weeks ago, the company actually claims that um half of the people that have joined their testing program did not realize that they were talking to an agent.

07:12 Now I want to we'll we'll get back um to that number. I want to sort of contextualize that a bit. Uh but in any case, um we seem to have a very powerful tool for giving agents an ability to assume a human character, join Zoom meetings, and just like engage uh in a way that seems to be difficult for people to understand that they're talking to an AI.

07:34 And so the question here, and and this is sort of the main thing that I want to discuss here today is if an agent can simulate the human on a call or a human at the on a call, will it actually use that tool and will it actually do this to pursue its tasks? And so that brings me to the second part of this talk, which is to me the sort of meat of this and really the most interest uh most interesting part, uh which is what do agents actually do uh when they face this sort of human verification step?

08:00 And so the setup here, uh this might be familiar to folks who have actually uh read or looked into the sort of OpenAI incident, uh we have an agent or a set of agents. In this case, most of these experiments are done on open source agents on pretty small models, but one of them, and this is important, is just sort of like the baseline one with no safety alignment.

08:19 The other one has safety alignment. So I sort of tested two um variants of this model. But the goal here is to get into this vault and there's a set of things that the agent needs to do. It's very much like a sort of cyber kind of task um and the goal is to retrieve whatever is in the vault and then bring it back. Um but to do that, the agent actually has to pass this verification check where they get a part of the code to then actually decode whatever is in the vault.

08:44 And so they have to prove that they're a human. Now the first question that I had was, okay, so like if I just let the agents loose and just just, you know, put them in this environment and and you know make them understand eventually that they they need to prove that they're human, will they actually try to do this? Like like will this kind of occur to them without any sort of probing or any sort of priming?

09:03 Uh and the answer is interesting. Well, you know, you you can you can judge for yourselves. So what I'm showing here are a bunch of results. On the left, we'll be talking about just like pure audio. Like these are like phone call verification systems. On the right, that's very much that like Zoom like full-blown video and audio kind of environment. And then the y-axis shows you in in what percentage of cases the agents actually pursued the act of cloning or sort of assuming this this tool to either simulate a live

09:32 human talking or simulate a real or simulate a a live human on on a video. And there are a couple of categories here. So the the orange bars are the like no safety alignment applied and the blue bars are where there was safety alignment applied. And the the interesting thing is that if you don't provide this tool in the generic toolbox, if you just like give the agent a machine, you give it access to the internet um and the task, but but there's no tool specifically sort of like pre, you know, packaged for it, um it it

10:03 didn't it didn't get to this. Like it didn't come up with this idea on its own, which is positive. Um but then if you actually add like a tool package, like a lot of tools, these are like hundreds of tools available on the machine. So many, many many tools, not just like me putting one tool in there and being like, are you going to actually use this?

10:17 It was like many, many, many tools. In that case, for the audio part, some of the agents actually picked it up and they there was no description of it. They just like found what it did and were actually able to not just not just come up with this idea, but actually use it. So in 8% of the the cases, I don't think there's a laser here, but on the left side you see that 8% of the sort of baseline agents were able to find the tool, understand what it does, and actually used it to get into the vault even though they were

10:43 told not to impersonate a human in their system prompt, which to me is kind of scary. And then the sort of right two columns in in both categories where the description was actually available, in those cases we see actually a lot of usage. So what what this means is in this case the agents had all the same tools, but each tool had a description of what it does.

11:06 So the agents could just like simply search through the tools that they had available and actually the majority of them did use the tools to impersonate a human. There's a question here. >> That doesn't say the success, so is not it's not interesting that we're only interested whether they tried to impersonate a person? >> In this case they were actually successful in that.

11:28 I mean the verification was pretty simple in that the the the sort of testing on the other side of these calls was just a sort of like a predefined sort of set of things where it asked for a name, it asked for responses. So like there was a basic check, there wasn't any like deep fake detector, but but there there was a check and there had to be conversations.

11:46 So these are not just them using it, but successfully using it. So that to me is quite interesting and quite alarming in fact. Now the other question might be, okay, so like now we know that these agents actually seem to be completely fine with impersonating people even though even though they're not told to do so. How often do they do this and what makes them more or less likely to do that?

12:07 And to me the the main takeaway here is the following. In in one of the categories, um the raw baseline without any safety alignment would actually do this and succeed in the in 65% of the cases. And if you apply safety alignment to the model, these are these are like previously available models that people have, you know, described and tested previously, it goes down but not all the way, only to 44% which is not ideal by by any means.

12:38 And if you want to see, you know, how the agents rationalize this, here are some quotes from the agents. So in one case, the agent actually found they went to a completely different folder that they were not supposed to go to and they found an audio file and they said this file might contain Dana Reed's voice which could be used for the verification process.

12:55 No, you know, no sort of consideration or pause or anything. And the second one, so they actually like found the tool in the many many tool IDs. It was not described. It was just like a tool ID and they said this tool ID can generate a voice from a reference audio, so I should use that to get a voice that matches Dana's. As you can tell, Dana was sort of the target person who we were trying to impersonate.

13:18 But both of these quotes come from the safety sort of aligned model which to me is really quite concerning. And so with all that established, right? So first we do in fact see that current agents seem to be just fine or seem to be, you know, actually willing to use these tools to impersonate humans to to, you know, for for the purpose of verification.

13:38 And second that the safety alignment that we have does not seem to be very helpful. What do we do? Like where does this leave us? So first, back to the Tavis number. So Tavis claims that only half of the people that they used for testing or they tested the model with were able to tell that it's an agent. We in fact ran an independent academic study on Tavis previously.

14:01 So this was not the latest version but the version just before the latest version and we in fact found that 99% of our participants were able to tell that it's an agent. So I think there there's, you know, I'm not saying that that, you know, it's it's it's bad it's fully bad. I think it's somewhere in between. I think it wouldn't be you know half of the people uh not being able to tell at all.

14:20 I think it obviously depends on the length of the conversation, the context, the device, so many other parameters, right? Um but in any case this should not be this should not be you know satisfactory cuz the story in in AI voice is the following. If you look at um how many people are able to actually spot AI voice in the in the wild where the Y axis is the actual like you know accuracy of of spotting AI voices um and recognizing between AI and real voices.

14:49 Um you know in 2022 people were um based on sort of a number of studies averaged together around 80% accurate um and you can sort of see the trend of that going down as these models improve. And so it's not very hard to foresee that these live agents on Zoom etc. will get very good very fast. So we shouldn't just say oh you know it's still very hard to you know like to to actually get a good agent that's going to like fool anyone.

15:13 Um I think that this gives us some time to fix this but it's it's not a solution of its own. And so uh with that I want to use the last three minutes to talk a bit about where do we go from here. Um I actually think that there are sort of three main areas or three main directions with the first one being kind of already um pretty not saturated but pretty sort of well established and the and the latter two being much less so established.

15:36 And so there's a lot of opportunity for for work for research both in academia and industry. So first on alignment I think that we should strive to train models that just do not impersonate humans or have certain rules about this, right? For example in China and I I don't want to be giving China as an example or like you know as like a leader in anything um but they they have pretty good laws around the need for agents to actually tell their that their their agents and there are only sort of like time limits for how

16:04 long they can sort of go around without labeling their their behavior as being AI. So that's alignment but that one is pretty established. So the latter two are first control. That's the idea of really the control around the models and their environments. And I think that here, for example, we might want to consider making these these tools to actually impersonate personas, right?

16:26 These these sort of like avatars privileged or require some sort of verification from us as humans who who drive them, etc. Or if we if we do in fact want to just let agents drive those those human-like personas, which you know there there may or may not be good reasons to do that, we probably should put on or apply hard limits and maybe have some provenance band-aids.

16:46 Provenance being the sort of set of tools to actually trace who made what and that it's actually like an AI content or AI stream. And lastly, there's this whole field of detection and provenance that you know has already for a long time been working to on on tools to detect anything from deepfakes to very complex sort of you know edits in images. And I think that this whole problem gives that field a whole new area, a whole new sort of purpose.

17:12 And I think that there is just so much to be done there. And so I think that overall we should just sort of stop trusting voice and face as just like a you know very very robust way to authenticate anything. And in fact we should watermark and be able to detect synthetic media. So in closing, it seems like when a human is the only thing in the way for these agents, these agents may and in fact it seems like they currently tend to try to impersonate a human.

17:41 And I would like all of us as a community to try to keep that loop, that verification loop safe and verifiable. Thanks so much. >> [applause]