The Problem with Jev: Can You Trust AI Decisions? Transcript, AI Summary & Key Points

00:01 Will it rain tomorrow? It is decidedly so. Oh, really? How can you be so sure? Reply hazy. Try again. Okay, I'm trying again. How do you know it's going to rain? You seem pretty confident about that. Um. Better not tell you now. Okay. That's not helping. This is clearly not going to give me the answers I need. You may have heard about a new approach to AI that's called JV, and my friend Martin did a really great explainer video here on the channel.

00:31 So go check that out if you're not familiar with this. It's a fascinating new technology, and it's designed to do something very different from what an LLM or large language model that we've all become familiar with. The big difference is an LLM generates words. Lots of them. Jev makes decisions, quick decisions, and that's potentially a big deal for automation, but I have a problem and it's a very cybersecurity guy kind of problem.

01:05 And yes, it's with trust. Yeah. I mean, who other than cyber security people would come up and spend so much time developing a concept we call zero trust. Yeah. My tribe is often accused of having trust issues, but that's a whole other story for another video. In other words, how do we know that the particular Jev instance that we're talking to isn't just some sophisticated fortune telling ball with probabilities attached to it?

01:34 It's essentially a black box and we can't see into it. We can't determine how it came up with its answers. So we don't know if they're wrong. And if they were, how would we know? Let's explore that question. Let's start with LLMs. So LLMs give us something to inspect. We've got input that goes in. So a prompt. Then we've got text that comes out. And with an LLM you get a well worded response.

02:01 Now that response may be wrong. It may be completely bogus. It may be even confidently wrong, and we call those hallucinations, but at least there's something there that you can inspect. So you can ask the question, why did you come up with this? You can challenge it. You can even have another LLM that goes in and inspects the results and looks over the model.

02:27 So you could look at all of those kinds of things. You could look at the tools that the LLM used. You could look at the data that was used to train the model. So all of those things are available to you, including a chain of evidence around that decision. So it's not perfect. Far from it, but there's something to observe. Now let's take a look at Jev here.

02:48 We've got input going in. And then we get a decision that comes out. Yes 97% okay. That's incredibly useful for software because it's not going to have to go through and interpret all of the stuff that came out. It gets a structured decision and gives you a probability of confidence. That's one of the things that makes Jev really interesting, this business here about the confidence.

03:18 The probability tells me something about the model's confidence, but it doesn't necessarily tell me the reason for the decision. And those are two very different things. Confidence isn't necessarily evidence and it certainly isn't proof. Here's a simple example. Let's say we start with a security alert. So something comes in and tells us we're under attack.

03:42 And Jev tells us that what you should do is contain that particular host as a response, and it gives us a 97% confidence. Okay. That's impressive. Saved us a lot of time. Told us what the problem is and even what we need to do, but suppose I'm the security guy responsible for that decision. I want to know why did you tell me that? Where did you come up with that?

04:02 Was it because the process matched some known malware? Was it that the IP address was known to be malicious? Was there some unusual behavior that we observed? Was the machine especially sensitive, and therefore we're going to build more protections around it as a result? Or was there some sort of authorization problem that the model recognized and didn't understand?

04:33 Or did somebody manipulate the information that went into the model or even the model itself? Because from a security perspective, the input and the model are part of the attack surface. Let's imagine we have an attacker who has figured out the information that Jev is going to be looking at, and they do something to manipulate that information that's going into the model.

04:59 Maybe they alter a log, maybe they manipulate some metadata, maybe they poison a data source that was used to train the model. Or maybe they exploit some weakness in the workflow feeding the model. Somewhere along the way, the model could make a perfectly consistent decision based on corrupted information. Then what do we have? Well, you know the old saying, garbage in, in this case, confident garbage out.

05:27 And here's the scary part. The model still will say 97% confidence. That doesn't mean that the decision is trustworthy. It means the model is confident that given what it saw, that's what the answer should be. But it could be confidently wrong, or the model could just be wrong for an entirely innocent reason, in the same way that LLMs hallucinate. But how would you know in any of these cases?

05:53 And this brings me to an important concept in cybersecurity. It's this idea of observability. When something goes wrong in a traditional system. I want to be able to reconstruct what happened. I want to know what came in. I want to know what changed. I want to see what rules fired in the system. I want to know what data was consulted to make the decision.

06:20 What decision was actually made and who caused it or what caused it, and ultimately, why. So what I need ultimately from all of this is I need an audit trail. I need something that I can investigate and verify that the operation is as we intended. That's how we investigate. That's how we detect attacks. That's how we establish accountability. If I'm going to take evasive actions that might shut down a system, I need to be sure the advice I'm getting is correct and not just confident.

06:58 So where's the visibility we JEV? It's going to give us structured outputs and answers, and it even gives us probabilities to tell us how confident it is, and JEV, specifically designed to be embedded inside structured workflows. That's all good. Easy to plug this in to a workflow. In fact, TypeSafe. The company behind JEV says something that I really like.

07:20 They say instead of asking one big question, it's actually better to break it up into a number of smaller questions. That way, you end up with a better designed set of of observed abilities. It built into the system. So here's an example of what I mean. Let's take our alert comes in scenario again. And maybe the first thing we ask is not just what do we do, but we say was this particular activity authorized or not?

07:50 And if it wasn't then maybe we say, what kind of evidence do you have to support that? And then do we have a risk calculation to go along with this and then adding all of those things up, maybe we're able to render a decision. And that's a lot better engineering, because now you can observe the individual decisions that went into making the final decision, instead of just treating the whole thing as one big black box that you can't see into.

08:21 So what we want to do is do improvement through refinement. and each one of these is a stepwise refinement of the previous. But there's still a fundamental question as a security guy, this is how I'm wired. It's verify then trust. It's not trust then verify. So here's the problem. If Jev tells me contain this machine 97% okay, we've got a number, but what am I exactly verifying?

08:52 I can't verify that Jev have produced the output. I mean, I know it came from Jev, but I don't know where it came from. I can verify that the output conforms to the expected type. I can record the probabilities, I can log the inputs. I can audit the surrounding workflows, all of that kind of stuff, but can I really independently verify the intelligence that produced that decision?

09:15 That's a lot harder to do because I don't have a proof. I have a prediction, and guess what? Attackers love things that we can't see. Think about traditional security controls. If an attacker compromises it, I get evidence. I get logs, events, changes, anomalies, indicators of compromise, all of those kinds of things that I can look at later and decide what actually happened.

09:41 But imagine a compromised AI decision system that simply starts with subtly different decisions. Not everything is compromised, just 1% more likely to approve the bad thing than it used to be, or for a particular type of target. Maybe it starts getting a little more inaccurate. So the thing is accurate. It's good, it's doing things, and then all of a sudden it's not.

10:05 I've built up confidence from these, so I don't even question the result in this case. That's potentially a lot harder to detect, because every individual answer may look perfectly reasonable by itself, and our belief in the system has become self-assessed with that confidence. We believe it because we've seen it again and again, and our confidence will increase over time, and we will become less critical in our analysis of those results.

10:38 So what's the answer? I'm not saying don't use Jev. Quite the opposite. I think this kind of capability can be extremely useful to us, but I would treat the AI itself as an untrusted component and then put security controls around it. We need things like input validation. I need to know what's going into it. I need to make sure that I have independent signals that are all coming in and not relying on a single source.

11:06 I need to audit the logs. I need to look for drift in the model's behavior. Look for anomalies and have humans involved in the escalation process with independent verification for some of the high risk decisions. So we keep the human in the loop, and critically, don't let Jev be the only thing that knows what happened. As an AI moves from just generating text to taking actions.

11:33 This question becomes even more important if an LLM writes a bad paragraph. Well, that's one thing. If an AI decides to block this account, or pay the invoice, or shut down the server or approve this particular transaction, now we're talking about an entirely different securit problem, because the question isn't just can AI make the right decision? It's can I know when it makes the wrong decision?

11:56 And even more importantly, can I tell whether somebody caused it to make the wrong decision? Observability is a big concern with Jev now, not that it's necessarily less accurate than an LLM, or that there's any particular problem with probabilistic decisions. They're not inherently bad. Actually, I think the structured approach is pretty compelling.

12:21 The concern is this, the more we allow AI to make decisions for us, the more important it becomes to observe, to audit and to verify those decisions. Because confidence isn't proof and a probability is not the same thing as an explanation. And just because the decision looks legitimate at first glance doesn't necessarily mean it can be trusted. Don't just ask what did AI decide? Ask how do I know what it decided? Verify. Then trust. So that way we aren't just left with magical AI decisions.