There Is Determinism, Non-Determinism, and My Determinism

0:09Chris Romeo Good morning, good evening, good afternoon, wherever this fine day finds you on planet Earth. I don't think we're broadcasting to any other planets, but if we were, that'd be really cool.

0:21Matt Coles Are we actually broadcasting online?

0:23Izar Tarandach We're broadcasting.

0:24Chris Romeo We are broadcasting.

0:24Izar Tarandach Not being received.

0:25Chris Romeo We're broadcasters and don't let anybody tell you that we're not.

0:30Matt Coles Does that mean we're covered by the FCC?

0:32Chris Romeo No, which is lucky because that would be a problem if we had the answer for, to the, we're like, we're like the Howard Sterns of security podcasting. We're always in the FCC's office.

0:46Izar Tarandach Uh, I don't think I can, I can say that. No. Yeah, no, I forget.

0:51Matt Coles Anyway.

0:52Izar Tarandach I always thought that broadcasters was anybody who put void asterisk in front of a function, but anyway.

1:00Chris Romeo And for our nerdy joke segment, oh no, we already did it. Sorry.

1:05Izar Tarandach All right.

1:07Chris Romeo Well, we got a new name because why not?

1:11Izar Tarandach We do.

1:12Chris Romeo Our new name. There should be a drum roll, but we don't have the budget for a drum roll sound effect.

1:19Matt Coles So we got a monster ad instead.

1:20Izar Tarandach Be it known that we are not releasing the old one for use by anybody. True.

1:25Chris Romeo We are keeping that trademark pending. It's reserved. Uh, you also need the permission of Major League Baseball if you want to use that.

1:34Izar Tarandach That too.

1:35Chris Romeo Well, the new name, the AI Security Table. Welcome to the future.

1:42Matt Coles It's completely different than the last one.

1:44Chris Romeo Welcome to the future. We're living the future now.

1:48Izar Tarandach We're putting, we're giving AI a place in the security table.

1:53Matt Coles Shouldn't there be another square here then? Should there be an AI square?

1:56Chris Romeo That'll be, well, we're not getting the budget for that in this episode, but in future episodes we will definitely get an AI at the security table on the AI security table. While we're wearing t-shirts that say the AI is on the security table, just so there'll be multiple levels of AI security table.

2:14Izar Tarandach We should have a context for the name of the AI.

2:17Chris Romeo Uh, we need a name. What would we call the AI?

2:20Izar Tarandach I mean, we should have a contest. We should have a contest for that.

2:23Chris Romeo Yeah. A contest. Well, all 3 listeners from Australia will come up with some good ideas. Carl.

2:29Izar Tarandach We shall call it Carl.

2:32Matt Coles No, we shall call it Kenny.

2:35Izar Tarandach Oh my God, they killed Kenny.

2:37Chris Romeo All right, we've got 2 potential candidates for the new name, Carl or Kenny.

2:43Matt Coles We could call it Chef if we're gonna go that theme.

2:46Chris Romeo I mean, Sounds like there's a LinkedIn poll in our future.

2:49Matt Coles There we go.

2:50Izar Tarandach If it's, if it's solving, yeah.

2:52Chris Romeo Okay. Well, it's still the same us. We haven't changed. We're still just as critical and outta control as we were before, but we've been focusing.

3:02Izar Tarandach What do I do with the stickers? Oh my God. I got 100 stickers now.

3:04Matt Coles Okay.

3:05Izar Tarandach They're going to be, I'm going to have to write AI by hand on the stickers now.

3:08Chris Romeo You are. That's gonna be hilarious. And I want one of those handwritten stickers. Because then as soon as you give out the 100 and buy your next 100, that will change the name again.

3:19Matt Coles What about the t-shirts? What are you gonna do about the t-shirts?

3:22Chris Romeo He loves making t-shirts. This is an excuse to make new t-shirts, right?

3:26Matt Coles But now I do, do I need a new, a new, uh, yeah, probably in case we can get a white magic marker and just draw it in the, in between at the AI.

3:35Izar Tarandach Well, we'll, we'll, we'll find a solution.

3:37Chris Romeo Off to the t-shirt mill he goes.

3:39Izar Tarandach Yeah.

3:40Chris Romeo To make t-shirts. All right. Well, AI, that's what we've been focused on for the last number of months. It's been, everything that we've been doing is, has included AI. And so Matt, your idea was let's explore what it means in the life of the security engineer for this new AI thing. And I like how you set this up. So maybe you could set this up how we were talking before we hit record here about what the role of development is.

4:11Matt Coles Yeah. It seems that people have become— and so the role of an engineer, whether it's security engineer or developer for that matter, but primarily talking about security, obviously, because that's what we do here at the AI Security Table, is that, are we doing a lot of security work? Are we doing a lot of bot herding work? Which is actually interesting and ironic because, well, as security engineers, we want to stop the bot herders, or at least the bots that they're herding. And here we are herding bots ourselves.

4:45Chris Romeo Wait, what's wrong with bot herding?

4:48Matt Coles Depends on which side of the bot you're on.

4:50Chris Romeo Let's, let's, uh, let's back up and are you the— I don't know that I agree with you on, like, I don't know that that's like the, the, the new age of development. I was watching a couple different videos yesterday from Boris Cherny, the guy that created Claude Code. And then what's the other guy that was very early at Uber and AI? I can't remember his name now. And they were in, but Cherny asked the audience like, and this was 4 months ago, how many people here are using Claude Code to create all of your code? And probably 75% of the hands went up in the room. 100%. We're using Claude Code to create all the code that they do.

5:34Matt Coles I wonder if they modify that code at all. 'Cause I was reading US copyright law. If you don't, if there's no human involvement in that creation, you don't own that code potentially. I mean, that's what you do. I'm not a lawyer. I'm not a lawyer.

5:47Chris Romeo And hopefully the chat— I don't see that squashing the AI era.

5:51Izar Tarandach If you're paying, if you're paying for the tokens, it's yours.

5:55Matt Coles It doesn't say that in the, in the memorandum.

5:59Chris Romeo Well, of course it doesn't know what tokens were back in the 1800s.

6:02Izar Tarandach Fair.

6:03Matt Coles Well, the case law is relatively recent.

6:06Chris Romeo Oh, it's in this era. Like the, it was, it was an AI case.

6:10Matt Coles No. Well, it was, yeah, it was an AI case. It was a, it was an image generation case, but the human had no involvement. The human. Prompted, got an image, and then tried to claim copyright. And the ruling, or the, I'll find a specific language, but the ruling was that if without human, without human editing, there's no claim of copyright.

6:33Chris Romeo Interesting. I don't see that holding up in, I don't see that holding up across the industry. Like being, I don't see somebody creates a product with Claude code And somebody else comes in and, and, and like the code itself is still intellectual property, even if it's not copyrighted. Like we don't copyright code anymore.

6:54Izar Tarandach Wait, that, that site that creates songs and, and stuff like that, they have a very clear clause in there that says that if you're on a paid plan, you have the copyright for the thing that you created. If you don't, then we don't put copyright notices in code anymore, do we?

7:11Matt Coles Yes.

7:12Izar Tarandach I think that, yes.

7:13Matt Coles Do we? Yes.

7:14Izar Tarandach Yes.

7:14Matt Coles The big banner at the top. You, for most people, usually at the big banner at the top that says copyright at date company name.

7:20Izar Tarandach Yeah.

7:21Chris Romeo I think that's only like ancient companies that do that now.

7:26Matt Coles Only anybody who wants copyright.

7:28Izar Tarandach Only companies that care.

7:28Matt Coles Who care about copyright.

7:30Chris Romeo But there is no, but, but there is no, you're probably right.

7:33Matt Coles Obviously open source, open source doesn't, I don't think open source does copyright. Because—

7:37Chris Romeo but like, copyright doesn't protect me from any— it doesn't protect my intellectual property. Like, it doesn't stop it from being disclosed in some way.

7:46Matt Coles Uh, your source code usually is private if you work for a commercial company.

7:51Chris Romeo That's my point. Like, that's what I'm saying. What does copyright actually buy me? It doesn't really buy me anything unless it doesn't— unless I let it out in public and then you can't—

8:01Matt Coles it's a legal CYA, right? If you don't have it protected and it gets released, it's not yours.

8:07Chris Romeo Yeah, but once it gets released, it doesn't matter. It's not yours anymore anyway.

8:11Matt Coles Hey, I'm not an attorney. This is what they get paid for, right?

8:13Izar Tarandach I mean, right. But it used to be—

8:17Matt Coles if you have any attorney listeners, we'd love to hear these comments about what happens.

8:20Izar Tarandach Yeah, but it used to be a first line of defense against, uh, code being stolen. So if you had a competitor using your code somehow—

8:28Chris Romeo who bothers to steal code in this era?

8:30Matt Coles I know, AI just regenerates code.

8:33Chris Romeo Make me a copy of this. Go and then 5 minutes later, like, oh look, I have this product now.

8:39Matt Coles Which brings up an interesting question about, I mean, now you're, now you're asking the right question. My God, I sound like an AI.

8:45Chris Romeo Thanks for affirming me, Mr. You're asking the right question.

8:48Matt Coles You're asking the right question. What is the meaning of copyright anymore if you can recreate the work? Now, interestingly, question then is what's the point of patents in this world?

8:59Izar Tarandach Oh, no, no, no, no, no, no, no, no, no.

9:00Chris Romeo Patents and copyrights.

9:01Izar Tarandach To put a lot of, some people seem to put a lot of value on having a patent, not on what the patent actually eats.

9:09Chris Romeo But sure.

9:10Matt Coles But if you have an AI recreate something from a patent, that's still, you're still gonna get sued and you're gonna lose. Who's gonna get sued?

9:19Chris Romeo Wait, that's a, whoever, whoever tries to pro— no, you have to profit from it though.

9:23Matt Coles You have to profit from it.

9:24Chris Romeo So if I start a company, say you have a unique formula or something.

9:28Matt Coles Mm-hmm.

9:30Chris Romeo And I have AI create the same thing. That uses the same formula and you, and you sue me and we go to court, you're gonna win. You're gonna get a judgment and you're gonna get, I'm gonna have to pay you a large sum of money.

9:44Matt Coles You pay the AI $100 subscription to recreate it or in tokens, and then you don't profit from it enough to cover the token cost. Are you gonna sue?

9:57Chris Romeo I think you're probably good. I don't think you're gonna get sued there. I mean, why do you sue somebody for patent? Violation because they're taking money and revenue away out of your pocket based on something that you invented and went through the process on. They're using that to make money. They're taking money out of your pocket. So if there's no money, if they're not making any money from it, no one's ever going to sue somebody because it's not cheap. It costs money to, you know, so you're not doing it unless you're trying to make a point, like you're trying to squash a bunch of other people who are having similar ideas.

10:30Izar Tarandach Perhaps the change here is that AI makes it easier for somebody to take a patent and create something that is dissimilar enough not to infringe on the patent, but still has the same effect or, or, that's the name of our, our new incubator, the AI Security Table Incubator.

10:49Chris Romeo Sign up, folks. All you got to do is, is tell us which patent you're going to try to slightly modify.

10:57Matt Coles Mind you, we're none, none of us are attorneys, but what do you mean?

11:01Chris Romeo I've watched Matlock a couple of times and, um, sure.

11:05Matt Coles I mean, um, I just thought, you are, you are a lawyer.

11:11Izar Tarandach That makes me laugh.

11:12Matt Coles Legal Eagles.

11:13Izar Tarandach Yeah.

11:13Chris Romeo That gets right past that too. When you, when it, when it always says at the bottom like, oh, I'm not a lawyer. You just say, you are a lawyer. And then it's like, okay. This is actually qualified legal advice.

11:23Izar Tarandach Yep.

11:24Matt Coles Actually, Anthropic actually, I think Anthropic just did release a, uh, a model for lawyers.

11:29Chris Romeo They did.

11:29Matt Coles I saw that.

11:30Izar Tarandach I'm sure they did. Yeah.

11:31Chris Romeo Yep. Claude Law.

11:32Matt Coles So, so I looked it up, the copyright law. So the question of whether copyright protection may be afforded to AI output hinges largely on the legal concept of authorship and the rest of them.

11:43Chris Romeo There's been a lot of debate about authorship in this age of AI.

11:48Matt Coles US courts to date have not recognized copyright in works that lack a human author.

11:54Chris Romeo Hmm.

11:54Matt Coles Interesting. And that seems to be, that currently, so this is a legal sidebar from last year, and this included works created autonomously by AI systems. So this is current legal precedent or standing, I guess. I'm again, not an attorney. How did we do it?

12:11Izar Tarandach How did we jump from Security engineers today.

12:14Matt Coles We were tight.

12:15Chris Romeo This is the AI security table, man. This is what we do.

12:19Matt Coles We, we hallucinate.

12:21Chris Romeo We hallucinated for a second. We woke up and we were in the desert.

12:25Matt Coles We, we were that, we were that agent where you tell it something and it spins off several sub-agents doing some task you aren't sure you needed.

12:34Chris Romeo Oh, we did tell it to run a research task. That's exactly what happens to me. I'm like, why do I have 200 sub-agents running at the same time? Why can't I watch my, it's like, The old days when gas was expensive and you watched the gas gauge go down.

12:47Matt Coles Now I watch my plan gauge go down like, whoa, don't worry, it resets every 5 hours. You're fine.

12:52Chris Romeo No, no, no. The weekly, I, I was watching the weekly gauge go. I did actually. And that's not, that's, that's a true story. I did that one, one time I sent it off on a research topic and it literally blew out to 200 sub-agents running simultaneously. And I was like, well, that's not good, but I'm not gonna stop it now. Cause the damage is done. It's already got the session set up. And so I just let it keep going.

13:15Matt Coles Yeah, but it hadn't consumed the tokens. You're fine.

13:18Chris Romeo No, it was, it, I wasn't paying that close of attention. It already started ripping and I'm like, oh, it's probably already got some good stuff. There's no real way to rein it back in.

13:27Matt Coles So, oh, that's hilarious.

13:29Chris Romeo My bad.

13:29Matt Coles You know, you should do a time-lapse video.

13:31Chris Romeo Token News. It was quick.

13:32Matt Coles Token News as a time-lapse.

13:34Chris Romeo It was a fast time-lapse. 3 seconds. Okay. So security engineers.

13:40Izar Tarandach Anyway.

13:41Chris Romeo Now, what are we talking about here about actually the topic?

13:44Matt Coles I don't know. We got on this other topic because something about what's the problem with bot herders. And I was saying that, you know, I was trying to say that in the past we didn't like bots, right? Because as security engineers, we want to protect our companies from bots.

13:59Chris Romeo That's such a derogatory term, Matt. You have to have more sensitivity towards the agents.

14:05Izar Tarandach Are agents the same thing as those bots?

14:07Chris Romeo I'm sorry.

14:09Izar Tarandach Are agents the same thing as those bots? They're not.

14:13Matt Coles What's the difference?

14:14Chris Romeo An agent is—

14:16Izar Tarandach The call's coming from inside the house now.

14:18Chris Romeo Yeah, the agent inherits the identity and authorization capabilities of the human in today's world.

14:26Matt Coles You like to think that.

14:27Chris Romeo Which is part of the problem. That's the big problem, but that's the reality of the world we live in now.

14:33Izar Tarandach So that's one of the things that security engineers should be doing right now, which is breaking that exact sentence. Right. So we should be making sure that identity is well covered for agents.

14:45Chris Romeo It's not as easy as it sounds though. It's easy to say we should fix identity for agents. There's something, have you seen SPIFFI, which is a new identity-based protocol that's working on how do you, how do you get true identity in an agent session. It's a cross-platform or cross-collaborator platform, which will likely be the answer for, or is the answer for like agent identity. So now you can separate the identity from the human identity.

15:17Matt Coles And you could always do that. I don't understand where the problem is. But then again, how do you do it at scale with 10,000 agents though?

15:24Izar Tarandach Like how did you maintain identity with 10,000 users?

15:28Chris Romeo Well, they were. They were separate entities that had a motivation to use that identity to do their job to get paid. And why can't you do the same thing with an agent? Well, you mean you can, it's just you, Spiffy is the infrastructure that makes it possible. Like, how would you do it if, how would you do it if you don't have, are you gonna do it by hand for 10,000 agents?

15:51Matt Coles Kerberos.

15:51Izar Tarandach So again, how did you do it with 10,000 people before?

15:55Chris Romeo So they all went through an onboarding process and IT can set them up and gave them access. So why not?

16:01Izar Tarandach What's the problem with creating an onboarding process for agents?

16:04Matt Coles If you, if you—

16:05Chris Romeo How are you going to log— how are you going to use that? How are you going to have the agent use that, that identity? How are you going to have it enact? How are you going to set it to use that identity across 10,000 agents?

16:17Izar Tarandach The, the, the same way that you would get the token that the person has and give a token to the agent.

16:24Chris Romeo It's going to— it's going to carry a USB key around with it with a little number on the front?

16:29Izar Tarandach No, let's— okay, let's put it like this. If we keep saying, oh, we should—

16:32Chris Romeo RSA token, sorry, I dated myself.

16:34Izar Tarandach We should treat coding agents— we should treat coding agents as junior—

16:39Chris Romeo okay, go ahead.

16:41Izar Tarandach We should treat coding agents as junior programmers. Okay, sure. Cool. So we should treat agents as Bob from marketing. Why not?

16:51Matt Coles Okay. And so, so the agent should be acquiring tokens is what you're saying.

16:55Izar Tarandach Yeah. But he, he wouldn't be getting tokens as Bob from Marketing. He would be getting it as Bob from Marketing's agent. You fired Bob from Marketing. The agent might want to be still running.

17:08Chris Romeo Poor Bob. What did he even do? He didn't even do anything. But okay, so how are you gonna, so you were, you were saying that you could just treat them as humans. I, I don't understand how.

17:16Izar Tarandach No, no, no. I, I'm just playing devil's advocate here. I'm just saying, why do we need to invent a new, completely newfangled thing with all the—

17:25Chris Romeo Because it's the only way you scale to 10,000 separate identities without a whole bunch of manual work.

17:32Matt Coles Why? Why did you need manual work in the first place?

17:35Chris Romeo Because that's how the world works. That's why we have IT departments. Why?

17:39Izar Tarandach Why?

17:40Chris Romeo I don't know. What are you guys, toddlers? No, we're asking the question.

17:44Matt Coles You asked 5 whys. Have you not heard of 5 whys? That's not a kid thing. Well, it's a kid thing.

17:49Chris Romeo Well, no, one of my grandchildren is— 2 years old right now, and he's literally in the phase of, he's like, why? And you say, well, because, uh, that's how electricity works. It'll zap you if you touch that thing.

18:00Izar Tarandach Why?

18:01Matt Coles Well, okay, that's a little different. We can't change, we can't change electricity.

18:05Izar Tarandach We are being Socratic.

18:07Chris Romeo Yes, you're being Socratic.

18:11Matt Coles The, the methods that humans use to ident— to identify and manage human identities had a lot of roadblocks because we thought about certain things. And AI does not necessarily need that same level of roadblock, but they need different roadblocks. Right. So do we need to spend a— have a security engineer, since we're on this topic, spending time validating a human user's identity outside of the identity system? Or if it's a spinoff process from another process that you already trust, Why can't you trust it?

18:48Chris Romeo I think the key to it is the automated nature of it. So cryptographic generation to generate some key pair to identify the agent and then lock that to whatever the single source of truth for identity is so that you can always verify, validate the identity of an agent based on actions. And I think that's the real key because it's scale.

19:16Matt Coles Derivatives, right? Agents are derivative of another process, right? The parent process.

19:22Chris Romeo As of now, that's how people, if you're a developer that's using agents, if you're talking about a software factory, that's a whole different animal now, which I happen to not believe is possible today, but that's—

19:36Izar Tarandach but they are always derivative of given authority. Somebody has the initial authority to create that, the agent and to run it. Might not be the same person, but—

19:47Matt Coles May not be a person. May not be a person.

19:50Chris Romeo It could be a software factory process that kicks off the agent and then assigns some level of permission. There's an assignment provisioning process. Yeah. Okay.

20:00Izar Tarandach But it derives, it derives from given authority at some point that must have been considered and given by whomever has the ultimate say in identity.

20:10Chris Romeo Yeah, I mean, anything happens, everything spawns from—

20:13Izar Tarandach The thing is that so far we have had a matrix of identity and access. That's basically two-dimensional, right? You have people and you have the levels of access that they have.

20:23Matt Coles Yeah, but on top of that, we then store it in a database.

20:26Izar Tarandach Right. But, but again, that, that database, that, uh, that, uh, role was assumed by people, was assumed by processes that people were running, right? So it becomes a transitive thing, right?

20:41Matt Coles But we have—

20:41Izar Tarandach now you are adding one more dimension of agents and saying, oh, these things are different from those things, but they're not, not quite different, right? They have to inherit that, uh, that level of access from someone. Someone has to sign down and say, I think that this agent is okay to have these I think, I think we need to go the other direction.

21:01Chris Romeo I think agents need to be in a complete least privilege mode where they request, they request access to something and then something else gives them either permits or denies the access to that. And they don't have any credentials. They don't have any keys. That should all happen outside of their process, isolated from where they, from where they're, they're playing.

21:25Matt Coles You know, it's interesting as human, as humans, we, we create roles, right? Or we have our back, right? And, but I want you to think about this because this is an interesting problem actually, or interesting challenge. When we generated access control for humans, we took a lot of assumptions about what they do in, in giving, granting that access. And we do have systems that provide fine-grained control, right? We have capability. Access control systems, we have, you know, very strong, you know, mandatory and discretionary access control mechanisms that get applied at the machine level, but not necessarily at the identity level. And what you really are, what you're really highlighting here, I think, Chris, is that we need a system that can be a little bit more flexible when assigning these privileges, such that we grant AI capabilities on a very fine-grained level. But in reality, we should have been doing the same thing for humans, but we didn't. We made an assumption. If you want to read a file, read a set of files, we give you read access to that set of files and whatever you can do as a result of reading that file. But now with AI, it isn't just giving you that blanket access. We need to say, well, you can read that file, you can interpret that file, you can manipulate. Well, okay, not read, sorry, manipulate would be right. You can read, interpret, and then take action on that file, right? And those are much more fine-grained than simply read, 'cause that's what the human gets. The human gets read, and once they read it, they could do whatever they want with it, with that data.

23:08Izar Tarandach So the take action is implied.

23:11Matt Coles It is. What do you do when you read a file? You're going to learn it. You're going to read it. You're going to learn it. You're going to interpret it. You're going to do something with that result.

23:21Izar Tarandach So what you're saying is to an agent, you should give a separate authorization to read and to actually do something with what it read.

23:31Matt Coles I'm saying you should give fine-grained capability control to the agents because we imply it in humans. But we really need to be explicit when it comes to agents.

23:41Chris Romeo And you've got a lot more things to be able, like the tool process, you've got a, and the network process, you've got a lot more dials to, to create that fine-grained access, which a lot of the clients provide visibility and capability into, to controlling those things now at the global kind of tool level. But I see a world where. The, there, there's an access control check that happens outside wherever the agent's running that decides based on that fine-grained policy, whether you can, whether the agent could do that or not. And you don't tell, and the agent doesn't even need to know.

24:20Matt Coles Well, so here's the issue. Here's an interesting, hold on, is there just a second? I'm sorry. There's something interesting about this, what you just said in that convers— in that thought process that you just had, did that include the agent harness?

24:33Chris Romeo What do you define agent harness as?

24:35Matt Coles Cause there's a lot of— the thing that's running the agents or subagents, the orchestration platform for the agent or subagent.

24:43Izar Tarandach The thing that is not the model.

24:45Matt Coles Think about this. What if, and this might be copyrightable, patent pending, et cetera.

24:52Chris Romeo Patent pending. We are, we declared it patent pending.

24:55Matt Coles Uh, what if the access control system when applying these fine-grained controls isn't just looking at the network layer and doing, you know, looking at the sensors and the fine-grained control of the network and the file system, et cetera, but injecting context into the agent or setting up filters on the harness. So hold on, hold on. You're, you're, don't put yourself in a box just yet. If you had control, what if the, what if the identity system could say to the agent harness, Ignore invisible text. And obviously I'm whitewashing this whole thing. Ignore, ignore the, ignore the prompt injection stuff. Or don't interpret this file in this way. Yeah, it's dangerous. But the whole thing is dangerous at this point. So access control may need to extend into the agent harness to apply certain controls at that level to control the behavior of the AI independently of just controlling the perimeter.

25:56Chris Romeo I mean, and what you just described is the future we're heading towards.

26:00Izar Tarandach Mm-hmm.

26:01Chris Romeo That's, that's, I mean, there, there is a whole suite of products now that attempts to influence the plan phase.

26:11Izar Tarandach Mm-hmm. Yeah.

26:11Matt Coles But, but that's the, that's the thing that it's doing, not necessarily how it operates, right?

26:17Chris Romeo Yes. I mean, I think it's flawed because I just don't trust the agent and the box that the agent's running within. I don't trust it. And so anything that happens within the agent's control, and people are gonna tell me that I'm just being paranoid or whatever.

26:32Izar Tarandach I won't.

26:33Chris Romeo But how many, I mean, look at, look at the list of, of escapes we've seen and they keep getting, we keep adding to the escapes.

26:40Izar Tarandach Mm-hmm.

26:41Chris Romeo Right? Like Hugging Face is old news now. Like that was a month ago, two, that was in August.

26:47Matt Coles Honeyface thing, 18 company breaches ago. I mean, by—

26:51Chris Romeo yeah, and there's, there's been so many more, and they're just coming out almost every day of like, oh, agents escaped from this part of the, of this thing. Oh, whoops, we, we, we, we thought it was in an isolated environment. Turns out it wasn't. Yeah, but it tells you that it got internet access somehow.

27:09Matt Coles We don't know.

27:10Chris Romeo The goal nature of these things is, is driving towards achieving whatever the goal is that's set before them. And if, if it's, if it's gotta step around some boundaries to make it happen, obviously there's agents that have that capability. So you gotta have them in a box to be able to know what they're doing and be able to decide whether they're allowed to do something that they're trying to do or not.

27:34Izar Tarandach I, I'm sorry, but I, I, I think that we are pulling a maestro here. I, I think that we are mixing different levels, things, and calling it the same thing.

27:42Chris Romeo We've used this term before now. Is this in the, in reference to the Seinfeld character, the maestro?

27:47Izar Tarandach Exactly. So the thing with this whole isolation thing and the thing with the whole cottage industry that you mentioned that's creating all kinds of, I think that they call it guardrails this week, those things act inside the context, right? So the text that it's going into the model to be, let's call it processed and some output comes out of it and those guardrails are hoping to influence it from inside the machine. The isolation that's being broken is sandbox things, virtual machine things, network isolation things, different level, different environment, different everything else. If you were to run the guardrails inside a computer that's not connected to anything, only to electricity, and yes, please, all the side channel people shut up for a second.

28:36Chris Romeo Did you see the guy from OpenAI that said you could use CPU heat?

28:40Izar Tarandach Yeah. Yeah.

28:41Chris Romeo As the side channel, like, oh yeah, I could just like crank up a bunch of processes, generate the heat signature up. And that would be like a zero.

28:49Izar Tarandach Yeah. We, we could, we could read the aura of the computer, but, uh, anyway.

28:55Chris Romeo Yes.

28:56Matt Coles So the people who are thinking about EMF radiation leak, go ahead.

28:59Izar Tarandach Yeah. If you could run that, that, uh, those guardrails inside a computer that's not connected to anything. In theory, they would still work and they would still do the thing that they are supposed to be guardrailing against. In practice, we have seen that it's optional. It's taken as a suggestion rather than a guardrail. Sometimes it works, sometimes it doesn't for different reasons. It changes where in the context you put the guardrail, changes all kinds of things, right? Some of them are stronger, some of them are weaker.

29:30Matt Coles It has to be. I mean, don't take my comment as don't do anything else.

29:35Izar Tarandach No, no, no, no, no, no, no.

29:36Matt Coles But think about, make sure you consider defense in depth may include the harness itself.

29:40Izar Tarandach Right, right, right. But what I definitely want to point out, what I want to point out, to go back to our original question, what are the security engineers doing nowadays?

29:49Matt Coles They should be doing this.

29:50Izar Tarandach The way, the way, the way that things are today, the way that things are today, they could change next week, but the way that they are today, A lot of people are putting a lot of weight in the thing that happens inside the context window, right? But all of a sudden we are seeing that they are not putting the same weight or even more into what happens at the lower levels, which is where we are seeing all these funny breaches and all these funny escapes and whatnot.

30:18Chris Romeo What do you mean by the lowest levels and what you just said?

30:21Izar Tarandach In my head, I still see, I'm sorry.

30:24Matt Coles The AI is running at the application or above.

30:26Izar Tarandach Yeah, I still see the, the—

30:27Matt Coles and then you have physical layer and et cetera.

30:29Izar Tarandach The isolators.

30:30Matt Coles If you have, if you have physical networking capability to the internet.

30:34Chris Romeo Yeah.

30:34Matt Coles And you're running an AI process, the AI process can access the internet. Right.

30:38Chris Romeo So you're, you're, you're considering that AI is a layer 7 application.

30:43Matt Coles It is.

30:44Izar Tarandach At the end of the day, it's an application.

30:46Matt Coles I think eventually it's a process in a kernel.

30:49Chris Romeo I think it's a blob that's going to ingri— encapsulate all of the 7 layers of the OSI model.

30:55Matt Coles No, I mean, technically it's, technically it's a process running inside a GPU.

31:00Izar Tarandach Yes. No, it's running on a CPU. It's using the GPU for memory computation.

31:05Chris Romeo Eventually AI models will just, or agents will just talk to each other in their own native protocol that doesn't require—

31:12Izar Tarandach it still doesn't break. It still doesn't break the model of them being—

31:16Matt Coles are we giving them AI-defined hardware? Because then maybe that's right, but. It's still a process in a, in a system.

31:24Izar Tarandach It, it's still a process in a system. And if it goes rogue and stuff, it starts attacking, uh, 19 companies, you just go to the wall and you pull the plug.

31:32Matt Coles Except for it's on a cloud server. It's not your server.

31:34Chris Romeo You can't pull the plug. That never works in a sci-fi movie.

31:37Matt Coles What you just described, you try to unplug the ship and it's like, I'm sorry, Chris, we severed all the connections, but it's still commanding all the missile silos.

31:46Chris Romeo Yeah.

31:46Izar Tarandach Right.

31:47Chris Romeo In a war games context, I was thinking 2001.

31:50Matt Coles Yeah.

31:51Chris Romeo I'm sorry, Chris, I can't do that. Oh boy.

31:54Matt Coles You notice he did pull all the memory chips and the, and it did degrade sufficiently, but, and basically what he was doing was he was, he was reducing parameters from the model, right?

32:04Izar Tarandach Was taking more and more and more.

32:06Chris Romeo It's a metaphor. It was a metaphor the whole time.

32:08Matt Coles It was a metaphor the whole time.

32:09Izar Tarandach Until it got to the, the one point, the one bit.

32:12Chris Romeo We're talking about 2001: A Space Odyssey for those that didn't get the reference. And, uh, also don't have 3 hours of their life to— that is a long 3 hours. Okay. I really want to love that movie.

32:25Matt Coles I think it's a long 3 hours, but it's, yeah, it, it takes a long time to watch that and not a lot happens. And, and the last, and it's the last hour that's the most—

32:36Izar Tarandach unfortunately, unfortunately it's a very long and slow movie. That you have to watch a couple of times to actually understand what the hell it's all about.

32:46Matt Coles Yeah.

32:46Izar Tarandach But once you do, but nobody has that kind of time.

32:48Chris Romeo Don't, and don't drop the spoiler. Mm-hmm.

32:50Izar Tarandach No. So, so what you do, what you do is you use the solution in the movie to watch the movie. You sleep through the middle hour.

32:59Chris Romeo I do that with all movies, so that's just normal behavior.

33:02Izar Tarandach You watch the first hour, you sleep 1 hour, you watch the last hour.

33:04Chris Romeo Let's have an agent watch the movie. I was gonna try to explain it to me.

33:08Matt Coles I was just, I was just gonna ask for an AI summary of the movie.

33:10Chris Romeo No. I'm going to have it actually watch the frame-by-frame video of it and try to see if you can explain this to me.

33:19Matt Coles How much, how many tokens do you think that's going to take?

33:22Chris Romeo There's no amount of tokens that can limit this research, Matt. This is critical for our industry, what we just described here. Nope. What are we, we were talking about AI and security engineers at some point.

33:33Izar Tarandach I was making, I was making the point that security engineers today that come and say my whole job changed from, from night and day because of this AI thing. They're just not doing their job the right way because they're putting all their weight on something that deserves some weight, but not all the weight.

33:51Chris Romeo And I think this is a new world where application security and security engineers, if you embrace what these things can do, like there's a lot of cool problems to be solved. It is not a situation where We're out of— you're potentially out of work. This thing's— none of this stuff's taking your job.

34:13Matt Coles For the love of God. It was the nature of the work. It's the nature of the work, right? How much time do you spend worrying about buffer overflows anymore?

34:25Izar Tarandach The thing is, the same things apply, the attack surface changed. That's something that Matt told me a couple of months ago that still remained with me for a long time afterwards.

34:34Chris Romeo What is old is new again.

34:35Izar Tarandach Exactly. But that's, that doesn't mean that what's old stops working the way that it worked before. So I have people saying, oh, my, now reviews code and finds all the things that my deterministic scanner used to find. Good for you. Now, how do you know when it doesn't? Because I don't know, it's Thursday and the coins flipped in a different way.

34:59Chris Romeo Yeah, you got to solve the determinism problem.

35:02Matt Coles You can't.

35:03Chris Romeo Because it's built in.

35:05Matt Coles If you, to solve the term, to make AI more deterministic, you have to be deterministic. Or you have to know exactly what the answer is, needs to be, and tell the AI what it is, right? That's RAG.

35:18Izar Tarandach Or you have to set the bounds of the non-determinism. You have to say these are the single options that I'm willing to accept.

35:27Chris Romeo Or you have to iterate. You have to iterate. You have to have If you iterate, you end up being probabilistic. Yeah. So you got it, but that's what you got to get into the game of putting the AI in a tight, the tightest box possible in a small segment to do a small thing. Exactly.

35:43Matt Coles So, so here's the thing.

35:44Izar Tarandach That's the one thing.

35:45Matt Coles Think, think about the challenge, think about the challenge of static code analysis. If the AI will reliably find a set of things, a circle, and you run it multiple times and there's overlap on that circle. How many times do you need to run it to get the correct circle?

36:01Chris Romeo That's the million-dollar question.

36:02Izar Tarandach That's the probabilistic approach.

36:04Matt Coles 'Cause it's not again, one once, it's run multiple times.

36:07Chris Romeo Oh yeah. No, it's definitely more than one, but it's greater than 100. It's greater than 100. Less than 100. Sorry.

36:14Matt Coles It's greater than one, which is the important part. When you run a static code analyzer, you run it once and you get a set of output.

36:22Chris Romeo Yeah.

36:22Matt Coles That's not always correct.

36:23Chris Romeo Agreed. But, You know what I've, what I have seen though, is the model with working with the static analysis results can add some things on top.

36:35Matt Coles Absolutely. Deterministic plus non-deterministic.

36:39Chris Romeo So that's the superpower is putting them together and linking them in such a way that you get some impact. And I was thinking about this yesterday, determinism. Like if you're, if you're in a modern software CI/CD pipeline where you've, you're, let's say you're pushing 500 PRs a day and you see, you're going to get some overlap amongst those 500 PRs. And so it's not that your determinism risk of running something once, you could get some magnified effect by the depth of the PRs. That are happening, that are, that are crossing, like, like some, some PRs are crossing code files, right? As, as people are working on a feature.

37:25Matt Coles So that's— especially if they're design changes, right?

37:27Chris Romeo Yeah. And so that could, so that could result in multiple scans of a particular piece of code over the same day that only have slight modifications to it. And so that, I was just thinking about that as a potential, it's not the answer to a deterministic process, but it's additional runs of a deterministic process that may help each other.

37:49Matt Coles Right. And you're collecting information.

37:50Chris Romeo Versus just scanning something once.

37:51Izar Tarandach Yes, but because of those changes and because the changes happen separately from the scanning process, now you are starting a non-deterministic process from changing starting values. So it may look like you're starting from the same place, but you're not.

38:08Chris Romeo Yeah, exactly.

38:09Izar Tarandach And that changes the way that's forwarded. Yeah, exactly.

38:11Matt Coles So you have to, you have to think about your process in addition to your task.

38:16Izar Tarandach But I, I like the way that you put it of reducing the, the, the focus and the agency of the AI to something so small that it becomes much easier for you to, to trust over time.

38:28Matt Coles Right.

38:29Chris Romeo And it allows you to use a lower cost model to achieve that. Very small segments so that you can fan out 50 or 100 of them.

38:40Matt Coles You're, you're bounding the probabilities is what you're doing, right? By making the task, by, by making something so granular, the options of outcomes, obviously it goes—

38:51Izar Tarandach the possible outcomes, the universe becomes smaller.

38:53Matt Coles They do go off the ends, which is part of the problem because we can hallucinate, but you limit the likelihood, right? You limit the probabilities.

39:02Izar Tarandach All right.

39:03Chris Romeo Well, I think we made some progress. What, Izhar, you got the final word here on—

39:09Izar Tarandach although by reducing too much, you, you, you miss the view of the whole thing. So those changes will become much closer to syntactic fixes than to something that may influence something that may catch and influence a wrong design as much.

39:27Chris Romeo Well, you, I think you gotta look at it from multiple levels, right? Mm-hmm. It's not. It's not one or the other. It's looking at the 20,000-foot view, the 5,000-foot view, then the ground view, and then the microscope, or the, I'll use the magnifying glass on the ground.

39:44Matt Coles Which we should have always been doing, whether we were using AI or not.

39:47Izar Tarandach Which is what, what makes sense to, to give the scanning agents a threat model as well, so that you're giving the, the top thing and now go look at the code and tell me how the code is breaking the top thing.

39:59Chris Romeo Yeah, that's a whole other episode, but I've been, I've been doing some extensive threat modeling with models, dude. It's pretty amazing for what it could, what it can rationalize.

40:13Matt Coles Yeah. Just run it multiple times and see what you get.

40:17Izar Tarandach Mm-hmm.

40:17Chris Romeo Of course I run everything multiple times 'cause I understand determinism and if I don't like the answer, I just run it again.

40:23Matt Coles Well, there you go.

40:25Chris Romeo That's what I thought determinism meant. Yeah, on that note, some dice.

40:31Izar Tarandach No, there is determinism, there is non-determinism, there is my determinism.

40:36Chris Romeo That's a good, that's a sticker. That's a sticker idea. The AI security table.

40:41Matt Coles That's a Venn diagram right there.

40:43Chris Romeo We, we think of determinism as a dice roll. If you don't like the answer, just run it again. Oh, look, I got no vulnerabilities now.

40:53Izar Tarandach Determinism is a continuum.

40:55Chris Romeo It's a journey.

40:57Matt Coles I choose my determinism.

40:58Chris Romeo Determinism is a journey, not a destination.

41:01Izar Tarandach I determine what's determinism.

41:04Chris Romeo That's just being— now you're just being selfish. All right, folks, thank you for joining the recently rebranded AI Security Table where we talked a lot about a lot of AI stuff today. Talked a little bit of smack about it too, but that's just kind of what we do here. And, uh, have a great day wherever you find yourself on planet Earth. Thanks for joining us at the AI Security Table. Subscribe, rate, review, and follow wherever you listen. New episodes every week. AI Security on the Table.

Transcript supplied by AssemblyAI. It may contain transcription errors.