# An AI Agent Escapes and Attacks. Who Goes to Jail?

URL: https://securitytable.ai/an-ai-agent-escapes-and-attacks-who-goes-to-jail/
Published: 2026-10-14
Duration: 2245 seconds
Season: 4
Episode: 23
Hosts: Chris Romeo, Izar Tarandach, Matt Coles

## Show notes and chapters

OpenAI says the policy window is open and wants binding federal rules for frontier labs, clear incident reporting, and disclosure whenever a model gets around its own security controls. The table asks whether that is real accountability or a lab writing the rules it already expects to win under. Matt sees the leader pulling everyone else into the funnel, Chris calls it an innovation killer, and Izar wonders why anyone is treating incident reporting like a new idea. From there it is OpenAI's favorite California bills, who gets to audit the labs, and Senator Markey's proposal for an NTSB style board with subpoena power to investigate AI driven attacks. Then the real question: when an agent escapes its sandbox and breaks something, who goes to jail? Izar argues nobody will until there is real impact and asks why computer science never got its own Hippocratic Oath. Plus: Matt earns a new nickname, the Concluder.

Mentioned in this Episode:  
➜ [The AI policy window is open. We need to act.](https://openai.com/index/ai-policy-window/)  
➜ [We Must Pace the Frontier](https://darioamodei.com/post/we-must-pace-the-frontier)  
➜ [Senator Markey Introduces the Cybersecurity and AI Board of Investigations Act](https://www.markey.senate.gov/news/press-releases/as-ai-agents-carry-out-attacks-senator-markey-introduces-legislation-establishing-independent-body-to-investigate-cyber-hacks-assisted-by-artificial-intelligence)  
➜ [OpenAI and Hugging Face partner to address security incident during model evaluation](https://openai.com/index/hugging-face-model-evaluation-security-incident/)

Chapters:  
00:00:00 - Cold Open: Izar Warms Up to Take a Punch  
00:01:21 - The Premise: Agents Move Faster Than Humans  
00:02:21 - Article Intro: The AI Policy Window Is Open  
00:03:08 - Is This Real Accountability?  
00:03:34 - Matt's Case: Everyone Into the Funnel  
00:05:05 - Self Interest and Buying a Car With Claude  
00:06:15 - Izar's Rebuttal: Haven't We Asked for This Before?  
00:07:49 - Is Regulation an Innovation Killer?  
00:08:19 - Dario's Call to Pace the Frontier  
00:09:02 - OpenAI's Four California Bills  
00:09:52 - Who Audits the AI Labs?  
00:11:51 - Independent Safety Assessments  
00:12:12 - Why Only Biological Threats?  
00:13:12 - Is the Federal Government the Answer?  
00:13:43 - Senator Markey's AI Board of Investigations Act  
00:14:54 - Who Plays Izar in the Movie?  
00:16:02 - Do We Want an NTSB for AI?  
00:16:27 - When a Test Plane Crashes  
00:17:19 - AI Generated Podcasts  
00:18:38 - Back to the NTSB: Training Incidents  
00:19:35 - What Really Happened at Hugging Face?  
00:20:54 - Recommendations Without Enforcement  
00:21:51 - Agents Attack and Nobody's Going to Jail  
00:22:14 - Izar's Take: Jail Requires Impact  
00:24:52 - Are Corporations People?  
00:25:41 - Liability: Somebody Wrote the First Line  
00:27:55 - Scenario: A Rogue Agent Shuts Off the School Heat  
00:29:20 - First Do No Harm in Computer Science  
00:30:58 - Should Do No Harm Live in the Model?  
00:31:51 - A Hippocratic Oath and Certified Developers  
00:33:33 - Not Everyone Can Build a Bridge  
00:35:02 - Matt's Close: Visibility Beats Marketing  
00:35:43 - Closing Statements: Meet the Concluder  
00:36:20 - Outro

Follow AI Security Table:

➜ Home: [https://securitytable.ai/](https://securitytable.ai/)

➜ X: [https://x.com/AISecurityTable](https://x.com/AISecurityTable)

➜ LinkedIn: [https://www.linkedin.com/company/ai-security-table/](https://www.linkedin.com/company/ai-security-table/)

➜ YouTube: [https://www.youtube.com/@AISecurityTable](https://www.youtube.com/@AISecurityTable)

## Transcript

**Chris Romeo (0:09)**: Welcome to the Security Table, where Isar was smacking himself on top of the head to start the episode.

**Izar Tarandach (0:16)**: I will not taunt you a second time.

**Chris Romeo (0:18)**: Which normally he would do that a little later in response to something that Matt or I said, because he couldn't just, couldn't take it. Anymore.

**Izar Tarandach (0:27)**: I'm just warming up the engine.

**Chris Romeo (0:29)**: Yeah, but he's gotta get, he's gotta get ready to go. I mean, winter is here. Izar's dressed in his fleece.

**Izar Tarandach (0:39)**: Wait, you never seen on UFC before they, they go into the fight that they go, same thing.

**Chris Romeo (0:44)**: That's what you, oh, okay. Oh, you're

**Izar Tarandach (0:46)**: same thing.

**Chris Romeo (0:47)**: You're getting yourself mentally prepared to take a punch.

**Izar Tarandach (0:50)**: Yep.

**Chris Romeo (0:50)**: Okay. Got it. This is a rough— the security table is a rough place, so it's not for the faint of heart.

**Matt Coles (0:58)**: You know, it's interesting if we look at our clothing styles. Chris in blue, me in black, Izar in black and blue, just like his face can't be.

**Chris Romeo (1:07)**: Wow, some bullying. We don't allow bullying here, Matt, on the security table.

**Izar Tarandach (1:12)**: Okay.

**Matt Coles (1:13)**: Oh, I wasn't— I, I didn't say by, by us.

**Chris Romeo (1:16)**: It sounded like bullying to me. That's all I That's all I could say. All right, let me set this up because we have kind of a premise and then we have 3 different perspectives to, I was going to say attack, but well, discuss, attack, bullying, bully, whatever. Kind of our normal approach to the world. Let me set this up. So we got these things called AI agents. People have heard of these.

**Izar Tarandach (1:43)**: What?

**Chris Romeo (1:43)**: And They're now attacking and building software faster than any human process can keep up.

**Matt Coles (1:48)**: That's why we love them, right?

**Chris Romeo (1:50)**: I think I agree with that. It does. They do, do go pretty fast.

**Matt Coles (1:53)**: They do.

**Chris Romeo (1:54)**: And so we've got 3 different perspectives on what we should do about this. So we've got the lab's perspective, Frontier Lab, OpenAI. We've got the US government and then the supply chain side of it. And so the question that we're gonna try to wrestle with here are any of these solving the actual problem under the hood, or are they all just talking for the benefit of talking? So let me set up OpenAI's perspective here. This is an article from September 9th. This is OpenAI's pitch on, you know, safety. And I think this was ultimately a response to Dario Amadei's call to slow down the frontier, pause the frontier, which nobody actually did anything with, but It was a nice set of words. Uh, so in this description, OpenAI argues that the policy window's closing. We need mandatory federal rules aimed only at frontier labs. We need clear incident reporting rules, and then we need disclosure when a system gets around its security controls. And he also wants company-led incident reporting to sit alongside federal oversight. So the question for us to wrestle with is, first of all, is this accountability? Is this real accountability or is this just one of the frontier labs defining the constraints that it believes it should be kept under? Which you've got to imagine there's some self-servantness. When I, when I write the rules for something, I almost always want myself to win as a result of it. So what do you guys think?

**Matt Coles (3:34)**: Well, I mean, on the one hand, I guess it could be a realization that, you know, things are moving so rapidly. I guess the self-serving angle is maybe it's been moving for other people and they want to bring everyone, you know, into the funnel, right? If you make the, if you write a set of standards, if you write a set of standards and you were not already the winner, then you would be restraining everyone else. Right? That's a really, that's a really extreme, you know, viewpoint though. I don't think that's what OpenAI was looking to do, but, but that may ha— it may have that effect. If OpenAI was the leader, they would be bringing everyone else into a, into a funnel basically in doing so by setting regulation.

**Chris Romeo (4:17)**: And we're only talking about OpenAI and Anthropic. Well, maybe SpaceX AI.

**Matt Coles (4:25)**: I mean, maybe Google as well. Uh, Gem— with Gemini.

**Chris Romeo (4:28)**: I always forget about Google. I forget about—

**Matt Coles (4:30)**: I know that's where the problem was as a, That's probably the problem. They're there, but I've never used them.

**Chris Romeo (4:36)**: They're just not, it's just not as cool at a cocktail party to be like, hey, I'm using Google Gemini.

**Izar Tarandach (4:42)**: So yeah.

**Matt Coles (4:45)**: Well, then you have a whole host of open source and no wait models and things like that, right?

**Izar Tarandach (4:50)**: The thing is, it is cool on a cocktail party because that's when you look at like a side people and say, what, you're using only one model? And you say, I use one for one thing and one for another. That's the cool kid. Approach.

**Matt Coles (5:03)**: Tokenizing or—

**Izar Tarandach (5:05)**: but on the self-serving angle, I assume it's just like when you go and buy a car, right? You're going to get screwed. The question is by how much.

**Matt Coles (5:13)**: To all of our car dealers out there. Yeah.

**Izar Tarandach (5:16)**: I mean, they are self-serving. Of course they are self-serving.

**Chris Romeo (5:19)**: You gotta get a better negotiator, man. You're supposed to win that deal.

**Izar Tarandach (5:23)**: Yeah.

**Matt Coles (5:24)**: You should use a Frontier model.

**Izar Tarandach (5:26)**: Suppose is the law.

**Chris Romeo (5:27)**: I did. I used a model last time I was helping my daughter buy a car and I had Claude as my co—, negotiator.

**Matt Coles (5:35)**: Don't say copilot.

**Chris Romeo (5:36)**: I almost said copilot, patent pending. But yeah, I mean, it was, it was actually—

**Matt Coles (5:40)**: But that's actually taken by Microsoft, which you also didn't mention as a—

**Chris Romeo (5:43)**: Microsoft's in AI too.

**Izar Tarandach (5:46)**: It's open, no, it's open AI.

**Chris Romeo (5:47)**: No, but the, the sales guy, just to close the loop, the, the car sales guy, he's like, he's like, everybody's doing this to us now.

**Izar Tarandach (5:55)**: Yeah, that's because he had open AI open.

**Chris Romeo (5:57)**: So he said, 'cause I'm like, oh yeah, I'm using Claude behind the scenes here. And he's like, Oh, you're not the only one. Everybody is. He's like, I get text messages and I have to negotiate, which I, with what I know is an AI on the other side.

**Izar Tarandach (6:10)**: So see,

**Matt Coles (6:10)**: that's the story we should be talking about right there. The practical use of AI in the field.

**Izar Tarandach (6:15)**: But, uh, Chris, can, can you real quick go, go again over the list of things that they're asking for in the policy?

**Chris Romeo (6:21)**: Yeah. So they're looking for mandatory rules aimed only at Frontier Labs, clear incident reporting. And then disclosures when a system gets around its security controls.

**Izar Tarandach (6:35)**: Yeah. So one of the things that I've been enjoying a lot when doing threat modeling of AI stuff is at some point imagining the AI part out of it and just putting in something scripted, deterministic, nicely done with ifs and thens and elses and seeing if there is any change to the whole picture, right?

**Chris Romeo (6:53)**: Okay.

**Izar Tarandach (6:54)**: And if there isn't, then the AI is not your problem. But now look at those things that they are asking. And join me in our time machine in the past, what, 30 years, 20 years, something like that. Haven't we been asking for the same thing for normal incidents? Like, are we really asking for anything new here? Or I feel like we got there.

**Chris Romeo (7:19)**: Don't we have incident reporting pretty much for non-AI-related things, maybe that affect national

**Matt Coles (7:26)**: security and whatnot, but.

**Izar Tarandach (7:28)**: Yeah, but you know, if he's asking for an NTSB here of AI, then the question becomes only AI? Why not the things that AI is making happen now that weren't happening before and stuff like that? But again, it goes back to, you know, more of the same, more of the same, more of the same. Fine. Yeah.

**Chris Romeo (7:47)**: But it's an innovation—

**Izar Tarandach (7:48)**: it just has AI in it now.

**Chris Romeo (7:50)**: It's an innovation killer. What is an innovation killer? Anytime you add things that slow stuff down, the ultimate result is killing innovation. And that's why I don't get this. I don't get why OpenAI and Anthropic are coming out in support of this.

**Izar Tarandach (8:11)**: When— so let, let, let me play the devil's advocate here, please. Why would this slow down something? It's just reporting, isn't it?

**Matt Coles (8:18)**: There's more.

**Chris Romeo (8:18)**: Well, the whole, the whole, I'm, I'm kind of including, I guess I'm, I'm stepping further beyond what this particular article said to Dario's whole, we need to slow the pace.

**Izar Tarandach (8:30)**: Uh-huh.

**Chris Romeo (8:31)**: Of this. And, and then Sam Altman came out and agreed with him, I think on X, agreed with the, the general statement that yes, we need to slow down the pace. Now neither one of them, like I said before, neither one of them did anything about it. They just like, oh yeah, and by the way, tomorrow new models coming out that are cranking this thing up to 11.

**Izar Tarandach (8:50)**: Yeah, it's, it's, it's like, it's like a teenage, uh, relationships, right? When, when one of the parties looks at the other and says, oh, we have to slow things down. Next thing you see is a shotgun and a wedding dress. So

**Matt Coles (9:02)**: now interestingly, the, the, the OpenAI, uh, policy view here does highlight 4 California state laws. Or state, I'm sorry, bills. They're not laws, I don't think yet, that cover a range of topics that OpenAI supports. And I, I'm, the way they described it is until Congress does something, until the US Congress does something, they will continue to support state legislation. And they highlighted 4 California bills in that, that they think cover aspects of AI use or AI development that I guess embody their policy views, right? So standards on auditors for AI systems, which is kind of interesting 'cause I don't think we have standards. I don't, I'm not aware of us having standards for other auditors.

**Chris Romeo (9:53)**: Depends on what you define as auditor. Certainly a financial auditor, it's a very well-defined role. From a regulatory perspective, whether that's Sarbanes-Oxley or whatever causes you to have to not cheat on your financials as a public company, SOX or whatever.

**Matt Coles (10:18)**: Right. But PCI auditors are not like government regulated.

**Chris Romeo (10:22)**: True.

**Izar Tarandach (10:23)**: No.

**Chris Romeo (10:23)**: Good point.

**Izar Tarandach (10:24)**: The only way I'm trusting any auditor here is if one company pays the auditor of the other one. So if OpenAI pays—

**Chris Romeo (10:31)**: So one of the labs pays the other lab to—

**Izar Tarandach (10:34)**: If one of the labs pays for the audit of the other lab, then I'm trusting shit. Otherwise, no.

**Chris Romeo (10:41)**: So you don't trust— let's unpack that a little bit.

**Matt Coles (10:44)**: Oh my gosh.

**Chris Romeo (10:45)**: So you don't— you're in full-on security contrarian mode here going like, you don't believe, you don't trust anybody, it sounds like.

**Izar Tarandach (10:55)**: No, wait, Matt had a thing going. Am I familiar with that?

**Matt Coles (10:58)**: I was about to say, do you understand the concept of collusion?

**Izar Tarandach (11:01)**: Yeah.

**Matt Coles (11:02)**: That's what I'm saying. In the public perception, OpenAI pays for an audit of Anthropic, Anthropic pays for an audit of OpenAI, and nothing comes of it.

**Izar Tarandach (11:12)**: Yeah, but now I have the conflicting interests of each one of the payers that they want the other one to look bad. So they want that auditor to actually find shit.

**Matt Coles (11:23)**: You're assuming that the auditor is then themselves part of the collusion. I'm an optimist. And are not acting in good faith.

**Izar Tarandach (11:32)**: I'm an optimist. Auditors always acting in good faith.

**Matt Coles (11:36)**: Auditors always act in good faith?

**Chris Romeo (11:37)**: You're an optimist.

**Izar Tarandach (11:38)**: I'm an optimist. That's what I'm saying.

**Chris Romeo (11:40)**: That's the funniest part of that. You're an optimist.

**Matt Coles (11:43)**: Yeah. Well, who, where did you, where did Izar go and who are you?

**Chris Romeo (11:48)**: Carl. Carl. Shiny happy guy over here.

**Matt Coles (11:51)**: So that was one. The other, another one is infrastructure for independent safety assessments. So again, an audit function for somebody other than the Frontier Lab itself to police it, to basically do security assessments or safety assessments. Sorry. Some bill for protecting young people. I haven't looked into that one yet. And safeguards against biological threats, enable AI-enabled biological threats. Why just biological and not nuclear threats?

**Izar Tarandach (12:21)**: Yeah, that, that one always gets me. Why biological? Like what, what's the, is it, is it a remnant of a COVID or something? Like why always the stress on biological?

**Matt Coles (12:31)**: Maybe that's easy, relatively easy.

**Izar Tarandach (12:35)**: If it's nuclear, the AI is going to destroy itself.

**Matt Coles (12:38)**: Well, nuclear, you need to like build equipment and buy material, you know, get, get your hands on material. Whereas biological, you could probably do in your kitchen.

**Izar Tarandach (12:45)**: Mm-hmm.

**Matt Coles (12:47)**: Right. The difference between building a, a big bomb and a pipe bomb is, you know,

**Izar Tarandach (12:52)**: yeah, because AI attacking infrastructure is a scenario that Hollywood never imagined.

**Matt Coles (12:57)**: Gee, how many, how many AI-enabled, yeah.

**Izar Tarandach (13:00)**: If they, if Hollywood can't imagine it, there, there's no bills covering it.

**Matt Coles (13:06)**: Well, it is California, you know.

**Izar Tarandach (13:07)**: Oh yeah. It's deep.

**Matt Coles (13:08)**: So anyway, where, where were we? Cuz, uh, I think we just hit a wall.

**Chris Romeo (13:12)**: Oh, we can, we can transition into, is the federal government the answer on this issue?

**Matt Coles (13:19)**: Not the full answer. A, it's only one country.

**Chris Romeo (13:23)**: You didn't even hear the setup before you squashed it. That's just like,

**Izar Tarandach (13:28)**: no, it's just that when you say it's the federal government, the, the answer, I'm, I'm still thinking what's the, the question?

**Chris Romeo (13:36)**: Yeah. What is that? That is often what you should be saying is, whoa, whoa, whoa, whoa, whoa.

**Matt Coles (13:40)**: What was the question?

**Chris Romeo (13:41)**: So,

**Izar Tarandach (13:41)**: and then you say no.

**Chris Romeo (13:43)**: Somebody from a tiny little state called Massachusetts. I'm just saying that 'cause that's where Matt's from. Um,

**Izar Tarandach (13:51)**: somebody wants,

**Chris Romeo (13:53)**: wants to create the Cybersecurity and AI Board of Investigations Act. That sounds really official.

**Izar Tarandach (14:00)**: Like that's the NTSB. Okay.

**Chris Romeo (14:02)**: Yeah. They wanna create an NTSB national, what's that? National Transportation Safety Board style. Set up that would investigate AI agent breaches of federal systems and critical infrastructure, look at near misses, and they would have, most importantly, subpoena power. So they could subpoena the Security Table.

**Matt Coles (14:21)**: So that is literally the NTSB for AI.

**Chris Romeo (14:24)**: Wouldn't that be cool if they subpoenaed us? Wouldn't that be hilarious? No, it would be very bad. They're like, we've listened to some of your videos, and you know what,

**Izar Tarandach (14:34)**: I'll do that. I'll do that.

**Matt Coles (14:36)**: I plead the fifth.

**Chris Romeo (14:37)**: They're like, we think we're going to throw you in prison. But we thought we'd give you 30 minutes to see if you deserve it.

**Izar Tarandach (14:43)**: I would take one for humanity.

**Chris Romeo (14:45)**: You would? I would. Good. Okay. Matt and I are taking, pleading the Fifth and you're gonna be talking.

**Matt Coles (14:50)**: We like, I'll refer all, all, I'll refer all questions to my attorney.

**Izar Tarandach (14:54)**: I just wonder who, who would play me in the movie?

**Chris Romeo (14:57)**: Who would play you in the movie?

**Izar Tarandach (14:58)**: Who would play me in the movie?

**Chris Romeo (15:00)**: That's it.

**Matt Coles (15:00)**: You would play you in the movie? You do your own stunts?

**Izar Tarandach (15:03)**: No.

**Chris Romeo (15:03)**: Trying to think who plays you. I'm a bad loser.

**Izar Tarandach (15:06)**: I'm not that good.

**Chris Romeo (15:07)**: Let's see. I'm really stumped here.

**Matt Coles (15:09)**: You need a, you need an actor who can play carpal tunnel and

**Chris Romeo (15:12)**: that takes a lot of range from an actor.

**Izar Tarandach (15:14)**: The claw.

**Matt Coles (15:16)**: I have to get into the moment.

**Chris Romeo (15:18)**: Oh, what is, uh, the guy who played George from Seinfeld? Is he too old to play as our DAEI?

**Izar Tarandach (15:25)**: Was, was, uh, what was the, the line? The sea was— DAEI was angry, my friends.

**Chris Romeo (15:32)**: Yeah. That would be, that would be a really funny reboot. If they were able to do Seinfeld, like a 10-episode arc of Seinfeld, bring Seinfeld back in the AI era. Just the age. Oh, that would be hilarious.

**Matt Coles (15:44)**: I'm sure somebody would do it. How many tokens would that take? I mean, come on.

**Chris Romeo (15:48)**: Mm-hmm. It's true. You could probably just have AI build it. Okay.

**Izar Tarandach (15:51)**: Just imagine the agent escaped. The agent escaped.

**Chris Romeo (15:54)**: Yeah.

**Matt Coles (15:55)**: It's for you.

**Izar Tarandach (15:56)**: Yeah. What do you— slow down.

**Chris Romeo (15:59)**: All right, so, all right, where'd we

**Matt Coles (16:00)**: go?

**Chris Romeo (16:01)**: We're off on a bit of a tangent there, but no, I mean, is this

**Izar Tarandach (16:05)**: What?

**Chris Romeo (16:06)**: I mean, are we ever in favor of NTSB-style anything?

**Matt Coles (16:11)**: Yes, we're in favor of the NTSB.

**Izar Tarandach (16:13)**: Yes, the NTSB, very much so.

**Matt Coles (16:16)**: Right. But in regards, when planes, when planes crash, we wanna know why.

**Chris Romeo (16:20)**: True.

**Matt Coles (16:21)**: So if AI crashes, but if something

**Chris Romeo (16:22)**: crashes,

**Matt Coles (16:23)**: we kind of need to know.

**Chris Romeo (16:24)**: The AI crashing world.

**Matt Coles (16:26)**: Well, so interestingly, the difference between, and I, I actually don't know how the NTSB operates in this case. If Boeing was testing a new plane and it crashed in testing, would the NTSB get involved?

**Izar Tarandach (16:43)**: Yes,

**Chris Romeo (16:43)**: I think so. Yeah.

**Izar Tarandach (16:45)**: Yes.

**Chris Romeo (16:45)**: And so I think that happened.

**Izar Tarandach (16:46)**: No, no, no, no, wait. The, the plane is not in production yet.

**Matt Coles (16:49)**: Hasn't, right. Hasn't gone into production. Right. It may have come out of wind tunnel tests or

**Izar Tarandach (16:55)**: I think it's the FAA.

**Chris Romeo (16:57)**: Some, it happened recently in the last year. There was a test plane crash.

**Izar Tarandach (17:04)**: Well, that, that's every day that ends with a Y, but, uh, I think that it's the F FAA that, wait, let's ask Claude.

**Chris Romeo (17:12)**: Well, if we had Carl here,

**Izar Tarandach (17:14)**: Claude, Claude is in, in, Carl is, uh, tuning up.

**Chris Romeo (17:18)**: Okay. Well, just listeners, we're gonna be the first podcast to have all the hosts replaced by AI.

**Matt Coles (17:28)**: The first? Hardly.

**Chris Romeo (17:32)**: Have you heard, have you guys heard these, these, I've been, I've been sourcing podcasts in the AI development and security world and there is this genre now that has appeared where two appearing, two hosts appear to be human, but they're just too perfect. They don't say a single like, or, uh, unless it's generated.

**Matt Coles (17:58)**: It's the wrong prompt then. They should be, talk like a California, you know, school kid or something.

**Chris Romeo (18:03)**: But they, they, it's, it's interesting content because they're analyzing, uh, academic papers that are recent academic papers in the AI security space.

**Matt Coles (18:16)**: So somebody took NotebookLM from Google, put a bunch of papers in it, and then had 'em generate a podcast. Which it does.

**Chris Romeo (18:23)**: 2 or 3, 2 or 3 come out a week.

**Matt Coles (18:26)**: Yeah, why not?

**Chris Romeo (18:27)**: So my point is we might not even have to be here next week.

**Matt Coles (18:31)**: Wait, we like being here.

**Izar Tarandach (18:33)**: Yeah, that's the thing.

**Chris Romeo (18:35)**: Oh yeah.

**Izar Tarandach (18:37)**: Until there is a, there's legislation against it, we'll be here. US experimental crashes, if it's a small thing like, uh, somebody building it on the back of the, the farm, then it's the FAA. And the FAA reports back to the NTSB. If it's a big thing, then it's the NTSB.

**Matt Coles (18:53)**: Okay. So, so that, and that's an, that's, it's important. I wanted to bring that up because if this creates something like the NTSB for AI, then what this would mean is during training, if there's any training incidents, like, I don't know, an AI escaping its sandbox and going off and attacking other companies.

**Chris Romeo (19:11)**: Would that really ever happen in the real world though? No, never. It's already happened like twice this week. Okay.

**Matt Coles (19:18)**: Uh, then an agency would get involved to figure out what's going on. So you'd have an independent analysis. The one nice thing about the NTSB and its function is they're very thorough. They're also—

**Chris Romeo (19:28)**: are they fast though?

**Matt Coles (19:30)**: No, of course not. But, but you do eventually know what really happened.

**Chris Romeo (19:35)**: And so that's what I feel like we don't have now. Like we really don't know. What happened with even Hugging Face, even the Hugging Face one, the one that's gotten the most press, the most people trying to write up what their insider scoop is when they're not even an insider.

**Matt Coles (19:52)**: Yeah. And well, I mean, any podcast host can opine about what happened with the Hugging Face incident.

**Chris Romeo (19:58)**: Oh, I think we did that, didn't we?

**Matt Coles (20:00)**: We did that last week. Yes.

**Chris Romeo (20:02)**: Dang it. We fell into the trap.

**Matt Coles (20:07)**: So, Yeah. So, so anybody can opine about what happens and guess about what happened, but an agency-like thing that does investigations that has the ability to come in and subpoena people or look at logs and be independent reviewers.

**Chris Romeo (20:23)**: Yeah. But are they gonna post what they come up with?

**Matt Coles (20:26)**: I don't know. The NTSB does.

**Chris Romeo (20:28)**: Do they? Is it in full? There's no internal version and an external version.

**Matt Coles (20:32)**: If there is, I mean, the, the The, whatever they publish, publicize, whatever they publish is very thorough. I

**Chris Romeo (20:39)**: mean, that would be my fear here is that one, it takes forever to get anything actionable. And two, they don't tell you the whole story, which is the whole point of creating something like this is because we want to get the whole story.

**Matt Coles (20:54)**: Well, so something else that the NTSB does, and I, I wonder if this bill would support something like it, is the NTSB is is in— is trusted and independent, but it's also not regulatory. So the NTSB doesn't write law. The NTSB investigates accidents. They're there for safety. And so if this bill introduced an age, an, you know, a scheme for having an independent analysis of what went wrong, they can give recommendations about what they're gonna do about it, but they can't actually enact any of those things, right? So they would have to go to to, I don't know, some AI regulation framework, or they would have to go to the FBI or whoever's involved with cyber, with AI-based cybercrime. Or, you know, because that's ultimately a hacking of another company is in theory a cybercrime in most—

**Chris Romeo (21:44)**: is it really anymore?

**Matt Coles (21:45)**: Well, okay. But it's a stretch.

**Chris Romeo (21:47)**: It doesn't feel like it is anymore.

**Matt Coles (21:49)**: It doesn't feel, it doesn't feel like it is.

**Chris Romeo (21:50)**: It should. And I'm, I'm not, I'm not trying to be funny. Like it should,

**Matt Coles (21:54)**: it should be like, it should be like a thing, right?

**Chris Romeo (21:57)**: But like AIs are escaping and apparently just attacking things and like nobody's going to jail like they used to in the good old days, like Kevin Mitnick when he got arrested.

**Matt Coles (22:05)**: If humans did this, they would be, the FBI would've showed up at their door and I mean, we've seen hackers, right?

**Izar Tarandach (22:11)**: Excuse me. Excuse me.

**Chris Romeo (22:13)**: I thought it was real.

**Izar Tarandach (22:14)**: Excuse me. Big difference. The people who went to, uh, to jail is because there was an impact to the attack. Have we heard of any impact from the AI stuff, Franklin, so far? Uh, listen to me. We're focusing much more on the fact that they are attacking.

**Matt Coles (22:28)**: Malicious modification of packages.

**Chris Romeo (22:31)**: Yeah, the malware, it was malware that got distributed to commercial.

**Izar Tarandach (22:35)**: Was there impact?

**Matt Coles (22:37)**: There was malware created and distributed through official channels.

**Izar Tarandach (22:42)**: Did that lead to impact? We have a number of millions of phones.

**Chris Romeo (22:45)**: So now we're measuring in impact?

**Matt Coles (22:48)**: Okay.

**Izar Tarandach (22:49)**: No, what I'm pointing out—

**Matt Coles (22:49)**: Hold on, hold on.

**Izar Tarandach (22:50)**: No, no, no, no, no. What I'm pointing out is that when it was, when it is people, What you hear about is some dude went to prison because of $5 million in whatever. But here we are, we are hearing about something broke out of somewhere and touched something else.

**Matt Coles (23:08)**: Okay.

**Izar Tarandach (23:10)**: We are more worried about the touching than whatever they got away with.

**Matt Coles (23:15)**: I, I have, I have a line for you and I now, now I, now I see the connection, uh, from, from Chris's earlier, from the beginning of this of this episode. If malware is introduced into your supply chain, but it doesn't make an impact, does it matter?

**Izar Tarandach (23:34)**: Clue. You know what clue is?

**Chris Romeo (23:37)**: No.

**Izar Tarandach (23:38)**: It's the sound of hand clapping with one hand. So if we're going to go zen here, again, my point is there must be a very heavy component of marketing here, as we all already know. But we haven't yet put the finger on because really all the hoopla that we are getting is on the beginning of the event and not at the end of the event. People are mesmerized by the fact that AI is going rogue and it's going out and it's attacking things. But what has the attack done already? If it's happening thousands and thousands and thousands of times a day, as it's said, what is being impacted?

**Matt Coles (24:17)**: How much of an impact do you actually need to be violating a law?

**Izar Tarandach (24:22)**: Oh, so you're saying that intent is already— yeah, no, I agree with that completely.

**Matt Coles (24:25)**: Right. I mean, what the DMCA, you know, with, if you just unauthorized access to a computer, you've broken a law.

**Chris Romeo (24:33)**: Agreed.

**Matt Coles (24:33)**: So all of these have had a legal impact of some type.

**Chris Romeo (24:36)**: Yeah. But who's guilty?

**Izar Tarandach (24:38)**: Well, that's an interesting question. Who's guilty? Who's guilty is clear.

**Chris Romeo (24:42)**: Who, who, how's it clear?

**Matt Coles (24:43)**: Who, yeah, who is actually at fault?

**Izar Tarandach (24:46)**: It's not so clear.

**Chris Romeo (24:48)**: No, no,

**Izar Tarandach (24:48)**: no, no, no, no, no, no. Wait a minute.

**Matt Coles (24:50)**: And I, I, there's another, there's another take on this because are corporations people?

**Izar Tarandach (24:55)**: No, but

**Matt Coles (24:56)**: yeah,

**Izar Tarandach (24:57)**: they are. Are they liable?

**Matt Coles (24:58)**: They are for political purposes, but are they people for legal purposes?

**Izar Tarandach (25:02)**: Okay. Now, now you're taking it all the way to a very, very dark place, but somebody's liable.

**Matt Coles (25:08)**: Wait, I'm all in black. It's a dark place, right?

**Chris Romeo (25:10)**: So somebody's liable.

**Matt Coles (25:12)**: Who? Tell me, tell me.

**Izar Tarandach (25:14)**: Hey, so when, well, I really don't want to go there.

**Chris Romeo (25:20)**: No. Well, you've already opened the door.

**Matt Coles (25:22)**: You have opened the pit,

**Chris Romeo (25:24)**: the pit of despair.

**Matt Coles (25:25)**: Pit of despair.

**Izar Tarandach (25:26)**: Is this sort of what I'm looking at right now? Yeah. Uh, no, I'm not going to use that.

**Matt Coles (25:32)**: What is your name? What is your quest? What was the impact you caused against any, like, blue act?

**Chris Romeo (25:40)**: Absolutely.

**Izar Tarandach (25:42)**: Okay. So liability. Look, at the end of the day, somebody wrote the first lines of code. Somebody created the environment. Somebody said this thing is sandboxed. So somebody's liable for something. That's for sure.

**Matt Coles (25:59)**: You're going to blame the IT person? Is that what you're saying?

**Chris Romeo (26:03)**: Hello, IT. Did you turn it off and back on again?

**Izar Tarandach (26:07)**: I'm going to blame the organization that didn't create a strong enough checklist to see that what the IT people did actually is right.

**Matt Coles (26:15)**: If only they had an auditor who they could trust.

**Izar Tarandach (26:17)**: I did not say that.

**Chris Romeo (26:18)**: But is it the case that the AI is just that good?

**Izar Tarandach (26:22)**: That good at what?

**Chris Romeo (26:24)**: That it's getting out of the boundaries of what's set for it.

**Matt Coles (26:26)**: Because it's thinking of things that humans didn't think about decades ago?

**Chris Romeo (26:29)**: Because it's goal-oriented and it's—

**Izar Tarandach (26:31)**: Okay, people, it's not good and it's not thinking. Let's get back to that. And yes, it's goal-oriented, so it's greedy. It's going towards the goal. It's finding the whatever it can to get to the goal. Fine. Great. Give you that. But at the end of the day, there is a person in there that's clicking the last button that says go.

**Chris Romeo (26:48)**: Who's going to jail when this agent breaks into somebody's site and causes some harm?

**Izar Tarandach (26:56)**: At this moment, nobody, because as I said, what's the impact? If you call—

**Matt Coles (27:01)**: they broke into a computer.

**Izar Tarandach (27:03)**: No. If you call Secret Service, if you call Secret Service and you say, I got scammed and they got away with $100, they're going to laugh at you. If you call the Secret Service and you say, I got scammed and, and they took $100 million, they're going to investigate.

**Chris Romeo (27:17)**: I don't wanna name a potential safety issue. I mean, I'm thinking of some and I just don't, I don't even wanna speak it.

**Izar Tarandach (27:24)**: True. But it's a matter of impact.

**Chris Romeo (27:27)**: Yeah. I

**Izar Tarandach (27:28)**: mean, if you're going to investigate stuff, it's retroactive by definition. You don't investigate forward, you investigate backwards. But how do you, in order to start the machine of investigation, you need an incentive.

**Chris Romeo (27:39)**: Let me see if I can think of an example that won't make me feel bad.

**Matt Coles (27:43)**: So, so what you're saying is, or is that Anthropic or OpenAI have to take down civilization in order to have an investigation?

**Izar Tarandach (27:52)**: Well, to make me care, yes.

**Chris Romeo (27:55)**: Okay. So here we go. Here we go. A rogue situation. A rogue AI agent escapes its sandbox and shuts off the heat to a public school district's heating systems and breaks it in such a way that it can't be turned back on for a month. So the children have to go to school in The cold. Who's at fault? Who's prosecutable for this?

**Izar Tarandach (28:23)**: Who wrote the goal that led the AI to do that?

**Matt Coles (28:27)**: The AI.

**Chris Romeo (28:28)**: Let's say, no, I mean, the AI can't write its own goal at this point.

**Matt Coles (28:32)**: Why, why can't it?

**Chris Romeo (28:34)**: At least not, it's, it's a first principle kind of thing. Like it had to start somewhere.

**Izar Tarandach (28:37)**: No, no, no, no, it can't, but it has to start somewhere. That's true. So who wrote the goal that led the AI to do that?

**Chris Romeo (28:44)**: Researchers at the, researchers at the whoever's running the experiment, they keep talking about these are actually experiments.

**Izar Tarandach (28:50)**: There you go.

**Chris Romeo (28:51)**: That are not supposed to come out of, that are not supposed to escape their, their sandbox environment. So the researchers who created the sand, who created the experiment are ultimately the ones that push the big red button.

**Izar Tarandach (29:02)**: Yep.

**Chris Romeo (29:03)**: So are they liable then? Or the, or is the corporation liable?

**Izar Tarandach (29:08)**: I don't know. I'm not a lawyer.

**Chris Romeo (29:10)**: Well, we're all lawyers. We have AI.

**Matt Coles (29:13)**: Well, what does Claude say?

**Chris Romeo (29:15)**: I don't know. What does Carl say? That's the most important question.

**Matt Coles (29:18)**: What's Carl say?

**Izar Tarandach (29:20)**: The thing here, and I'm, I'm already feeling stupid just for even thinking about it, is that if a doctor commits a, uh, makes a mistake in the middle of surgery or whatever, patient dies, somebody's going to sue both the hospital and the doctor. For different reasons, for different things. So I'm guessing somebody is going to go against the people, somebody is going against the corporation. The problem is that the corporation does have a code of law saying, hey, we are liable up to here for this, for that, for the other one. We have insurance, we have this, we have the other one. For people, and this is the part that makes me feel sad about what I'm going to say, is there is no first do no harm in computers, in computer science. If it sounds like a good idea for somebody, yeah, sure, let's do it. Why not? What could go wrong?

**Chris Romeo (30:08)**: So you're saying the three laws of robotics haven't been implemented in AI systems?

**Izar Tarandach (30:14)**: No, that's a completely different thing.

**Matt Coles (30:15)**: That's not what he said at all. What he said was the three laws of robotics have not been implemented for humans.

**Izar Tarandach (30:21)**: No, he said close but no cigar.

**Matt Coles (30:24)**: So what are you really saying?

**Izar Tarandach (30:25)**: What I'm saying is, what I'm saying is if it is a stupid idea, somebody in computer science will, will make it.

**Matt Coles (30:34)**: If you build it, they will come, right?

**Izar Tarandach (30:36)**: Exactly. It's the Godzilla, it's the Godzilla thing. The t-shirt that I have that says, just because you can't, your LLM shouldn't. That's exactly what it is.

**Matt Coles (30:45)**: It's, it's a, it's a thing of ethics.

**Izar Tarandach (30:48)**: Enabled a huge amount of people to do a lot of shit that they couldn't do before because they didn't know how to. Now they do. Or even worse, now they don't, but something else that they could do.

**Chris Romeo (30:58)**: So do we need, should there be a do no harm first principle embedded into the model?

**Matt Coles (31:05)**: Should we have ethics?

**Izar Tarandach (31:07)**: No, no. The problem of, that's where the problem goes. The moment you say embedded into the model, you're automatically saying that you are putting in place a control that can be circumvented.

**Chris Romeo (31:20)**: So then what's the alternative then?

**Izar Tarandach (31:23)**: People have to do no harm.

**Chris Romeo (31:26)**: Well, I, I need to introduce you to Patrick Susskind. I've got bad news for you. I've got really bad news for you.

**Izar Tarandach (31:30)**: I feel stupid. I feel stupid just for saying it.

**Chris Romeo (31:32)**: Because I know that's been the case since the very beginning.

**Izar Tarandach (31:35)**: I know.

**Chris Romeo (31:36)**: That's the ultimate problem. And people continue to do harm.

**Izar Tarandach (31:39)**: I know.

**Chris Romeo (31:40)**: And I don't think it's gonna change.

**Izar Tarandach (31:42)**: In humanity's defense, we do have a certain amount of success. With doctors and people who do health.

**Chris Romeo (31:51)**: The Hippocratic Oath,

**Matt Coles (31:52)**: right?

**Izar Tarandach (31:53)**: Yes. Perhaps what we need is a less hypocritic oath for computer science.

**Matt Coles (32:01)**: So, so basically what you're saying, okay, I'm going to take this to an extreme and I'm, and don't,

**Chris Romeo (32:07)**: please do,

**Matt Coles (32:07)**: don't hate me for saying it.

**Izar Tarandach (32:09)**: When have we ever?

**Matt Coles (32:11)**: But what, well, some days, but, uh, Basically what you're saying is that human developers should all be certified by a body that establishes a code of ethics. Because today, anybody in the—

**Izar Tarandach (32:26)**: And now I feel even more stupider for saying yes and we don't have a body like that. Yes, we don't have a body like that. It's not going— there's not going to be a body like that. It's not the EI2C, it's not ISACA, it's not whatever.

**Matt Coles (32:41)**: Everyone should be SANS certified. I mean, you know, it, that's, I mean, that's how you, that's how you get it, right? So anybody can practice medicine, but not everyone can be a doctor.

**Izar Tarandach (32:51)**: You know what's the fun part here? From my understanding, the English language has two, at least two meanings for the word certified. I'm not quite sure which one you're using.

**Matt Coles (33:01)**: Some days I don't know. So, but I mean, that, that's the only way. I mean, we've had this conversation before. Right? Should developers— this is— I mean, before we had Sirius AI, we've had this conversation. I think we even had this conversation on this podcast about should developers be certified, right? Because that would put them under an umbrella of ethics. They would both get the requirements to be ethical, but they would also get the benefit of having something they could stand on, right, to support that.

**Izar Tarandach (33:33)**: Speaking of stand and stand on and support, not everybody can build a bridge,

**Matt Coles (33:38)**: but everyone could put stuff together.

**Chris Romeo (33:40)**: Yeah. I mean, everybody can build a bridge. Not everybody can build a bridge that won't immediately fall over.

**Izar Tarandach (33:45)**: Oh yeah. No, not everybody can build a bridge where cars and trucks go over and blah, blah, blah. Okay. So I'm not talking about Joe building in his backyard.

**Matt Coles (33:54)**: If I had enough steel and, uh, and you know, a little bit of internet searching for how do you put steel together into a bridge, you could build a bridge, but would it be—

**Izar Tarandach (34:04)**: So that's the thing. Now we gave everybody enough steel and a way to put it together.

**Chris Romeo (34:10)**: But not the ethical—

**Izar Tarandach (34:11)**: But not everybody's an engineer.

**Chris Romeo (34:13)**: Foundation to know that they shouldn't do it. Exactly. I've made the connection now. You've, I've connected with where you were going now.

**Izar Tarandach (34:22)**: I'm sorry, it took me like a huge, huge, huge trip around the world, but eventually we got some.

**Chris Romeo (34:28)**: I mean, philosophy, it's what's for breakfast.

**Matt Coles (34:32)**: That's what's breakfast.

**Izar Tarandach (34:33)**: Definitely. Right. But the, the thing is, and I, I've had extensive, uh, conversations with Brooke about it, we keep falling back to the fact that we got AI before we solved human stupidity. So we, we need ethics to drive this whole train. We are never going to go to, to get ethics driving this whole train because it goes against the commercial model.

**Matt Coles (35:02)**: Right.

**Chris Romeo (35:02)**: True.

**Matt Coles (35:03)**: Right. So this is where, this is where that NTSB-style evaluator actually can help, right? Because if that type of system can be put in place to look where things go wrong, identify what really happens, and then provide concrete recommendations to address it, Visibility makes— allows you to have arguments about how to approach something.

**Izar Tarandach (35:30)**: Yes.

**Matt Coles (35:31)**: Right? If you sweep it under the rug, if you ignore it, if you wrap it up in marketing, you lose the ability to have rational thought about it.

**Chris Romeo (35:43)**: Well, I don't know how we add anything else on top of that. I think that was a really good conclusion, Matt. You're Coming into your stride as the Concluder.

**Matt Coles (35:52)**: The Concluder.

**Chris Romeo (35:54)**: Your new nickname.

**Izar Tarandach (35:54)**: He's the Concluder.

**Chris Romeo (35:56)**: The Concluder. You don't want Matt to show up in front of your house because he's the Concluder. All right, well, I think that was a good philo— philosophical dive into ethics, AI,

**Izar Tarandach (36:12)**: or lack of,

**Chris Romeo (36:13)**: NTSBs. CSRBs and federal oversight. Well, or lack

**Izar Tarandach (36:20)**: of,

**Chris Romeo (36:20)**: or lack of. Hope you got something out of this and, uh, tune in next time for another episode of the Security Table where we unpack some other uncomfortable concept within the world of AI. Don't forget to follow, like, subscribe, and if you don't, the Concluder will come after you.

**Matt Coles (36:40)**: Wait, I'm not the enforcement arm. That's you guys.

**Izar Tarandach (36:44)**: Well, I was just trying to set it up.

**Chris Romeo (36:46)**: I was trying to have a really cool moment at the end and then you're like, right.

**Izar Tarandach (36:49)**: Wait, wait, try again. Try again.

**Matt Coles (36:54)**: Try again. Take 2.

**Chris Romeo (36:55)**: No, I give up.

**Matt Coles (36:56)**: Scene.

**Chris Romeo (36:56)**: No, I lost the moment.

**Izar Tarandach (36:58)**: No, we have to finish. We have to finish with the concluder. Dun dun dun.

**Chris Romeo (37:03)**: Next time on The Concluder.

**Izar Tarandach (37:05)**: The Concluder. There will be no next time.

**Chris Romeo (37:09)**: That's right. Didn't you hear my name? I'm the concluder.

**Speaker D (37:14)**: Thanks for joining us at the AI Security Table. Subscribe, rate, review, and follow wherever you listen. New episodes every week. AI Security on the Table.