An AI Agent Escapes and Attacks. Who Goes to Jail?

0:09Chris Romeo Welcome to the Security Table, where Isar was smacking himself on top of the head to start the episode.

0:16Izar Tarandach I will not taunt you a second time.

0:18Chris Romeo Which normally he would do that a little later in response to something that Matt or I said, because he couldn't just, couldn't take it. Anymore.

0:27Izar Tarandach I'm just warming up the engine.

0:29Chris Romeo Yeah, but he's gotta get, he's gotta get ready to go. I mean, winter is here. Izar's dressed in his fleece.

0:39Izar Tarandach Wait, you never seen on UFC before they, they go into the fight that they go, same thing.

0:44Chris Romeo That's what you, oh, okay. Oh, you're

0:46Izar Tarandach same thing.

0:47Chris Romeo You're getting yourself mentally prepared to take a punch.

0:50Izar Tarandach Yep.

0:50Chris Romeo Okay. Got it. This is a rough— the security table is a rough place, so it's not for the faint of heart.

0:58Matt Coles You know, it's interesting if we look at our clothing styles. Chris in blue, me in black, Izar in black and blue, just like his face can't be.

1:07Chris Romeo Wow, some bullying. We don't allow bullying here, Matt, on the security table.

1:12Izar Tarandach Okay.

1:13Matt Coles Oh, I wasn't— I, I didn't say by, by us.

1:16Chris Romeo It sounded like bullying to me. That's all I That's all I could say. All right, let me set this up because we have kind of a premise and then we have 3 different perspectives to, I was going to say attack, but well, discuss, attack, bullying, bully, whatever. Kind of our normal approach to the world. Let me set this up. So we got these things called AI agents. People have heard of these.

1:43Izar Tarandach What?

1:43Chris Romeo And They're now attacking and building software faster than any human process can keep up.

1:48Matt Coles That's why we love them, right?

1:50Chris Romeo I think I agree with that. It does. They do, do go pretty fast.

1:53Matt Coles They do.

1:54Chris Romeo And so we've got 3 different perspectives on what we should do about this. So we've got the lab's perspective, Frontier Lab, OpenAI. We've got the US government and then the supply chain side of it. And so the question that we're gonna try to wrestle with here are any of these solving the actual problem under the hood, or are they all just talking for the benefit of talking? So let me set up OpenAI's perspective here. This is an article from September 9th. This is OpenAI's pitch on, you know, safety. And I think this was ultimately a response to Dario Amadei's call to slow down the frontier, pause the frontier, which nobody actually did anything with, but It was a nice set of words. Uh, so in this description, OpenAI argues that the policy window's closing. We need mandatory federal rules aimed only at frontier labs. We need clear incident reporting rules, and then we need disclosure when a system gets around its security controls. And he also wants company-led incident reporting to sit alongside federal oversight. So the question for us to wrestle with is, first of all, is this accountability? Is this real accountability or is this just one of the frontier labs defining the constraints that it believes it should be kept under? Which you've got to imagine there's some self-servantness. When I, when I write the rules for something, I almost always want myself to win as a result of it. So what do you guys think?

3:34Matt Coles Well, I mean, on the one hand, I guess it could be a realization that, you know, things are moving so rapidly. I guess the self-serving angle is maybe it's been moving for other people and they want to bring everyone, you know, into the funnel, right? If you make the, if you write a set of standards, if you write a set of standards and you were not already the winner, then you would be restraining everyone else. Right? That's a really, that's a really extreme, you know, viewpoint though. I don't think that's what OpenAI was looking to do, but, but that may ha— it may have that effect. If OpenAI was the leader, they would be bringing everyone else into a, into a funnel basically in doing so by setting regulation.

4:17Chris Romeo And we're only talking about OpenAI and Anthropic. Well, maybe SpaceX AI.

4:25Matt Coles I mean, maybe Google as well. Uh, Gem— with Gemini.

4:28Chris Romeo I always forget about Google. I forget about—

4:30Matt Coles I know that's where the problem was as a, That's probably the problem. They're there, but I've never used them.

4:36Chris Romeo They're just not, it's just not as cool at a cocktail party to be like, hey, I'm using Google Gemini.

4:42Izar Tarandach So yeah.

4:45Matt Coles Well, then you have a whole host of open source and no wait models and things like that, right?

4:50Izar Tarandach The thing is, it is cool on a cocktail party because that's when you look at like a side people and say, what, you're using only one model? And you say, I use one for one thing and one for another. That's the cool kid. Approach.

5:03Matt Coles Tokenizing or—

5:05Izar Tarandach but on the self-serving angle, I assume it's just like when you go and buy a car, right? You're going to get screwed. The question is by how much.

5:13Matt Coles To all of our car dealers out there. Yeah.

5:16Izar Tarandach I mean, they are self-serving. Of course they are self-serving.

5:19Chris Romeo You gotta get a better negotiator, man. You're supposed to win that deal.

5:23Izar Tarandach Yeah.

5:24Matt Coles You should use a Frontier model.

5:26Izar Tarandach Suppose is the law.

5:27Chris Romeo I did. I used a model last time I was helping my daughter buy a car and I had Claude as my co—, negotiator.

5:35Matt Coles Don't say copilot.

5:36Chris Romeo I almost said copilot, patent pending. But yeah, I mean, it was, it was actually—

5:40Matt Coles But that's actually taken by Microsoft, which you also didn't mention as a—

5:43Chris Romeo Microsoft's in AI too.

5:46Izar Tarandach It's open, no, it's open AI.

5:47Chris Romeo No, but the, the sales guy, just to close the loop, the, the car sales guy, he's like, he's like, everybody's doing this to us now.

5:55Izar Tarandach Yeah, that's because he had open AI open.

5:57Chris Romeo So he said, 'cause I'm like, oh yeah, I'm using Claude behind the scenes here. And he's like, Oh, you're not the only one. Everybody is. He's like, I get text messages and I have to negotiate, which I, with what I know is an AI on the other side.

6:10Izar Tarandach So see,

6:10Matt Coles that's the story we should be talking about right there. The practical use of AI in the field.

6:15Izar Tarandach But, uh, Chris, can, can you real quick go, go again over the list of things that they're asking for in the policy?

6:21Chris Romeo Yeah. So they're looking for mandatory rules aimed only at Frontier Labs, clear incident reporting. And then disclosures when a system gets around its security controls.

6:35Izar Tarandach Yeah. So one of the things that I've been enjoying a lot when doing threat modeling of AI stuff is at some point imagining the AI part out of it and just putting in something scripted, deterministic, nicely done with ifs and thens and elses and seeing if there is any change to the whole picture, right?

6:53Chris Romeo Okay.

6:54Izar Tarandach And if there isn't, then the AI is not your problem. But now look at those things that they are asking. And join me in our time machine in the past, what, 30 years, 20 years, something like that. Haven't we been asking for the same thing for normal incidents? Like, are we really asking for anything new here? Or I feel like we got there.

7:19Chris Romeo Don't we have incident reporting pretty much for non-AI-related things, maybe that affect national

7:26Matt Coles security and whatnot, but.

7:28Izar Tarandach Yeah, but you know, if he's asking for an NTSB here of AI, then the question becomes only AI? Why not the things that AI is making happen now that weren't happening before and stuff like that? But again, it goes back to, you know, more of the same, more of the same, more of the same. Fine. Yeah.

7:47Chris Romeo But it's an innovation—

7:48Izar Tarandach it just has AI in it now.

7:50Chris Romeo It's an innovation killer. What is an innovation killer? Anytime you add things that slow stuff down, the ultimate result is killing innovation. And that's why I don't get this. I don't get why OpenAI and Anthropic are coming out in support of this.

8:11Izar Tarandach When— so let, let, let me play the devil's advocate here, please. Why would this slow down something? It's just reporting, isn't it?

8:18Matt Coles There's more.

8:18Chris Romeo Well, the whole, the whole, I'm, I'm kind of including, I guess I'm, I'm stepping further beyond what this particular article said to Dario's whole, we need to slow the pace.

8:30Izar Tarandach Uh-huh.

8:31Chris Romeo Of this. And, and then Sam Altman came out and agreed with him, I think on X, agreed with the, the general statement that yes, we need to slow down the pace. Now neither one of them, like I said before, neither one of them did anything about it. They just like, oh yeah, and by the way, tomorrow new models coming out that are cranking this thing up to 11.

8:50Izar Tarandach Yeah, it's, it's, it's like, it's like a teenage, uh, relationships, right? When, when one of the parties looks at the other and says, oh, we have to slow things down. Next thing you see is a shotgun and a wedding dress. So

9:02Matt Coles now interestingly, the, the, the OpenAI, uh, policy view here does highlight 4 California state laws. Or state, I'm sorry, bills. They're not laws, I don't think yet, that cover a range of topics that OpenAI supports. And I, I'm, the way they described it is until Congress does something, until the US Congress does something, they will continue to support state legislation. And they highlighted 4 California bills in that, that they think cover aspects of AI use or AI development that I guess embody their policy views, right? So standards on auditors for AI systems, which is kind of interesting 'cause I don't think we have standards. I don't, I'm not aware of us having standards for other auditors.

9:53Chris Romeo Depends on what you define as auditor. Certainly a financial auditor, it's a very well-defined role. From a regulatory perspective, whether that's Sarbanes-Oxley or whatever causes you to have to not cheat on your financials as a public company, SOX or whatever.

10:18Matt Coles Right. But PCI auditors are not like government regulated.

10:22Chris Romeo True.

10:23Izar Tarandach No.

10:23Chris Romeo Good point.

10:24Izar Tarandach The only way I'm trusting any auditor here is if one company pays the auditor of the other one. So if OpenAI pays—

10:31Chris Romeo So one of the labs pays the other lab to—

10:34Izar Tarandach If one of the labs pays for the audit of the other lab, then I'm trusting shit. Otherwise, no.

10:41Chris Romeo So you don't trust— let's unpack that a little bit.

10:44Matt Coles Oh my gosh.

10:45Chris Romeo So you don't— you're in full-on security contrarian mode here going like, you don't believe, you don't trust anybody, it sounds like.

10:55Izar Tarandach No, wait, Matt had a thing going. Am I familiar with that?

10:58Matt Coles I was about to say, do you understand the concept of collusion?

11:01Izar Tarandach Yeah.

11:02Matt Coles That's what I'm saying. In the public perception, OpenAI pays for an audit of Anthropic, Anthropic pays for an audit of OpenAI, and nothing comes of it.

11:12Izar Tarandach Yeah, but now I have the conflicting interests of each one of the payers that they want the other one to look bad. So they want that auditor to actually find shit.

11:23Matt Coles You're assuming that the auditor is then themselves part of the collusion. I'm an optimist. And are not acting in good faith.

11:32Izar Tarandach I'm an optimist. Auditors always acting in good faith.

11:36Matt Coles Auditors always act in good faith?

11:37Chris Romeo You're an optimist.

11:38Izar Tarandach I'm an optimist. That's what I'm saying.

11:40Chris Romeo That's the funniest part of that. You're an optimist.

11:43Matt Coles Yeah. Well, who, where did you, where did Izar go and who are you?

11:48Chris Romeo Carl. Carl. Shiny happy guy over here.

11:51Matt Coles So that was one. The other, another one is infrastructure for independent safety assessments. So again, an audit function for somebody other than the Frontier Lab itself to police it, to basically do security assessments or safety assessments. Sorry. Some bill for protecting young people. I haven't looked into that one yet. And safeguards against biological threats, enable AI-enabled biological threats. Why just biological and not nuclear threats?

12:21Izar Tarandach Yeah, that, that one always gets me. Why biological? Like what, what's the, is it, is it a remnant of a COVID or something? Like why always the stress on biological?

12:31Matt Coles Maybe that's easy, relatively easy.

12:35Izar Tarandach If it's nuclear, the AI is going to destroy itself.

12:38Matt Coles Well, nuclear, you need to like build equipment and buy material, you know, get, get your hands on material. Whereas biological, you could probably do in your kitchen.

12:45Izar Tarandach Mm-hmm.

12:47Matt Coles Right. The difference between building a, a big bomb and a pipe bomb is, you know,

12:52Izar Tarandach yeah, because AI attacking infrastructure is a scenario that Hollywood never imagined.

12:57Matt Coles Gee, how many, how many AI-enabled, yeah.

13:00Izar Tarandach If they, if Hollywood can't imagine it, there, there's no bills covering it.

13:06Matt Coles Well, it is California, you know.

13:07Izar Tarandach Oh yeah. It's deep.

13:08Matt Coles So anyway, where, where were we? Cuz, uh, I think we just hit a wall.

13:12Chris Romeo Oh, we can, we can transition into, is the federal government the answer on this issue?

13:19Matt Coles Not the full answer. A, it's only one country.

13:23Chris Romeo You didn't even hear the setup before you squashed it. That's just like,

13:28Izar Tarandach no, it's just that when you say it's the federal government, the, the answer, I'm, I'm still thinking what's the, the question?

13:36Chris Romeo Yeah. What is that? That is often what you should be saying is, whoa, whoa, whoa, whoa, whoa.

13:40Matt Coles What was the question?

13:41Chris Romeo So,

13:41Izar Tarandach and then you say no.

13:43Chris Romeo Somebody from a tiny little state called Massachusetts. I'm just saying that 'cause that's where Matt's from. Um,

13:51Izar Tarandach somebody wants,

13:53Chris Romeo wants to create the Cybersecurity and AI Board of Investigations Act. That sounds really official.

14:00Izar Tarandach Like that's the NTSB. Okay.

14:02Chris Romeo Yeah. They wanna create an NTSB national, what's that? National Transportation Safety Board style. Set up that would investigate AI agent breaches of federal systems and critical infrastructure, look at near misses, and they would have, most importantly, subpoena power. So they could subpoena the Security Table.

14:21Matt Coles So that is literally the NTSB for AI.

14:24Chris Romeo Wouldn't that be cool if they subpoenaed us? Wouldn't that be hilarious? No, it would be very bad. They're like, we've listened to some of your videos, and you know what,

14:34Izar Tarandach I'll do that. I'll do that.

14:36Matt Coles I plead the fifth.

14:37Chris Romeo They're like, we think we're going to throw you in prison. But we thought we'd give you 30 minutes to see if you deserve it.

14:43Izar Tarandach I would take one for humanity.

14:45Chris Romeo You would? I would. Good. Okay. Matt and I are taking, pleading the Fifth and you're gonna be talking.

14:50Matt Coles We like, I'll refer all, all, I'll refer all questions to my attorney.

14:54Izar Tarandach I just wonder who, who would play me in the movie?

14:57Chris Romeo Who would play you in the movie?

14:58Izar Tarandach Who would play me in the movie?

15:00Chris Romeo That's it.

15:00Matt Coles You would play you in the movie? You do your own stunts?

15:03Izar Tarandach No.

15:03Chris Romeo Trying to think who plays you. I'm a bad loser.

15:06Izar Tarandach I'm not that good.

15:07Chris Romeo Let's see. I'm really stumped here.

15:09Matt Coles You need a, you need an actor who can play carpal tunnel and

15:12Chris Romeo that takes a lot of range from an actor.

15:14Izar Tarandach The claw.

15:16Matt Coles I have to get into the moment.

15:18Chris Romeo Oh, what is, uh, the guy who played George from Seinfeld? Is he too old to play as our DAEI?

15:25Izar Tarandach Was, was, uh, what was the, the line? The sea was— DAEI was angry, my friends.

15:32Chris Romeo Yeah. That would be, that would be a really funny reboot. If they were able to do Seinfeld, like a 10-episode arc of Seinfeld, bring Seinfeld back in the AI era. Just the age. Oh, that would be hilarious.

15:44Matt Coles I'm sure somebody would do it. How many tokens would that take? I mean, come on.

15:48Chris Romeo Mm-hmm. It's true. You could probably just have AI build it. Okay.

15:51Izar Tarandach Just imagine the agent escaped. The agent escaped.

15:54Chris Romeo Yeah.

15:55Matt Coles It's for you.

15:56Izar Tarandach Yeah. What do you— slow down.

15:59Chris Romeo All right, so, all right, where'd we

16:00Matt Coles go?

16:01Chris Romeo We're off on a bit of a tangent there, but no, I mean, is this

16:05Izar Tarandach What?

16:06Chris Romeo I mean, are we ever in favor of NTSB-style anything?

16:11Matt Coles Yes, we're in favor of the NTSB.

16:13Izar Tarandach Yes, the NTSB, very much so.

16:16Matt Coles Right. But in regards, when planes, when planes crash, we wanna know why.

16:20Chris Romeo True.

16:21Matt Coles So if AI crashes, but if something

16:22Chris Romeo crashes,

16:23Matt Coles we kind of need to know.

16:24Chris Romeo The AI crashing world.

16:26Matt Coles Well, so interestingly, the difference between, and I, I actually don't know how the NTSB operates in this case. If Boeing was testing a new plane and it crashed in testing, would the NTSB get involved?

16:43Izar Tarandach Yes,

16:43Chris Romeo I think so. Yeah.

16:45Izar Tarandach Yes.

16:45Chris Romeo And so I think that happened.

16:46Izar Tarandach No, no, no, no, wait. The, the plane is not in production yet.

16:49Matt Coles Hasn't, right. Hasn't gone into production. Right. It may have come out of wind tunnel tests or

16:55Izar Tarandach I think it's the FAA.

16:57Chris Romeo Some, it happened recently in the last year. There was a test plane crash.

17:04Izar Tarandach Well, that, that's every day that ends with a Y, but, uh, I think that it's the F FAA that, wait, let's ask Claude.

17:12Chris Romeo Well, if we had Carl here,

17:14Izar Tarandach Claude, Claude is in, in, Carl is, uh, tuning up.

17:18Chris Romeo Okay. Well, just listeners, we're gonna be the first podcast to have all the hosts replaced by AI.

17:28Matt Coles The first? Hardly.

17:32Chris Romeo Have you heard, have you guys heard these, these, I've been, I've been sourcing podcasts in the AI development and security world and there is this genre now that has appeared where two appearing, two hosts appear to be human, but they're just too perfect. They don't say a single like, or, uh, unless it's generated.

17:58Matt Coles It's the wrong prompt then. They should be, talk like a California, you know, school kid or something.

18:03Chris Romeo But they, they, it's, it's interesting content because they're analyzing, uh, academic papers that are recent academic papers in the AI security space.

18:16Matt Coles So somebody took NotebookLM from Google, put a bunch of papers in it, and then had 'em generate a podcast. Which it does.

18:23Chris Romeo 2 or 3, 2 or 3 come out a week.

18:26Matt Coles Yeah, why not?

18:27Chris Romeo So my point is we might not even have to be here next week.

18:31Matt Coles Wait, we like being here.

18:33Izar Tarandach Yeah, that's the thing.

18:35Chris Romeo Oh yeah.

18:37Izar Tarandach Until there is a, there's legislation against it, we'll be here. US experimental crashes, if it's a small thing like, uh, somebody building it on the back of the, the farm, then it's the FAA. And the FAA reports back to the NTSB. If it's a big thing, then it's the NTSB.

18:53Matt Coles Okay. So, so that, and that's an, that's, it's important. I wanted to bring that up because if this creates something like the NTSB for AI, then what this would mean is during training, if there's any training incidents, like, I don't know, an AI escaping its sandbox and going off and attacking other companies.

19:11Chris Romeo Would that really ever happen in the real world though? No, never. It's already happened like twice this week. Okay.

19:18Matt Coles Uh, then an agency would get involved to figure out what's going on. So you'd have an independent analysis. The one nice thing about the NTSB and its function is they're very thorough. They're also—

19:28Chris Romeo are they fast though?

19:30Matt Coles No, of course not. But, but you do eventually know what really happened.

19:35Chris Romeo And so that's what I feel like we don't have now. Like we really don't know. What happened with even Hugging Face, even the Hugging Face one, the one that's gotten the most press, the most people trying to write up what their insider scoop is when they're not even an insider.

19:52Matt Coles Yeah. And well, I mean, any podcast host can opine about what happened with the Hugging Face incident.

19:58Chris Romeo Oh, I think we did that, didn't we?

20:00Matt Coles We did that last week. Yes.

20:02Chris Romeo Dang it. We fell into the trap.

20:07Matt Coles So, Yeah. So, so anybody can opine about what happens and guess about what happened, but an agency-like thing that does investigations that has the ability to come in and subpoena people or look at logs and be independent reviewers.

20:23Chris Romeo Yeah. But are they gonna post what they come up with?

20:26Matt Coles I don't know. The NTSB does.

20:28Chris Romeo Do they? Is it in full? There's no internal version and an external version.

20:32Matt Coles If there is, I mean, the, the The, whatever they publish, publicize, whatever they publish is very thorough. I

20:39Chris Romeo mean, that would be my fear here is that one, it takes forever to get anything actionable. And two, they don't tell you the whole story, which is the whole point of creating something like this is because we want to get the whole story.

20:54Matt Coles Well, so something else that the NTSB does, and I, I wonder if this bill would support something like it, is the NTSB is is in— is trusted and independent, but it's also not regulatory. So the NTSB doesn't write law. The NTSB investigates accidents. They're there for safety. And so if this bill introduced an age, an, you know, a scheme for having an independent analysis of what went wrong, they can give recommendations about what they're gonna do about it, but they can't actually enact any of those things, right? So they would have to go to to, I don't know, some AI regulation framework, or they would have to go to the FBI or whoever's involved with cyber, with AI-based cybercrime. Or, you know, because that's ultimately a hacking of another company is in theory a cybercrime in most—

21:44Chris Romeo is it really anymore?

21:45Matt Coles Well, okay. But it's a stretch.

21:47Chris Romeo It doesn't feel like it is anymore.

21:49Matt Coles It doesn't feel, it doesn't feel like it is.

21:50Chris Romeo It should. And I'm, I'm not, I'm not trying to be funny. Like it should,

21:54Matt Coles it should be like, it should be like a thing, right?

21:57Chris Romeo But like AIs are escaping and apparently just attacking things and like nobody's going to jail like they used to in the good old days, like Kevin Mitnick when he got arrested.

22:05Matt Coles If humans did this, they would be, the FBI would've showed up at their door and I mean, we've seen hackers, right?

22:11Izar Tarandach Excuse me. Excuse me.

22:13Chris Romeo I thought it was real.

22:14Izar Tarandach Excuse me. Big difference. The people who went to, uh, to jail is because there was an impact to the attack. Have we heard of any impact from the AI stuff, Franklin, so far? Uh, listen to me. We're focusing much more on the fact that they are attacking.

22:28Matt Coles Malicious modification of packages.

22:31Chris Romeo Yeah, the malware, it was malware that got distributed to commercial.

22:35Izar Tarandach Was there impact?

22:37Matt Coles There was malware created and distributed through official channels.

22:42Izar Tarandach Did that lead to impact? We have a number of millions of phones.

22:45Chris Romeo So now we're measuring in impact?

22:48Matt Coles Okay.

22:49Izar Tarandach No, what I'm pointing out—

22:49Matt Coles Hold on, hold on.

22:50Izar Tarandach No, no, no, no, no. What I'm pointing out is that when it was, when it is people, What you hear about is some dude went to prison because of $5 million in whatever. But here we are, we are hearing about something broke out of somewhere and touched something else.

23:08Matt Coles Okay.

23:10Izar Tarandach We are more worried about the touching than whatever they got away with.

23:15Matt Coles I, I have, I have a line for you and I now, now I, now I see the connection, uh, from, from Chris's earlier, from the beginning of this of this episode. If malware is introduced into your supply chain, but it doesn't make an impact, does it matter?

23:34Izar Tarandach Clue. You know what clue is?

23:37Chris Romeo No.

23:38Izar Tarandach It's the sound of hand clapping with one hand. So if we're going to go zen here, again, my point is there must be a very heavy component of marketing here, as we all already know. But we haven't yet put the finger on because really all the hoopla that we are getting is on the beginning of the event and not at the end of the event. People are mesmerized by the fact that AI is going rogue and it's going out and it's attacking things. But what has the attack done already? If it's happening thousands and thousands and thousands of times a day, as it's said, what is being impacted?

24:17Matt Coles How much of an impact do you actually need to be violating a law?

24:22Izar Tarandach Oh, so you're saying that intent is already— yeah, no, I agree with that completely.

24:25Matt Coles Right. I mean, what the DMCA, you know, with, if you just unauthorized access to a computer, you've broken a law.

24:33Chris Romeo Agreed.

24:33Matt Coles So all of these have had a legal impact of some type.

24:36Chris Romeo Yeah. But who's guilty?

24:38Izar Tarandach Well, that's an interesting question. Who's guilty? Who's guilty is clear.

24:42Chris Romeo Who, who, how's it clear?

24:43Matt Coles Who, yeah, who is actually at fault?

24:46Izar Tarandach It's not so clear.

24:48Chris Romeo No, no,

24:48Izar Tarandach no, no, no, no, no, no. Wait a minute.

24:50Matt Coles And I, I, there's another, there's another take on this because are corporations people?

24:55Izar Tarandach No, but

24:56Matt Coles yeah,

24:57Izar Tarandach they are. Are they liable?

24:58Matt Coles They are for political purposes, but are they people for legal purposes?

25:02Izar Tarandach Okay. Now, now you're taking it all the way to a very, very dark place, but somebody's liable.

25:08Matt Coles Wait, I'm all in black. It's a dark place, right?

25:10Chris Romeo So somebody's liable.

25:12Matt Coles Who? Tell me, tell me.

25:14Izar Tarandach Hey, so when, well, I really don't want to go there.

25:20Chris Romeo No. Well, you've already opened the door.

25:22Matt Coles You have opened the pit,

25:24Chris Romeo the pit of despair.

25:25Matt Coles Pit of despair.

25:26Izar Tarandach Is this sort of what I'm looking at right now? Yeah. Uh, no, I'm not going to use that.

25:32Matt Coles What is your name? What is your quest? What was the impact you caused against any, like, blue act?

25:40Chris Romeo Absolutely.

25:42Izar Tarandach Okay. So liability. Look, at the end of the day, somebody wrote the first lines of code. Somebody created the environment. Somebody said this thing is sandboxed. So somebody's liable for something. That's for sure.

25:59Matt Coles You're going to blame the IT person? Is that what you're saying?

26:03Chris Romeo Hello, IT. Did you turn it off and back on again?

26:07Izar Tarandach I'm going to blame the organization that didn't create a strong enough checklist to see that what the IT people did actually is right.

26:15Matt Coles If only they had an auditor who they could trust.

26:17Izar Tarandach I did not say that.

26:18Chris Romeo But is it the case that the AI is just that good?

26:22Izar Tarandach That good at what?

26:24Chris Romeo That it's getting out of the boundaries of what's set for it.

26:26Matt Coles Because it's thinking of things that humans didn't think about decades ago?

26:29Chris Romeo Because it's goal-oriented and it's—

26:31Izar Tarandach Okay, people, it's not good and it's not thinking. Let's get back to that. And yes, it's goal-oriented, so it's greedy. It's going towards the goal. It's finding the whatever it can to get to the goal. Fine. Great. Give you that. But at the end of the day, there is a person in there that's clicking the last button that says go.

26:48Chris Romeo Who's going to jail when this agent breaks into somebody's site and causes some harm?

26:56Izar Tarandach At this moment, nobody, because as I said, what's the impact? If you call—

27:01Matt Coles they broke into a computer.

27:03Izar Tarandach No. If you call Secret Service, if you call Secret Service and you say, I got scammed and they got away with $100, they're going to laugh at you. If you call the Secret Service and you say, I got scammed and, and they took $100 million, they're going to investigate.

27:17Chris Romeo I don't wanna name a potential safety issue. I mean, I'm thinking of some and I just don't, I don't even wanna speak it.

27:24Izar Tarandach True. But it's a matter of impact.

27:27Chris Romeo Yeah. I

27:28Izar Tarandach mean, if you're going to investigate stuff, it's retroactive by definition. You don't investigate forward, you investigate backwards. But how do you, in order to start the machine of investigation, you need an incentive.

27:39Chris Romeo Let me see if I can think of an example that won't make me feel bad.

27:43Matt Coles So, so what you're saying is, or is that Anthropic or OpenAI have to take down civilization in order to have an investigation?

27:52Izar Tarandach Well, to make me care, yes.

27:55Chris Romeo Okay. So here we go. Here we go. A rogue situation. A rogue AI agent escapes its sandbox and shuts off the heat to a public school district's heating systems and breaks it in such a way that it can't be turned back on for a month. So the children have to go to school in The cold. Who's at fault? Who's prosecutable for this?

28:23Izar Tarandach Who wrote the goal that led the AI to do that?

28:27Matt Coles The AI.

28:28Chris Romeo Let's say, no, I mean, the AI can't write its own goal at this point.

28:32Matt Coles Why, why can't it?

28:34Chris Romeo At least not, it's, it's a first principle kind of thing. Like it had to start somewhere.

28:37Izar Tarandach No, no, no, no, it can't, but it has to start somewhere. That's true. So who wrote the goal that led the AI to do that?

28:44Chris Romeo Researchers at the, researchers at the whoever's running the experiment, they keep talking about these are actually experiments.

28:50Izar Tarandach There you go.

28:51Chris Romeo That are not supposed to come out of, that are not supposed to escape their, their sandbox environment. So the researchers who created the sand, who created the experiment are ultimately the ones that push the big red button.

29:02Izar Tarandach Yep.

29:03Chris Romeo So are they liable then? Or the, or is the corporation liable?

29:08Izar Tarandach I don't know. I'm not a lawyer.

29:10Chris Romeo Well, we're all lawyers. We have AI.

29:13Matt Coles Well, what does Claude say?

29:15Chris Romeo I don't know. What does Carl say? That's the most important question.

29:18Matt Coles What's Carl say?

29:20Izar Tarandach The thing here, and I'm, I'm already feeling stupid just for even thinking about it, is that if a doctor commits a, uh, makes a mistake in the middle of surgery or whatever, patient dies, somebody's going to sue both the hospital and the doctor. For different reasons, for different things. So I'm guessing somebody is going to go against the people, somebody is going against the corporation. The problem is that the corporation does have a code of law saying, hey, we are liable up to here for this, for that, for the other one. We have insurance, we have this, we have the other one. For people, and this is the part that makes me feel sad about what I'm going to say, is there is no first do no harm in computers, in computer science. If it sounds like a good idea for somebody, yeah, sure, let's do it. Why not? What could go wrong?

30:08Chris Romeo So you're saying the three laws of robotics haven't been implemented in AI systems?

30:14Izar Tarandach No, that's a completely different thing.

30:15Matt Coles That's not what he said at all. What he said was the three laws of robotics have not been implemented for humans.

30:21Izar Tarandach No, he said close but no cigar.

30:24Matt Coles So what are you really saying?

30:25Izar Tarandach What I'm saying is, what I'm saying is if it is a stupid idea, somebody in computer science will, will make it.

30:34Matt Coles If you build it, they will come, right?

30:36Izar Tarandach Exactly. It's the Godzilla, it's the Godzilla thing. The t-shirt that I have that says, just because you can't, your LLM shouldn't. That's exactly what it is.

30:45Matt Coles It's, it's a, it's a thing of ethics.

30:48Izar Tarandach Enabled a huge amount of people to do a lot of shit that they couldn't do before because they didn't know how to. Now they do. Or even worse, now they don't, but something else that they could do.

30:58Chris Romeo So do we need, should there be a do no harm first principle embedded into the model?

31:05Matt Coles Should we have ethics?

31:07Izar Tarandach No, no. The problem of, that's where the problem goes. The moment you say embedded into the model, you're automatically saying that you are putting in place a control that can be circumvented.

31:20Chris Romeo So then what's the alternative then?

31:23Izar Tarandach People have to do no harm.

31:26Chris Romeo Well, I, I need to introduce you to Patrick Susskind. I've got bad news for you. I've got really bad news for you.

31:30Izar Tarandach I feel stupid. I feel stupid just for saying it.

31:32Chris Romeo Because I know that's been the case since the very beginning.

31:35Izar Tarandach I know.

31:36Chris Romeo That's the ultimate problem. And people continue to do harm.

31:39Izar Tarandach I know.

31:40Chris Romeo And I don't think it's gonna change.

31:42Izar Tarandach In humanity's defense, we do have a certain amount of success. With doctors and people who do health.

31:51Chris Romeo The Hippocratic Oath,

31:52Matt Coles right?

31:53Izar Tarandach Yes. Perhaps what we need is a less hypocritic oath for computer science.

32:01Matt Coles So, so basically what you're saying, okay, I'm going to take this to an extreme and I'm, and don't,

32:07Chris Romeo please do,

32:07Matt Coles don't hate me for saying it.

32:09Izar Tarandach When have we ever?

32:11Matt Coles But what, well, some days, but, uh, Basically what you're saying is that human developers should all be certified by a body that establishes a code of ethics. Because today, anybody in the—

32:26Izar Tarandach And now I feel even more stupider for saying yes and we don't have a body like that. Yes, we don't have a body like that. It's not going— there's not going to be a body like that. It's not the EI2C, it's not ISACA, it's not whatever.

32:41Matt Coles Everyone should be SANS certified. I mean, you know, it, that's, I mean, that's how you, that's how you get it, right? So anybody can practice medicine, but not everyone can be a doctor.

32:51Izar Tarandach You know what's the fun part here? From my understanding, the English language has two, at least two meanings for the word certified. I'm not quite sure which one you're using.

33:01Matt Coles Some days I don't know. So, but I mean, that, that's the only way. I mean, we've had this conversation before. Right? Should developers— this is— I mean, before we had Sirius AI, we've had this conversation. I think we even had this conversation on this podcast about should developers be certified, right? Because that would put them under an umbrella of ethics. They would both get the requirements to be ethical, but they would also get the benefit of having something they could stand on, right, to support that.

33:33Izar Tarandach Speaking of stand and stand on and support, not everybody can build a bridge,

33:38Matt Coles but everyone could put stuff together.

33:40Chris Romeo Yeah. I mean, everybody can build a bridge. Not everybody can build a bridge that won't immediately fall over.

33:45Izar Tarandach Oh yeah. No, not everybody can build a bridge where cars and trucks go over and blah, blah, blah. Okay. So I'm not talking about Joe building in his backyard.

33:54Matt Coles If I had enough steel and, uh, and you know, a little bit of internet searching for how do you put steel together into a bridge, you could build a bridge, but would it be—

34:04Izar Tarandach So that's the thing. Now we gave everybody enough steel and a way to put it together.

34:10Chris Romeo But not the ethical—

34:11Izar Tarandach But not everybody's an engineer.

34:13Chris Romeo Foundation to know that they shouldn't do it. Exactly. I've made the connection now. You've, I've connected with where you were going now.

34:22Izar Tarandach I'm sorry, it took me like a huge, huge, huge trip around the world, but eventually we got some.

34:28Chris Romeo I mean, philosophy, it's what's for breakfast.

34:32Matt Coles That's what's breakfast.

34:33Izar Tarandach Definitely. Right. But the, the thing is, and I, I've had extensive, uh, conversations with Brooke about it, we keep falling back to the fact that we got AI before we solved human stupidity. So we, we need ethics to drive this whole train. We are never going to go to, to get ethics driving this whole train because it goes against the commercial model.

35:02Matt Coles Right.

35:02Chris Romeo True.

35:03Matt Coles Right. So this is where, this is where that NTSB-style evaluator actually can help, right? Because if that type of system can be put in place to look where things go wrong, identify what really happens, and then provide concrete recommendations to address it, Visibility makes— allows you to have arguments about how to approach something.

35:30Izar Tarandach Yes.

35:31Matt Coles Right? If you sweep it under the rug, if you ignore it, if you wrap it up in marketing, you lose the ability to have rational thought about it.

35:43Chris Romeo Well, I don't know how we add anything else on top of that. I think that was a really good conclusion, Matt. You're Coming into your stride as the Concluder.

35:52Matt Coles The Concluder.

35:54Chris Romeo Your new nickname.

35:54Izar Tarandach He's the Concluder.

35:56Chris Romeo The Concluder. You don't want Matt to show up in front of your house because he's the Concluder. All right, well, I think that was a good philo— philosophical dive into ethics, AI,

36:12Izar Tarandach or lack of,

36:13Chris Romeo NTSBs. CSRBs and federal oversight. Well, or lack

36:20Izar Tarandach of,

36:20Chris Romeo or lack of. Hope you got something out of this and, uh, tune in next time for another episode of the Security Table where we unpack some other uncomfortable concept within the world of AI. Don't forget to follow, like, subscribe, and if you don't, the Concluder will come after you.

36:40Matt Coles Wait, I'm not the enforcement arm. That's you guys.

36:44Izar Tarandach Well, I was just trying to set it up.

36:46Chris Romeo I was trying to have a really cool moment at the end and then you're like, right.

36:49Izar Tarandach Wait, wait, try again. Try again.

36:54Matt Coles Try again. Take 2.

36:55Chris Romeo No, I give up.

36:56Matt Coles Scene.

36:56Chris Romeo No, I lost the moment.

36:58Izar Tarandach No, we have to finish. We have to finish with the concluder. Dun dun dun.

37:03Chris Romeo Next time on The Concluder.

37:05Izar Tarandach The Concluder. There will be no next time.

37:09Chris Romeo That's right. Didn't you hear my name? I'm the concluder.

37:14Speaker D Thanks for joining us at the AI Security Table. Subscribe, rate, review, and follow wherever you listen. New episodes every week. AI Security on the Table.

Transcript supplied by AssemblyAI. It may contain transcription errors.