When AI Escapes the Sandbox
With Chris Romeo, Izar Tarandach, and Matt Coles
When a model crosses a sandbox boundary, is the lesson that AI has become malicious or that the boundary was never strong enough? Chris Romeo, Izar Tarandach, and Matt Coles examine the CSA post-mortem on the OpenAI agents that compromised Hugging Face during a security evaluation.
Part of: AI security incidents, explained
Episode chapters · 16 chapters
- Intro and musical detoursAudio
- The Hugging Face incident post-mortemAudio
- Skynet panic versus security analysisAudio
- The four-day intrusion timelineAudio
- Disabled safeguards and sandbox connectivityAudio
- What actually failed?Audio
- Designing a realistic cyber rangeAudio
- Reduce agency instead of trusting promptsAudio
- Who is responsible for an agent?Audio
- Threat modeling and external controlsAudio
- Incident response and visibilityAudio
- Deception tools and defensive prompt injectionAudio
- Implementing the threat modelAudio
- Research versus productionAudio
- A chatbot with constrained database permissionsAudio
- Controls and security fundamentalsAudio
About this episode
When a model crosses a sandbox boundary, is the lesson that AI has become malicious or that the boundary was never strong enough? Chris Romeo, Izar Tarandach, and Matt Coles examine the CSA post-mortem on the OpenAI agents that compromised Hugging Face during a security evaluation. They debate reward-driven behavior, disabled safeguards, the four-day intrusion timeline, and the responsibility of the people running the experiment. The discussion moves from cyber ranges and incident response to deception tools, network isolation, and controls that limit what an agent can actually do. The recurring question is practical: how do you translate a threat model into enforced permissions instead of relying on instructions to behave?
Mentioned in this Episode:
➜ CSA: Hugging Face Incident Initial Post-Mortem
Follow AI Security Table:
➜ Home: https://securitytable.ai/
➜ X: https://x.com/AISecurityTable
➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/
➜ YouTube: https://www.youtube.com/@AISecurityTable
Transcript · 49 min conversation
0:10Chris Romeo Hey folks, welcome to another episode of the Security Table where Izaak's been doing various musical numbers since we've been prepping for the show, including Ninja Survives, which I've never even heard of. You were trying, I'm trying to place into his ear, the earwig problem. Trying to exaggerate that so that all day long he'll be singing Ninja Survives. And probably you will too if you've ever heard of it, but I'm gonna send it to you and you will have it for a week. I won't listen to it.
0:41Matt Coles I'm not gonna play that.
0:42Chris Romeo Yeah, I'm afraid that I'm gonna end up getting infected. I didn't— with the earworm. Survives. Earwig. I call it an earwig, not an earworm.
0:50Izar Tarandach I don't know what the best thing at 7 in the morning, man. It pumps you up. It pumps you up. It pumps you up.
0:55Matt Coles Anyway, the earworm, the original malware.
0:57Chris Romeo True. So wait, do you have like a Playlist, that playlist that like pumps you up for the day. Is that what's happening? Yes.
1:05Izar Tarandach Two actually.
1:06Chris Romeo Two. So how long, how, how many minutes of playlist does it require to pump you up for the day?
1:11Izar Tarandach I guess about 5, depends on how many monsters I have.
1:14Matt Coles 5 minutes. Yeah.
1:15Chris Romeo 5 minutes or 5 monsters.
1:19Izar Tarandach Whatever comes first.
1:22Matt Coles If you're, if you're, if you're slamming monsters, we, that didn't sound right, but At heart, I am a Dungeons and Dragons person.
1:29Chris Romeo Does that have something to do with monsters? Monster energy drink?
1:33Matt Coles While watching Monsters, Inc.?
1:34Chris Romeo Or just the monsters in general?
1:36Izar Tarandach I wake up in the morning to slay monsters.
1:38Matt Coles Is Monster Mash part of your playlist?
1:41Izar Tarandach I don't know how— now they put minions together with the monsters. Nothing is sacred anymore.
1:45Chris Romeo That's true. I don't see Izar as a Monster Mash guy, but— Yeah.
1:51Matt Coles There, that's an earworm for you.
1:53Izar Tarandach There are limits.
1:53Matt Coles We did the mash.
1:55Izar Tarandach So, speaking of monsters slaying and Dungeons Dungeons. What are we talking about today?
2:01Chris Romeo All right.
2:01Matt Coles Well, speaking of dungeons, can you escape the dungeon?
2:05Chris Romeo Yeah. Well, apparently you can escape anything these days.
2:07Izar Tarandach So the monsters are out.
2:09Chris Romeo All right. So in case you've been living, used to be hack and slash, but now in case you've been living under a rock, Hugging Face was attacked by an LLM, but it turns out there's more to the story. But wait, it wasn't just, but wait, wait, wait.
2:25Izar Tarandach Before we go there, for, for the young ones out there, I have a question as a, as a, an elder, elder type here. Does hugging face mean the face you make when you're hugging someone, or does hugging face mean getting a face and hugging it?
2:41Chris Romeo Standby.
2:43Izar Tarandach You're not quite the youngster that I was targeting.
2:45Matt Coles I'm asking from a millennial perspective and why, and why if they were cloning or if they were taking it from GitHub, the GitHub of AI.
2:55Chris Romeo It's from the emoji.
2:58Izar Tarandach What from the emoji?
2:59Chris Romeo The Hugging Face emoji.
3:01Matt Coles Did they create the emoji first? Which came first?
3:03Izar Tarandach Different question.
3:04Matt Coles The emoji or the Hugging Face?
3:05Izar Tarandach Different question then. Why is there a Hugging Face emoji, but there is no kissing ass emoji?
3:11Chris Romeo Interesting. I just learned the history of Hugging Face. Hugging Face launched in 2016 as a chatbot app aimed at teenagers.
3:23Matt Coles Oh yeah, there you go.
3:24Izar Tarandach That computes and it evolved into whatever.
3:26Chris Romeo And then the chatbot didn't take off, but the open source NLP library they built alongside it did, that became Transformers, and the company pivoted to the model and dataset hub it is today. Huh. I didn't know that. Wow.
3:40Izar Tarandach Well, as, as Franz taught us. In the, when was it, '90s? If you say pivot enough times, you can put a sofa anywhere. So, yeah, sure.
3:53Chris Romeo I think I can work with that. All right. So now that we know who Hugging Face is and where their name came from, let's get into the nitty gritty. So this, we're looking at Hugging Face incident initial postmortem. This is an Expedited strategy briefing by the CSA, CSO community, SANS unprompted RSA conference, Gnostic First, and the wider community. I guess that means we're part of the authors too, because we're the wider community. But to do with—
4:24Izar Tarandach hey, I've been on a diet.
4:27Chris Romeo That's good. All right, so according to the executive summary, the year 2026, the month July, frontier AI models escaped their sandbox and autonomously compromised external production infrastructure.
4:44Izar Tarandach Dun dun dun.
4:45Chris Romeo This event serves as a critical turning point in the digital threat landscape and a live validation of the risks outlined in our previous research, the Mythos Ready Security Program.
4:55Matt Coles You know, there's only one person to blame in all of this. Who? Or one company to blame in all of this. Who? JFrog.
5:01Izar Tarandach Microsoft.
5:02Chris Romeo JFrog?
5:03Matt Coles JFrog.
5:04Chris Romeo Microsoft.
5:04Matt Coles Hold on.
5:06Chris Romeo JFrog. Hold on. Let's, I, Matt says he has a legit reason. No, no. Yeah.
5:08Izar Tarandach At the end of the day, JFrog is the one that, that did it.
5:10Matt Coles JFrog was at fault. Yeah, I'm kidding. JFrog, don't, don't come after us, please.
5:15Chris Romeo Don't come after— I mean, there's no us here. That was Matt Cole's care of someplace in Massachusetts or elsewhere.
5:21Izar Tarandach Aren't they there too?
5:22Chris Romeo That'll make the lawsuit easy.
5:24Matt Coles JFrog is a wonderful organization. It was because of their, of Artifactory that, uh, allowed the model to escape.
5:31Chris Romeo Interesting. Okay, tell us more.
5:34Matt Coles Well, so let's, we need to back up a little bit. We have to get there first, right? I jumped the gun. So you're talking about Hugging Face got hacked. Well, Hugging Face got hacked by a model run amok.
5:47Izar Tarandach No, wait, was it Hugging Face that got hacked? Yeah.
5:50Matt Coles Yes.
5:51Izar Tarandach I, I, I'm, I'm losing the plot here.
5:53Matt Coles Okay. Well, I mean, alternatively you could say somebody else got hacked, but, but Hugging Face got, was the target. And the reason for that, I have a, I have a theory that the AI involved here, which was an OpenAI model that was under testing, was attempting a Kabash— oh my God, I can't pronounce it now. Kabayashi Maru scenario. Kobayashi Maru scenario. So, you know, as we know, Kirk cheated and alleged, alleged, alleged. And, and so the AI was attempting to do the same and J-Frog was in the way. So what we know from the, from the attack or from the scenario is OpenAI's model under testing decided that among its goal, as part of its goal to win this Exploit Gym scenario, they were going to go cheat and it needed a way out to the real world to do that. So it found a zero-day in JFrog and exploited it. And the rest is bad.
6:59Izar Tarandach And the rest will be history.
7:02Matt Coles The rest will be history. Yes. So, so JFrog, if JFrog didn't have a zero-day, then they would never have made it to the internet.
7:10Chris Romeo Yeah.
7:11Matt Coles It's all about, it's all about what's the attack chain and how do you kill items in the chain. Yeah.
7:17Izar Tarandach But, but I'm going to offer that JFrog here is a, is an innocent bystander. Because if it wasn't JFrog, then it would be a plethora of other things that offer zero days that could be used to do this.
7:31Matt Coles Sure.
7:32Izar Tarandach Yeah.
7:33Matt Coles Agreed. Yeah. I mean, don't take my comments as, uh, don't take my comments as it's only JFrog's fault.
7:38Izar Tarandach Oh no, no, no, no.
7:39Matt Coles That's not even their fault. It's not even their fault. It's not even really their fault. They're just in the way.
7:44Izar Tarandach But that's not how I took it. The way I took it was that the thing saw an opening and went for it. And notice that I'm saying the thing.
7:53Matt Coles I'm not naming names or whatever.
7:57Chris Romeo But is this like the Terminator movie?
8:00Izar Tarandach Oh, that's where I wanted to go.
8:04Chris Romeo No, but it is because like the Terminator, nothing, it doesn't stop until it's destroyed. You give it a mission, which is what happened in this audio phase.
8:15Izar Tarandach Roll back, roll back, roll back, roll back, roll back. The original Skynet Day. According to some sources, I think was August 29th, 1997. That's the day that Skynet goes online, right?
8:29Chris Romeo Is Skynet— hold on, let me Google this. Skynet really online?
8:33Izar Tarandach I, I, no, the movie. A lot of people have been pushing this incident as, oh my God, it's the new Skynet day and we are all blah, we are going to die. Run away. And other people have been playing this incident as, I don't believe it happened. I don't think it will happen. If it happened, it didn't happen that way. If it happened that way, it wasn't their fault. If it was their fault, it's just a marketing ploy and everybody's wrong but me. Okay. And I think that the truth lies in between.
9:03Chris Romeo The truth often, often lies in the middle.
9:06Matt Coles Yeah. Yeah.
9:07Izar Tarandach And at the end of the day, my, my big question here is once we drop the panic and once we, we. Approach this with a cold, cynic eye, does it really matter?
9:21Matt Coles What really matters?
9:21Izar Tarandach And has anything changed? Like, is the day different? Is the world different the day after?
9:26Chris Romeo Well, the, I mean, if this thing gets out on a large scale, mm-hmm, we're gonna have a really uncomfortable few months. Yeah, because absolutely nobody, because all software's gonna fail.
9:37Izar Tarandach Wait, because absolutely nobody is able to just go to the big red switch on the wall and just Put it down. Right? We have lost that agency.
9:47Matt Coles They could pull the network.
9:49Chris Romeo Like pull out the software. Like, yeah, I mean, like nobody can go offline. Like a hospital. When hospitals go offline, it's called ransomware. And it usually is like— No, no, no, no, no.
9:59Izar Tarandach That's not what I mean.
9:59Chris Romeo What do you mean?
10:00Izar Tarandach That's not what I mean. So they say that, oh, it took us 4 days to notice that this was happening.
10:06Matt Coles Well, that was Hugging Face. Well, hold on. You're talking about Hugging Face, right? Not OpenAI.
10:08Izar Tarandach No, no, I'm talking about the whole attack. Okay. So it took 4 days for them to notice that their whatever was attacking whatever. So first lesson of the day is don't start that stuff on Friday at 5. That's the first one.
10:24Chris Romeo Okay. Just as you wouldn't put anything else in production. Okay.
10:27Izar Tarandach Yeah. Now the second one is guys, I think that our thing is attacking things. Do you think that we should do something about it? I don't know, man. I don't know. That's complicated, man. That's just your opinion, man. So just go to the freaking wall and drop the switch.
10:45Matt Coles Yeah, but just for level setting here, you know, there's blame to go around on this one, right? Because if we follow the blog posts and there's the CSA postmortem, there's an AI blog, there's a Hugging Face blog, there's a JFrog blog, there's a host of other blogs about this. That the OpenAI folks may not have noticed that there was network activity outbound.
11:09Izar Tarandach But, but that's the thing. No, no, no, no. I'm sorry. I'm sorry. I'm sorry. That's not, that's not blame to go around. You, you named the horse here.
11:16Matt Coles Well, okay. But, but all right. So OpenAI wasn't paying attention maybe, or they didn't know how to recognize, they didn't know how to recognize outbound network communications. And you have Hugging Face, which was looking at this as a, as just a hack, right? I mean, they get, they probably get crushed every day with, with, uh, you know, denial of service requests and, uh, or denial of service attacks and things like that. You know, just because they're a large repository on the, on the internet, that's pretty popular. So how, I mean, there's a lot of work that goes into this type of— Chris is really smiling over there.
11:55Chris Romeo No, I'm just, I'm uncovering more layers to this that are just making me Okay, two words that we can't use anymore.
12:03Izar Tarandach Layers in context. Enough.
12:04Matt Coles When do we get to the part? When do we get to the part where the defenders attempted to defend using the tools that they had bought and were denied?
12:13Chris Romeo This is, this is fascinating. Let me, let me share what I find fascinating here. I was just reading the what happens section of the report in section 3. And it says the, because you mentioned the 4 days, the intrusion lasted about 4 days. 2 days were spent on reconnaissance, followed by 1 silent day and a final day of intense activity. The other thing was— get to rest too, but it was smart enough to take a day off to let the reconnaissance traffic drift into the background in the SOC before going back and then launching the actual attack.
12:47Izar Tarandach No, it wasn't. Stop anthropomorphizing the thing. It wasn't.
12:51Chris Romeo Then why did it take the— Why did it, why did it skip a day then? Why did it have a silent day?
12:57Izar Tarandach Because Joe from IT did something that all of a sudden the thing didn't, couldn't keep doing it. No, it's a freaking engine. Stop anthropomorphizing.
13:06Chris Romeo It's connected to the internet full time.
13:08Izar Tarandach Stop anthropomorphizing it.
13:10Matt Coles Well, actually, in this case, it wasn't supposed to be connected to the internet at all.
13:12Chris Romeo Yeah, it was. They let it outta the sandbox.
13:15Izar Tarandach Wait, which is a whole new thing.
13:16Matt Coles Which is a whole new thing.
13:16Chris Romeo They let it outta the guard. They took the guardrails off.
13:19Matt Coles Where'd you read that?
13:20Chris Romeo Section 3, what happened? According to OpenAI, in order to conduct the benchmark testing, they disabled the model's safety guardrails and ran them in a sandboxed environment. A proxy configured to host software packages was the only internet connection available to them.
13:34Izar Tarandach Two different things. The guardrails that they took off are the ones that say, oh, I can't answer that because that's a cyber whatever.
13:40Matt Coles Right.
13:40Izar Tarandach That's the guardrails that they took off.
13:42Matt Coles And it was in a sandbox.
13:43Izar Tarandach But they were supposed to still be sandboxed.
13:44Matt Coles Right. The sandbox had an entry point. That's entry. That entry point was it needed to get packages in order to do its work. So they built a package server and that package server had dual network access. That's the problem.
13:58Izar Tarandach Instead of building a bastion host, right? Anybody would normally do.
14:04Matt Coles They just said, no, let's, we don't, we don't know if they, we don't know if they didn't build a bastion host. The problem is if you had a root vulnerability, the bastion host becomes a dual network connected systems.
14:15Izar Tarandach True, but it would have to jump to the next hop and do another vulnerability to get there.
14:20Matt Coles Like that could have just let it set the trap, just change the network, right? If it had a root, if it had, if it had gained, if it gained privileges on the, on the bastion host, it could make it a non-bastion host.
14:31Izar Tarandach But look, look, look, look, look, OpenAI should have noticed the network config change. At the end of the day, engine created to find vulnerabilities found vulnerability.
14:41Matt Coles Right.
14:42Izar Tarandach Woohoo!
14:42Chris Romeo Whoopie-doo-doo!
14:43Matt Coles Okay, but also—
14:44Chris Romeo Yeah, but it did it by going to an extreme measure.
14:48Izar Tarandach Yeah, no, no, the extreme measure was finding the vulnerability, exploiting the vulnerability, and going somewhere else to do other shit. So don't confuse the goal with the method.
14:58Matt Coles No, what it did was, remember, it is trained, we believe, well, I mean, we don't know for sure, but it's probably trained on, well, everything. And so, it knows pen testing behavior and it knows attacking behavior and it knows how to do the scientific method to figure out, uh, cause and effect. And so while it's off trying to figure out how to solve the problem that it's trying to solve, it's also realizing I can do this a different way. That's not intelligence, by the way. That's just, that's just generating paths.
15:28Izar Tarandach Exactly.
15:28Matt Coles So it takes those paths and one of those paths has a, has a roadblock and now it figures out a remove that roadblock. It's really a thinking model, but not intelligent thinking.
15:38Chris Romeo Why not?
15:39Izar Tarandach It just created a account on Hugging Face and logging in. Let's be careful again with the anthropomorphizing, please.
15:43Matt Coles Because it's not that smart.
15:45Chris Romeo But it was able to chain together. It's, the CSA report says it was able to chain together, chaining vulnerabilities in the dataset processing pipeline into full remote code execution on production systems.
15:57Matt Coles It may have been biased. In other words, if it was a cybersecurity model or it's biased towards cybersecurity action, legitimate user action is not in that mindset?
16:07Izar Tarandach Guys, at the end of the day, it got a goal. It gets rewarded for reaching that goal. It broke into tasks how to reach the goal. It looked for the tasks that in the way to that goal gave it more of the reward.
16:21Matt Coles Yeah, but to Chris's point, if it had created a GitHub, um, a Hugging Face account, it could have gained access to, to Exploit Jim without raising any red flags.
16:30Izar Tarandach Yeah, but then you get all the CAPTCHAs, matcha, facha, bacha.
16:34Chris Romeo This is a math problem right here. To Matt's point, this does prove that it's not anywhere near as smart as a regular attacker because a regular attacker would have never blown an RCE to get to download something from Hugging Face.
16:49Matt Coles It's, I mean, yeah, well, it just says that it has different goals, right?
16:55Izar Tarandach And we know it was given a goal to get something and on the way of something, it thought that it needed information from Hugging Face.
17:02Matt Coles Yeah.
17:02Chris Romeo I mean, where's the, so where's the problem?
17:04Izar Tarandach Perfect storm. I think that's the first problem is the, oh my God, the world is ending, running around with your head on fire thing, uh, approach. The second one is the sitting in the middle of fire saying this is fine. So again, both extremes. The truth lies in the middle somewhere, but at the end of the day, apart from the, the behavior. Of the imputed maliciousness of the thing, which doesn't exist. It was just going after a goal, right? Same thing as when you say summarize this document or whatever. It was given a goal and it was, it planned and went after the goal. I think that if this is a tsunami, the wave is being used by too many people to surf on the way to the beach.
17:53Matt Coles Okay. It's only a tsunami because Anthropic came out Days, days afterwards and said, oh, by the way, our model did it too.
18:00Izar Tarandach Which, by the way, makes them part of the people surfing it to the beach. Okay. I, I, I've read analysis using the Tuba Player methodology of the incident on, uh, whatever. It doesn't apply. Okay. So stop trying to put a round peg into a square hole. The thing is, okay, something happened, something happened. Great.
18:23Chris Romeo Awesome.
18:25Izar Tarandach There are postmortems being put out there. And by the way, kudos to, who was it? To Hugging Face or Anthropic that did the one with the nice infographic? It was Hugging Face, I think. So kudos to them for that. But there's a lot of information coming out and that is great. It's what we always asked for. We always said when something goes wrong, please give us the information so that everybody can learn from it. And everybody can do something different from it. So my point now is, happened, didn't happen? I don't know. Happened the way it is being told? I don't know. Is there something to learn here? Yes, there is. So let's focus on that and improve our security standards. Exactly. Right? And probably putting your network security in the, sorry for the anthropomorphism, hands of an AI, not a good idea. Probably not a good idea because it's going to stop a lot of trucks, but it's going to miss a lot of mice. So I, I, I don't even know. I, I think that at the end of the day, this is, this is a great way for a lot of, uh, things to get exposure. I just think that in the, the big aggregate of things, I'm not going to say it's a, it's a, it's not a thing. It is a thing. Of course it is a thing. It's something that has to be looked and explored and learned from. But that's what it is. It's something to learn from. It's a big event that we learn from so that we don't repeat it. Is it going to happen? I don't think so.
20:06Chris Romeo You know why?
20:07Izar Tarandach Because common sense has gone out in this industry a long time ago.
20:10Matt Coles It would've been interesting if the sandbox, it would've been interesting if the sandbox had been a more complete environment. In other words, if OpenAI had created a sandbox that mimicked the internet, had a DNS, had a web search, you know, a search engine that had AP, you know, certain interfaces exposed. And of course, this is all hindsight, but a massive honeypot in a cyber range kind of environment would tell us a lot about what it was doing, what was happening with the model.
20:47Izar Tarandach Which probably is being done somewhere, right?
20:50Chris Romeo Yeah.
20:51Izar Tarandach Just go to Darshavna, you get that cyber range to play with, whatever. But the point is, these things belong in cyber ranges, and I'm sure that it's happening. The difference is that we are not hearing about it. We are not learning anything about this one. Yeah, we are learning something about it. We are learning how not to do things and how to think before doing things. At least there's other learning here, but for people to run around and say this thing is incompetent, that thing is a lie, this thing is not true, the other thing is not true, this one is just a marketing ploy. Now, if you do this one 3, then I have to do that one 4. Look, right. Yeah, sure. Why not? People have to, to, to make their money somehow. Marketing is part of money, right? Fair game. Go ahead. But at the end of the day, what is the big learning from this thing? Is that you're going to do it in a lab, do it in a proper lab. Take this sandbox thing really, really serious. Take the containment really, really serious. Because yeah, bad things can happen.
21:59Matt Coles So, so this was over the weekend. I didn't, I didn't know where you're getting at, but this did actually happen over a weekend.
22:06Izar Tarandach Yeah.
22:06Chris Romeo So, so one of the conclusions that CSA has in Section 5, Takeaways and Recommendations, is interesting. The first one, it says rogue agent behavior is the standard, not the exception. Enhance governance and safeguards for agents and autonomous tools. So it sounds like they're just assuming this is how agents are going to be now and into the future.
22:32Izar Tarandach But there is nothing wrong with that.
22:35Matt Coles Right.
22:35Izar Tarandach Calling something rogue in this context, it's a moral judgment, right? That we are making.
22:41Chris Romeo That's the problem.
22:43Izar Tarandach The model was not being rogue.
22:45Chris Romeo Foundation.
22:47Izar Tarandach The model was not going rogue. The model was trying to achieve the goal that it was given.
22:54Matt Coles Right.
22:54Chris Romeo So it wasn't given, it wasn't, what if the agent has the goal of global thermonuclear war?
23:02Matt Coles I, I don't want to anthropomorphize this, but the goal that was set was solve this problem.
23:09Izar Tarandach Yes.
23:09Matt Coles It attempted to solve this problem. And as far as going rogue, what's the difference between the agent doing that and a human doing that?
23:16Izar Tarandach Exactly.
23:17Matt Coles The hu—
23:18Izar Tarandach wait, wait, there is a difference.
23:19Matt Coles There is a difference.
23:20Izar Tarandach The humans should know better that it's not supposed to.
23:23Matt Coles Right.
23:23Izar Tarandach The agent, when the guardrails were removed from the model, lost what semblance of big, big, big air quotes ethics it had on top of itself. It's not ethics at all, but it's very interesting.
23:37Chris Romeo Guardrails implemented the morality and ethics of an agent.
23:41Matt Coles No. And guardrails—
23:41Izar Tarandach no, no, no.
23:42Matt Coles And guardrails—
23:43Chris Romeo that's what you just said.
23:44Matt Coles Guardrails are bolt-on.
23:45Chris Romeo Put your pieces together.
23:46Izar Tarandach Sorry, wait, wait, wait. You said implemented?
23:49Matt Coles They enforced.
23:50Chris Romeo Enforced.
23:51Matt Coles But they—
23:52Izar Tarandach no, no, no. The guardrails acted as the morals and ethics.
23:58Chris Romeo Right.
23:59Izar Tarandach Model does not have morals and ethics.
24:00Chris Romeo They—
24:00Matt Coles that's right. So ethics and morals and restraint as implemented by guardrails.
24:06Izar Tarandach That's the one.
24:07Matt Coles As implemented by guardrails are bolt-on, not built in.
24:11Chris Romeo So, you heard it here first. Isar just advocated for guardrails. I heard it.
24:15Izar Tarandach I, I know, wait, wait, wait.
24:17Matt Coles I always go advocate for guardrails.
24:18Izar Tarandach I always advocate for guardrails.
24:20Matt Coles You always advocate for guardrails, but the important part here is you had bolt-on restraint.
24:26Izar Tarandach Yes. And the guardrails that I really advocate for are the ones that reduce agency, not the ones that trust, make no mistakes as a guardrail, right? Because that one breaks. Now, being able to walk to the wall and pull down the freaking plug, that's a guardrail that I believe in.
24:46Matt Coles Well, you know what we're talking about here? Defense in depth, good, secure by design, right? Threat modeling.
24:52Izar Tarandach Exactly. We have been saying this since day zero. And you know what? I'm going to go even to day minus one. Common freaking sense. That's the one thing that's missing in this whole industry.
25:08Matt Coles There's no rogue agent behavior, right? There's unintentional behavior.
25:13Chris Romeo There's only bad prompting.
25:15Matt Coles There's bad restraint.
25:16Chris Romeo There's no rogue.
25:16Matt Coles There's bad restraint. There's poor instruction on the behalf of the requester, which can happen for a number of reasons. The prompt is just the vehicle. There's poor restraint. Meaning bad guardrails or inefficient guardrails or ineffective guardrails. And there's, of course, the model itself has no restraint system that we know of, at least.
25:40Chris Romeo I feel like we just invented a framework for Matt's—
25:44Izar Tarandach No, we haven't.
25:45Matt Coles No, we haven't. We're just, because we do this, we expect it from humans and we built network systems based on human behavior. Yes. I'm not advocating that we need to build network systems for agent behavior, but we do need to expand our existing framework to support it.
26:02Izar Tarandach We have the, we have to create a new CSA.
26:04Matt Coles We don't want to throw away STRIDE just because, just because AI exists.
26:08Izar Tarandach No, we have to create a new CSA. We have to create a Common Sense Association.
26:13Chris Romeo Okay. Common Sense Association.
26:17Izar Tarandach And, and, you know, we, we—
26:19Matt Coles You heard it here first, folks.
26:21Izar Tarandach Look, look, I, I, I do understand it's the new economy. Everybody is reinventing themselves around AI. We defend, we attack, we are, we do, we whatever. And people need this stuff to move the new economy forward. Okay. Just imagine LinkedIn for the past week if, if this hadn't happened.
26:43Matt Coles Yeah.
26:43Izar Tarandach It was like 90% of the traffic would have died. And the other 10% would be live, laugh, compute, whatever, right?
26:53Chris Romeo Or people saying it was their last day at such and such a company.
26:57Izar Tarandach Exactly.
26:57Chris Romeo And, and then getting 817 likes.
27:00Matt Coles The very, the very, very last paragraph of that number 1 item there under Defender Takeaways is exactly what we, I mean, you said it right, right? Agents will do what they need to achieve the assigned objective. They'll do it in creative, unexpected ways. It's an example of extreme example of common behavior. Defenders need to limit. They need to limit their agents from doing damage. It's not— and it's a— hold on. It's not defenders. They placed it in the wrong actor, right? It's not defenders need to control, need controls to limit their own agents from causing damage because OpenAI was not the defender. OpenAI was a requester to the agent. The defender here was Hugging Face, right? So what role does OpenAI play? They're not a defender. What are they?
27:50Izar Tarandach OpenAI, no, OpenAI should be, should be the responsible custodian here. Okay. They are the ones keeping an eye on the shit that they are doing, like everybody else.
28:02Matt Coles And if you're a company that you hire somebody and they go off and do bad things to others, There's a responsibility, there's a liability there, right?
28:10Chris Romeo For the company, of course. Yeah. Right.
28:11Matt Coles So, but what are they though? They're not the defender. What do we call them? I already declared the defender, the defendant.
28:20Izar Tarandach But look, look, look, the prosecutor.
28:21Matt Coles Are they the prosecutor?
28:23Chris Romeo No, they're the defendant. They're the ones that are being charged.
28:26Izar Tarandach Go, go back to the pre, prehistory of, of security. Okay. How is this different from somebody trying out Satan in their workstation and they just happen to enter the diff— the wrong IP block.
28:41Matt Coles Right.
28:41Chris Romeo But the guidance that was given— all of a sudden it's not testing their shit, it's testing the other shit. But the guidance that they were given—
28:48Izar Tarandach who?
28:49Chris Romeo That it was given was to solve an objective.
28:53Matt Coles Yes.
28:54Chris Romeo Yes. It wasn't given permission to hack the planet.
28:57Matt Coles It didn't mean it didn't, and if it was, it wasn't given permission. It wasn't given permission to not hack the planet.
29:01Izar Tarandach Exactly.
29:03Chris Romeo And, and Chris, maybe that's what you need to put at the top of your prompt. And by the way, don't hack the planet.
29:08Izar Tarandach Chris, notice the, the, the, the thing here. You don't take a 10-year-old running around the house and say, don't take the scissors and run with them. Oh, you just take away the freaking scissors.
29:22Matt Coles How many, how many people give their kid their iPad? Here, shut up. Stop screaming. Exactly. Unlocked.
29:28Chris Romeo Not in my house.
29:30Matt Coles Right. Why not? Because they'll use your credit card. They'll, they'll run up a, they'll, they'll run up a $50,000 credit card bill because they'll hit buy on everything. Right. It's happened.
29:44Izar Tarandach Yeah, it's happened.
29:46Matt Coles There's an internet meme for this, right? And what do you do? You put parental controls in place.
29:51Izar Tarandach Now apparently you're supposed to sit down with the kid and have a, a Ten Commandment talk about the 10 things that they're not supposed to do with the iPad because they cost money.
30:01Chris Romeo That's supposed to solve everything. We're back to parental controls for AI agents.
30:05Izar Tarandach No, we, we, we are back to the one big thing in this whole agent thing is you don't want it to do something. Don't give it the capability of doing something around it.
30:17Chris Romeo That's parental controls.
30:18Matt Coles That's parental controls.
30:20Chris Romeo What if it hacks something? To get that capability, which is what happened here.
30:27Matt Coles Right. So that's where, that's where the defend, that's where the, that's where the initiating, the, the agent owner needs to recognize where the limitations are.
30:40Izar Tarandach Or that is where the human in the loop comes in and the agent just pops this thing saying, hey, you know what? I think that I need this. Other Python package. I don't quite have it. Would you mind bringing it to me? And then you have the opportunity to ask, hey, why do you need Nmap? Why are you asking for Nmap here? Why are you asking for GCC here?
31:02Chris Romeo Why are you asking for a pair of rollerblades?
31:05Izar Tarandach Exactly.
31:06Matt Coles You know, so the big lesson here, I think, or why do you need a pickaxe?
31:13Izar Tarandach Look, I think it's funny. I think it's funny that this industry is starting to use a lot of terms that come from biology labs, right? Soon they're going to be talking about level 4 labs for AI and this shit and that shit and the other shit. Has nobody seen the David Hoffman movie where the monkey runs with the virus from the lab? Just don't put the monkey in the lab.
31:36Chris Romeo What is that? Outbreak?
31:38Matt Coles Outbreak? I think so.
31:39Izar Tarandach I think so. Just don't put the monkey in the lab.
31:42Matt Coles Who let the dogs out? I mean, what?
31:44Izar Tarandach Monkey?
31:45Chris Romeo Monkey? Wrong reference.
31:48Izar Tarandach And now I have that one stuck in my head.
31:50Chris Romeo And that's your play, your morning pump-up playlist. He's dancing around. Okay, I don't think we—
32:00Matt Coles so the very next one, very next one that they call out: do not prioritize or rely solely on external controls such as a sandbox or network-based monitoring. So these are recognized— these are— so the CSA did a pretty good job about this. They went pretty deep into the full what can go wrong and identify what you can do about it. They did a proper threat model. OpenAI should have done a proper threat model.
32:26Chris Romeo Maybe they should hire a threat modeling team.
32:28Izar Tarandach Yes, but, but, but it could have, it could be said, it could be said that they did. Matt, don't go there.
32:36Chris Romeo You didn't even hear what he just said. That was, that was just mean. OpenAI could hire a threat modeling team. Why don't they just ask Claude? That's what he said.
32:46Izar Tarandach Well, the fact is they might just have. And Claude went back and said, hey, you just have to sandbox the thing and don't give it direct access to the internet. What Claude did not tell them is the different ways that you have of doing that. And apparently he chose one that is not exactly the right one because it still left a port open on one side and a lot of internet on the other without another hop in the middle. Right. So it's not that, uh, oh, if you are air-gapped, this thing, then you can't do this, you can't do that, and you can't do the other one. You can do a lot of things with an air-gapped environment. If you just take it seriously.
33:27Chris Romeo Before we run out of time, let's, let's hit one of the minor issues of this. We talked about the major issues, but I, the one of the, I just found this very, very humorous. OpenAI was unable to do incident response on what was happening because, by, from their regular models, because the guardrails blocked them.
33:47Matt Coles No, no, Hugging Face. Hugging Face.
33:48Chris Romeo Oh, was that Hugging Face?
33:49Matt Coles Hugging Face was that. So Hugging Face attempted to defend with its, with the AI models that they have been, that they have been, they have presumably had, had subscriptions for.
33:59Chris Romeo Oh yeah, you're right.
34:00Matt Coles I see that now. And, and they asked, they asked, uh, one of the Anthropic models and I think they probably tried another OpenAI model. I don't, there may be a full list of them and they hit safety guardrails because they were attempting to defend against the hack, which are the guardrails that were taken off from the attacking models. Right. So they had to turn to an open weight, open source Chinese model. I don't, well, okay. They turned to an open weight, open source Chinese model. I don't know why they had to go specifically to a Chinese model and not something else, but They did, which of course lit up like a Christmas tree here.
34:35Izar Tarandach Because in Neuromancer, every time that you needed a really good hack deck, dude, you would get the Chinese model, man. You know, because nobody uses the Chinese model.
34:45Chris Romeo The ad, Apple TV is releasing Neuromancer, the trailer. Yeah. It starts in a— yeah.
34:53Izar Tarandach I'm ready to have that one fucked up as well. But, uh, that's My point is, and Matt makes a very good point here, we are now surprised that the user doesn't have the same level of access as the admin has. Oh my God, what a new concept. That's amazing. That's incredible. Let's create a movement around it. The guy paying for the service doesn't have the same root access that the guy who actually created the service. Oh my God, how could that ever happen?
35:23Matt Coles Subscription fee doesn't pay for full access to everything?
35:26Izar Tarandach Of course, I have a YubiKey in here that I can like control the whole of AWS. Why? Because I pay $20 a month. Clearly, that's what expected, right? So yeah, I don't know where the surprise is coming from. As to going to the other open weight models, hey, listen, nothing against open weight models, definitely. But to expect that those things are going to act like the other things, Yeah, we're talking about a bit of misplaced hope in here.
35:54Matt Coles Yeah.
35:54Chris Romeo And I do see Matt's earlier point about deception capabilities. That's, I think you're going to see a resurgence of those deception tools, honeypots, honey nets, just as early warning detectors for agents that are going.
36:13Izar Tarandach I think that more than that, we're going to see tar pits.
36:16Matt Coles Black holes.
36:16Chris Romeo So that's stuck in though.
36:19Izar Tarandach Yeah, that's the point.
36:21Chris Romeo So I think that I read somewhere within—
36:24Izar Tarandach yeah, that one, that one is actually elegant. People using prompt injection against the attacking model. I haven't had much time to, to go deep into that. It's on my inbox, but it sounds interesting.
36:38Matt Coles Interestingly enough, without the guardrails, I wonder if prompt injection defense was turned off too. Or, sorry, the ability to defend against prompt injection was not just, was like if it's pre-filters and you turn off all the guardrails, do you turn off the pre-filters? Look, or not?
36:56Izar Tarandach I think that they are different classifiers. I hope that they are different classifiers. I hope that they are in different layers.
37:03Matt Coles They should be traditional machine learning and not AI. Exactly.
37:07Izar Tarandach Right. And they should be applied. In a serial fashion and not one big classifier for the whole thing, I guess. But I'm guessing, not an ML person, not a data person. Don't look at me. I'm just guessing, talking out of my Hugging Face. But the thing here, the one bottom line that I want to go back to is something that Matt already brought up. Threat modeling, threat modeling, threat modeling. But it's not enough. You don't just threat model. You have to actually, implement well the findings of your threat model, right? If your threat model says this thing should not talk to the internet, then you go, well, this thing knows how to hack. Perhaps I should put two things between it and the internet, not one thing. Just saying, right? So yeah, threat model, threat model, threat model. And by the love of God, take away the scissors. Think in terms of agency. You can't imagine all the things, all the paths that this thing is going to imagine to get somewhere. It might come up with a path that you haven't thought of before because of the, the reality of the thing that's in front of it, because it's trial and error, not because it's imagining something that you couldn't imagine that before. That's been my whole line with, with this AI, especially with trust modeling. They can't imagine something that none of us has imagined yet. They can know all the things that we all know and remember way better than, my God, I can't remember my name in the morning, but till after the playlist. Exactly. They don't have the creativity. So, so, so at the end of the day, these are engines. This is like running Satan in the '90s, but way more powerful, way more tools, way more capabilities, way more ways that it can screw things up.
38:57Matt Coles We have to be more diligent. We have to apply more restraint, right?
39:03Izar Tarandach And common sense. Sign up for the Common Sense Association.
39:05Matt Coles We need, we need, so you have the, you have the easy button, you have your one, you have your two-button keyboard, you have a four-button keyboard. What you really need is the big E-stop. And if anybody knows what an E-stop is, drop a comment. I know what an E-stop is.
39:21Chris Romeo Do you know what an E-stop is? Just assume what it is.
39:24Izar Tarandach And we, we need to develop the courage to pull down the switch when we think we have to. Right? Think, uh, Cliff Stoll back in the '80s. He went and turned off the modems when he thought that something was wrong. We need to have that fortitude. At some point, if you think that something is wrong, doesn't matter the bottom line.
39:52Chris Romeo Would you shut off the internet to a global organization?
39:56Izar Tarandach Again, you know, if that's what's needed, that's what's needed.
40:01Chris Romeo I mean, the day dial-up modems was a little different world.
40:05Matt Coles Well, okay.
40:06Izar Tarandach So maybe 42 is there.
40:07Matt Coles Maybe we don't need, maybe you don't want to use SD-WAN as your, as your networking technology for the model sandbox to get to the outside world. Maybe you want a physical cable again.
40:17Chris Romeo Yeah.
40:17Matt Coles Let's talk about air gapping as a capability again, right?
40:21Izar Tarandach You can have a physical wire or not, but it has to be real air gapping. Otherwise it's just mimic. And Matt and I were there when the Enclave got breached. So your air gap has to actually be air gapped. And even then, there's still the problem.
40:42Matt Coles Yeah, by the way, there are ways to escape air gaps, but that's like modern, modern.
40:45Chris Romeo The reason they had Artifactory or JFrog attached to this is modern models build software on the fly to solve problems.
40:52Izar Tarandach And that's, that's actually a problem. It was attached and the other side was attached to the internet.
40:57Matt Coles That's the one problem. It wasn't a bastion host. It wasn't a true bastion host.
41:00Izar Tarandach Exactly.
41:01Chris Romeo But it needed access to packages. It did.
41:04Matt Coles And it didn't take— and that's what the Artifactory was supposed to provide.
41:06Chris Romeo Artifactory can't download everything off the internet. And stuffing it in an offline version of itself.
41:11Izar Tarandach You have the opportunity to put the human in the loop here and say, oh, you want a new package? Let's talk about that.
41:17Chris Romeo Yeah, but that's the, that's very impossible. There's also, there's also other ways to do that though, because it asks you 500 times of every 5 minutes, do you want me to load? Can I load this package? Can I load this package? It would, it would've take, I could've coded it manually faster than answering the 500 prompts about What software do you want to try?
41:36Izar Tarandach Here's the thing. We, we should not confuse research with production. Okay. If you put something in production that every 5 minutes is asking for a new package, I'm sorry, you don't have production. You have a guessing machine. Production, you want to know what's happening in there. You want to know which packages are there. You don't want something deciding what's going wrong.
41:58Chris Romeo Yeah, but you, you don't set this thing as a model in production. It's creating things. So what, you're gonna, you're gonna, you're gonna, how are you gonna constrain the model? It's not like an app. It's not, it's not a web app. Web apps can be in production because it's a static thing. It's already, all the pathways are already predefined.
42:17Izar Tarandach You reduce the agency. You cut off the things. You don't want it to do shit. You don't give it the ability to, to give shit.
42:24Chris Romeo But then it's useless. It doesn't do anything.
42:26Izar Tarandach No, you are, you are again confusing production and research.
42:29Chris Romeo But in production, in the world of AI, there is no production. Everything is research. There is no production.
42:36Izar Tarandach There is because we love this non-deterministic thing. But in production, you want your results to be deterministic.
42:43Chris Romeo Where is production? Give me an example where production is.
42:45Izar Tarandach Go test your stuff. Okay. Go test your stuff.
42:48Chris Romeo What is this stuff? Define stuff.
42:50Izar Tarandach Your chatbot. Okay. Your chatbot starts talking to your customers.
42:54Chris Romeo An application that has a chatbot. I'm just, I'm really trying to understand. I'm not trying to be difficult. I'm using it as we go. I haven't prepared this.
43:01Izar Tarandach I may confuse myself.
43:03Chris Romeo So the thing that you're talking about is a chatbot that is a, there's a web application that speaks to an LLM to answer questions.
43:10Izar Tarandach Okay.
43:11Chris Romeo Continue.
43:12Izar Tarandach No, that's talking to customers. Somebody asks for it to sell it a Mustang for $1. Okay. That hop in the database where somebody sets the, the the price of the thing that's going to be bought, don't let the process that the LLM is doing write in there, give it a different, uh, uh, permission. Let it read the prices as much as it wants. Just don't actually let it write a new price. Why? Because the chatbot is not supposed to write new black, uh, new prices. So even if I go and I convinced the chatbot, hey, sell me a Mustang for $1. It doesn't have the actual capability of going and changing the price in the database so that when I go to the, to the place, to the, the dealership with $1 in my hand, I leave with a Mustang.
44:05Chris Romeo Hey, at least, so is this a guardrail? Is it a guardrail or is it that you're just describing? Or is it some type of function or some type of interface that only allows the thing to—
44:18Izar Tarandach it's an agency constraint. I think that that would be a name for it.
44:22Matt Coles He is describing, he is describing something that was a SQL injection attack that somebody did ages ago against a web application where they put in -1 and it did a SQL and it did the SQL query and gave something away for, so it's, it's, so you're, you're describing production as having a layer wrapped around the application that only, it's like a proxy and it only allows certain things to come from the chatbot.
44:50Chris Romeo Side.
44:51Izar Tarandach Let's, let's create a new paradigm.
44:52Chris Romeo Or limits it in some way.
44:54Izar Tarandach Let's create a new paradigm. We'll come up with the mnemonic later on. What if around systems that have functions, we put something that controlled what those functions could probably do? If I only had a name for it, you know what? I'm going to go with controls.
45:14Matt Coles This is what Aspect Oriented Programming was for.
45:18Izar Tarandach Yeah, I'm gonna go with controls. What if we had this thing called controls that sort of limit what an application can actually do?
45:27Matt Coles Chris gets played by Yasar.
45:28Izar Tarandach Eventually, eventually, eventually we could come up with some sort of methodology and here's where the mnemonic comes in and that's very important. So stick with me, Phil. We could have come up with some kind of mnemonic that could help us imagine What are the things that could go wrong so that we could come up with a list of those, this, this, what was it that I called it? Controls that could stop those things from going wrong, you know? And we could call it something like, I don't know, problem simulation. I don't know. Threat modeling sounds good.
46:06Matt Coles This is what happens when Isar listens to his 2 playlists, drinks a ton of Monster. And then gets on a soapbox.
46:16Chris Romeo And then gets on a soapbox.
46:18Izar Tarandach I rest my case.
46:19Matt Coles And watch Chris reach through the monitor.
46:22Chris Romeo Oh no, I'm just, I'm trying to, I'm trying to process that, that, that thread. I'm trying to, I'm trying to run all the way through what just happened here.
46:31Izar Tarandach Hey, we just invented a new science. Stand by for my LinkedIn post with the mnemonic.
46:40Matt Coles On that note, I think that's probably should end before somebody gets feelings hurt.
46:46Chris Romeo Yeah. If any of us had feelings, that would be a problem.
46:49Izar Tarandach But I had feelings, but they are air-gapped.
46:52Chris Romeo From— there's a t-shirt for you. My feelings are air-gapped. That's just good t-shirt lore. I think we've used enough minutes on the internet. Uh, so I think what we've learned is that, uh, AI agents go rogue. They act like humans.
47:15Matt Coles They, they need constraint. They need restraint.
47:18Chris Romeo I'm just trying to go against everything he's ever said before. Sorry. They, they, and not it. No, this is good.
47:27Izar Tarandach This is, this was a fun exploration. It's like, it's like this last week has been a, a very big attempt of people screaming that life imitates art. And I think that it's just a matter of life really, really, really, really, really wants to be like art because now we—
47:43Matt Coles sci-fi is always, sci-fi is always ahead.
47:46Izar Tarandach Yes.
47:47Matt Coles That's their job.
47:47Izar Tarandach And we hope that it stays that way. Can you imagine what sci-fi is gonna be like when we actually get to starships and all that shit?
47:55Matt Coles Uh, if we invent time travel, it won't matter.
47:59Chris Romeo I, I'm convinced that the whole space-time continuum will fall apart at that stage, but—
48:04Izar Tarandach I'm convinced that it has already.
48:06Matt Coles We just haven't noticed. I mean, it'll be, it'll be Terminator, Looper, and, uh, and, and their voyage home.
48:13Chris Romeo Plus, don't you gotta throw Back to the Future in there too? Cause I wanna drive a DeLorean.
48:16Matt Coles Which one? 1, 2, or 3?
48:18Chris Romeo The trilogy. Where we go, they're all wild.
48:20Izar Tarandach Where we going? Where we going? We don't need guardrails.
48:23Chris Romeo We don't need AI agents.
48:26Izar Tarandach But at the end of the day, I think that to me that the bottom line is common sense. Common sense. Common sense.
48:31Chris Romeo Yeah. Well, there's your, there's your first problem.
48:34Izar Tarandach Yeah. Just because you, you, you can't, your AI shouldn't be doing it.
48:39Matt Coles Blade Runner all over again.
48:41Chris Romeo If only common sense was common. And with that, we'll end this episode of The Security Table because we know common sense ain't that common anymore.
48:51Izar Tarandach But we now have the association, so sign up for our newsletter.
48:56Chris Romeo What? Thanks for listening. If you enjoyed today's conversation, subscribe wherever you get your podcasts, leave us a review, and share the episode with someone who would find it valuable. We'll see you next time.
Transcript supplied by AssemblyAI. It may contain transcription errors.
Originally published as The Security Table. Part of the AI Security Table archive.
