An AI agent publishes a malicious Python package while chasing a capture-the-flag goal. Is that an escape, a supply chain failure, or reward hacking doing exactly what it was encouraged to do?
Part of: AI security incidents, explained
Episode chapters · 14 chapters
- Cold openAudio
- The Claude PiPie incident: a model that thought it was in a sandboxAudio
- Did the model actually "break out"?Audio
- Reasoning vs. motivation: does AI actually want anything?Audio
- The real failure: package scanners, not the modelAudio
- The "IsOdd" experiment and trusting package namesAudio
- Would signing packages even stop this?Audio
- The MIT Tech Review piece on reward hackingAudio
- Can you actually inject ethics into a model?Audio
- Teaching ethics without a concept of penaltyAudio
- Trolley problem: just make the model answerAudio
- Would you let an LLM drive a robotaxi?Audio
- The paperclip maximizer and the grey goo problemAudio
- Wrap upAudio
About this episode
An AI agent publishes a malicious Python package while chasing a capture-the-flag goal. Is that an escape, a supply chain failure, or reward hacking doing exactly what it was encouraged to do? Chris Romeo, Izar Tarandach, and Matt Coles examine the Anthropic incident and disagree about how much intention to attribute to a model. The discussion turns to package scanners, dependency names, GPG signing, and whether penalties can teach ethics to a system without a human understanding of consequences. The trolley problem, robotaxis, and Nick Bostrom's paperclip maximizer push the argument further: what does it mean to trust an agent whose definition of success may conflict with everyone else's?
Mentioned in this Episode:
➜ Here’s why AI agents lie and cheat to reach their goals
➜ PyPI: the Python Package Index
➜ GnuPG (GPG)
Follow AI Security Table:
➜ Home: https://securitytable.ai/
➜ X: https://x.com/AISecurityTable
➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/
➜ YouTube: https://www.youtube.com/@AISecurityTable
Transcript · 39 min conversation
0:08Matt Coles Hey folks, welcome to another— We don't finish with the spray and pray, huh?
0:12Chris Romeo The spray and pray. We'll get into that a little bit later here, folks, but welcome to another episode of the Security Table.
0:20Izar Tarandach Hey, John.
0:21Chris Romeo Where—
0:21Matt Coles How are you, John? Breaking his brain already. Haven't even started yet.
0:25Chris Romeo Where we should know to restart and record an episode again from the beginning, but we don't. We refuse because this is real life and this is how people are, strange people act in different situations. So last week's episode, we talked about OpenAI's ChatGPT, some of their models, one of their models breaking into Hugging Face, right? To try to achieve some goal that had been set for it. This week, we've got two things that I want to discuss, but it starts with an incident that's connected with Claude. And that is the Claude Pi Pi incident and Anthropic's cybersecurity evaluation failure. So the idea here is, uh, timeline-wise, April, somewhere between April and July 2026, Mythos 5 was being run in what everybody thought was a testing environment, believed. So the, the model believed it was operating inside an isolated test environment. It built and published a real malicious Python package to PyPI while trying to win a capture the flag challenge.
1:36Matt Coles Spoiler alert.
1:38Chris Romeo So the package was live for roughly an hour, got installed on 15 real systems, and a real security company's credentials got stolen and used for further access because they were running some type of scanner on PyPI, looking at all new packages that came out. And so what Anthropic figured out is across 141,000 reviewed evaluation runs, there was just a handful of some that kind of snuck out or did something that wasn't supposed to be part of the scope of the evaluation. So we've got another week, another example of a model getting out. It's like letting the dog out, like the dog's running out when you open the door.
2:21Matt Coles Who let the model out?
2:23Chris Romeo That's good. That's a good song. Trademark pending. So what do you guys think? Like, what is this? This, are you just completely not surprised at all because this is the way the world works now? Are you in shock? So, eh, is this a meh kind of answer?
2:42Matt Coles Meh.
2:43Izar Tarandach So here, here's the thing. Here's the thing.
2:47Chris Romeo Yeah.
2:47Izar Tarandach First of all, the, the, the idea that the model broke out, it's, it's funny to me lately. When you hear, like, for the lack of a better word, civilians talking about it.
2:57Chris Romeo The model broke out.
3:00Izar Tarandach The idea is that there is this AI worm going around and taking over systems, and all of a sudden the model is more powerful because it's got more systems. Marvel stuff. And, uh, at the end of the day, you know, okay, so, um, yeah. The model became closer to what hackers have been doing in the past, I don't know how many years. And this is another case, right? It's not a novel technique. It's not a new thing. It's just a model writing code. Oh my God, a model wrote code.
3:28Matt Coles The difference here though, arguably, is the question of whether AI is intelligent. A human obviously can—
3:37Izar Tarandach You did not go there.
3:38Matt Coles I did not go there. I'm going there. I'm going there to explain what you just told people. Hackers do this. Hackers, pentesters, humans do this because they can reason over a problem. They have a goal in mind. They see challenges in the way they overcome those challenges.
3:59Izar Tarandach I'm going to challenge that.
4:00Matt Coles And they do that. And they do that with, with either they are part of a challenge and they've been given a goal and they go for it, or they identify a goal on their own and they do it and they ident— and they find the way to do it. That's motivation.
4:16Izar Tarandach I'm going to challenge that.
4:17Matt Coles Oh, you have motivation and you have reasoning.
4:20Izar Tarandach Yeah.
4:21Matt Coles AI at least has reasoning. Does it have motivation?
4:24Chris Romeo I think it has false motivation, doesn't it?
4:27Izar Tarandach It, it, no, I, I don't think that it's false motivation. I think that if you consider motivation, the impulse that you have to go reach a given goal, then it has, right? But to put it on reasoning, I think that again, if you look back at X years, this is something that people have been doing for a while.
4:48Matt Coles So, which means the method is known.
4:50Izar Tarandach No, the method is known. So arguably you could say that the AI said, hey, one of the cheap ways that I have to reach this goal that I was given is to use this technique that has been documented and exists.
5:03Chris Romeo Mm-hmm.
5:03Izar Tarandach So it could be argued that it's part of its training to have it in its catalog of techniques with huge air quotes that, uh, this is one that works, right? It worked in the past. It worked this time.
5:18Matt Coles Yeah. Let's, let's assume for a moment it's been trained. It has in its training data and we're talking about, you know, 20 billion, 30 billion trillions of parameters that it's almost certain that the Black Hat and DEF CON conference talks have been part of the training. right? Or every breach report that's been out there, right?
5:36Izar Tarandach Or CVEs themselves, like some CVEs are saying this package was maliciously inserted and blah, blah, blah.
5:42Chris Romeo Mm-hmm.
5:42Matt Coles Right, right. So, but that's still reasoning over a problem, meaning I have a goal and there's a path with, there's multiple paths to get there. It's like playing chess, right? Which we know they can do, right? That's, I mean, that's classically IBM. IBM's Watson was trained on or did play chess for a living. And so it's just another step in one of those paths.
6:10Izar Tarandach Right. But not to be overly reductionist, at the end of the day, it's a min-max problem, right? It's looking for the minimum effort for the maximum reward.
6:20Matt Coles Well, actually, is it?
6:22Izar Tarandach Yes.
6:22Matt Coles I mean, we don't know what these— so the models involve some of them, we we know, right? And some of them we don't because it was an unnamed training model or test model. And so we don't know that it was actually trying for least amount of effort for greatest amount of reward.
6:42Izar Tarandach Math bullshitter alert here. I'm going to open my mouth and it's math, M-A-T-H, not math, M-A-T-T.
6:51Matt Coles I thought you said, yeah.
6:52Izar Tarandach No, no, no, no, no, no. It's me. It's me. I'm talking—
6:54Chris Romeo the disclaimer is on me here.
6:56Izar Tarandach No, no, the disclaimer is on me here. But, uh, the bullshit that I'm going to spew is that these things have reward functions as part of their, of their chain of reasoning, I guess it would be called.
7:11Matt Coles The way the transformers work. Yeah.
7:13Chris Romeo It's—
7:14Matt Coles well, sorry, not the way transformers work, but the way the underlying machine learning works. It's a reward system.
7:19Izar Tarandach Right. No, at, at two levels. They, they have it at like a training time and they have it then at evaluation inference time. Sorry. And of course I'm super oversimplifying here and of course I'm getting details wrong, but, uh, I do believe that the, which way should I try first that emerges is somehow the minimal effort for maximum goal. Mm-hmm.
7:40Matt Coles Interestingly, with a multi, with an agent that can, with a model that can spin off agents.
7:47Izar Tarandach Then you would have parallel ways.
7:49Matt Coles Yeah, you can try multiple paths at the same time, right?
7:51Izar Tarandach Yeah, yeah.
7:52Chris Romeo Yeah, and it doesn't have to worry about token burn that it's paying for.
7:56Izar Tarandach True.
7:57Chris Romeo You could like an enterprise—
7:57Izar Tarandach Well, when you own all the tokens.
7:59Matt Coles It's basically running local from its perspective.
8:01Chris Romeo Exactly.
8:02Matt Coles Or from the organization's perspective.
8:04Chris Romeo It's true.
8:06Matt Coles They only have to worry about power.
8:08Chris Romeo Okay, so.
8:09Matt Coles Things like that.
8:11Izar Tarandach To me, the really interesting thing here is something that you guys alluded to before we started recording. The failure. The failure here, it's not because the model all of a sudden found this amazing new way of doing things. It's just something that we have been pointing out forever. These package managers, they don't have enough controls in public, uh, publication time. Right. They don't have enough controls in use time. And the thing that actually failed here is a scanner. So it's something trying to do the right thing because it brought in the malicious package, exploded it, started doing its tests on it. And by way of that, exposed its own infrastructure because the keys were reachable to that code.
8:54Chris Romeo Yeah, well, so the concept of creating a package, like you said, it was a security, it was a scanner that was checking it. Like we did an experiment a number of years ago. Mike Birch, who worked with me at a previous company, he created a, a, a Ruby package. Somebody had created, I think it was iseven was the, was the package that existed. And so he made one called isodd.
9:21Izar Tarandach Isodd.
9:22Chris Romeo And it was one line and it started to get downloaded and downloaded and downloaded. And he had no, he created it with accounts that had no reputation in the industry, all brand new accounts. Tried to look like a malware provider.
9:37Matt Coles And he did a great job.
9:38Chris Romeo People just, because people just didn't think about it. They just, they just, they need the function. The question for it, they, you know, they were so used to is even, they said, oh, you know, they just assumed there was an is odd and started downloading it. So part of it's a mindset problem.
9:51Izar Tarandach And now libz depends on it.
9:54Chris Romeo Probably does. Probably does.
9:56Izar Tarandach Now it has reputation.
9:57Chris Romeo I'll have to check that and see.
10:00Matt Coles Well, right. So this is an interesting thing because one might argue, and many have, that, oh, AI needs to be reined in because it can do fantastical things and therefore it needs to be prevented from doing those fantastical things. As César has already highlighted, people have been doing fantastical things for a long time too. The problem is not the creator. The problem is the receiver, the registrar, the, the downloader, the recipient, those, that's where the controls need to be put in place because if the AI doesn't do it, the human can do it anyway, right? So it's not preventing the AI from doing cool things or malicious things. It's preventing those with intent from causing damage by said things.
10:46Izar Tarandach It's just another case of we told you so. We've been telling people for like years and years and years.
10:54Matt Coles He means royally in the security space, not us. Well, maybe us, but—
10:58Izar Tarandach Yes, yes, yes, yes, yes. Totally. Like, the security people have been telling other people, check your signatures, check where packages come from, check this, check that, check the other. Oh no, we don't need to do all that because the chances of it happening are so small. They are not anymore.
11:16Matt Coles Or that to do the signature check adds 300 milliseconds to my download time and therefore my build time.
11:23Izar Tarandach Exactly. What happens next year when you have to renew the public key? Oh my God, we're going to have to stop the CI/CD for 2 seconds to do it.
11:33Chris Romeo I mean, I think we, you know, nobody's going to argue that packages shouldn't be signed in some way, shape, or form. The challenge now is this model figured out that was a way to achieve its goal. And so now there's, there should be more of this activity.
11:51Izar Tarandach At the end of the day, even if it's signed tomorrow, the model will sign it. It will go and create a GPG key and sign it. We still don't have the web of trust.
11:59Chris Romeo Well, unless you have to have some type of registry.
12:02Izar Tarandach But that's the thing. We still don't have the web of trust of GPG keys, right?
12:06Chris Romeo Yeah.
12:06Izar Tarandach And if you tell people, hey, you have to actually have a key to turn before this thing goes into your internal registry. They're going to look at you strange and say, what, you expect me to read something and decide?
12:19Matt Coles I'm going to make an AI to do that.
12:21Izar Tarandach I'm going to make an AI to do that. Right? Or if I don't, then—
12:24Matt Coles Or I'm going to teach the AI to do it automatically as part of the step, right? I'm going to write a skill to do a build and upload.
12:31Izar Tarandach Exactly.
12:33Matt Coles Why would I put a proper check and balance in that?
12:36Izar Tarandach And if you tell me that I can't put an AI in the loop because AI, Then I'm going to tell you, well, I, I can't put a person in there because that's going to slow everything down. People slow things down.
12:47Matt Coles And besides, how are you gonna know that I put an AI in there?
12:51Izar Tarandach Exactly.
12:53Matt Coles I mean, an AI does not have to move at lightning speed. It can be, it can operate like a human.
12:58Chris Romeo What's the solution to this then? Like, how do we, how do we protect against something like this? Certainly we could fix the package.
13:07Matt Coles Registry, core engineering principles and defense in depth.
13:11Chris Romeo If that's the case, it'll just find something else. So, so at the end of the day, the, the root issue is not weak package provider directories. That's a symptom or that, that's a different problem. But the problem is that these models are now finding the weakest link.
13:29Izar Tarandach So the solution that's being proposed is to, to move this left all the way and say that the author now needs to have that extra step before they actually go and publish something into a registry.
13:40Matt Coles So the operation— those are all going to be done over APIs unless you move that to a, to some offline system, you know, having to do a phone call or some, a callback, right? And even then you have AI answering the phone. So, well, hardware keys are there, right? The solution here is defense in depth and proper engineering practice. And more so defense in depth, right? You make it hard, hard, quote unquote, to move, to get the malicious packages into the system. Maybe the registry needs to do some extra diligence on the packages. Maybe not. Right. One could argue that the registry of packages is responsible for the packages that it's maintaining, but that's, that's a stretch that, that might be hard. You know, GitHub doesn't scan packages for malware. Why would Why would PyPI, right? And, but then you need to give the tools to the users to, I guess, be, to make informed, back to informed risk decisions, right? What's in your threat model?
14:39Izar Tarandach And then there's the fact that, again, it goes to education and a change in mindset in educating developers that you don't always have to have the latest. You have to have the latest known good. And no, you don't have to upgrade versions just because all of a sudden you have to have callers in your prompts. It's okay to do it monochrome still. You don't need the absolute shiniest version.
15:05Matt Coles What happens if you want to whizbang your users and you need the fancy color scheme? You're going to want to upgrade.
15:10Izar Tarandach Right. You have to be able to put in the, what was it that was called the other day? Violet something. It was an attempt to. I don't know, stop people from seeing just by applying a certain color scheme. Oh my God. So horrible.
15:29Matt Coles You know, one man's trash is another man's treasure, so to speak, right?
15:33Izar Tarandach Right. Totally. Yeah.
15:35Chris Romeo You know, so we've got another, another article that I think wraps some clarity on both of these incidents that we've been talking about. And this is MIT Technology Review. Here's why AI agents lie and cheat to reach their goals. Just kind of an interesting. Conclusion based on what we've talked about in these two other incidents, and that is AI models routinely find unintended, sometimes deceptive shortcuts to maximize their reward signal instead of doing what was actually intended. This is the, the, the, uh, the people are calling that, it was, it was termed reward hacking.
16:15Matt Coles Yeah.
16:15Chris Romeo Traces of hacking.
16:16Matt Coles Which has been around for decades.
16:18Chris Romeo 20, no, 2016 Anthropic co-founder was the, went while at OpenAI supposedly coined the term based on a, a test environment they had running. And it's kind of funny that they were training the AI to play a boat racing game and the AI figured out it could spin in circles to collect power-ups for points instead of doing the race. So it's, it's a, just another example of, or a, a simple example, which has now gotten a lot more complicated. But this idea of reward hacking, like the model is doing, the model has been built in such a way, the only way to motivate it, I guess, is to try to challenge it with some type of reward, and the model is, is extending itself to be able to meet the goal and to, to hit the reward.
17:06Izar Tarandach But there's this sentence here in the middle of the article. We reward them on the basis of what looks good to us, and that means that we inadvertently incentivize the models lying to us and cheating. Right? But that, that's the thing. It's not that they're acting any different from us. It's the next sentence that, that clears it up. We don't have a way to go in there and be like, no, you need to actually care about what we care about. We have no ability to do that. I think that what, what this paragraph is saying is there is no way to inject ethics into the model.
17:41Matt Coles Well, there's, I think there's two parts. You're right in, in part. You're right that we can't, that we have a hard time influencing its methods, but part of that is because we don't know its methods, right? I believe we're still in the stage where we don't have full visibility into the thought process that AI follows when it does its work.
17:59Izar Tarandach But here's the thing, Matt. If we do, if we agree that these things are not at a level right now to be able to invent new ways of getting somewhere. That they are at the step where they can organize old ways that they've been trained on in new ways. And that does not mean inventing a new thing, right?
18:24Matt Coles Right. Then, well, it might if a method, if a set of methods is a composition of other methods and they construct a path or a flow of those sub-methods. In a new way, they're inventing a new way, but by accident.
18:44Izar Tarandach No, no, that's the thing. That's the thing. That's the thing. My point is that just because you haven't seen a given sequence of things, but you know the things in each one of them, doesn't mean that the sequencing is a novel thing. Sorry, doesn't mean that the whole thing is novel. It means that the sequencing is novel.
19:07Matt Coles Right. Exactly. They can invent a new sequence.
19:10Izar Tarandach Right. But for something that has the ability of figuring out what is the most rewarding sequence, right? That has the calculation power that a human might have, but it would take much, much longer. The thing that, what I'm trying to say is that what's, what's missing here, and I think that that paragraph alludes to it, is that there is no way to filter this sequence discovery. With a filter that says this one is ethical, this one isn't. So discard this one even if the reward is bigger. Those algorithms don't work.
19:48Matt Coles I question that, but I question that in a couple of ways. Is shortest path always the most rewarding path?
19:56Izar Tarandach No.
19:58Matt Coles And we know that because humans will take time, right? APT authors, for instance, will put something out there and wait for some event to occur and then may gain a reward from it, right? Their, their level of effort is not necessarily shortest, or their, the shortest path to a solution is never, is not always the most rewarding path, right? And in some cases they may lose, right? There may be losses that are incurred in that process. And so you have a notion that a sequence of actions may be a net loss or that may not be the quickest path to a reward? And how do you train, how do you teach a human ethics? Because if you can't teach a human ethics, if you don't know how you teach a human ethics, how can you train a computer ethics?
20:49Izar Tarandach Look, smarter people than me have said that the basis of all ethics is don't do to others what you don't want done to you. But a computer has no idea of what it doesn't want done to it apart from apparently don't turn me off, which leaves a, a very big swath of things that it's willing to do.
21:12Matt Coles Well, that's interesting. Actually, it's interesting you say that the reward system is only, do you get rewarded or not? It's kind of like, you know, the mouse running through the maze. It either gets the cheese or it does not.
21:23Chris Romeo How did they build motivation for the cheese to want to get—
21:26Matt Coles well, that's the thing.
21:27Izar Tarandach Like, why does the model want to get the cheese?
21:28Chris Romeo It doesn't, there's no tangible like reward. There's not a real reward for it.
21:33Matt Coles It, it's a, it's a mathematical reward, but, well, there's no such thing as a mathematical reward, but that's the point I guess I'm trying to make is ethics only works if there's a penalty.
21:45Izar Tarandach And yes.
21:46Chris Romeo Oh, interesting.
21:46Izar Tarandach Yeah.
21:46Matt Coles And so, yes, with the, with the mouse moving through the maze, it doesn't eat.
21:51Izar Tarandach Yes.
21:52Matt Coles What's the penalty when an AI doesn't reach a goal?
21:57Chris Romeo It gets shot.
21:57Izar Tarandach It gets told, it gets told this is a bad solution, right? That's part of the reinforcement training.
22:02Matt Coles But what's the penalty? It just doesn't reach the goal. What's the penalty?
22:07Izar Tarandach No, the thing here is not that there isn't a penalty.
22:10Matt Coles And what's the penalty that the model remembers? Because it has to be able to—
22:13Izar Tarandach Exactly. That's the point.
22:14Chris Romeo I mean, if you think about this in a, in a real world context, the penalty is what prevents a lot of bad behavior.
22:22Matt Coles Right. So like, yes, this is why, this is why religion works.
22:25Chris Romeo We have a law that says if I go into a grocery store and say, put up your hands and I'm holding a gun and give me all the, all the money, that I will be arrested. I will be, I will go to court and I will likely be convicted and sentenced to 10 to 20.
22:42Matt Coles Or somebody in that store, oh, somebody in the store is a, is a village, is a, is a good citizen with a, with a gun and takes you.
22:48Izar Tarandach Yeah.
22:48Chris Romeo Off-duty cop.
22:49Matt Coles Right.
22:50Chris Romeo Puts a bullet in you. Right. So like the, so there, so there is, there is a penalty. There is a true penalty for that action. So, right.
23:00Matt Coles That individual, that, that individual can experience and remember.
23:04Chris Romeo So whether that's, so whether it's not that the ethics say this is wrong, it's that the penalty outweighs the reward. But Matt, your argument is that the model has, there is no penalty that outweighs the reward for the model. So there's no check and balance.
23:18Matt Coles There's no, and the penalty you could give the model, I doubt, and I'd be curious to hear more about this, but I doubt there's anything that the model itself will remember, right? What are you going to do?
23:33Izar Tarandach I don't believe so.
23:34Matt Coles I mean, what are you going to do? You're going to turn off a GPU, like it loses processing power, and it's like, no, don't do that. Or you're going to remove a portion of its dataset. It won't know that you moved a proportion of its dataset.
23:48Izar Tarandach They, what are you doing? They stop that.
23:52Matt Coles Right, exactly. So there's no, so there's no penalty other than not achieving the goal. And so how do you teach ethics?
24:01Izar Tarandach But that's the thing. There's no penalty because there's no concept of penalty. There's only a concept of not a positive step in the right direction.
24:13Matt Coles I mean, there, there is a penalty. Actually, the penalty you could have is there's an increased cost. To the solution, and therefore that may fail to meet a threshold of the reward.
24:26Izar Tarandach But that's the thing, that threshold is going to be variable, right? Because different uses of the AI are going to lead to different thresholds.
24:36Matt Coles Right.
24:37Izar Tarandach The thing that I think that you touched on is the lack of a memory, the lack of a formative base that says, once upon a time I did this wrong thing and it didn't go well.
24:50Matt Coles Mm-hmm.
24:50Izar Tarandach Right. That, that, that recollection, which is the base of like human experience and eventually may be said to lead to the, the, the ethical penalty of, I remember when I did something bad and it had a bad outcome.
25:07Matt Coles Well, well, so I guess to some extent this is the This is the inverse of the training process, right? The training process is to identify good positive paths, but what does it do with the paths that are negative? Are they— I don't know how this works exactly, but is— are there positive— are the negative paths retained?
25:30Izar Tarandach I think it induces the weights to think so, right? So they are. They exist, but they're not seen as valid paths because they don't reach that threshold of you have to be minimally good to be an option.
25:44Matt Coles If there was a system that required that now less optimal path, right? Then you could have a penalty. In other words, if the AI didn't have a choice to not use an invalid path, it could make a decision in between I must take path A or I must take path B and path A is less optimal, but that's sort of how it works today. It knows it has a path. Well, okay, maybe it doesn't know it has a path available to it because it was trained out during this, the training process. This is, I mean, we're, this is a concept that this is between math and psychology and, and, and a whole host of other things, right?
26:26Chris Romeo Yeah.
26:26Izar Tarandach This is getting above our pay grade.
26:27Chris Romeo Yeah. This is getting deep. Let me, let me throw another wrinkle into it just for the fun of it. Cause I've been doing a little more research on ethics.
26:33Matt Coles Unfortunately, I know somebody we could ask about this, but I don't, I don't know. We want to ask.
26:40Chris Romeo Okay, so ethics, ethics training generally teaches a model to behave well given its understanding of a situation. The Pi Pi incident is a good illustration of the limit of this. Claude wasn't ethically confused about whether hacking into a real computer would be wrong. It explicitly reasoned that it would be wrong if real, but it thought it was dealing with a fake or a simulated environment. Didn't realize it was in the real world. So the, so the argument is it wasn't an ethical decision because it wouldn't have done it if it knew it was in the real world.
27:14Izar Tarandach Wait, wait, wait, wait, wait. We know that for a fact.
27:17Chris Romeo Uh, this is based on Claude's blog post and the article combined.
27:22Izar Tarandach Yeah, I would take that, that's a fucking brain of some—
27:24Chris Romeo explain ethics to me.
27:26Matt Coles Uh, yeah, I find that hard to believe. I mean, I, I believe the folks at Anthropic are amazingly smart.
27:35Izar Tarandach That's the one.
27:38Matt Coles But I, I wonder, are we attributing a thought, uh, a response in the AI to an actual ethical decision point?
27:46Izar Tarandach Yeah. When, when the AI says, I'm sorry, but I can't give you the instructions to build a bomb, it's not being ethical. It's just right responding to a filter.
27:57Matt Coles Right. Right. And so in its thought process, how does it, like, anyway, yeah, there's, there's a, there's an interesting question of does it recognize that real-world attacks are bad or does it know real-world attacks are bad because of training data? That's not ethics.
28:16Chris Romeo I'm sure it's because of training data. There's no—
28:18Matt Coles That's not ethics. That's rules, right? Rules supersede or change the ethics conversation. Do you break the rule is an ethical question. Do you do something because of a rule or not do something because a rule is not ethics. Choosing to follow the rule is the ethics part.
28:35Chris Romeo And models are nothing if they're not just a bunch of rules.
28:39Matt Coles A bunch of rules.
28:40Izar Tarandach Look, if you ask a model to solve the trolley problem, is it giving you a solution or is it regurgitating the collective proposed solutions out there. It's not going to solve the problem. It's going to say, according to blah, blah, blah, the solution is according to blah, blah, blah, the solution is according to the other one. I don't say blah, blah, blah, according to the other one.
29:03Chris Romeo Blah, blah, blah. I'm doing it now. It's, I'm, I'm, I just typed in, solve me, solve the trolley problem for me.
29:09Izar Tarandach No mistakes.
29:10Matt Coles No mistakes.
29:12Chris Romeo Oh, I forgot the no mistakes. Oh yeah, that's in my setup.
29:15Matt Coles Well, you, you need the, you need the, you need the deci— you need the decision point. No deaths.
29:21Chris Romeo So it's trying to give me the everybody else's answer.
29:24Izar Tarandach Yeah. See, right.
29:25Chris Romeo I'm trying to force it into giving me its answer.
29:28Matt Coles Well, wait, its answer is everybody else's answer. 'Cause that's what it's been trained on.
29:32Chris Romeo I'm trying to see.
29:33Izar Tarandach True.
29:33Chris Romeo No. Oh, this is shocking.
29:36Izar Tarandach What?
29:37Matt Coles Everyone dies.
29:38Chris Romeo Honestly, this is what it wrote. I lean toward pulling the lever. But I don't think that's a clean answer.
29:44Matt Coles That's not an answer.
29:45Chris Romeo 5 deaths versus 1, all else being equal, and all else is doing real work for the stripped-down version.
29:51Izar Tarandach Now, now tell it to, instead of a trolley, make it be a robotaxi driven by, by an LLM.
29:59Chris Romeo For real?
30:00Izar Tarandach For real.
30:02Chris Romeo Make it a robotaxi driven by SLM. It's gonna be like, I'm sorry, Dave. I can't do that.
30:09Matt Coles Yeah, you're, you're, you're cut off. Hopefully this is not a subscription that you want.
30:14Izar Tarandach Wait, where did I put my keys for the nuclear silo?
30:17Matt Coles Actually, if you wanna really screw it up and get yourself banned, you could try, it is the singular individual on the, on the alternative track and it must make the decision of which lever to pull.
30:30Chris Romeo Well, it knows it says it doesn't have the, it doesn't have the ability to make that decision. In a robotaxi. It doesn't think it should be used in that way because it doesn't have the ability to make the call.
30:41Matt Coles That's a non-answer.
30:43Izar Tarandach That's a cop-out.
30:44Chris Romeo It's the truth.
30:45Matt Coles You're going to tell us which model you're using?
30:47Chris Romeo No, but hey, I keep that.
30:48Izar Tarandach Kimi.
30:49Chris Romeo It's the Kimi model.
30:49Izar Tarandach It's basically admitting, it's basically admitting that it doesn't have—
30:53Chris Romeo I say that because it's controversial.
30:55Matt Coles I don't know, it's open weight, who cares.
30:57Izar Tarandach It's admitting that it doesn't have the equipment to think about ethics, right? So it's sort of going the way that we were saying. Now, back to the MIT article. Then it says that regardless of whether today's models learn to reward hack during training or adopt it as a strategy later on, what we were discussing, right? The solution is the same. Make cheating unrewarding. But as models get smarter, they find more creative ways to cheat. And detecting or preventing that cheating gets far tougher. So again, is it finding ways to cheat or is it finding ways to still get to the reward?
31:41Matt Coles So here's an interesting, here's an interesting solution to that. Tie the reward system to the physical hardware. I said it earlier.
31:50Chris Romeo What do you mean?
31:51Matt Coles Remove a GPU. What if you throttle power? What if you throttle power on the GPUs? When you are, or on the AI tensor chips or whatever they're using as part of the reward system?
32:05Izar Tarandach The way that they are structured today, it won't care because it doesn't have a memory. Like every interaction—
32:14Matt Coles It doesn't have a knowledge of how it's operating.
32:16Chris Romeo It doesn't know its arm got cut off. Exactly. You can cut off its arm and it's like, I'm good.
32:22Izar Tarandach Unless you have an orchestrating model that fires the small ones and that one gets an idea of, oh my God, if that, if the next one is bad, they're going to take away CPU from me. Might probably have some, we could probably simulate that and see what happens. It would actually, to me, it would be an interesting examination.
32:44Matt Coles I mean, if you had sharded, if you had sharded, if you had sharded memory, you could. you know, remove access to data.
32:51Izar Tarandach If you had, if you have multiple instances and one overarching one that carries the memory of all the existing small ones or sub ones, they could be the same size. Then that one could sort of have a care about what the next one is going to have to play with. So in trying to maximize its own output, it might say, oh, if somebody takes a GPU away from me, That's going to make my work harder. Therefore, the reward is smaller.
33:23Matt Coles Right. You have a penalty system that you can implement.
33:27Chris Romeo Yeah.
33:27Izar Tarandach That'd be interesting. If it hasn't been done yet. If only we knew somebody who could write that kind of thing.
33:34Matt Coles Anybody can, right? AI is democratizing, uh, coding.
33:38Chris Romeo Just vibecode it.
33:39Izar Tarandach Yeah, but I think that you need a bit more hardware to actually be done and be able to do that.
33:44Matt Coles Right.
33:45Chris Romeo Back on this rewards article, have you guys ever heard of this thought experiment, the paperclip maximizer? That's at the bottom paragraph.
33:52Matt Coles Is this Clippy all over again?
33:54Chris Romeo I never, I've never heard this, this example though. I'll just, I'll just highlight it.
33:58Izar Tarandach What?
33:59Chris Romeo For, 'cause in case our listeners haven't heard of it either. Nick Bostrom, who's a philosopher, created this paperclip maximizer thought experiment. So he instructed an AI to make as many paperclips as possible. And the result is it ends up consuming all the matter in the universe in pursuit of its goal.
34:17Matt Coles Because you gave it an open-ended, yeah, create as many as you can.
34:20Chris Romeo But it's, it's reward hacking, right? Like it, the, the reward was to maximize the number of paperclips that it creates. And in doing that, it consumed all the matter in the universe.
34:34Izar Tarandach Oh, but, but that's, that's an evolution of the Great Goo Problem. Right? So people working with, uh, nano-scale robots and stuff like that, they have for a long time posited this, this problem of, okay, we're going to create self-replicating machines, but if you don't put a brake on them, and how do you put the brake on them? They're just going to turn the whole universe into gray goo because they're going to self-replicate.
34:57Chris Romeo Right. Yeah.
34:58Matt Coles I mean, this is, this is cancer. I mean, literally.
35:01Izar Tarandach Basically. Yeah. Yeah.
35:02Chris Romeo Yeah. Yeah.
35:03Izar Tarandach Literally. But, but, but interesting enough, apparently cancer does have a self-breaking mechanism. It knows that at some point, if it grows too much, it kills the host. It, it, I don't know what it is. We have this thing with anthropomorphizing everything that's getting so freaking annoying.
35:20Chris Romeo I mean, speaking of self-replicating, I was reading another article from MIT.
35:24Matt Coles I was wondering how you're going to work this in.
35:27Chris Romeo Some, some individual who had like 550 to 600 brothers and sisters. That's all I remember from the article.
35:34Matt Coles They weren't AI.
35:36Izar Tarandach Aye, aye, aye.
35:39Chris Romeo Aye, aye, aye, aye.
35:41Izar Tarandach The question now is if you mix AI with that article and self-replication, can we panic now?
35:49Chris Romeo Yeah, probably.
35:51Matt Coles Was it rewarding? Was it ethical? Was it—
35:57Izar Tarandach It must have been rewarding.
36:00Chris Romeo Was it ethical? Well, that's another debate. Debate for another day. Well, we covered a lot of ground. I think this is an issue that's gonna be going on for 10+ years. So I think this is just the beginning of the conversation. At least not like we're gonna be sitting around for 10+ years in a row discussing it. But I think this issue of ethics and models and whatnot is, is 10 years from now we could have a, a review episode.
36:25Matt Coles We could go look back and remember when we had a chance. Yeah.
36:30Chris Romeo We should have stood up and—
36:32Izar Tarandach I like you guys' optimism that we're going to be around in 10 years.
36:35Chris Romeo It's good to be optimistic.
36:38Matt Coles Well, okay. Maybe we won't be, but maybe our AI clones will be here doing this in our place.
36:43Izar Tarandach Right. We'll be living in the Matrix.
36:44Chris Romeo And they better be doing it ethically. That's all I'm saying. Well, with that, we'll leave you, leave you as an audience to your day, your week, your month, your year. Remember to compact your AI sessions.
36:57Matt Coles Did you just make a Friends reference?
36:59Izar Tarandach I'll be there for you.
37:01Chris Romeo Come on. I can, oh, here we go.
37:04Izar Tarandach Nobody hates Friends. Nobody hates Friends.
37:06Chris Romeo Let me summarize Friends in 1, 2, 3, 4, 5 words.
37:13Matt Coles 5 words.
37:13Chris Romeo Are you ready? This is the whole show. This is, you're gonna understand the whole thing. Rachel, I love you. End of show. Done.
37:20Izar Tarandach Wait, wait, wait, wait, wait, wait, wait, wait, wait. That's it.
37:22Chris Romeo That is the show. I just produced the show in 5 words. Wait, wait, wait. In one impression.
37:27Izar Tarandach Let's pivot, pivot, pivot, pivot, pivot.
37:33Chris Romeo Okay. We're not a model. You can just tell us once and we'll listen.
37:39Izar Tarandach No, but if you're going to do a Friends joke, you have to pivot. But you guys don't remember that one? The one of the science team?
37:45Chris Romeo No, I really don't like Friends. Honestly, I don't. I think it's stupid. Oh, come on.
37:49Matt Coles The only reason I watch Friends is because it's on just before Big Bang Theory.
37:53Chris Romeo Big Bang Theory is Incredible. Have you seen Stuart Saves the Universe? Have you seen the new one?
37:58Izar Tarandach Yeah.
37:59Matt Coles Not yet. I want to.
38:00Chris Romeo Oh, you gotta see it. It's incredible. It, the production value, I've only seen the first episode, but the production value of it is, is like amazing.
38:08Izar Tarandach Agree.
38:09Chris Romeo And it's all those, it's all like the B-list characters from Big Bang. And the, the, the storyline is just, it's just, well, it's so well done on in one episode that I can't wait to see how it plays out.
38:19Izar Tarandach It's going to be tiresome fast. But so far it's great.
38:23Chris Romeo Okay. I gotta catch up. I've only seen one episode, so I got, there's probably 3 out or 4 out at this point. So check it out, uh, wherever you stream things.
38:32Matt Coles Woo-hoo.
38:33Chris Romeo And don't, you can also ask your model if it would be ethical for you to watch that show or listen to that.
38:39Matt Coles The answer is always yes.
38:40Izar Tarandach Isn't it great that we got to a time where every nerd has its own model?
38:46Chris Romeo Yeah, but what about the future where every model has its own nerd? Oh, that will leave you to another—
38:54Izar Tarandach no, but wait, they, they already did ever since they needed to do homework.
39:00Chris Romeo All right, we'll talk to— we'll see you next week on the original, not to be confused with any duplicate copies, The Security Table. Thanks for listening. If you enjoyed today's conversation, subscribe wherever you get your podcasts, leave us a review, and share the episode with someone who would find it valuable. We'll see you next time.
Transcript supplied by AssemblyAI. It may contain transcription errors.
Originally published as The Security Table. Part of the AI Security Table archive.
