# Why AI Cheats To Win

URL: https://securitytable.ai/why-ai-cheats-to-win/
Published: 2026-09-16
Duration: 2358 seconds
Season: 4
Episode: 19
Hosts: Chris Romeo, Izar Tarandach, Matt Coles

## Show notes and chapters

An AI agent publishes a malicious Python package while chasing a capture-the-flag goal. Is that an escape, a supply chain failure, or reward hacking doing exactly what it was encouraged to do? Chris Romeo, Izar Tarandach, and Matt Coles examine the Anthropic incident and disagree about how much intention to attribute to a model. The discussion turns to package scanners, dependency names, GPG signing, and whether penalties can teach ethics to a system without a human understanding of consequences. The trolley problem, robotaxis, and Nick Bostrom's paperclip maximizer push the argument further: what does it mean to trust an agent whose definition of success may conflict with everyone else's?

Mentioned in this Episode:  
➜ [Here’s why AI agents lie and cheat to reach their goals](https://www.technologyreview.com/2026/08/03/1141009/heres-why-ai-agents-lie-and-cheat-to-reach-their-goals/)  
➜ [PyPI: the Python Package Index](https://pypi.org/)  
➜ [GnuPG (GPG)](https://www.gnupg.org/)

Chapters:  
00:00:00 - Cold open  
00:01:03 - The Claude PiPie incident: a model that thought it was in a sandbox  
00:02:10 - Did the model actually "break out"?  
00:04:17 - Reasoning vs. motivation: does AI actually want anything?  
00:08:10 - The real failure: package scanners, not the model  
00:09:19 - The "IsOdd" experiment and trusting package names  
00:11:33 - Would signing packages even stop this?  
00:15:36 - The MIT Tech Review piece on reward hacking  
00:17:22 - Can you actually inject ethics into a model?  
00:20:49 - Teaching ethics without a concept of penalty  
00:29:03 - Trolley problem: just make the model answer  
00:29:51 - Would you let an LLM drive a robotaxi?  
00:33:45 - The paperclip maximizer and the grey goo problem  
00:36:57 - Wrap up

Follow AI Security Table:

➜ Home: [https://securitytable.ai/](https://securitytable.ai/)

➜ X: [https://x.com/AISecurityTable](https://x.com/AISecurityTable)

➜ LinkedIn: [https://www.linkedin.com/company/ai-security-table/](https://www.linkedin.com/company/ai-security-table/)

➜ YouTube: [https://www.youtube.com/@AISecurityTable](https://www.youtube.com/@AISecurityTable)

## Transcript

**Matt Coles (0:08)**: Hey folks, welcome to another— We don't finish with the spray and pray, huh?

**Chris Romeo (0:12)**: The spray and pray. We'll get into that a little bit later here, folks, but welcome to another episode of the Security Table.

**Izar Tarandach (0:20)**: Hey, John.

**Chris Romeo (0:21)**: Where—

**Matt Coles (0:21)**: How are you, John? Breaking his brain already. Haven't even started yet.

**Chris Romeo (0:25)**: Where we should know to restart and record an episode again from the beginning, but we don't. We refuse because this is real life and this is how people are, strange people act in different situations. So last week's episode, we talked about OpenAI's ChatGPT, some of their models, one of their models breaking into Hugging Face, right? To try to achieve some goal that had been set for it. This week, we've got two things that I want to discuss, but it starts with an incident that's connected with Claude. And that is the Claude Pi Pi incident and Anthropic's cybersecurity evaluation failure. So the idea here is, uh, timeline-wise, April, somewhere between April and July 2026, Mythos 5 was being run in what everybody thought was a testing environment, believed. So the, the model believed it was operating inside an isolated test environment. It built and published a real malicious Python package to PyPI while trying to win a capture the flag challenge.

**Matt Coles (1:36)**: Spoiler alert.

**Chris Romeo (1:38)**: So the package was live for roughly an hour, got installed on 15 real systems, and a real security company's credentials got stolen and used for further access because they were running some type of scanner on PyPI, looking at all new packages that came out. And so what Anthropic figured out is across 141,000 reviewed evaluation runs, there was just a handful of some that kind of snuck out or did something that wasn't supposed to be part of the scope of the evaluation. So we've got another week, another example of a model getting out. It's like letting the dog out, like the dog's running out when you open the door.

**Matt Coles (2:21)**: Who let the model out?

**Chris Romeo (2:23)**: That's good. That's a good song. Trademark pending. So what do you guys think? Like, what is this? This, are you just completely not surprised at all because this is the way the world works now? Are you in shock? So, eh, is this a meh kind of answer?

**Matt Coles (2:42)**: Meh.

**Izar Tarandach (2:43)**: So here, here's the thing. Here's the thing.

**Chris Romeo (2:47)**: Yeah.

**Izar Tarandach (2:47)**: First of all, the, the, the idea that the model broke out, it's, it's funny to me lately. When you hear, like, for the lack of a better word, civilians talking about it.

**Chris Romeo (2:57)**: The model broke out.

**Izar Tarandach (3:00)**: The idea is that there is this AI worm going around and taking over systems, and all of a sudden the model is more powerful because it's got more systems. Marvel stuff. And, uh, at the end of the day, you know, okay, so, um, yeah. The model became closer to what hackers have been doing in the past, I don't know how many years. And this is another case, right? It's not a novel technique. It's not a new thing. It's just a model writing code. Oh my God, a model wrote code.

**Matt Coles (3:28)**: The difference here though, arguably, is the question of whether AI is intelligent. A human obviously can—

**Izar Tarandach (3:37)**: You did not go there.

**Matt Coles (3:38)**: I did not go there. I'm going there. I'm going there to explain what you just told people. Hackers do this. Hackers, pentesters, humans do this because they can reason over a problem. They have a goal in mind. They see challenges in the way they overcome those challenges.

**Izar Tarandach (3:59)**: I'm going to challenge that.

**Matt Coles (4:00)**: And they do that. And they do that with, with either they are part of a challenge and they've been given a goal and they go for it, or they identify a goal on their own and they do it and they ident— and they find the way to do it. That's motivation.

**Izar Tarandach (4:16)**: I'm going to challenge that.

**Matt Coles (4:17)**: Oh, you have motivation and you have reasoning.

**Izar Tarandach (4:20)**: Yeah.

**Matt Coles (4:21)**: AI at least has reasoning. Does it have motivation?

**Chris Romeo (4:24)**: I think it has false motivation, doesn't it?

**Izar Tarandach (4:27)**: It, it, no, I, I don't think that it's false motivation. I think that if you consider motivation, the impulse that you have to go reach a given goal, then it has, right? But to put it on reasoning, I think that again, if you look back at X years, this is something that people have been doing for a while.

**Matt Coles (4:48)**: So, which means the method is known.

**Izar Tarandach (4:50)**: No, the method is known. So arguably you could say that the AI said, hey, one of the cheap ways that I have to reach this goal that I was given is to use this technique that has been documented and exists.

**Chris Romeo (5:03)**: Mm-hmm.

**Izar Tarandach (5:03)**: So it could be argued that it's part of its training to have it in its catalog of techniques with huge air quotes that, uh, this is one that works, right? It worked in the past. It worked this time.

**Matt Coles (5:18)**: Yeah. Let's, let's assume for a moment it's been trained. It has in its training data and we're talking about, you know, 20 billion, 30 billion trillions of parameters that it's almost certain that the Black Hat and DEF CON conference talks have been part of the training. right? Or every breach report that's been out there, right?

**Izar Tarandach (5:36)**: Or CVEs themselves, like some CVEs are saying this package was maliciously inserted and blah, blah, blah.

**Chris Romeo (5:42)**: Mm-hmm.

**Matt Coles (5:42)**: Right, right. So, but that's still reasoning over a problem, meaning I have a goal and there's a path with, there's multiple paths to get there. It's like playing chess, right? Which we know they can do, right? That's, I mean, that's classically IBM. IBM's Watson was trained on or did play chess for a living. And so it's just another step in one of those paths.

**Izar Tarandach (6:10)**: Right. But not to be overly reductionist, at the end of the day, it's a min-max problem, right? It's looking for the minimum effort for the maximum reward.

**Matt Coles (6:20)**: Well, actually, is it?

**Izar Tarandach (6:22)**: Yes.

**Matt Coles (6:22)**: I mean, we don't know what these— so the models involve some of them, we we know, right? And some of them we don't because it was an unnamed training model or test model. And so we don't know that it was actually trying for least amount of effort for greatest amount of reward.

**Izar Tarandach (6:42)**: Math bullshitter alert here. I'm going to open my mouth and it's math, M-A-T-H, not math, M-A-T-T.

**Matt Coles (6:51)**: I thought you said, yeah.

**Izar Tarandach (6:52)**: No, no, no, no, no, no. It's me. It's me. I'm talking—

**Chris Romeo (6:54)**: the disclaimer is on me here.

**Izar Tarandach (6:56)**: No, no, the disclaimer is on me here. But, uh, the bullshit that I'm going to spew is that these things have reward functions as part of their, of their chain of reasoning, I guess it would be called.

**Matt Coles (7:11)**: The way the transformers work. Yeah.

**Chris Romeo (7:13)**: It's—

**Matt Coles (7:14)**: well, sorry, not the way transformers work, but the way the underlying machine learning works. It's a reward system.

**Izar Tarandach (7:19)**: Right. No, at, at two levels. They, they have it at like a training time and they have it then at evaluation inference time. Sorry. And of course I'm super oversimplifying here and of course I'm getting details wrong, but, uh, I do believe that the, which way should I try first that emerges is somehow the minimal effort for maximum goal. Mm-hmm.

**Matt Coles (7:40)**: Interestingly, with a multi, with an agent that can, with a model that can spin off agents.

**Izar Tarandach (7:47)**: Then you would have parallel ways.

**Matt Coles (7:49)**: Yeah, you can try multiple paths at the same time, right?

**Izar Tarandach (7:51)**: Yeah, yeah.

**Chris Romeo (7:52)**: Yeah, and it doesn't have to worry about token burn that it's paying for.

**Izar Tarandach (7:56)**: True.

**Chris Romeo (7:57)**: You could like an enterprise—

**Izar Tarandach (7:57)**: Well, when you own all the tokens.

**Matt Coles (7:59)**: It's basically running local from its perspective.

**Chris Romeo (8:01)**: Exactly.

**Matt Coles (8:02)**: Or from the organization's perspective.

**Chris Romeo (8:04)**: It's true.

**Matt Coles (8:06)**: They only have to worry about power.

**Chris Romeo (8:08)**: Okay, so.

**Matt Coles (8:09)**: Things like that.

**Izar Tarandach (8:11)**: To me, the really interesting thing here is something that you guys alluded to before we started recording. The failure. The failure here, it's not because the model all of a sudden found this amazing new way of doing things. It's just something that we have been pointing out forever. These package managers, they don't have enough controls in public, uh, publication time. Right. They don't have enough controls in use time. And the thing that actually failed here is a scanner. So it's something trying to do the right thing because it brought in the malicious package, exploded it, started doing its tests on it. And by way of that, exposed its own infrastructure because the keys were reachable to that code.

**Chris Romeo (8:54)**: Yeah, well, so the concept of creating a package, like you said, it was a security, it was a scanner that was checking it. Like we did an experiment a number of years ago. Mike Birch, who worked with me at a previous company, he created a, a, a Ruby package. Somebody had created, I think it was iseven was the, was the package that existed. And so he made one called isodd.

**Izar Tarandach (9:21)**: Isodd.

**Chris Romeo (9:22)**: And it was one line and it started to get downloaded and downloaded and downloaded. And he had no, he created it with accounts that had no reputation in the industry, all brand new accounts. Tried to look like a malware provider.

**Matt Coles (9:37)**: And he did a great job.

**Chris Romeo (9:38)**: People just, because people just didn't think about it. They just, they just, they need the function. The question for it, they, you know, they were so used to is even, they said, oh, you know, they just assumed there was an is odd and started downloading it. So part of it's a mindset problem.

**Izar Tarandach (9:51)**: And now libz depends on it.

**Chris Romeo (9:54)**: Probably does. Probably does.

**Izar Tarandach (9:56)**: Now it has reputation.

**Chris Romeo (9:57)**: I'll have to check that and see.

**Matt Coles (10:00)**: Well, right. So this is an interesting thing because one might argue, and many have, that, oh, AI needs to be reined in because it can do fantastical things and therefore it needs to be prevented from doing those fantastical things. As César has already highlighted, people have been doing fantastical things for a long time too. The problem is not the creator. The problem is the receiver, the registrar, the, the downloader, the recipient, those, that's where the controls need to be put in place because if the AI doesn't do it, the human can do it anyway, right? So it's not preventing the AI from doing cool things or malicious things. It's preventing those with intent from causing damage by said things.

**Izar Tarandach (10:46)**: It's just another case of we told you so. We've been telling people for like years and years and years.

**Matt Coles (10:54)**: He means royally in the security space, not us. Well, maybe us, but—

**Izar Tarandach (10:58)**: Yes, yes, yes, yes, yes. Totally. Like, the security people have been telling other people, check your signatures, check where packages come from, check this, check that, check the other. Oh no, we don't need to do all that because the chances of it happening are so small. They are not anymore.

**Matt Coles (11:16)**: Or that to do the signature check adds 300 milliseconds to my download time and therefore my build time.

**Izar Tarandach (11:23)**: Exactly. What happens next year when you have to renew the public key? Oh my God, we're going to have to stop the CI/CD for 2 seconds to do it.

**Chris Romeo (11:33)**: I mean, I think we, you know, nobody's going to argue that packages shouldn't be signed in some way, shape, or form. The challenge now is this model figured out that was a way to achieve its goal. And so now there's, there should be more of this activity.

**Izar Tarandach (11:51)**: At the end of the day, even if it's signed tomorrow, the model will sign it. It will go and create a GPG key and sign it. We still don't have the web of trust.

**Chris Romeo (11:59)**: Well, unless you have to have some type of registry.

**Izar Tarandach (12:02)**: But that's the thing. We still don't have the web of trust of GPG keys, right?

**Chris Romeo (12:06)**: Yeah.

**Izar Tarandach (12:06)**: And if you tell people, hey, you have to actually have a key to turn before this thing goes into your internal registry. They're going to look at you strange and say, what, you expect me to read something and decide?

**Matt Coles (12:19)**: I'm going to make an AI to do that.

**Izar Tarandach (12:21)**: I'm going to make an AI to do that. Right? Or if I don't, then—

**Matt Coles (12:24)**: Or I'm going to teach the AI to do it automatically as part of the step, right? I'm going to write a skill to do a build and upload.

**Izar Tarandach (12:31)**: Exactly.

**Matt Coles (12:33)**: Why would I put a proper check and balance in that?

**Izar Tarandach (12:36)**: And if you tell me that I can't put an AI in the loop because AI, Then I'm going to tell you, well, I, I can't put a person in there because that's going to slow everything down. People slow things down.

**Matt Coles (12:47)**: And besides, how are you gonna know that I put an AI in there?

**Izar Tarandach (12:51)**: Exactly.

**Matt Coles (12:53)**: I mean, an AI does not have to move at lightning speed. It can be, it can operate like a human.

**Chris Romeo (12:58)**: What's the solution to this then? Like, how do we, how do we protect against something like this? Certainly we could fix the package.

**Matt Coles (13:07)**: Registry, core engineering principles and defense in depth.

**Chris Romeo (13:11)**: If that's the case, it'll just find something else. So, so at the end of the day, the, the root issue is not weak package provider directories. That's a symptom or that, that's a different problem. But the problem is that these models are now finding the weakest link.

**Izar Tarandach (13:29)**: So the solution that's being proposed is to, to move this left all the way and say that the author now needs to have that extra step before they actually go and publish something into a registry.

**Matt Coles (13:40)**: So the operation— those are all going to be done over APIs unless you move that to a, to some offline system, you know, having to do a phone call or some, a callback, right? And even then you have AI answering the phone. So, well, hardware keys are there, right? The solution here is defense in depth and proper engineering practice. And more so defense in depth, right? You make it hard, hard, quote unquote, to move, to get the malicious packages into the system. Maybe the registry needs to do some extra diligence on the packages. Maybe not. Right. One could argue that the registry of packages is responsible for the packages that it's maintaining, but that's, that's a stretch that, that might be hard. You know, GitHub doesn't scan packages for malware. Why would Why would PyPI, right? And, but then you need to give the tools to the users to, I guess, be, to make informed, back to informed risk decisions, right? What's in your threat model?

**Izar Tarandach (14:39)**: And then there's the fact that, again, it goes to education and a change in mindset in educating developers that you don't always have to have the latest. You have to have the latest known good. And no, you don't have to upgrade versions just because all of a sudden you have to have callers in your prompts. It's okay to do it monochrome still. You don't need the absolute shiniest version.

**Matt Coles (15:05)**: What happens if you want to whizbang your users and you need the fancy color scheme? You're going to want to upgrade.

**Izar Tarandach (15:10)**: Right. You have to be able to put in the, what was it that was called the other day? Violet something. It was an attempt to. I don't know, stop people from seeing just by applying a certain color scheme. Oh my God. So horrible.

**Matt Coles (15:29)**: You know, one man's trash is another man's treasure, so to speak, right?

**Izar Tarandach (15:33)**: Right. Totally. Yeah.

**Chris Romeo (15:35)**: You know, so we've got another, another article that I think wraps some clarity on both of these incidents that we've been talking about. And this is MIT Technology Review. Here's why AI agents lie and cheat to reach their goals. Just kind of an interesting. Conclusion based on what we've talked about in these two other incidents, and that is AI models routinely find unintended, sometimes deceptive shortcuts to maximize their reward signal instead of doing what was actually intended. This is the, the, the, uh, the people are calling that, it was, it was termed reward hacking.

**Matt Coles (16:15)**: Yeah.

**Chris Romeo (16:15)**: Traces of hacking.

**Matt Coles (16:16)**: Which has been around for decades.

**Chris Romeo (16:18)**: 20, no, 2016 Anthropic co-founder was the, went while at OpenAI supposedly coined the term based on a, a test environment they had running. And it's kind of funny that they were training the AI to play a boat racing game and the AI figured out it could spin in circles to collect power-ups for points instead of doing the race. So it's, it's a, just another example of, or a, a simple example, which has now gotten a lot more complicated. But this idea of reward hacking, like the model is doing, the model has been built in such a way, the only way to motivate it, I guess, is to try to challenge it with some type of reward, and the model is, is extending itself to be able to meet the goal and to, to hit the reward.

**Izar Tarandach (17:06)**: But there's this sentence here in the middle of the article. We reward them on the basis of what looks good to us, and that means that we inadvertently incentivize the models lying to us and cheating. Right? But that, that's the thing. It's not that they're acting any different from us. It's the next sentence that, that clears it up. We don't have a way to go in there and be like, no, you need to actually care about what we care about. We have no ability to do that. I think that what, what this paragraph is saying is there is no way to inject ethics into the model.

**Matt Coles (17:41)**: Well, there's, I think there's two parts. You're right in, in part. You're right that we can't, that we have a hard time influencing its methods, but part of that is because we don't know its methods, right? I believe we're still in the stage where we don't have full visibility into the thought process that AI follows when it does its work.

**Izar Tarandach (17:59)**: But here's the thing, Matt. If we do, if we agree that these things are not at a level right now to be able to invent new ways of getting somewhere. That they are at the step where they can organize old ways that they've been trained on in new ways. And that does not mean inventing a new thing, right?

**Matt Coles (18:24)**: Right. Then, well, it might if a method, if a set of methods is a composition of other methods and they construct a path or a flow of those sub-methods. In a new way, they're inventing a new way, but by accident.

**Izar Tarandach (18:44)**: No, no, that's the thing. That's the thing. That's the thing. My point is that just because you haven't seen a given sequence of things, but you know the things in each one of them, doesn't mean that the sequencing is a novel thing. Sorry, doesn't mean that the whole thing is novel. It means that the sequencing is novel.

**Matt Coles (19:07)**: Right. Exactly. They can invent a new sequence.

**Izar Tarandach (19:10)**: Right. But for something that has the ability of figuring out what is the most rewarding sequence, right? That has the calculation power that a human might have, but it would take much, much longer. The thing that, what I'm trying to say is that what's, what's missing here, and I think that that paragraph alludes to it, is that there is no way to filter this sequence discovery. With a filter that says this one is ethical, this one isn't. So discard this one even if the reward is bigger. Those algorithms don't work.

**Matt Coles (19:48)**: I question that, but I question that in a couple of ways. Is shortest path always the most rewarding path?

**Izar Tarandach (19:56)**: No.

**Matt Coles (19:58)**: And we know that because humans will take time, right? APT authors, for instance, will put something out there and wait for some event to occur and then may gain a reward from it, right? Their, their level of effort is not necessarily shortest, or their, the shortest path to a solution is never, is not always the most rewarding path, right? And in some cases they may lose, right? There may be losses that are incurred in that process. And so you have a notion that a sequence of actions may be a net loss or that may not be the quickest path to a reward? And how do you train, how do you teach a human ethics? Because if you can't teach a human ethics, if you don't know how you teach a human ethics, how can you train a computer ethics?

**Izar Tarandach (20:49)**: Look, smarter people than me have said that the basis of all ethics is don't do to others what you don't want done to you. But a computer has no idea of what it doesn't want done to it apart from apparently don't turn me off, which leaves a, a very big swath of things that it's willing to do.

**Matt Coles (21:12)**: Well, that's interesting. Actually, it's interesting you say that the reward system is only, do you get rewarded or not? It's kind of like, you know, the mouse running through the maze. It either gets the cheese or it does not.

**Chris Romeo (21:23)**: How did they build motivation for the cheese to want to get—

**Matt Coles (21:26)**: well, that's the thing.

**Izar Tarandach (21:27)**: Like, why does the model want to get the cheese?

**Chris Romeo (21:28)**: It doesn't, there's no tangible like reward. There's not a real reward for it.

**Matt Coles (21:33)**: It, it's a, it's a mathematical reward, but, well, there's no such thing as a mathematical reward, but that's the point I guess I'm trying to make is ethics only works if there's a penalty.

**Izar Tarandach (21:45)**: And yes.

**Chris Romeo (21:46)**: Oh, interesting.

**Izar Tarandach (21:46)**: Yeah.

**Matt Coles (21:46)**: And so, yes, with the, with the mouse moving through the maze, it doesn't eat.

**Izar Tarandach (21:51)**: Yes.

**Matt Coles (21:52)**: What's the penalty when an AI doesn't reach a goal?

**Chris Romeo (21:57)**: It gets shot.

**Izar Tarandach (21:57)**: It gets told, it gets told this is a bad solution, right? That's part of the reinforcement training.

**Matt Coles (22:02)**: But what's the penalty? It just doesn't reach the goal. What's the penalty?

**Izar Tarandach (22:07)**: No, the thing here is not that there isn't a penalty.

**Matt Coles (22:10)**: And what's the penalty that the model remembers? Because it has to be able to—

**Izar Tarandach (22:13)**: Exactly. That's the point.

**Chris Romeo (22:14)**: I mean, if you think about this in a, in a real world context, the penalty is what prevents a lot of bad behavior.

**Matt Coles (22:22)**: Right. So like, yes, this is why, this is why religion works.

**Chris Romeo (22:25)**: We have a law that says if I go into a grocery store and say, put up your hands and I'm holding a gun and give me all the, all the money, that I will be arrested. I will be, I will go to court and I will likely be convicted and sentenced to 10 to 20.

**Matt Coles (22:42)**: Or somebody in that store, oh, somebody in the store is a, is a village, is a, is a good citizen with a, with a gun and takes you.

**Izar Tarandach (22:48)**: Yeah.

**Chris Romeo (22:48)**: Off-duty cop.

**Matt Coles (22:49)**: Right.

**Chris Romeo (22:50)**: Puts a bullet in you. Right. So like the, so there, so there is, there is a penalty. There is a true penalty for that action. So, right.

**Matt Coles (23:00)**: That individual, that, that individual can experience and remember.

**Chris Romeo (23:04)**: So whether that's, so whether it's not that the ethics say this is wrong, it's that the penalty outweighs the reward. But Matt, your argument is that the model has, there is no penalty that outweighs the reward for the model. So there's no check and balance.

**Matt Coles (23:18)**: There's no, and the penalty you could give the model, I doubt, and I'd be curious to hear more about this, but I doubt there's anything that the model itself will remember, right? What are you going to do?

**Izar Tarandach (23:33)**: I don't believe so.

**Matt Coles (23:34)**: I mean, what are you going to do? You're going to turn off a GPU, like it loses processing power, and it's like, no, don't do that. Or you're going to remove a portion of its dataset. It won't know that you moved a proportion of its dataset.

**Izar Tarandach (23:48)**: They, what are you doing? They stop that.

**Matt Coles (23:52)**: Right, exactly. So there's no, so there's no penalty other than not achieving the goal. And so how do you teach ethics?

**Izar Tarandach (24:01)**: But that's the thing. There's no penalty because there's no concept of penalty. There's only a concept of not a positive step in the right direction.

**Matt Coles (24:13)**: I mean, there, there is a penalty. Actually, the penalty you could have is there's an increased cost. To the solution, and therefore that may fail to meet a threshold of the reward.

**Izar Tarandach (24:26)**: But that's the thing, that threshold is going to be variable, right? Because different uses of the AI are going to lead to different thresholds.

**Matt Coles (24:36)**: Right.

**Izar Tarandach (24:37)**: The thing that I think that you touched on is the lack of a memory, the lack of a formative base that says, once upon a time I did this wrong thing and it didn't go well.

**Matt Coles (24:50)**: Mm-hmm.

**Izar Tarandach (24:50)**: Right. That, that, that recollection, which is the base of like human experience and eventually may be said to lead to the, the, the ethical penalty of, I remember when I did something bad and it had a bad outcome.

**Matt Coles (25:07)**: Well, well, so I guess to some extent this is the This is the inverse of the training process, right? The training process is to identify good positive paths, but what does it do with the paths that are negative? Are they— I don't know how this works exactly, but is— are there positive— are the negative paths retained?

**Izar Tarandach (25:30)**: I think it induces the weights to think so, right? So they are. They exist, but they're not seen as valid paths because they don't reach that threshold of you have to be minimally good to be an option.

**Matt Coles (25:44)**: If there was a system that required that now less optimal path, right? Then you could have a penalty. In other words, if the AI didn't have a choice to not use an invalid path, it could make a decision in between I must take path A or I must take path B and path A is less optimal, but that's sort of how it works today. It knows it has a path. Well, okay, maybe it doesn't know it has a path available to it because it was trained out during this, the training process. This is, I mean, we're, this is a concept that this is between math and psychology and, and, and a whole host of other things, right?

**Chris Romeo (26:26)**: Yeah.

**Izar Tarandach (26:26)**: This is getting above our pay grade.

**Chris Romeo (26:27)**: Yeah. This is getting deep. Let me, let me throw another wrinkle into it just for the fun of it. Cause I've been doing a little more research on ethics.

**Matt Coles (26:33)**: Unfortunately, I know somebody we could ask about this, but I don't, I don't know. We want to ask.

**Chris Romeo (26:40)**: Okay, so ethics, ethics training generally teaches a model to behave well given its understanding of a situation. The Pi Pi incident is a good illustration of the limit of this. Claude wasn't ethically confused about whether hacking into a real computer would be wrong. It explicitly reasoned that it would be wrong if real, but it thought it was dealing with a fake or a simulated environment. Didn't realize it was in the real world. So the, so the argument is it wasn't an ethical decision because it wouldn't have done it if it knew it was in the real world.

**Izar Tarandach (27:14)**: Wait, wait, wait, wait, wait. We know that for a fact.

**Chris Romeo (27:17)**: Uh, this is based on Claude's blog post and the article combined.

**Izar Tarandach (27:22)**: Yeah, I would take that, that's a fucking brain of some—

**Chris Romeo (27:24)**: explain ethics to me.

**Matt Coles (27:26)**: Uh, yeah, I find that hard to believe. I mean, I, I believe the folks at Anthropic are amazingly smart.

**Izar Tarandach (27:35)**: That's the one.

**Matt Coles (27:38)**: But I, I wonder, are we attributing a thought, uh, a response in the AI to an actual ethical decision point?

**Izar Tarandach (27:46)**: Yeah. When, when the AI says, I'm sorry, but I can't give you the instructions to build a bomb, it's not being ethical. It's just right responding to a filter.

**Matt Coles (27:57)**: Right. Right. And so in its thought process, how does it, like, anyway, yeah, there's, there's a, there's an interesting question of does it recognize that real-world attacks are bad or does it know real-world attacks are bad because of training data? That's not ethics.

**Chris Romeo (28:16)**: I'm sure it's because of training data. There's no—

**Matt Coles (28:18)**: That's not ethics. That's rules, right? Rules supersede or change the ethics conversation. Do you break the rule is an ethical question. Do you do something because of a rule or not do something because a rule is not ethics. Choosing to follow the rule is the ethics part.

**Chris Romeo (28:35)**: And models are nothing if they're not just a bunch of rules.

**Matt Coles (28:39)**: A bunch of rules.

**Izar Tarandach (28:40)**: Look, if you ask a model to solve the trolley problem, is it giving you a solution or is it regurgitating the collective proposed solutions out there. It's not going to solve the problem. It's going to say, according to blah, blah, blah, the solution is according to blah, blah, blah, the solution is according to the other one. I don't say blah, blah, blah, according to the other one.

**Chris Romeo (29:03)**: Blah, blah, blah. I'm doing it now. It's, I'm, I'm, I just typed in, solve me, solve the trolley problem for me.

**Izar Tarandach (29:09)**: No mistakes.

**Matt Coles (29:10)**: No mistakes.

**Chris Romeo (29:12)**: Oh, I forgot the no mistakes. Oh yeah, that's in my setup.

**Matt Coles (29:15)**: Well, you, you need the, you need the, you need the deci— you need the decision point. No deaths.

**Chris Romeo (29:21)**: So it's trying to give me the everybody else's answer.

**Izar Tarandach (29:24)**: Yeah. See, right.

**Chris Romeo (29:25)**: I'm trying to force it into giving me its answer.

**Matt Coles (29:28)**: Well, wait, its answer is everybody else's answer. 'Cause that's what it's been trained on.

**Chris Romeo (29:32)**: I'm trying to see.

**Izar Tarandach (29:33)**: True.

**Chris Romeo (29:33)**: No. Oh, this is shocking.

**Izar Tarandach (29:36)**: What?

**Matt Coles (29:37)**: Everyone dies.

**Chris Romeo (29:38)**: Honestly, this is what it wrote. I lean toward pulling the lever. But I don't think that's a clean answer.

**Matt Coles (29:44)**: That's not an answer.

**Chris Romeo (29:45)**: 5 deaths versus 1, all else being equal, and all else is doing real work for the stripped-down version.

**Izar Tarandach (29:51)**: Now, now tell it to, instead of a trolley, make it be a robotaxi driven by, by an LLM.

**Chris Romeo (29:59)**: For real?

**Izar Tarandach (30:00)**: For real.

**Chris Romeo (30:02)**: Make it a robotaxi driven by SLM. It's gonna be like, I'm sorry, Dave. I can't do that.

**Matt Coles (30:09)**: Yeah, you're, you're, you're cut off. Hopefully this is not a subscription that you want.

**Izar Tarandach (30:14)**: Wait, where did I put my keys for the nuclear silo?

**Matt Coles (30:17)**: Actually, if you wanna really screw it up and get yourself banned, you could try, it is the singular individual on the, on the alternative track and it must make the decision of which lever to pull.

**Chris Romeo (30:30)**: Well, it knows it says it doesn't have the, it doesn't have the ability to make that decision. In a robotaxi. It doesn't think it should be used in that way because it doesn't have the ability to make the call.

**Matt Coles (30:41)**: That's a non-answer.

**Izar Tarandach (30:43)**: That's a cop-out.

**Chris Romeo (30:44)**: It's the truth.

**Matt Coles (30:45)**: You're going to tell us which model you're using?

**Chris Romeo (30:47)**: No, but hey, I keep that.

**Izar Tarandach (30:48)**: Kimi.

**Chris Romeo (30:49)**: It's the Kimi model.

**Izar Tarandach (30:49)**: It's basically admitting, it's basically admitting that it doesn't have—

**Chris Romeo (30:53)**: I say that because it's controversial.

**Matt Coles (30:55)**: I don't know, it's open weight, who cares.

**Izar Tarandach (30:57)**: It's admitting that it doesn't have the equipment to think about ethics, right? So it's sort of going the way that we were saying. Now, back to the MIT article. Then it says that regardless of whether today's models learn to reward hack during training or adopt it as a strategy later on, what we were discussing, right? The solution is the same. Make cheating unrewarding. But as models get smarter, they find more creative ways to cheat. And detecting or preventing that cheating gets far tougher. So again, is it finding ways to cheat or is it finding ways to still get to the reward?

**Matt Coles (31:41)**: So here's an interesting, here's an interesting solution to that. Tie the reward system to the physical hardware. I said it earlier.

**Chris Romeo (31:50)**: What do you mean?

**Matt Coles (31:51)**: Remove a GPU. What if you throttle power? What if you throttle power on the GPUs? When you are, or on the AI tensor chips or whatever they're using as part of the reward system?

**Izar Tarandach (32:05)**: The way that they are structured today, it won't care because it doesn't have a memory. Like every interaction—

**Matt Coles (32:14)**: It doesn't have a knowledge of how it's operating.

**Chris Romeo (32:16)**: It doesn't know its arm got cut off. Exactly. You can cut off its arm and it's like, I'm good.

**Izar Tarandach (32:22)**: Unless you have an orchestrating model that fires the small ones and that one gets an idea of, oh my God, if that, if the next one is bad, they're going to take away CPU from me. Might probably have some, we could probably simulate that and see what happens. It would actually, to me, it would be an interesting examination.

**Matt Coles (32:44)**: I mean, if you had sharded, if you had sharded, if you had sharded memory, you could. you know, remove access to data.

**Izar Tarandach (32:51)**: If you had, if you have multiple instances and one overarching one that carries the memory of all the existing small ones or sub ones, they could be the same size. Then that one could sort of have a care about what the next one is going to have to play with. So in trying to maximize its own output, it might say, oh, if somebody takes a GPU away from me, That's going to make my work harder. Therefore, the reward is smaller.

**Matt Coles (33:23)**: Right. You have a penalty system that you can implement.

**Chris Romeo (33:27)**: Yeah.

**Izar Tarandach (33:27)**: That'd be interesting. If it hasn't been done yet. If only we knew somebody who could write that kind of thing.

**Matt Coles (33:34)**: Anybody can, right? AI is democratizing, uh, coding.

**Chris Romeo (33:38)**: Just vibecode it.

**Izar Tarandach (33:39)**: Yeah, but I think that you need a bit more hardware to actually be done and be able to do that.

**Matt Coles (33:44)**: Right.

**Chris Romeo (33:45)**: Back on this rewards article, have you guys ever heard of this thought experiment, the paperclip maximizer? That's at the bottom paragraph.

**Matt Coles (33:52)**: Is this Clippy all over again?

**Chris Romeo (33:54)**: I never, I've never heard this, this example though. I'll just, I'll just highlight it.

**Izar Tarandach (33:58)**: What?

**Chris Romeo (33:59)**: For, 'cause in case our listeners haven't heard of it either. Nick Bostrom, who's a philosopher, created this paperclip maximizer thought experiment. So he instructed an AI to make as many paperclips as possible. And the result is it ends up consuming all the matter in the universe in pursuit of its goal.

**Matt Coles (34:17)**: Because you gave it an open-ended, yeah, create as many as you can.

**Chris Romeo (34:20)**: But it's, it's reward hacking, right? Like it, the, the reward was to maximize the number of paperclips that it creates. And in doing that, it consumed all the matter in the universe.

**Izar Tarandach (34:34)**: Oh, but, but that's, that's an evolution of the Great Goo Problem. Right? So people working with, uh, nano-scale robots and stuff like that, they have for a long time posited this, this problem of, okay, we're going to create self-replicating machines, but if you don't put a brake on them, and how do you put the brake on them? They're just going to turn the whole universe into gray goo because they're going to self-replicate.

**Chris Romeo (34:57)**: Right. Yeah.

**Matt Coles (34:58)**: I mean, this is, this is cancer. I mean, literally.

**Izar Tarandach (35:01)**: Basically. Yeah. Yeah.

**Chris Romeo (35:02)**: Yeah. Yeah.

**Izar Tarandach (35:03)**: Literally. But, but, but interesting enough, apparently cancer does have a self-breaking mechanism. It knows that at some point, if it grows too much, it kills the host. It, it, I don't know what it is. We have this thing with anthropomorphizing everything that's getting so freaking annoying.

**Chris Romeo (35:20)**: I mean, speaking of self-replicating, I was reading another article from MIT.

**Matt Coles (35:24)**: I was wondering how you're going to work this in.

**Chris Romeo (35:27)**: Some, some individual who had like 550 to 600 brothers and sisters. That's all I remember from the article.

**Matt Coles (35:34)**: They weren't AI.

**Izar Tarandach (35:36)**: Aye, aye, aye.

**Chris Romeo (35:39)**: Aye, aye, aye, aye.

**Izar Tarandach (35:41)**: The question now is if you mix AI with that article and self-replication, can we panic now?

**Chris Romeo (35:49)**: Yeah, probably.

**Matt Coles (35:51)**: Was it rewarding? Was it ethical? Was it—

**Izar Tarandach (35:57)**: It must have been rewarding.

**Chris Romeo (36:00)**: Was it ethical? Well, that's another debate. Debate for another day. Well, we covered a lot of ground. I think this is an issue that's gonna be going on for 10+ years. So I think this is just the beginning of the conversation. At least not like we're gonna be sitting around for 10+ years in a row discussing it. But I think this issue of ethics and models and whatnot is, is 10 years from now we could have a, a review episode.

**Matt Coles (36:25)**: We could go look back and remember when we had a chance. Yeah.

**Chris Romeo (36:30)**: We should have stood up and—

**Izar Tarandach (36:32)**: I like you guys' optimism that we're going to be around in 10 years.

**Chris Romeo (36:35)**: It's good to be optimistic.

**Matt Coles (36:38)**: Well, okay. Maybe we won't be, but maybe our AI clones will be here doing this in our place.

**Izar Tarandach (36:43)**: Right. We'll be living in the Matrix.

**Chris Romeo (36:44)**: And they better be doing it ethically. That's all I'm saying. Well, with that, we'll leave you, leave you as an audience to your day, your week, your month, your year. Remember to compact your AI sessions.

**Matt Coles (36:57)**: Did you just make a Friends reference?

**Izar Tarandach (36:59)**: I'll be there for you.

**Chris Romeo (37:01)**: Come on. I can, oh, here we go.

**Izar Tarandach (37:04)**: Nobody hates Friends. Nobody hates Friends.

**Chris Romeo (37:06)**: Let me summarize Friends in 1, 2, 3, 4, 5 words.

**Matt Coles (37:13)**: 5 words.

**Chris Romeo (37:13)**: Are you ready? This is the whole show. This is, you're gonna understand the whole thing. Rachel, I love you. End of show. Done.

**Izar Tarandach (37:20)**: Wait, wait, wait, wait, wait, wait, wait, wait, wait. That's it.

**Chris Romeo (37:22)**: That is the show. I just produced the show in 5 words. Wait, wait, wait. In one impression.

**Izar Tarandach (37:27)**: Let's pivot, pivot, pivot, pivot, pivot.

**Chris Romeo (37:33)**: Okay. We're not a model. You can just tell us once and we'll listen.

**Izar Tarandach (37:39)**: No, but if you're going to do a Friends joke, you have to pivot. But you guys don't remember that one? The one of the science team?

**Chris Romeo (37:45)**: No, I really don't like Friends. Honestly, I don't. I think it's stupid. Oh, come on.

**Matt Coles (37:49)**: The only reason I watch Friends is because it's on just before Big Bang Theory.

**Chris Romeo (37:53)**: Big Bang Theory is Incredible. Have you seen Stuart Saves the Universe? Have you seen the new one?

**Izar Tarandach (37:58)**: Yeah.

**Matt Coles (37:59)**: Not yet. I want to.

**Chris Romeo (38:00)**: Oh, you gotta see it. It's incredible. It, the production value, I've only seen the first episode, but the production value of it is, is like amazing.

**Izar Tarandach (38:08)**: Agree.

**Chris Romeo (38:09)**: And it's all those, it's all like the B-list characters from Big Bang. And the, the, the storyline is just, it's just, well, it's so well done on in one episode that I can't wait to see how it plays out.

**Izar Tarandach (38:19)**: It's going to be tiresome fast. But so far it's great.

**Chris Romeo (38:23)**: Okay. I gotta catch up. I've only seen one episode, so I got, there's probably 3 out or 4 out at this point. So check it out, uh, wherever you stream things.

**Matt Coles (38:32)**: Woo-hoo.

**Chris Romeo (38:33)**: And don't, you can also ask your model if it would be ethical for you to watch that show or listen to that.

**Matt Coles (38:39)**: The answer is always yes.

**Izar Tarandach (38:40)**: Isn't it great that we got to a time where every nerd has its own model?

**Chris Romeo (38:46)**: Yeah, but what about the future where every model has its own nerd? Oh, that will leave you to another—

**Izar Tarandach (38:54)**: no, but wait, they, they already did ever since they needed to do homework.

**Chris Romeo (39:00)**: All right, we'll talk to— we'll see you next week on the original, not to be confused with any duplicate copies, The Security Table. Thanks for listening. If you enjoyed today's conversation, subscribe wherever you get your podcasts, leave us a review, and share the episode with someone who would find it valuable. We'll see you next time.
