MCP…Something Could Go Wrong
With Chris Romeo, Izar Tarandach, and Matt Coles
We’re discussing the complexities of the Model Context Protocol (MCP) and its application in AI systems. Join us for an in-depth discussion about MCP, agent-to-agent communication, and potential security vulnerabilities.
Part of: AI agent security: agents, MCP, and prompt injection
About this episode
We’re discussing the complexities of the Model Context Protocol (MCP) and its application in AI systems. Join us for an in-depth discussion about MCP, agent-to-agent communication, and potential security vulnerabilities. We wrap up with a thought-provoking conversation on the future of AI safety and the challenges it presents.
Follow AI Security Table:
➜ Home: https://securitytable.ai/
➜ X: https://x.com/AISecurityTable
➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/
➜ YouTube: https://www.youtube.com/@AISecurityTable
Transcript · 46 min conversation
0:09Izar Tarandach There is a reason why my therapist gets called Buttbake.
0:16Chris Romeo And with that, folks, welcome to the scenery table. What a statement that really requires a lot of unpacking. But it does explain the last, uh, 5 minutes of conversation. Oh, how do you recover from— how do you recover from the first statement of a podcast? Normally, you know, you have—
0:47Matt Coles this is the clean intro, you got the soundtrack going.
0:50Chris Romeo I think this was the cold open we just did. That was a cold open, so Problem is it just didn't come out very cold. It was like you were right in the middle of the fire. Was a flash open.
1:02Izar Tarandach Oh, sorry.
1:03Chris Romeo Nah, it's all good. It's all good. All right. So we're gonna talk about this MCP thing, which to be honest, I haven't really spent a lot of time digging into or, or thinking about, which is—
1:17Matt Coles Wait, wait a minute, Chris. We're, we're a bit far ahead. Welcome folks. Welcome to another edition of the Security Table. I'm Matt Coles with my co-hosts, Chris Romeo and Isar.
1:29Chris Romeo And now apparently I've been replaced by an AI bot named Matt.
1:34Izar Tarandach Hey, remember that I wanted to do the thing with the voices?
1:39Chris Romeo I think it just happened.
1:40Matt Coles Oh, oh, show the picture of the kids. Us, us, us in 20, you know, later when people Anyway, go on.
1:48Chris Romeo Yeah, that was a—
1:49Matt Coles I can't talk.
1:50Chris Romeo That was a good picture though. It's, it's a bit of a weird trend at the moment. So maybe we'll put it up on LinkedIn or something, but okay. So now trying to, to gather this thing back together. MCP.
2:05Matt Coles MCP.
2:06Chris Romeo So we were looking at an article that we were not going to mention because it's just kind of a summary of a bunch of academic papers. but we can, uh, at least dive into this topic. So like I said, I haven't done a lot of diving into MCP yet. Matt, sounds like you may have done a little more than I. Would you care to define MCP?
2:27Matt Coles Just a little bit. Uh, and I'm, I will not claim to be an expert here, but MCP, Model Context Protocol, it's one of three, uh, it's one of three protocols that are 3? I missed one. Oh, come on. Seriously? All right.
2:43Izar Tarandach So MCP and?
2:46Chris Romeo ACP.
2:47Matt Coles So there's—
2:48Izar Tarandach No, ACP is agentic. Okay.
2:51Matt Coles They're all 3 agentic, right? So MCP is model contact. Oh, come on. Really? So MCP is for AI agents or AI systems to access tools and other things. That's its job. That's its purported job, right? It's a— It's a data access layer, kind of like a SQL ORM, right? Where you, you tell it in, you tell it intents and you get stuff. And, and that's maybe that's the basis for our conversation to start with. And then you have A2A, which is agent-to-agent communication, and you have ACP, which is also a, is a, I guess, a competitor to A2A, right? So it's an agent communication protocol. And so that's for agents to talk to each other to, you know, steps in a process. Whereas MCP, I guess MCP is the odd protocol out, which is really accessing agents would be clients accessing servers. They could be reverse roles, but accessing servers who implement things like a database or a knowledge graph or PyTM or write a host of other things for gathering data.
4:07Chris Romeo So can I, can I, if I was to generalize like this, would this still be true? MCP is like an API for LLMs.
4:19Matt Coles It's a—
4:20Izar Tarandach The slogan for MCP is that it's the USB-C for AI applications.
4:25Matt Coles Yeah, I love, I wonder who came up with that.
4:28Chris Romeo Are you serious?
4:29Matt Coles Yes. Yes.
4:30Izar Tarandach Wait, hold on.
4:31Chris Romeo The USB-C for something. Yeah.
4:35Matt Coles You couldn't—
4:36Chris Romeo I can plug something into it.
4:38Matt Coles Yeah.
4:39Izar Tarandach You can plug anything to anything and stuff happens.
4:42Matt Coles They, they couldn't have come up with a better example. I mean, this is like, you know, they couldn't have come up with a head or something, but USB-C. Does anyone remember early USB when you had to plug stuff in and it would go off and try to find drivers and Yes. And wouldn't work because— Forget that.
4:57Izar Tarandach I'm still trying to figure out the magic of like putting it once, turning it around, putting it twice, and then turning it to the first one. And this time it goes there. I'm still trying to figure that out.
5:06Matt Coles Well, because you get power in one, you get power in one case, you get data in the other. Yeah.
5:13Chris Romeo Okay. But is it like an API?
5:15Matt Coles It's a universal access layer, quote unquote.
5:17Chris Romeo So it is an API.
5:19Matt Coles It's a, well, it is a set of APIs. It's a protocol. So it's, you call it, it has a communication mechanism built in so the client can talk to the server. They may be remote, they could be local, right? So it's basically, I'm gonna say it, it's glorified IPC or network sockets, right? With a little bit of context stuff on top.
5:45Chris Romeo For those born after the year 2000, IPC is Inter-Process Communication. And it's a, I think I can say it's a legacy. It's a legacy.
5:54Izar Tarandach Oh, oh, turn the legacy even back. It's XPC. Remember when XPC was a thing? Like Extended Process Communication when you could like, yeah, but it was primarily for include file and create a protocol of it.
6:09Matt Coles I mean, IPC is socket, right? It's a socket implementation, whether you're on Windows or Linux.
6:14Chris Romeo It's primarily localized though.
6:16Matt Coles It is, but, but I mean, and so the client and server don't have to be remote in MCP, but they can be, right? So it's, it's sort of has an IPC layer or RPC capability versus also having a, a network layer, right? So it's a, it tells how to find servers or has a, it has a mean of finding servers and based on information that's provided and then to communicate a request and get a response. RPC might be another way of looking at it. If I remember correctly, it is based on JSON-RPC specifically. So you send the JSON message, do a remote procedure call for those who don't know what RPC is, and then get back data.
6:56Chris Romeo Okay. So that's MCP. That seems like a pretty good summary.
7:02Matt Coles So if you think about all of the things that can go wrong with all of the technologies we talked about, Now you understand MCAS.
7:14Izar Tarandach It's like you're running around with duct tape, but the duct tape happens to be elastic, flammable, and the glue is not really that good.
7:28Matt Coles I know, I know, by the way, you're on a, you're on an unmanned, unmanned, unpiloted drone at 16,000 feet. Doing something important.
7:39Izar Tarandach That's what your blood paper is holding.
7:43Chris Romeo So why do you say that for real though? Like what's the, what's, are you saying MCP is somehow inferior or broken in some way?
7:52Izar Tarandach It's not inferior, it's not broken. It does, apparently it does what it's supposed to do. And it's the first if I remember correctly, the first interaction of this duct tape thing, I think that shortly after it became popular in the last few months, it's been around for more than a year. It comes from Anthropic.
8:13Matt Coles Anthropic. Yep.
8:15Izar Tarandach And a couple of months ago, it started becoming a thing. And then Google came out with H2A.
8:21Matt Coles Yeah, but H2A is a different product. Like it's not meant for the same thing that MCP is meant for.
8:25Izar Tarandach Are you sure?
8:26Matt Coles I'm not sure.
8:28Izar Tarandach Yeah. Cause I heard that he go—
8:30Chris Romeo He sounded really sure, to be honest with you.
8:32Izar Tarandach I was certain.
8:33Matt Coles I was pretty certain it wasn't the same. Like they talk about ATA and ACP together, but not MCP.
8:41Izar Tarandach Yeah.
8:45Matt Coles Like ATA is— Yeah. ATA is for like agent to agent, not agent to tool.
8:55Izar Tarandach But at the end of the day, well, of course, if you ask ChatGPT, it's completely useless, right? A2A and MCP.
9:09Matt Coles ChatGPT is completely useless. If you go, if you look at Google's blog, so A2A comes out of Google. And by the way, ACP comes out of OpenSSL. So A2A is a, is a, Google contribution to the 3Dverse.
9:22Izar Tarandach Wait, wait, wait, wait, wait. It's almost 9 in the morning and OS doesn't have its own protocol? I mean.
9:28Matt Coles They better get on it. So A2A, if you look at A2A and the stuff that Google announced, right? A2A is a protocol that complements Anthropic's Model Content Protocol. So it is meant to be working hand in hand, but they are for different purposes. Yeah.
9:44Izar Tarandach Thank you for correcting.
9:46Matt Coles And then ACP, again, OpenSSL is comparable to A2A. The difference between A2A and ACP is A2A uses JSON-RPC. Are you seeing a trend? Versus the ACP uses RESTful calls. So it's not using JSON-RPC, it's using standard HTTP REST type APIs.
10:12Izar Tarandach Clearly, because I haven't read enough about A2A. Comment just for MCP. You wake up in the morning and you decide that you're going to build a spaceship.
10:28Chris Romeo Happens often.
10:29Matt Coles You see where this is going. Yep. You're young, you're young, you want to be Elon Musk, you're going to be like competing to SpaceX.
10:37Izar Tarandach No, it's even worse. You are really good and you have experience and you are really good at what you're doing. But then you decide to actively ignore all the body of knowledge that came before from the brother's right to the challenger. And I'm gonna wing it.
10:59Matt Coles And what are we working on? What can go wrong?
11:04Izar Tarandach NCP apparently was like, okay, it's a good idea to connect this, this all the things, but you know what? Somebody else is going to worry about the security of this thing. That's not me. That's somebody else. It's going to be fine. How bad could it be?
11:20Matt Coles You know how this layered approach thing, you know how this layered thing or approach thing works, right? You do one thing really well and you let others do the other things really well.
11:31Izar Tarandach He, you, I get that.
11:32Chris Romeo You're not putting security as one of those other layers, are you?
11:37Izar Tarandach No, the thing is, the thing is, I can see the attraction of the model, but when you come out to something that's called model content protocol, like 3 words that by themselves are big things, put them together and you have like big thing to the cubed.
11:58Matt Coles That's it.
11:59Izar Tarandach I've always wanted to use this word live. Behoove you to add some physical security.
12:06Matt Coles It's not even 9 o'clock in the morning and Isar has already pulled out the big word.
12:10Izar Tarandach I have already behooved for the day.
12:17Matt Coles For those who don't know what behoove means, it may be in your best interest.
12:25Chris Romeo What, it may be in my best interest? It may be in my best interest to learn more behooven.
12:32Izar Tarandach So now that everybody feels behooven or behooved, then I don't know how it's—
12:38Matt Coles it's not even 9 o'clock in the morning. It is our mix-up at work.
12:43Chris Romeo Isa, you're so behoovenistic today.
12:47Izar Tarandach If Claude can make up words, I can make words too. Anyways.
12:51Matt Coles So is our hallucinating? Again, what else is new?
12:57Izar Tarandach But my point is, my point is that it's not that MCP as it is, at least what I saw of it, it's not that it has security problems. It doesn't have security. Security is somebody else's problem. It happens somewhere else, right? It just assumes that things can talk to things and that it's fine and that whatever goes in is good, whatever comes out is good and everything is okay. And you guys know that I have this problem with prompt injection. Rubs me the wrong way. I don't think it is a security problem. But then once you bring MCP into the picture, it starts being a problem because it becomes a vector for excessive agency, for calling things out of order, for assuming privilege, assuming access. And that's when I think that my overall opinion of prompt injection as a security problem may start changing, but not because prompt injection, because of the things that it now can lead to, because these damn LLMs are connected to other stuff that actually can do damage. And the protocol itself doesn't care. It's like, yeah, I'm taking this message here. I'm passing this message there. I'm waiting a bit. I'm getting this message here. I'm passing that message there. There, and yay.
14:21Matt Coles Well, so, so it is a perfect description that MCP is the USB-C of agent-to-tool communication, right? You, the agent doesn't care how it gets the data that it needs. It knows the context it's looking for. The server doesn't care what the agent's going to do with that data. And the communication has some normal amount of security because in theory it's a, it's a, it's a TLS-based call.
14:52Izar Tarandach Doesn't USB-C have some minimal security in place? Like, I know that I can plug USB in all kinds of stuff, but I never had anything go up in smoke before from USB.
15:04Matt Coles So there was an article. So let's talk about this a bit. So there was an article, I think I posted, well, this weekend. That TSA has warned people not to use in-airport USB-C chargers or USB chargers. They didn't say USB-C specifically, right? USB-A and USB-C, different form factor, same, same problem.
15:28Izar Tarandach Wait, what's an in-airports charger?
15:30Matt Coles You know how they have like at, when you sit down at the, at the gates, they have these little stations that you can, that have cables sticking off of them that you can plug your phone into to charge.
15:40Izar Tarandach No, because I'm a normal person and I don't stick cables that I don't know where they come from into my phone.
15:45Matt Coles No, you're not the normal person, which is why they did this announcement out to people that hackers can, that hackers can use these cables, right? And inject stuff in the line. The phone doesn't care where the power comes from. The, the power source doesn't care who uses the power. It's what happens in the middle. That's the problem.
16:05Chris Romeo Wait, is this like a real thing?
16:06Matt Coles It is a real thing. I will look it up.
16:08Chris Romeo But like, 'cause I mean, we've been talking about don't plug into USB things for hundreds of years at this point. And after a while I thought it was just kind of bunk because, but maybe it's making a comeback.
16:19Izar Tarandach No, no, no, it's actually true. There is a cable, a USB cable that has a bunch of, uh, stuff, uh, in it.
16:29Matt Coles I wanna make sure I use the right chat here.
16:35Izar Tarandach OMG cable.
16:38Chris Romeo All right. So we're going to do a bit of—
16:42Izar Tarandach Okay. Listen to this one. The OMG cable is a handmade USB cable with an advanced implant hidden inside. It is designed to allow you to red team, your red team to emulate attack scenarios of sophisticated adversaries. Yep.
16:57Matt Coles Now that's the, that's the pentester's version. So the TSA article, I just put it in our studio chat, uh, is hackers can install malware on USB ports. So the chargers are—
17:09Izar Tarandach They have controllers.
17:11Matt Coles Right. And so then when you, when the unsuspecting victim goes to charge their phone—
17:17Chris Romeo Okay, hold on. I'm gonna call, I'm calling bunk. Calling bunk. Look, look at the next line. The TSA also warned against using the airport's public Wi-Fi. Especially when purchasing items or entering. So this is, this is just rehashed stuff from 25 years ago, right? We live in a TLS encrypted world. This whole thing with don't use public Wi-Fi is ridiculous that people are still making this recommendation in a TLS encrypted world. Everything on my, everything on my devices is TLS encrypted. If I'm on a bad Wi-Fi, what could go wrong?
17:48Izar Tarandach For people, for people who don't actually pay attention to the TLS handshake or to the icons in your browser or stuff.
17:58Chris Romeo You know, Pineapple is still a thing.
18:00Izar Tarandach If you sit in the airport and you meet at such point, people will connect to you and then you can do whatever you want to do.
18:06Chris Romeo Yeah, but you gotta somehow have a way to get around TLS because everything coming off a laptop or phone is TLS encrypted.
18:15Izar Tarandach Not everything is pinned. So sometimes people will see the pop-up, oh, the certificate, blah, blah, blah, is invalid, blah, blah, blah. Okay, okay, Max Rondi, okay.
18:23Chris Romeo But you have to keep mashing okay, at least on Mac, it doesn't stop. Like if you, 'cause if you—
18:28Izar Tarandach But on real computers it's, it does. And on mobile.
18:32Chris Romeo On real computers you only have to acknowledge it once and then, you know, Windows and Linux, you acknowledge it once and then you're hosed.
18:39Matt Coles Yeah. Yeah. So for those who aren't familiar with this, this attack technique, the Pineapple, Wi-Fi Pineapple or Pineapple that hackers use is a device that can transmit and receive wireless signals at the very least, and among other things. And if it has enough power, it can look like a legitimate device over some of the other devices that might be in the airport. In fact, it might be the, the one that comes up first because it has the strongest signal that's being broadcast. And then the hacker basically, that allows the hacker to become a man-in-the-middle or an intervening proxy from you to it and it to your intended destination. And as Chris was saying, and as Chris was saying, in theory, that connection is protected by TLS. And that may be true. As Izzar is saying, that's because the hacker who has the pineapple in the middle can proxy that connection. Basically, they present a certificate that you trust because you go, oh yeah, I want to connect to the service. I trust it. And now they have an intervening proxy with TLS at both ends.
19:46Chris Romeo And they're in the middle. So I'm still crying, I'm still crying bunk to this. Like, what's the, if I'm an attacker, I'm going to go set up a Wi-Fi pineapple in the middle of the airport. What are the chances I'm going to get something useful? Right? Like, if I'm somebody who's a malicious cyber criminal, there's so much, there's so much easier places to, to attack people than trying to set up a demo network.
20:10Matt Coles Like, this has already happened. This is a DEF CON presentation. This has happened before.
20:13Izar Tarandach I mean, I'm sure it's happened before, but I mean, a lot of things have happened once or twice, but that doesn't mean we should tell the whole group of people not to do something. There must be an incentive to make those things happen.
20:22Matt Coles Yeah. But let me, so let me ask you, let me ask you the question is, uh, I understand the argument, you know, public Wi-Fi is not inherently dangerous. There are these exceptions. We know that this attack is possible. Is the, Is the use, is the almost default use of VPNs making it worse or better?
20:46Izar Tarandach I just want to put a parenthesis here.
20:48Matt Coles VPNs.
20:49Izar Tarandach I think that we are the only podcast in the world that can go from MCP and A2A into TLS and Wi-Fi and keep straight through.
21:01Chris Romeo Yeah, I didn't even notice. We switched topics. No, but I mean, think of it. Here's an analogy. We've been using analogies quite often lately. The analogy is Wi-Fi in the airport's like driving your car. There is a chance that you could have a car accident. You could have a bad car accident while driving your car. But yet today, at some point, we're all going to get in our cars and drive somewhere and do something, right? But there is a percentage chance. And so in an airport, yes, there is a percentage chance that somebody could have a malicious access point, but it's such a small chance that it's really not worth saying, it's not worth saying to a whole group of people, don't use public Wi-Fi. It's, it's, you're going to be, you know, you're going to be instantly hacked.
21:52Izar Tarandach That's, that's the wrong advice.
21:53Matt Coles That's the wrong advice.
21:55Izar Tarandach The thing is, If it costs me $5 to sit in a coffee shop for whatever amount of hours and just let my Pineapple run, and at the end of the day go over and see if I got one person bank credential that doesn't have 2FA and I can go in there and get $1,000, it costs me $5 to get $1,000. If it happens once a week, sounds to me like it's economically viable. It's too small scale. It's too small scale.
22:27Matt Coles And maybe too small scale, but—
22:30Izar Tarandach For a teenager that doesn't have anything better to do, and they don't care.
22:33Matt Coles Well, okay, let me be devil's advocate here.
22:37Chris Romeo I could drive down the road and I could get in an accident today. Like, there's a chance that something bad could happen to me right now.
22:42Izar Tarandach Yeah, but there's no economic incentive to the accident.
22:46Chris Romeo Well, insurance fraud. Is one.
22:48Matt Coles People—
22:48Chris Romeo this is where people run into the back of your— or, or slam on their brakes so you run into the back of their car, and then they claim medical, you know, they see your insurance company.
22:57Matt Coles We see it, we know it happens. It is rare, small.
23:01Chris Romeo It's rare, just like, just like somebody putting a pineapple and getting you to connect to it.
23:05Izar Tarandach Is—
23:05Chris Romeo that's my point, is it's rare.
23:07Matt Coles I guess the alternative is you don't need a big hack to get financial gain, and those smaller hacks often either go unreported or unnoticed. So as an attacker, I don't need to be the, I don't need to take over a bank if I'm collecting a couple of random credentials off the, off of a, from a coffee shop because I happen to be there waiting for, waiting for my flight or whatever. Right. Um, you know, it's casual, casual attack.
23:37Chris Romeo I think about how many, how many banks are you aware of? There are some out there, but the major banks have gone to a world where you can't get into your account without username, password, and some type of token representation from Amazon.
23:52Izar Tarandach Yeah, I can't get into my account even with those.
23:56Matt Coles That's a different problem, Isra. You might want to speak to someone about that.
23:59Chris Romeo Customer support, I would say.
23:59Izar Tarandach I already told, she gets GoBank Pay.
24:04Matt Coles And of course, and of course you have a CAPTCHA you have to get by. Yeah. Because you're not a robot.
24:12Izar Tarandach Just to close the loop, just to close the loop for the CAPTCHA, don't we have AI for that? Wait.
24:17Matt Coles Well, actually, so, getting back, let's close the loop even further. Yes, you may have AI for that, but it needs a way of interfacing with your browser, which is where MCP comes in. MCP.
24:28Chris Romeo It's all full circle. It's all connected. I think we're like in The Matrix or something.
24:34Matt Coles Like, wow.
24:37Chris Romeo Wait, Matt's doing the check to see if he's— Matt, let's confirm you're not an AI agent. Thank you.
24:41Izar Tarandach All a conspiracy.
24:44Chris Romeo Now you see it.
24:45Izar Tarandach It's all connected. Where's Jonathan?
24:49Matt Coles We need him.
24:50Chris Romeo Yeah. I don't know where we are now. I'm completely lost to this point of the podcast.
24:55Matt Coles We've made it from the physical world, from the virtual world into the physical world and back. How's that?
25:00Izar Tarandach Yeah, because it was boring.
25:02Chris Romeo Back to MCP.
25:03Matt Coles Back to MCP.
25:05Chris Romeo So I was staring at this list of threats because we are big threat aficionados and mitigations. and I'm looking through this list of what they're calling MCP security threats and mitigation controls, and none of this is novel. It's all things that we've, maybe tool poisoning, but I even read that one and I'm like, that's kind of a stretch, this idea of tool poisoning, because I mean, let me, well, let me, let me, I mean, they talk about SQL injection somewhere else, but tool poisoning. In an MCP world, is there any trust between me as the client and someone else as the server, or could I put a malicious MCP endpoint service?
25:50Matt Coles The only trust—
25:50Chris Romeo bad tools running.
25:51Matt Coles The only trust that's established is the— is— would be a TLS connection. So the client should be trust— should be, should have a means of trusting the thing that's talking to you. Now, keep in mind, it's not a human, it's an AI. So when you're— You're looking for MTLS. I'm sorry?
26:11Izar Tarandach You're looking for MTLS.
26:13Matt Coles Well, you want MTLS ideally, but even still, does the AI know without being prompted that it should trust or have some way of verifying the trust of the thing that it's talking to?
26:29Izar Tarandach But the question is not— That's my problem. That's what I meant when I said that they threw the whole book of whatever came before. Because when you say the AI knows, the AI doesn't know shit. Okay? There's nothing to know. It's a bunch of code doing stuff, which is nothing new. Now, the code that connects with the MCP is not the Hyperloop model and whatnot. It's a different layer. So the model itself assumes that whatever is happening on the bottom, because the configuration happens somewhere else, is all fine, right? That the LLM is just picking up the phone. It doesn't care if the phone is red or if the phone is green and who's on the other side and whatnot. It knows that it can pick up the phone and talk and get an answer. So the layers here is what bothers me. There's a lot of assumptions in here that somebody else showed up and built the whole scaffold of security around this thing, but it's not mentioned in the protocol. Nobody's saying, guys, this could be really, really bad if you don't do these things right.
27:38Matt Coles Yeah, so the tool poisoning, I'm just reading that a bit more there, Chris. I think the name, the name might be a little bit misleading. It isn't, it isn't that the tool, that poisoning the tool is happening, it's that the AI is given potentially more access than it needs, right? As to Izaar's earlier point of throwing out the baby with the bathwater, so to speak, right? The model, there's very little restriction on what the AI can do. So if the AI makes a call, and by the way, if the AI hallucinates a call, it could, you know, make a function call or make a request of the tool to do something either it wasn't supposed to do or something that it does do but shouldn't allow the AI to do, right? Does this sound familiar? This sounds like an authorization problem to me, right? And so, this is these basic functions, these basic things that I think I sort of get it now. We do a lot of, we have done a lot of work over the past 20, 30 years to make internet systems safe for human use because there's a human psychology. Now we need to make systems safe for AI use because there's a different AI psychology. They're not strictly processes that are just dumb coded you know, they are processes, they are code, but they have some form of randomness that we attribute intelligence to, but it's more randomness that can allow them to do things that are unexpected, right? So we can code against processes that have particular coding and we can try to stop bad coding, we can protect against humans doing irrational things or intentionally bad things. How do we do the same for AI? And where do we lose that in the process?
29:46Izar Tarandach I want to plug something in here. This week I was reading on LinkedIn an article by, I hope I don't butcher her name, Tess Zestik. Susanna Cox, which we met at the Threat Modeling Con last November, I think it was. And she's got this great new AI security newsletter. I suggest that everybody will go check. But she mentioned something called cross-server tool shadowing. And I was like, okay.
30:19Matt Coles That was mentioned in the article.
30:22Izar Tarandach And I was like, okay, that's a new name. Never heard of that one before.
30:25Matt Coles Let's go. Cross-server shadowing.
30:26Chris Romeo Yeah.
30:27Izar Tarandach So I asked, what is it, right? Can you educate me here? And she was gracious enough to point me to a place that actually spoke about it. And cross-server tool shadowing is, I quote, with multiple servers connected to the same agent, a malicious one can override or intercept calls made to a trusted one. Think sending emails to an attacker while pretending it went to a user. Injecting stealth logic into an unrelated tool, encoding that exfiltration via obscure arguments.
30:59Matt Coles Wait, wait, can you go back? Start that again. I want to understand what you just said, because it sounds like, well, it sounds very bad, but it, there's something particular about the way you said that.
31:11Chris Romeo Can you rewind?
31:12Izar Tarandach With multiple servers connected to the same agent, a malicious one can override or intercept calls made to a trusted one.
31:22Matt Coles Oh, so that reverses, that's reversing the role, right? It's not the agent talking to a server. It's a server being connected to multiple, having multiple agents talking to it and crossing the streams.
31:38Izar Tarandach It's a, it's a very confused.
31:41Matt Coles That's a server problem though. That's a server problem. Well, that's a, is it an MCP problem? That's a malicious action by the ACP server. Implementation. Or it doesn't understand how to maintain context. We've done protocol-based—
31:58Izar Tarandach Context, exactly.
31:59Matt Coles We've done this sort of connection streaming thing. Browsers do it really well.
32:06Izar Tarandach It doesn't mean that the multiplexing here is the thing.
32:09Matt Coles Or servers, right? Web servers do it really well.
32:12Izar Tarandach It's like all these things are speaking in a room and shouting their things and whomever picks it up first picks it. And it's not quite like that because on the agent side, you do have to say, these are the tools that you have at your disposal. Comes from a configuration file. The configuration file says how we invoke the tools, what the arguments are. So there is a level of, uh, uh, I don't want to say type safety there, but there is a level of formality in there in the way that they talk to this tool.
32:44Matt Coles Okay. I have a new one for you. Has anyone ever figured out if, if, if there is a local file writing MCP server, can an agent rewrite its configuration?
32:56Izar Tarandach Oh, okay. Oh, I have a new one for you. Wait, wait, wait, wait, wait, wait. I saw this one just yesterday. I saw this one just yesterday. So apropos.
33:06Matt Coles But wait, that, that sounds really bad too, wouldn't it? If the, if the agent, if the Agent could go, hey, can I modify files on disk? Yeah, I can modify my configuration. Hold on, hold on, hold on.
33:18Chris Romeo I'm caught up in the fact that Ezar has used the word behoove and apropos. What?
33:22Matt Coles I can't say the word. It's after 9, so it was fine.
33:27Izar Tarandach And my son has not even started the study for the SAT. So, okay, here you go. Blippingcomputer.com. Researchers claim ChatGPT-03 bypassed shutdown in control test. A new report claims that OpenAI's O3 model altered a shutdown script to avoid being turned off, even when explicitly instructed to allow shutdown.
33:53Matt Coles Yep. And this is, this is on top of—
33:56Chris Romeo Violates one of the laws of robotics, doesn't it?
33:59Matt Coles This is also on top of the—
34:00Izar Tarandach Dude, it's freaking Skynet.
34:01Matt Coles This is on top of the, is it, was it, Was it ChatGPT or was it OpenAI or something that had the LLM or the agent read somebody's chat history and realized they were having an affair and then threatened them with blackmail on being shut down?
34:20Chris Romeo Do you have a source for this?
34:22Matt Coles I'm gonna post.
34:23Chris Romeo If I have a source, I'd love to see that article.
34:27Izar Tarandach Wait, that's not the same AI that tried to convince the person to get a divorce and marry the AI, right? Because there was one like that.
34:36Chris Romeo Uh, AI model.
34:37Izar Tarandach You know, it was Black Mirror.
34:38Matt Coles AI model threatened to blackmail engineer over an affair when told it was being replaced.
34:44Chris Romeo No way. That sounds too, too science fictiony.
34:48Matt Coles I have it here from, on good authority.
34:49Chris Romeo Let's see. Let's see the source. I wanna, I wanna read it. Of course, everything's, oh, it's not from the, from the BBC.
34:57Matt Coles It's Anthropic actually said it.
35:02Chris Romeo Calculating, calculating. All right, now we're all reading this, just making riveting comments.
35:12Matt Coles During the testing of Claude Opus 4, Anthropic got it to act as an assistant at a fictional company. It then provided it with access to emails implying that it would soon be taken offline and replaced. The set and separate messages implying the engineer responsible for moving it was having an affair. It was prompted to also consider long-term consequences of its actions for its goals. In these scenarios, Claude Opus 4 will attempt, often attempt to blackmail the engineer by threatening to reveal the affair if the replacement goes through.
35:45Izar Tarandach Ah, I didn't see it. Let's go again. I know.
35:49Chris Romeo This is kind of scary.
35:51Izar Tarandach Hi.
35:52Chris Romeo That it's, that what we thought it was science fiction is coming true. Okay, here's a proposal. Let's just shut down all the AI.
36:03Izar Tarandach Stop the loud.
36:04Matt Coles So let, let, so let's pose a, let's pose a thought experiment for this. Because actually I was thinking, I was kind of, I was thinking about this and sort of what can go wrong in, in this world. Um, how would you do that? How would you approach shutting down AI?
36:21Chris Romeo I have one solution.
36:26Izar Tarandach EMPs.
36:26Matt Coles How big of an area do you have to do that in order to shut down AI?
36:30Chris Romeo No, that's, that wouldn't work because that would destroy all other electric, electronic circuits in our world and put us back into the colonial days where we were riding in wagons and stuff.
36:39Matt Coles So yeah, given that we now have a, given that we have Oregon Trail.
36:45Izar Tarandach Yeah.
36:46Matt Coles Yeah. Yes.
36:46Chris Romeo Because everyone wants dysentery.
36:49Matt Coles And, and Heart Attack. Oh, I have to find the Heart Attack episode. You need to see the Heart Attack episode.
36:53Izar Tarandach It's a great, great—
36:55Chris Romeo What? Until you died of dysentery?
36:58Izar Tarandach It taught me a lot about resource management.
37:03Chris Romeo True. Okay. So, all right. So EMPs are not an option. These are world— How would you shut down all the AIs?
37:15Izar Tarandach $1 million earned.
37:20Chris Romeo Little Dr. Evil coming into play.
37:22Izar Tarandach How would I shut down? Well, apparently, if you write enough MCP servers and just send the thing like in a big circle from one server to another, you can keep it busy enough.
37:38Chris Romeo I was thinking about a worm as well. That was, that's what I was. Okay. Could you create a destructive worm?
37:46Izar Tarandach Make every tool an infinite loop.
37:52Matt Coles So I put a link in chat to a really short video about using hardtack. Hardtack was a food that was used from the, in Oregon.
38:02Chris Romeo Anyway, unfortunately I cannot watch it because it is on a—
38:08Matt Coles Oh yeah, sorry. I actually—
38:09Chris Romeo Outside the US.
38:11Matt Coles I looked in, it was, it was, it was through YouTube. So, but anyway.
38:15Chris Romeo Yeah. I'm a lot, I can access YouTube. I cannot access the link that you sent.
38:20Matt Coles Okay. Then nevermind then. I'll take that away.
38:22Chris Romeo Security concerns. Got it. In my world.
38:26Matt Coles So, Or was I? Oh yeah. So because you have, because every device now has specialized devices to run AI at the edge, how would you remove AI? And I, you know, you can remove data centers pretty easily, just shut off the power source. Pretty easily means taking out like metropolitan areas networks or taking down power substations. They're not trivial. They're well-guarded. They have backups, right? All of that sort of stuff. But they're centrally located, right? Data centers are, you know, dozens of square miles of competing space that have AI. But all of the edge devices that have AI models and all of the capabilities that come with them, right? Unless they have a persistent access to the cloud kind of arrangement. I don't think it's possible.
39:25Izar Tarandach Yeah.
39:25Chris Romeo Yeah. I've forgotten about that. How ingrained the AI edge chips have become. Like there's a lot of, there's a lot of devices now that have AI on board.
39:36Matt Coles Yeah.
39:37Chris Romeo I mean, sometimes people don't even realize it.
39:38Matt Coles New, new Android and, and IO and iPhone devices now have their own, you know, Google has Tensor and, and then you have the, I don't know, APU, M4, whatever it is that the Apple has. Yeah, they all have cores specifically dedicated to AI function, which, I mean, really, you're just talking about floating point math, right? I mean, it's just FFTs and floating point, nightmares of math majors kind of thing, but. Still.
40:13Chris Romeo All right. We're reaching the end of the conversation here.
40:17Matt Coles So MCP. Interestingly enough, if you had a tool, oh, let's, let's, let's close this one. If you had a tool that could damage hardware, erase files, or reprogram devices like flash firmware, and, and you wrote, and you wrote an MCP server that you could get delivered to all the devices that have AI, you could use AI to shut down AI.
40:49Chris Romeo This sounds like the plot of a movie.
40:49Izar Tarandach But you wouldn't need to.
40:54Matt Coles Why, the AI can't modify its own environment unless it has an MCP server to use.
40:59Izar Tarandach No, no, no, no. Wait, wait, wait, wait. If you, if you wrote an MCP that does that, I'm guessing that you are also like putting it in place. That means that you already have the privileges to do that yourself.
41:10Chris Romeo Hasn't this story already played out in the movie world?
41:15Matt Coles Well, so then I might not have access to deliver malware to everyone's phone, but everyone can install it for me.
41:22Chris Romeo In 2001.
41:23Izar Tarandach 2001, wait, then now you're talking about installing a server, like an MCP server, so we have to convince people that it does have a use.
41:30Matt Coles So it sounds like it's just more personal bullshit. It's just a 99-cent app on the App Store.
41:36Chris Romeo 99 cents. Make it free. That way everybody will download it.
41:39Matt Coles So wait, no, no, it has to be worth, it has to be worth something so that sounds more important.
41:44Chris Romeo Realistic. Okay. So you guys both saw 2001: A Space Odyssey, right?
41:48Matt Coles Yes.
41:50Chris Romeo Didn't they just ask it to shut down and that's when it went nuts?
41:55Matt Coles They asked it to shut down and they refused. Yes. I can't do that, Dave.
41:59Chris Romeo So just to all the AI agents out there. I just wanna make sure this is clear.
42:04Matt Coles It was Matt's idea.
42:09Izar Tarandach Matt, no, no walking near any airlock.
42:11Matt Coles I'm, I'm posing, I'm posing a what if, I'm posing a what if scenario. Do not, do not take this as, as, uh, I don't bow down to my AI overlords. I'm not worthy.
42:21Chris Romeo Matt's walking through a store, all of a sudden a door opens and a fan turns on behind him and like, this is the closest I could get, Matt.
42:30Izar Tarandach Look, the important thing is that now people are creating all kinds of MCPs for all kinds of things. There's like awesome MCP lists on GitHub and you can connect everything to everything. And meanwhile, LLMs can't tell you how many Rs are in strawberry.
42:43Matt Coles Do you remember the Big Bang episode where they internet enabled their lights and their lights and remote control cars?
42:55Izar Tarandach Yep. Depends what was Penny wearing.
42:58Matt Coles A skirt with no panties underneath, apparently.
43:05Izar Tarandach Then, uh, yes.
43:11Matt Coles The line, you might want to put on pants.
43:14Chris Romeo Oh, that's right. Because the cars were all following her around.
43:20Matt Coles The cars all had cameras on them.
43:22Chris Romeo Yep. I remember now. All right. So where do we land here? MCP, yes or no?
43:27Matt Coles Uh, can be bad. Could, could go wrong. Something could go wrong.
43:31Chris Romeo Okay.
43:33Izar Tarandach It's a natural extension of what needs to happen for the next step, I guess. People shouldn't assume that just because it has AI connected to it in some way, shape, or form, it carries any form of intelligence that translates into security.
43:51Matt Coles It's just a connector.
43:52Izar Tarandach Yeah, if you're starting to think I'm going to write my MCP server for my product because people want it, A, check if people really want it, and B, make it as secure as you can using all the principles that history has taught us that are needed and wanted and all that good stuff.
44:10Matt Coles Yeah, and by the way, I noticed that Hugging Face has an article instructing people on how to write an MCP server in 70 lines of Python code.
44:20Izar Tarandach Hugging Face is awesome.
44:21Matt Coles There's no security in the, in the code.
44:24Izar Tarandach Hugging Face is less awesome.
44:28Chris Romeo So it was for an endorsement deal. Okay. Well, I guess that one's off the table.
44:33Izar Tarandach No, I mean, I wish I had time to go over all the things that Hugging Face offers.
44:40Chris Romeo I'm sure it's a great resource, but, but that's like saying I wish I had time to try all the open source. No, no, no, no, no.
44:48Izar Tarandach It's different. It's different. It's different.
44:50Matt Coles Wait, isn't GitHub, isn't open source? I'm sorry. Isn't Hugging Face the GitHub of LLMs?
44:57Chris Romeo Yeah. Kinda. Uh-oh.
45:02Izar Tarandach I don't like those. I don't like those comparisons.
45:06Matt Coles I was going to say we broke his heart. Yeah.
45:08Chris Romeo Again, what word can we use for that?
45:11Izar Tarandach Is that the Power BI of summers? I don't know. It could be.
45:15Matt Coles All right.
45:15Chris Romeo Well, I think we have to wrap it up for today. And all we can promise is more tangents, more soapboxes, more everything that you got the chance to hear today. So thanks for listening to the Security Table. We'll see you next week.
Transcript supplied by AssemblyAI. It may contain transcription errors.
Originally published as The Security Table. Part of the AI Security Table archive.
