# XZ and the Trouble with Covert Identities in Open Source

URL: https://securitytable.ai/xz-and-the-trouble-with-covert-identities-in-open-source/
Published: 2024-05-02
Duration: 2634 seconds
Season: 2
Episode: 13
Hosts: Chris Romeo, Izar Tarandach, Matt Coles

## Show notes and chapters

Matt, Izar, and Chris delve into the complexities of open source security. They explore the topics of trust, vulnerabilities, and the potential infiltration by malicious actors. They emphasize the importance of proactive security measures, the challenges faced by maintainers, and propose solutions like improved funding models and behavior analysis for enhancing security within the open source ecosystem.

Follow AI Security Table:

➜ Home: [https://securitytable.ai/](https://securitytable.ai/)

➜ X: [https://x.com/AISecurityTable](https://x.com/AISecurityTable)

➜ LinkedIn: [https://www.linkedin.com/company/ai-security-table/](https://www.linkedin.com/company/ai-security-table/)

➜ YouTube: [https://www.youtube.com/@AISecurityTable](https://www.youtube.com/@AISecurityTable)

## Transcript

**Chris Romeo (0:09)**: We're starting this episode right in the middle. Not really in the middle, but we're going to continue that thread as Izar began the episode singing Pinky and the Brain that you didn't get to hear. All right, that's copyrighted. It's cop— okay, sorry. Uh, before we go deep into the realms of security and trust and everything else that we always talk about. Best '80s cartoon, because I think that's the vintage for Pinky and the Brain, wasn't it? Was it '90s?

**Izar Tarandach (0:39)**: Oh, '90s.

**Chris Romeo (0:39)**: '90s. Okay. '80s. '80s slash '90s cartoon. I can take either one. Hey Man.

**Matt Coles (0:45)**: Hey Man.

**Chris Romeo (0:46)**: Okay.

**Matt Coles (0:46)**: Good choice.

**Chris Romeo (0:47)**: You're going with DM, Danger Mouse? Okay. British accent.

**Izar Tarandach (0:51)**: I'm going with Hey Man.

**Chris Romeo (0:52)**: I don't remember that. What was, what was Hey Man?

**Izar Tarandach (0:55)**: Hey, He-Man. That's the one.

**Chris Romeo (0:57)**: Yeah, He-Man. We have the power.

**Matt Coles (1:00)**: Yeah, yeah, that might have been '70s, '80s.

**Izar Tarandach (1:04)**: No, no, no, no, no.

**Chris Romeo (1:05)**: Yeah, because I remember watching that on Saturday morning cartoons when that used to be a thing. Okay, but they would always watch Scooby-Doo.

**Matt Coles (1:12)**: I used to watch Scooby-Doo and that was the '60s, but I wasn't alive then, so you could get Scooby.

**Chris Romeo (1:16)**: Yeah, but He-Man, He-Man was later. It was— it was—

**Izar Tarandach (1:21)**: I could never do Scooby-Doo.

**Chris Romeo (1:25)**: Zoinkers!

**Izar Tarandach (1:25)**: It was always the same story every time.

**Chris Romeo (1:28)**: Every story. There's only 3 stories possible for anything that you create. Okay? There's boy meets girl, there's love triangle, and there's the quest. The hero's quest. Think about every story— I'm about to— your mind's about to just implode or explode. Every movie you've ever seen fits into one of those three categories.

**Matt Coles (1:51)**: Well, yeah, but what's your point?

**Chris Romeo (1:54)**: I don't know, I didn't have a point.

**Matt Coles (1:56)**: As a kid, do you care?

**Chris Romeo (1:57)**: No, he said Scooby-Doo. He said Scooby-Doo repeats itself over and over again, and that's my point, is everything repeats itself.

**Izar Tarandach (2:02)**: Sure, but one thing is to repeat the pattern, the other one is to repeat the actual story. I mean, how many times can the butler be the bad guy?

**Chris Romeo (2:10)**: Butlers are apparently bad people.

**Izar Tarandach (2:12)**: Yeah.

**Matt Coles (2:13)**: It's just different ways of tricking people, right?

**Chris Romeo (2:16)**: That's right.

**Matt Coles (2:16)**: Kids learn a lot, a lot of good lessons, right?

**Chris Romeo (2:18)**: How to— all right, my answer is going to be Ren Stimpy. I don't know why, I just loved Ren Stimpy. It was on Nickelodeon.

**Izar Tarandach (2:26)**: Stimpy, you idiot!

**Chris Romeo (2:27)**: I just, I don't know, it's just a good choice.

**Matt Coles (2:29)**: That's a good choice.

**Izar Tarandach (2:30)**: Ren Stimpy is the one that came from The Simpsons? No, like it's the one inside of The Simpsons?

**Chris Romeo (2:35)**: No, that's the cat— that's the rat or the mouse or something.

**Matt Coles (2:40)**: The Simpsons would be a good choice.

**Chris Romeo (2:42)**: Simpsons have been around for 37 years or something. Like, I'm like almost as, you know, long as like it's gonna be one of those shows that we can like measure our age against. Like, I think I was about, uh, 12 when The Simpsons came on TV. Sounds about right.

**Matt Coles (2:56)**: You could do the '90s, then you could do South Park, because South Park would just be like—

**Chris Romeo (3:01)**: I never got into South Park, but Beavis and Butt-Head was my show. Like, come on. Yeah, that would be cool. Come on, ESR, what's wrong with you?

**Izar Tarandach (3:12)**: ESR.

**Chris Romeo (3:14)**: And then they tried to remake it, and the remake is just not so good. Like, it's—

**Matt Coles (3:19)**: they tried to remake Scooby-Doo. They tried to remake Scooby-Doo, and the voice actors are just— they're just not—

**Chris Romeo (3:24)**: it's still Mike Judge, like, the— that's doing both voices. And the premise was great, like, them making fun of YouTube and TikTok videos should have been Should have been hilarious. And then it was just a total fail.

**Izar Tarandach (3:37)**: Like I was just like, I was very disappointed in where it went, but I'll give you Scooby-Doo was a great thing to prepare us for a life of InfoSec.

**Chris Romeo (3:47)**: Ooh, interesting.

**Izar Tarandach (3:49)**: Because the culprit is always the same thing.

**Chris Romeo (3:57)**: Isar always making that transition to the actual reason that this show exists to talk about security and privacy.

**Izar Tarandach (4:03)**: Somebody needs to be the adult around here.

**Matt Coles (4:05)**: I was gonna say, the other thing is I really have strived to emulate Captain Caveman.

**Izar Tarandach (4:13)**: Captain Caveman!

**Chris Romeo (4:20)**: Oh man, all of the flashbacks here of all these different shows.

**Izar Tarandach (4:23)**: It's better than looking like Cousin It.

**Matt Coles (4:27)**: True.

**Chris Romeo (4:27)**: From The Addams Family.

**Izar Tarandach (4:28)**: He wore clothes but didn't have the superpowers.

**Chris Romeo (4:31)**: Yes. Oh wait. Wasn't— when he— Cousin It from where?

**Matt Coles (4:34)**: From The Addams Family.

**Chris Romeo (4:35)**: Oh, it is Addams Family. Okay. I thought that's— but then he made me think it was a Captain Caveman person, like somebody that was on the show with it. Okay. Yeah, yeah. I remember that now. All right. Yeah. Enough witty banter. It's time to start talking about security. So the topic that we have for today is this general problem that now has been introduced in the software supply chain world. Where we have, what do we call them, actors? Let's just call them actors for now, introducing themselves into individual open source projects and over a period of time building trust amongst the teams to get to the point where they take on leadership roles or maybe even just the ability to commit and approve code. And then they become bad actors and they start injecting code into an open source project with the intention of having it go downstream into other projects. And so XZ was the, the item of the hour or day or month, I guess, like that's been the hot ticket item. But Matt, you had another article that OpenSSL came out and was pointing at. This is, is likely going to be a bigger problem than just a single package.

**Matt Coles (5:52)**: Sorry, OpenJS, sorry. Well, OpenSSL was part of it, but OpenJS, I think, was the original originator of this, posing the question about, well, is XE the only one that we caught, right? And that maybe there's, what's the potential for others to exist, right? And so, Because, I mean, and first off, this is not strictly an open source problem, right? This can happen. We believe it probably has happened, you know, over the many decades of software development, or not just software, system development, that, you know, commercial companies could have been infiltrated by bad actors to gain trusted positions and inject malicious things into a code base or hardware or whatever. And here now, when it comes to open source, there's already a lack of trust with open source components, right? Because it's somebody in their basement or some kid in high school or not even potentially not even in high school or not even a kid anymore with AI, building something that suddenly becomes that that critical component that everyone relies on. And so, what do you do? That's, I think, where, how do you establish trust? The question really comes down to how do you establish trust in the people or the processes when it comes to component development so that you can say, I'm using something that's good, it works and it doesn't have a side effect.

**Izar Tarandach (7:41)**: So that, that's one thing, but then there's another. The amount of investment that goes into building that second, that cover story, that cover identity that you're trying, you're using to try to infiltrate some team, group, organization, or whatever in order to get whatever payoff you get from this. So in this case, I said private key. You forge your keys, you can— if I understood correctly, you can go anywhere that's running that version of, uh, that has been infected with that version of XZ. But at the same time, there will always be that one guy that gets half a second delay on their SSH connection and decide that they're going to figure it out. No matter what. We, we are a bunch that, that does this kind of things, right? And we had enough precedent so that people are sort of like, yeah, if it looks strange, then perhaps it is strange, so let's look into it. So my question here is, is it really worth it to put all that investment, all those years of investment, money, time, people, into building a backdoor that is basically exposed to the whole world and could be as flimsy as somebody thinking, hey, I got a half a second delay in here.

**Chris Romeo (9:02)**: I mean, 100%, it's worth it, right? Like, there's a, there's a whole underground community of people that buy exploits and buy access into specific devices, and some of these exploits have gone for millions of dollars. And so The better we get at security over time, the less and less of those types of opportunities are going to be available for exploits that are that widespread. But if I can build one myself, especially if I'm a nation-state actor, if I can create one, if I can plant somebody in an open-source project, I don't even have to send them to the country of origin. I can just put them somewhere else. Because like the one point I wanted to bring back around here is like, This is not a new problem. It's just the first time it's happened in open source. People have gotten jobs inside of companies for the sole purpose of exfiltrating data through their normal access that they have to enterprise systems.

**Izar Tarandach (10:03)**: But look at the paradox here. You're putting this amazing investment in there for something that once you use it, it's probably going to be a once and done. Because few things go unnoticed to— I don't know, few, but most things go unnoticed today just because of the large volume of things that we are looking at. Right. And you know, you're going to Reddit, there is a one subreddit of, uh, one of the many cell phone providers and there's constantly somebody that works at the provider coming in and saying, oh, today somebody told me that if I help them on a SIM swapping scheme. They're gonna give me $5K per SIM. So millions of dollars, years of building a cover identity and everything versus going and paying some, somebody who's probably not being well paid a couple of Ks to do what you need.

**Matt Coles (11:06)**: So I, I would, let's be careful about, we don't know, I mean, gaining access to an open, to an open source project and, and, uh, gaining the trust of the maintainer of the maintainer, right? In this case of the libzma or, yeah, libzma? No, that's not right. But the XE, whatever the utility was, sorry. LZMA, sorry, that's what it was called. LZMA, yeah. LZMA. That maintainer, there was one maintainer who owned the thing and then somebody infiltrated that by gaining their trust. That doesn't— probably doesn't require much funds, right? As Chris said, they don't need to be geographically located in any, you know, anywhere. They just need to have the ability to talk intelligently and have— and actually, it may not— we don't know. It may not actually be their code, right, that they're contributing. They just have to be a face, right, or a or a, an identity.

**Izar Tarandach (12:14)**: And that's a big investment.

**Matt Coles (12:16)**: Not necessarily.

**Izar Tarandach (12:18)**: Well, yeah, because we don't, we're not running background checks on anybody.

**Matt Coles (12:21)**: Yeah, exactly. That individual exists as a GitHub, as a GitHub user, and that GitHub user just needs an email address.

**Chris Romeo (12:27)**: We're talking about nation state actors here.

**Matt Coles (12:30)**: But it doesn't matter.

**Chris Romeo (12:31)**: But it doesn't matter if these are, these are kind of coming at it this saying, oh, it's too expensive. It's, it's, it's not worth the resources put out. But when you're a nation state and you have buildings filled with people that are just trying to break into other nation states, critical infrastructure, computing infrastructure, whatever it is, this is a blip on their radar, on their budget of having—

**Matt Coles (12:54)**: it doesn't require— but, but then I hear nation state. I don't think it doesn't require—

**Chris Romeo (12:58)**: it does. But I'm saying that, I'm saying that it, when we're, if, but nation states are certainly one of the ones I'm most afraid of doing this. Because they have resources to be able to do this. They have— they could have a thousand— there could be a thousand field operatives right now in a building somewhere, somewhere in the world, who are trying to infiltrate open source projects using this same playbook. And it could be a playbook, they could just flood it.

**Izar Tarandach (13:22)**: At that point, you're going to start seeing a pattern developing. And I think that's looking at the, the, the exploit, we see that it was really elegant the way that it got injected in there.

**Chris Romeo (13:34)**: Mm-hmm.

**Izar Tarandach (13:35)**: So you're going to have to keep coming up with more and more and more elegant and covert ways of injecting yourself into this code.

**Chris Romeo (13:44)**: So nation state needs to do— No, but that's the thing.

**Izar Tarandach (13:48)**: No, just, just being nation state doesn't turn you into a, an uber hacking, uber as in super, not uber as in—

**Chris Romeo (13:56)**: Hold on, hold on. Stop the presses. Nations of, and I say, I, I don't, I'm not even calling any particular nations out. I'm gonna say nations of the world have covert organizations that are solely involved in breaking stuff.

**Matt Coles (14:19)**: Yes.

**Chris Romeo (14:19)**: Including our own.

**Izar Tarandach (14:19)**: Some of them are more successful and some of them are less successful. Just like thousands and thousands of people that do pentesting every day.

**Chris Romeo (14:26)**: I know a handful of them that came out of the United States version of this, and they're some of the scariest people. Yes, that I know on the planet.

**Izar Tarandach (14:35)**: Completely agree.

**Chris Romeo (14:36)**: So what I'm saying is, this is something we should be— this is a threat we should be very concerned about, because they do have the ability to create things that nobody's ever seen before over and over again, because it's not— they're not just using AI agents to try to create exploits. They're using some of the best minds on the planet to be able to do that. And so to say that there's, that they don't have the ability to do this, I disagree 100%.

**Izar Tarandach (15:05)**: So let, let me ask you this. Why is it that in the past 14 years that we know of, there have been no other instances of something at the level of Stuxnet?

**Chris Romeo (15:20)**: Coming back around to Stuxnet.

**Matt Coles (15:22)**: So, um, that was the tampering of the Siemens PLC.

**Chris Romeo (15:26)**: Yeah.

**Izar Tarandach (15:27)**: Right.

**Chris Romeo (15:28)**: The Iranian nuclear program, basically.

**Matt Coles (15:30)**: Right.

**Chris Romeo (15:30)**: So one, we may not know. I, I'm, I'm, I'm willing, I'm not willing to say there hasn't been, I'm, I'm willing to say 99.999% sure that other things have happened that are classified and we will never know about the same amount of, uh, same amount of Somebody made it, somebody made a decision to talk about that one, to bring it into the public's eye, right? Because a lot of this stuff happens behind the scenes and is never, it's, it's never unless they get caught, it's never brought into the public's eye, which I would say Stuxnet, there was a, the parties involved were caught that something happened, like when they figured it out, and then they broadcast that to the world. Same thing with this XZ. So I think there's a lot of other stuff like this that's happened in the world that we just don't have visibility into.

**Matt Coles (16:23)**: Agreed. And in theory, that's what I mean. So in theory, that's what folks like Snowden, for instance, right, was, was, was for right or wrong and mostly wrong, you know, he, you know, that attempting to reveal behaviors, right? So we don't know.

**Chris Romeo (16:46)**: But you say, you say right and wrong. Mostly wrong. I would say somewhere in the middle, right? Because the US government was doing some things that Snowden pointed out that were not even legal. And, you know, against United States citizens. And so, yeah, but so that's why I say I don't, I don't think of, I'm not gonna, I'm not gonna land on the side of Snowden is wrong. I'm gonna land, I'm gonna land, I'm gonna, I've always concluded about him that yes, he did, he is wrong in that he broke the conditions of his security clearances and everything else and his employment with the United States government. But there was also, the United States government was breaking the law at the same time. And so I'm somewhere in the middle of that whole discussion.

**Izar Tarandach (17:33)**: The point is that somebody injected themselves into one of the most secretive mechanisms of this nation and was able to exfil. We don't know. I don't know if he went in with the head of doing that or if it's something that happened while he was there.

**Chris Romeo (17:49)**: Yeah.

**Izar Tarandach (17:50)**: But it happened with backgrounds, background checks and all that.

**Chris Romeo (17:53)**: Yeah.

**Izar Tarandach (17:53)**: And then coming back to the open source stuff, we are just a bunch of jokes on the internet. Nobody knows you're a dog. So what is it that open source maintainers can do on their side to green or what consumers can do?

**Matt Coles (18:10)**: Right. I mean, let's break down the—

**Izar Tarandach (18:12)**: The first line of defense here is the maintainer. Sorry. Because he's the one giving the access, giving the rights.

**Chris Romeo (18:17)**: He or she, yeah, is the one—

**Izar Tarandach (18:18)**: He or she.

**Chris Romeo (18:19)**: Or they.

**Matt Coles (18:20)**: Or they. Because it might be multiple.

**Chris Romeo (18:22)**: So, but yeah, so what can they do? Well, I mean, against a well-funded adversary, they're probably going to lose eventually, which I hate to say, but that's just the reality of— I'm a realist, right?

**Izar Tarandach (18:38)**: That's people—

**Chris Romeo (18:40)**: People haven't listened to this show and, and thought that I'm not a realist. I'm always a realist.

**Matt Coles (18:44)**: Wait, wait, wait.

**Izar Tarandach (18:45)**: They're gonna lose on the infiltration point of the thing.

**Chris Romeo (18:49)**: Mm-hmm.

**Izar Tarandach (18:49)**: But once that infiltration happened, they, unless they give up all rights as maintainers, they still have the responsibility of checking that code that comes in.

**Chris Romeo (18:59)**: Yeah.

**Izar Tarandach (18:59)**: And seeing if it's quality code.

**Chris Romeo (19:01)**: So I think you have to get to the point as an open source project, and I'm, I'm completely spitballing this off the top of my head. This is not some, something that I've planned. But I think you have to reach the point.

**Matt Coles (19:11)**: This is new.

**Chris Romeo (19:12)**: Yeah.

**Izar Tarandach (19:12)**: This is just about a hat.

**Matt Coles (19:14)**: Yes.

**Chris Romeo (19:14)**: The hat contains the thoughts and then I just let them out. So I think you have to build your projects in such a way that you have to basically, and I'm going to go to the end and then kind of reverse engineer my way back. You have to have a collection of multiple individuals that are all colluding together to make this thing. Happen, like what happened with XZ. And in the XZ case, there was one nefarious party who was driving it under the trust of the other people. As far as what I've seen, there was no collusion amongst the team. So basically the only way we can have some semblance of protection and trust in open source is to have a bigger group of people that would have to collude together. But then it gets into then how do we push up individual packages? OpenSSL has their scorecards and stuff. How do you push this then so that there's a check and balance between, okay, the whole team would have to go off the rails and agree we're taking this thing down and we're, we're causing a big problem. But then there also, also has to be a check and balance of like what OpenSSL does with the scorecard at a small scale right now so that there's somebody else has to be looking for something that could clue us into this.

**Matt Coles (20:25)**: It's, it's more, I think it's a little bit more involved than that. So first off, think about identities, right? If you have a collection of people who associate with an open source project, how do you know they're the same person or not? How do you know if they are the same person or not?

**Chris Romeo (20:43)**: Blockchain.

**Matt Coles (20:44)**: Hmm?

**Chris Romeo (20:45)**: Blockchain.

**Matt Coles (20:46)**: What's that?

**Izar Tarandach (20:47)**: Blockchain.

**Chris Romeo (20:48)**: That was a joke. I'm sorry. I didn't mean to— that was a joke.

**Matt Coles (20:52)**: Yeah. So unless you tie identities to an individual through biometrics or some other means, Mm-hmm. You know, that you could get instant collusion by one person claiming to be multiple people.

**Chris Romeo (21:06)**: Right. Or, I mean, but that's more of a classic style of attack of compromising. Like I could compromise the accounts of an open source maintainer and then try to use their access that, you know, I'm not, there's no collusion, I'm just compromising. So that's been a bigger problem for a long time.

**Matt Coles (21:23)**: If I have a dozen email addresses and, and choose some names? Why can't I create multiple GitHub accounts and then become multiple people online?

**Chris Romeo (21:31)**: I mean, if you were going to build, you'd have to build the project from scratch though.

**Matt Coles (21:35)**: No, not a project. Just go to Linus, you know, go to, go to kernel.org and say, hey, I'm here to help. Hey, I'm here to help. Hey, I'm here to help. Now I have 3, 3 contributors.

**Chris Romeo (21:46)**: True. It's the same person. But you mean over the period of time, you would have to, you would have to run this playbook for a period of time to try to build the trust. And maybe that means open source maintainers need to have a better set of criteria. For before you give somebody this level of access, like you've met them in person. If I haven't met you in person, sorry, you can't be part of this. And we're having a discussion. So at least if an attacker is going to try to do this, they're going to have to put somebody that's got the CS chops and development chops to actually sit at the table and be a part of the discussion and not appear like they've got a little thing in their ear that they're getting the answers. Hold on one second. Here's how it works.

**Matt Coles (22:25)**: Do you remember, does anyone remember the, the, the theory behind GPG or PGP? PGP. Key signing parties.

**Izar Tarandach (22:37)**: The chain of trust.

**Matt Coles (22:39)**: The chain of trust where you trusted individuals because you meet them in person and you establish trust.

**Chris Romeo (22:45)**: You were supposed to meet.

**Izar Tarandach (22:45)**: And apart from MIT, apart from the MIT signing parties, I, I never saw it actually being instantiated. Yeah.

**Chris Romeo (22:51)**: You never, people never got together to do it, unfortunately.

**Izar Tarandach (22:54)**: But again, guys, it's interesting that you're pulling this so far towards the identity. And Matt, for example, the example of the Linux kernel. Do you know how difficult it is to get code in there? Like really, really, really difficult.

**Matt Coles (23:10)**: I don't. I don't.

**Izar Tarandach (23:11)**: Even if you are a maintainer, you don't just show up with a file and say, guys, this is going into the tree, right? There's like layers and layers and layers of check and people reading code. Character by character, it's not even line by line. And religious discussions of what goes in, which is proper and great because imagine if somebody does one of these at the Linux kernel, right? Let's just close shop and go home. Same thing with SSL. And I don't think that the, the, the solution lives in figuring out who's sending the code. We will never know if it's not a dog. Love dogs, by the way.

**Chris Romeo (23:53)**: Yeah.

**Matt Coles (23:54)**: But I'm just, by the way, we're not saying this is a only this. No, no.

**Izar Tarandach (23:59)**: Right, right, right, right. But my, my point is that that's such a, a flimsy place to go put controls because those are going to be so easily circumvented. On the other hand, for the past 60 weeks, we have been banging on this thing that, hey, open source projects. They need the tools to succeed. They need investment. They need people with the right knowledge, right? They need the right tools. They need the application of those right tools. And I don't think that it's an exaggeration to say that we are sort of, uh, up to the point where we have to define an open source SSDLC, saying these are the tools that are available for you guys out there. This is how you apply them. This is the ones that you're not going to pay for. These are the ones that you get for free because you're using, I don't know, GitHub or whatever, right? I, I'm not aware of anybody having written something at that level, something that can be used for that, for any open source project.

**Matt Coles (24:57)**: But would that help? But that would not have helped in this case because the individuals responsible for making things happen who would have been using those tools can just ignore the results or lie or whatever, right? They can subvert the process.

**Izar Tarandach (25:13)**: I'm not sure if I know all the details, but I don't think that this pseudo maintainer became the only maintainer.

**Matt Coles (25:22)**: He was not, or he or she was not, but, uh, but still had maintainer access. And so you're suggesting that, again, this goes back to the, the individuals involved one of them, at least one of them, with the power to stop a build or stop a release would need to have to do the right thing.

**Izar Tarandach (25:41)**: So proper stewardship of projects should be part of this, this definition of this best, best behavior guide or whatever, right? And explain to people that, yeah, you may want to abandon your project, but if you are a foundational thing, then you have a responsibility. So make sure that you pass that to someone with responsibility.

**Matt Coles (26:04)**: So now you're introducing the new, a new part. So the part, the other piece about this XZ issue wasn't that it was XZ, not that XZ itself, not, not just the fact that XZ itself was, was compromised, but the fact that XZ was a component of a component of a component.

**Izar Tarandach (26:20)**: Mm-hmm.

**Matt Coles (26:21)**: Ah, so XZ was compromised. SSH was the target.

**Izar Tarandach (26:28)**: Yep.

**Matt Coles (26:28)**: Right. And so, and through some intermediary, systemd, that allowed this chain of events to come together to allow this backdoor to exist.

**Izar Tarandach (26:38)**: It's always systemd, isn't it? Systemd.

**Matt Coles (26:42)**: But this, so this introduces a new problem. So now we're not talking just a maintainer of Xe that needed to discover that there was injection, right? That there was tampering of the codebase. But that the side effect or the net effect was in this other component. That other component, should they have known that this thing could have been brought in by systemd and therefore I need to make sure that all my transitive dependencies are good, are safe? And in the case of pluggable architecture like this, my understanding is these are modules that can be loaded by systemd sort of at runtime, I think. I'm not really up to speed on all the details, but I believe that this is something that gets plugged in sort of, you know, loaded. I'm thinking like PAM or something, you know, something that gets loaded at runtime. So would they even know? That XE was a component in the path.

**Izar Tarandach (27:55)**: Hmm.

**Chris Romeo (27:56)**: Yeah. I mean, there is, there is a— there's always been a, a danger with transitive dependencies because you add something and you don't really know. And that's one of the hidden dangers. And that's one of the things that we saw in the Sonatype State of the Supply Chains report from last year was that 7 out of 8 dependencies were transitive in applications. And so it's a big problem just from the general vulnerability issues that exist in the open source ecosystems. But when you have somebody trying to nefariously introduce a problem, you could, if they pick the right library, it could ripple through the entire—

**Matt Coles (28:39)**: Yep.

**Chris Romeo (28:40)**: Applications, everybody, like everybody, you could find something that, that, and that's an interesting analysis that I don't know if anybody's ever done, but I'd love to see like where, what are the packages that are the most used in other packages? So like where could you go where you could pick something that would be foundational that would flow through almost everything? Like OpenSSL.

**Izar Tarandach (29:01)**: That's for Node.

**Chris Romeo (29:03)**: Oh, for Node specifically.

**Izar Tarandach (29:04)**: For other stuff, yeah.

**Chris Romeo (29:05)**: I mean, think about the OpenSSL and just going back to the OpenSSL vulnerability from Heartbleed. It has nothing to do with any of this supply chain stuff. But what we learned was how reliant we were on OpenSSL as an industry because it was in every security product. Anybody that was doing any type of TLS or protocol-based cryptography was using OpenSSL to do it.

**Izar Tarandach (29:29)**: LibC or a compiler or a JIT interpreter. Any of those foundational things, get something in there, you're owning the whole world.

**Matt Coles (29:41)**: Yeah. Yeah. libc would be your, I mean, and imagine how, I mean, I don't know how many lines of code are in libc. I imagine it's millions.

**Chris Romeo (29:50)**: Yeah.

**Izar Tarandach (29:50)**: I'm getting depressed by the minute here. Yeah.

**Chris Romeo (29:53)**: I mean, what do we, what do we do about it though? Like what are, what are like coming back to your original question that I tried to answer about what, what should an open source maintainer do about this? Like what can they do about it?

**Matt Coles (30:06)**: Well, I think we, it isn't just, well, and to add to this, we need to be careful. We collectively, the industry, need to be careful about putting all of this burden on solely on the maintainers. As we saw with XZ specifically, the burnout of the primary maintainer led to an erosion of the trust but verify or assurance case and allowed somebody with— there were warning signs, I think, just based on the information that, you know, in the, in the sort of the analysis that was done, there were some interesting things were done and a on-the-ball maintainer who wasn't like ready to give up the ship anyway, um, may have noticed or may not have noticed, but, but may have noticed. Right? And so, this needs to be a group effort. And on top of that, as an open source system, this is the nature of open source. In theory, it should be inspectable by everybody.

**Chris Romeo (31:14)**: Well, that brings us back around to the funding problems. And I'm thinking about this because I just did an AppSec podcast episode with Mark Kerfee and Simon Bennetts talking about— how ZAP has now, Mark's company Crash Override is now paying for Simon and the other guy. I'm sorry, I don't remember the other gentleman's name who's part of the core team for ZAP because the OpenSSL thing, OpenSSL thing just didn't work out however that was going. So you can tune in for all the details, but he really, they really got me thinking about continuously about this problem of we're not funding open source. Companies that rely upon open source are not funding open source. So Matt, in the example you just gave, you said, oh, well, the maintainer was like at the end of their rope and they just couldn't do, they just couldn't do it anymore. Well, part of that is that they never had any funding to do what they're doing. Yet people relied upon this component to do other things that they made money for. And so if you have a world where there's a better funding model, maybe that maintainer doesn't get burned out because they've been doing this for free 40 hours a week in their free time.

**Izar Tarandach (32:24)**: That, that's, that's an interesting one, but, uh, it's not only an open problem, it's an open sore for some people because there have been so many ideas on how to do that funding and nothing picked up. But let me offer a different view, and, and I could be, I could be talking out of my behind here, probably am, but what else is new? Yeah, what else is new? But the thing is, we, we're focusing on still on the making pipeline. And my gut feeling is that nowadays, especially in this golden age of, uh, AI this, AI that, and AI the other one, we are finally at a point where it's much easier to do behavior analysis on a system than it was a couple of years ago. So rather than trying to prevent something that offers so many bumps on the road to actually get prevented, and, and some frankly, very hard to resolve problems. Right. I was going somewhere with this.

**Matt Coles (33:31)**: Yeah.

**Izar Tarandach (33:32)**: Why don't we focus on actually making sure that the behavior of the systems is expected and that you're not getting, I don't know, SSH connections from some places where you weren't getting from the past week. Well, so let's call that a problem.

**Matt Coles (33:48)**: Let's talk about that for a moment. So first off, you're talking about—

**Izar Tarandach (33:51)**: are you talking about half a second on your SSH connection? Sorry.

**Chris Romeo (33:54)**: Yeah. Yeah.

**Matt Coles (33:55)**: Well, I guess, are you, are you talking about the runtime behavior or are you talking about the project behavior?

**Izar Tarandach (34:02)**: I'm talking about the deployed system. The thing is out there.

**Matt Coles (34:05)**: Yep. And so you know about fuzzing, right?

**Izar Tarandach (34:09)**: Yeah, I'd like to. Yeah.

**Matt Coles (34:11)**: How many iterations have How many iterations of fuzzing is enough to know that you've covered all of the bases?

**Izar Tarandach (34:19)**: Well, it depends how fuzzy it is, but no. Oh, damn.

**Chris Romeo (34:22)**: Sorry.

**Izar Tarandach (34:22)**: Did we get you? It's my dispatch thing saying that I have water in my— forget about it. Anyway, it's silent. Why is it ringing?

**Chris Romeo (34:32)**: Yeah, I don't know.

**Izar Tarandach (34:35)**: It's electronics. I can deal with this stuff. So, my point is, it's not a matter of forcing the behavior with fuzzing or something, but it's actually observing this huge, immense, gigantic— I don't have enough adjectives— volume of data that we have coming from our systems, looking for that half a second delay on an SSH.

**Chris Romeo (35:07)**: Connection. But the challenge when you go down that road is the people that are trying to hide the thing that you're looking for, no, they, they, they come to learn the criteria that you're using and they just find ways to not introduce a half a second SSH.

**Izar Tarandach (35:27)**: It's an arms race.

**Matt Coles (35:29)**: Or introduce a new half a second, right? If, if I were a savvy attacker, maybe I would have introduced a crypto function that would've required entropy and needed to delay for a little bit to get that entropy.

**Chris Romeo (35:40)**: Maybe he introduced it 6 months ago so that when you put the actual attack into place, it all lines up in the behavioral. So I mean, that's the reactive, these are, I think I'm thinking of what you're describing is the reactive side of it.

**Izar Tarandach (35:54)**: It's the detective side. The point is that you have to be detecting the right things. You have to detect the right anomaly. So over time, anomalies that repeat themselves and get explained, become baseline behavior, right? So you're not going to be looking at them. But if at the end of the day what you have is to exploit these things, somebody can get that private key, generate keys, and go into your systems, and all of a sudden you have SSH connections coming in from the moon, that's not really something that usually happens around here, guys. Perhaps we have to, to look at it a bit more.

**Chris Romeo (36:30)**: I mean, I would— my expectation in the modern enterprise now is if I have SSH connections coming in today from any place that's outside of the norm, my SOC and my incident response team better be running that thing down and crushing it.

**Matt Coles (36:45)**: Whatever.

**Izar Tarandach (36:45)**: Not everybody has a SOC.

**Matt Coles (36:47)**: But also, who's responsible for that? Like, so let's, let's talk, let's talk through that scenario. So XE gets the backdoor, gets injected the SSH process. The SSH process is built into a system that is deployed at a company. At a lot of companies. Who's responsible to do that anomaly detection? SOC of most probably of every individual company for their own individual use cases to find anomalies.

**Chris Romeo (37:22)**: Yep.

**Matt Coles (37:22)**: At what point do they say, wait, this is a problem in SSH?

**Chris Romeo (37:26)**: I mean, I think that's, that's what we see happen through the ISAC processes where people are doing information sharing vertically in industries. Like there are some channels for communication, which you probably know better than I do 'cause you're closer to those channels and things, right? And I know they're not gonna be perfect, but I think they will do, they'll give us something, right? It will bubble up to the top at some point across multiple companies.

**Matt Coles (37:55)**: But will it be timely, right? And will it be appropriate attribution? I guess, yes. Multiple companies may say, hey, I'm seeing random inappropriate traffic on SSH. And some of those are persistent connections. Okay, great. That sounds like an attack in progress. Does that scream vulnerability in a component?

**Chris Romeo (38:25)**: Well, that's where your incident response team is going to have to jump in. And dig into it, right? Like, and this is kind of Isar's, he's calling it detect, the detection work phase.

**Matt Coles (38:36)**: It is detective work, but it's—

**Chris Romeo (38:37)**: I'm calling it a reactive measure. It's more reactive though, 'cause we're looking after the fact, right? It's not proactive where we're on the front side and in the open source project looking for it. And maybe we need more—

**Matt Coles (38:49)**: It's root cause analysis. It's root cause analysis, right? Effectively, it's looking at behavior Recognizing that behavior as inappropriate or, or incorrect, and then figuring out why is it incorrect. But the problem is, I think in this scenario, those connections would be, would be chalked up as somebody scanning your network or somebody is credential stuffing.

**Chris Romeo (39:12)**: Yeah.

**Matt Coles (39:12)**: Not somebody introduced a backdoor in a component that, you know, that is used by this thing to, um, to, to gain access.

**Chris Romeo (39:20)**: So I think we, I think the, the, Where I'm landing here as far as a potential solution is there is a proactive angle to this where open source maintainers need to get better at and have better criteria for how they allow people to access the code. Izar mentioned the Linux kernel is a good example of a model. Now, not every project has the depth of resources that the Linux kernel does, but there's a model and there's probably a simpler thing. There's some lessons you could extract from their approach that could apply to everybody. And then on more of the other side of the coin, having this information-sharing style approach where things that we didn't catch in the proactive phase of when somebody slips one of these things through, you do still have to have a reactive nature or detective nature to find a new vulnerability that's been introduced, you know, no matter where— at that point, no matter where the source came from. Right? Like, they'll trace it back eventually.

**Izar Tarandach (40:18)**: And in the middle, there's the step where nowadays we, we have some great companies out there who make their products available for free for open source projects. But not every open source project knows that this exists or thinks that they need to make use of it. So there's that bit of hygiene in the middle that we have to somehow enforce to open source maintainers. And tell them, hey, listen, if people depend on you, then that's a responsibility. You have to have some hygiene in place.

**Chris Romeo (40:47)**: Yeah. But there's no governance that makes them do anything with it.

**Matt Coles (40:52)**: And also, I think that there's a danger there. Again, you're putting a lot of pressure on those maintainers, not necessarily inappropriately, but if I'm a struggling maintainer, like I'm doing this because I really enjoy doing it, not because anything else, and somebody says, well, You're the linchpin to the internet. Oh, okay. I got it. Someone else did you buy, right? I mean, how many people do you push out of open source development as a result of that?

**Izar Tarandach (41:24)**: Yeah, true. Very true.

**Chris Romeo (41:26)**: Which we can't— I mean, yeah. Yeah. So yeah, we can't— we don't even have time today to get into a governance argument about open source. Like there is no governing body. There is some loose attempts. There are attempts in some organizations, but there isn't really anybody who's governing open source right now. But that's, that's going to be an issue for another day. So I don't know if we made any progress on this issue or not. I feel like we at least threw all the pieces out on the table for people to, to wrestle with and understand. And who knows, maybe somebody will come up with something better than we did as a possible solution. If you do, let us know so we can take credit for it. Just kidding.

**Izar Tarandach (42:03)**: We'll actually—

**Chris Romeo (42:04)**: we'll—

**Matt Coles (42:05)**: sorry. Before you, before you end, I think there is one thing we haven't talked about. And I'm— actually, we have, we have talked about it, but we haven't talked about how to make it work. So, as I mentioned, the Linux kernel as an example, they do character by, you know, effectively character by character code reviews. If, if every commit and every bit binary or text binary ASCII content of a, of a project gets reviewed, where there's a decision tree, right, or decision log. Why does this exist? Where did this come from? All of those things can be analyzed. Where does it go, right? What is it used for? If all of those things can be, can be looked at, then you're probably 70% of the way there. Now, can a human do that? Maybe not. Can a human— sorry, can a human alone do that? Probably not. Can multiple humans as a crowdsourced effort or an AI assistant-assisted effort do that? Maybe. Maybe that's what we need to look at.

**Chris Romeo (43:15)**: Yeah. Now, that's— it could be an interesting intersection when we think about all the things we've talked about in the past months about AI and how it's going to come into play. Maybe this is an angle. That AI can have some real impact with. So folks, thanks for listening once again to the Security Table. We hope you enjoyed this episode. And if you did enjoy it, tell a friend. If you didn't, don't tell anybody.

**Izar Tarandach (43:41)**: And now I don't want to run anything anymore.
