# Why 100X Isn't the Answer

URL: https://securitytable.ai/why-100x-isn-t-the-answer/
Published: 2024-11-07
Duration: 2694 seconds
Season: 2
Episode: 31
Hosts: Chris Romeo, Izar Tarandach, Matt Coles

## Show notes and chapters

A good discussion today covering two different articles, the first covers CISA's list of product security "bad practices", questioning whether it provides real value or is just content marketing. Then the discussion moves onto an article about Shift Left. The group debates whether it is truly more expensive to fix design flaws versus implementation bugs, noting the difficulty of quantifying the cost difference. They argue that the focus should be on providing proper training and incentives for developers to build secure software, rather than just adding more security tools. 

Articles discussed in the episode:

[Product Security Bad Practices](https://www.cisa.gov/resources-tools/resources/product-security-bad-practices)  
  
[Shift Left Pushback Triggers Security Soul Searching](https://www.darkreading.com/application-security/shift-left-pushback-triggers-security-soul-searching)

  
  

Follow AI Security Table:

➜ Home: [https://securitytable.ai/](https://securitytable.ai/)

➜ X: [https://x.com/AISecurityTable](https://x.com/AISecurityTable)

➜ LinkedIn: [https://www.linkedin.com/company/ai-security-table/](https://www.linkedin.com/company/ai-security-table/)

➜ YouTube: [https://www.youtube.com/@AISecurityTable](https://www.youtube.com/@AISecurityTable)

## Transcript

**Matt Coles (0:09)**: Google newsletter. Trying to sign up for the Google newsletter with an ad email valid email address from Google and it doesn't let me with my own email address. I think this is insane.

**Izar Tarandach (0:20)**: Did you try with two ads? Like your first one and then a second one at Gmail.com? and it drops the second.

**Matt Coles (0:28)**: I tried the plus sign with a, with a tag so I can filter and it doesn't like them.

**Chris Romeo (0:33)**: So welcome to an episode of Matt's Inbox.

**Matt Coles (0:36)**: Oh yeah.

**Chris Romeo (0:37)**: And setup process.

**Izar Tarandach (0:39)**: We have an MLM for that.

**Matt Coles (0:40)**: We do.

**Chris Romeo (0:41)**: This is a segment we do once a, once a month where we, uh, fix Matt's IT problems.

**Matt Coles (0:50)**: Fix. That's a, that's a funny word. You, I, You keep using that word. I don't think it means what you think it means.

**Chris Romeo (0:56)**: Hello? Hello, IT? Did you turn it off and then back on again?

**Matt Coles (1:00)**: Hello, computer? Computer?

**Izar Tarandach (1:02)**: I'm sorry, but as somebody whose initials is IT, I think that's very—

**Chris Romeo (1:09)**: That did mess me up one time when you joined a call and I was like, who, why is the IT department here? I'm like, IT, get the heck out of here. We don't need you. We fix our own computer problems and like—

**Izar Tarandach (1:21)**: Oh, but that's, people keep telling me even if my initials were something else. Go the way you're built.

**Matt Coles (1:27)**: If, uh, if nobody gets the reference to, uh, to that, to that quote, or computer, computer, it has to be in, it has to be in a Scottish accent. Uh, so I can't do a Scottish accent. I can only do a British accent. Is that from Her?

**Chris Romeo (1:42)**: Is it Her? The movie Her? No. What is it from?

**Matt Coles (1:45)**: It's The IT Crowd. The IT Crowd?

**Izar Tarandach (1:49)**: No, but wait, wait, wait. Where did it come from before? Ah, ah, ah, ah, sorry, you're missing it.

**Matt Coles (1:55)**: Uh-oh.

**Izar Tarandach (1:56)**: It comes from Star Trek, from one of the movies.

**Chris Romeo (1:59)**: Oh, really?

**Izar Tarandach (2:00)**: Where they travel to the past and they are shown a computer. And the first thing that Scotty does is grab the mouse and go, computer.

**Matt Coles (2:09)**: Hey, computer.

**Chris Romeo (2:10)**: That's right, I remember that.

**Matt Coles (2:12)**: Okay, it was funnier still on the crowd though.

**Izar Tarandach (2:14)**: Oh yeah, that was better.

**Chris Romeo (2:16)**: The IC crowd is something that the security table recommends that you check out, especially if you've ever worked at a help desk job.

**Izar Tarandach (2:28)**: Especially if you want to understand why we are so cynical sometimes.

**Matt Coles (2:32)**: Yes. And if you like British comedy, of course.

**Chris Romeo (2:34)**: Oh, of course. It's brilliant. Brilliant. I laugh every time. I've probably watched it 5 times at this point.

**Matt Coles (2:41)**: That was the first episode too, I think. The good one.

**Izar Tarandach (2:44)**: I think so. But that's a reboot waiting to happen.

**Chris Romeo (2:47)**: They tried. They were going to bring Moss to America. It was going to be an American version of it, and they did the pilot, which I saw somewhere. I watched the pilot and it wasn't good at all. They lost the magic. Like, what's that guy's name? The guy that plays Royke, first something, the Irish guy. Um, without him, he's like the, the, the match to Moss. Like, you need his over-giant personality with Moss's like lack of personality.

**Izar Tarandach (3:16)**: But the boss, the boss, the boss stayed.

**Chris Romeo (3:19)**: Which one?

**Izar Tarandach (3:19)**: I mean, the big boss, the guy that later on went to what we do in the shadows.

**Chris Romeo (3:25)**: Yeah, he's a big Broadway, uh, London theater performer. Keith Olsef. Yeah, I can't remember his name. It's Renom. Renom was the original boss, and then I don't remember what that What's that guy's name?

**Matt Coles (3:39)**: Okay.

**Chris Romeo (3:40)**: It'll come to me later while we're talking.

**Izar Tarandach (3:42)**: Oh, what we going to do today?

**Chris Romeo (3:45)**: We're not just going to analyze The IT Crowd? No. For the whole episode? Okay.

**Izar Tarandach (3:49)**: And this is not going to turn into Pinky and the Brain.

**Chris Romeo (3:52)**: That's true.

**Matt Coles (3:53)**: All right.

**Chris Romeo (3:53)**: Well, our first item on the docket is apparently the good folks at CISA produced a set of product security bad practices.

**Izar Tarandach (4:05)**: Something we've been doing for thousands of years already. We have best practices everywhere. We produce them every day.

**Chris Romeo (4:12)**: You've been building products for thousands of years?

**Matt Coles (4:15)**: It feels like it.

**Izar Tarandach (4:15)**: Accumulated, accumulated.

**Matt Coles (4:17)**: Accumulated. Number of products, number of years per product. Yeah, yeah. Thousand years sounds about right.

**Izar Tarandach (4:22)**: You know what? Tell you what, sometimes it feels like we have.

**Chris Romeo (4:27)**: So I think that the challenge here with our product security best practices or bad practices, sorry, is it's a bit of a cop-out. And the reason I say that is because all they did was take their document and flip it around and put not, and now they produced a new piece of content. So do we, do we just see content marketing at work here? Or is there any actual new value to what they've come up with as their list of bad practices?

**Izar Tarandach (4:55)**: It's basically formal proofing, right? They, they, they gave you like, one part and then they inverted it and said she's still valid so that thing works.

**Matt Coles (5:07)**: Hold on a sec guys I have to go there's a woodpecker at my door look right back welcome to the medical zoo—

**Chris Romeo (5:16)**: Matt let the record show Matt is literally leaving the recording room to tangle with a woodpecker! All of a sudden we hear Matt go, "Oh, stop, move." And then you hear this.

**Izar Tarandach (5:38)**: Tune in next week as we see Matt chase a coyote.

**Matt Coles (5:44)**: So normally I'm muting because the dogs are barking like crazy, but today it's a woodpecker that's in my, well, Wooden house and woodpeckers, you know, don't mix.

**Chris Romeo (5:55)**: So what is his name? What have you named him?

**Matt Coles (5:57)**: I don't care.

**Chris Romeo (6:00)**: Poke?

**Matt Coles (6:01)**: Poke.

**Izar Tarandach (6:01)**: So just—

**Matt Coles (6:04)**: So we don't get in trouble with the animal rights folks, I won't say dead woodpecker, but chased off woodpeckers.

**Chris Romeo (6:13)**: Yes. No woodpeckers were harmed in the filming of this podcast.

**Matt Coles (6:16)**: Right. All right, so bad practice. Some copper siding maybe. Bad practices, yeah. Shielding, clouding your house in copper siding.

**Chris Romeo (6:27)**: So how could you use the product security bad practices to prevent woodpecker attacks?

**Izar Tarandach (6:32)**: Exactly, you know, that's such a good analogy, like putting those strips in the corners of your house, hoping that's to drive the woodpeckers away. It's so much like some of the stuff here.

**Matt Coles (6:44)**: Well, and of course, the bad practices paper doesn't call out that you're not doing threat modeling, of course.

**Chris Romeo (6:54)**: Yeah. So that would be a bad practice to not do. So that's where this gets confusing is it's like all the negative.

**Matt Coles (7:00)**: Yeah. But they're not, but they're not calling out that they're being very selective of what they're calling out. Right. So they're calling out lack of memory safety, right? They're on, on the memory safety train. We could dive into that. They did, they did actually expand that one. We could talk about that a bit more. But then they called out, you know, OS command injection and SQL injection, but not like cross-site scripting or other vulnerabilities. Why, why that one? Very selective. So in this inversion, it wasn't simply, you know, not having defenses against, you know, common vulnerabilities. It was specifically against a specific vulnerability.

**Izar Tarandach (7:46)**: But it is, because later on they say presence of known exploited vulnerabilities.

**Matt Coles (7:51)**: Oh, sorry. I should say weakness, not a vulnerability, right? So calling it an OS kernel injection specifically.

**Izar Tarandach (7:58)**: It's like, do we really, are we really at the stage that we need somebody to pop up and say, guys, having known exploited vulnerabilities is a bad thing, okay? Let's not do it. By the way, can you sign here and pledge that you won't? It's, you know.

**Matt Coles (8:15)**: Well, I mean, alternatively, alternatively, you could go the way the EU is going, right? So if you, if you're familiar with, say, the CRA, I think it's called, the Computer Resilient Act or something, I forget what CRA stands for. It calls out having no vulnerabilities when you ship something. I mean, so you have CISA saying you shouldn't have vulnerabilities, that's a bad practice, you know, voluntary sign up for doing something about it. And then you have the CRA saying, if you want to ship to Europe, no varnishes.

**Chris Romeo (8:52)**: And it's just— That just shows that their legislation was created by people who have no idea how systems work, how software works at this stage. I would love for us to have a world where we could say this piece of software has no vulnerabilities, but it's just not possible. It's just not even, it's not even, it's just not even in the realm of possibility.

**Izar Tarandach (9:15)**: I'm sorry, it goes even deeper than that. We do live in a world where people say this piece of software has no vulnerabilities with absolutely no base on truth. They just say it because it became a marketing thing. Yeah, we checked ourselves and we found that we are fine.

**Matt Coles (9:31)**: Who we read as, you know, that doesn't work against the piece of technology that we use and therefore didn't find anything.

**Izar Tarandach (9:37)**: So in this world of promises and pledges, you know, we got to a point where people can, they feel good enough to go and declare themselves safe and that apparently is enough for some people.

**Chris Romeo (9:49)**: Yeah, but it makes you back it up. Right. But then, well, SolarWinds is a great example of this, who they would have told you they were safe. A month before their famed incident. And then they lost a lot of customers as a result of their supply chain or their build pipelines being—

**Izar Tarandach (10:08)**: But to go back to our last two episodes, they still have a lot of customers because people are saying, okay, so they had a bad— fine, cool. They still need the functionality. Let's move on.

**Matt Coles (10:19)**: Yeah, they had a bad day. Everybody has a bad day.

**Izar Tarandach (10:22)**: But on the chapter of things written by people who really have possibly not a lot of experience in what they are supporting. When you go down and when you scroll down, you get to the organizational processes and policies. So they made two choices to put in there, failing to publish timely CVEs with CWEs and failing to publish a vulnerability disclosure and disclosure policy. Now, step back and remember, to me, 15 minutes ago, When were you involved in organizational processes and policies of publishing hopefully secure software? Seriously, of all the problems that are in there, these two were the ones that we should go to first?

**Matt Coles (11:06)**: Well, so I have a theory about the CWE one, but the vulnerability disclosure practices is actually a good common sense one, I think, right? If you don't tell your customers how you treat vulnerabilities and you're not upfront about that, there's a lot of FUD that can happen, and having been in industries where vulnerability reporting was not really a thing, it led to a lot of consternation, like, hey, I found a bug in your software. I rely on this software for critical function XYZ. What am I going to do? Is this a problem for you? Right? You go to a support forum. You go to a support forum and you submit an article to a support forum.

**Chris Romeo (11:44)**: Everybody's got that now. Though the bad thing again, I'm not saying that it's bad.

**Izar Tarandach (11:48)**: I'm not even saying that everybody's got it. A lot of people don't. But if you are creating a foundational document, something that people are going to carry for years to come, are these the two hills you are choosing to die on? Is this the impact that you can make industry-wise?

**Matt Coles (12:04)**: I have a theory about the CWE one. I have a feeling that it's really a, we don't, we, and I say we collectively, you know, CISA or somebody in the government or other places, we don't know how big the mountain really is. And so let's add data, let's get companies to add data that we can use to do better reporting later. So by, so today, if you do a vulnerability and you get a CVE, you provide a description, and that description has been historically pretty horrible. Right, has not provided a lot of details about what the vulnerability is. You get, maybe you get a CVSS vector that provides some information that you can pull, you know, does it, is it remotely exploitable and is it disastrous if it's exploited, that sort of thing. But the underlying root cause is never really brought out in those descriptions or very rarely brought out in those descriptions. And so by adding the CWE or CWEs that are involved, you now have root cause. And so if you have this approach where you want to reduce, say, injection and memory safety vulnerabilities, and there's no, but there's no weaknesses, but there's no way to know that you're actually getting memory safety as a root cause of a CVE, adding a CWE to a CVE is a good way to do that.

**Izar Tarandach (13:30)**: Yes, but going back to the, to the documents, They say that for products used in service of critical infrastructure, NCFs, it is dangerous and significantly elevates risk to national security, national economic security, and national public health and safety for the software manufacturer to not issue CVEs in a timely manner. Nobody says what timely manner it means. For a minimum of critical and high-impact vulnerabilities, right? Additionally, it is dangerous and significantly elevates risk to national security, national economic security, and National Public Health and Safety to not include the CWE field in every CVE record. What I'm missing here is how.

**Chris Romeo (14:14)**: Well, there's no teeth. That's been my whole challenge with everything that they've done here. There are no teeth in this thing, and the requirement is wishy-washy.

**Matt Coles (14:24)**: I signed it first though.

**Izar Tarandach (14:25)**: But wait, it's not even teeth. It's we should have data to support this assertion. Tell me, when was the last time that national security got impacted because somebody didn't put a CWE in a CVE?

**Matt Coles (14:38)**: Okay, so here's the thing, right? If you have just— so if you have just a CVE with a description, and let's just say for a moment NIST is still bogged down in trying to get— doing CVE reviews and doesn't have a vector associated with that severity. So you have a severity, but you may not have a vector. You may not even have a severity, you may just have a description of a vulnerability. How do you prioritize that? And how do you prioritize that effectively? I'm not saying this is the best answer, but this is my theory, right? It adds data to a description which historically has been weak in providing information to a defender.

**Izar Tarandach (15:15)**: I'm sorry, I watched all of 24. I watched all of the Tom Clancy movies.

**Chris Romeo (15:21)**: Okay.

**Izar Tarandach (15:22)**: I watched a bunch of Clint Eastwood, not Clint Eastwood, Harrison Ford ones.

**Chris Romeo (15:27)**: What about Jack Ryan?

**Izar Tarandach (15:28)**: Jack Ryan, that too, and that's like the newest one.

**Matt Coles (15:32)**: Before or after Clancy was alive?

**Izar Tarandach (15:35)**: I haven't heard one. Don't get me started there. I haven't heard one single line of dialogue that begins with, "Qui, what is the CWE? This is a matter of national security." I mean, they're saying here it is dangerous and significantly elevates risk to national security. So we probably have something to point back and say, oops, this happened if we only had the CWE.

**Chris Romeo (16:00)**: Yeah, no, you've pointed that out. You've pointed that out. I didn't catch that the first time I read this. And so I am, as you continue to talk, I've been reading and rereading that statement. It's pretty darn funny that That basically the world is going to end if you don't put the C-W-E field in every C-V-E record. The world will end. Society as we know it will disappear. Zombies will roam the earth. Armageddon, apocalypse, whatever. Oh, he's just— the end is beginning now. He's not able to talk.

**Matt Coles (16:37)**: We saw that.

**Chris Romeo (16:38)**: I heard that. Nope. It's the beginning of the end.

**Matt Coles (16:51)**: One minute he's there and the next minute Cees is listening and going, oh, I blocked him.

**Chris Romeo (16:56)**: Look, I, I, I, what can I say?

**Izar Tarandach (16:59)**: They love me at the NSA.

**Matt Coles (17:02)**: Hoots. It's not the NSA you're worried about.

**Izar Tarandach (17:05)**: And by the way, guys, I love you back. I love you both.

**Matt Coles (17:09)**: Do you still start every Zoom call as, hello, whoever you are on the line? Yeah.

**Izar Tarandach (17:16)**: Plus I have black helicopters over here all the time.

**Matt Coles (17:21)**: Well, that's because you have a famous person running through the woods who you don't know what she's doing in there.

**Izar Tarandach (17:25)**: No, it's because I'm the line between the Hudson and the Sikorsky factory. It go by one day.

**Chris Romeo (17:32)**: All right, we've probably talked about this one enough. For the amount of new material.

**Matt Coles (17:37)**: We've just gotten started, but, uh, Shift Left is something else.

**Chris Romeo (17:42)**: All right, second item on the docket is the Stark Reading article, Shift Left Gets Pushback, Triggers Security Soul Searching. So I feel like we've been throwing some noise at Shift Left. I've been throwing noise at Shift Left for at least a year or two because I actually wrote a post about making fun of, I went and Googled Shift Left. Found a bunch of vendors that were using it in different ways and then made fun of each one of them individually. So I feel like that shows I was not, you know, I was one of the first people to turn on the whole shift left idea, but apparently there's a movement now. We've started something. We should get t-shirts.

**Matt Coles (18:21)**: So, I mean, my mantra has always been start left, don't shift left, start left. But if you were breaking down the bear, if you're breaking down the concept that shift left is an invalid concept, then start left is invalid too, because it fundamentally goes to who's responsible for what. And then that's highlighted in this article, right? So this is an article by, mostly by Acquia, but written by Dark Reading. And, you know, the question is, so shift left is, of course, the notion that security needs something done and tells developers to go do it.

**Chris Romeo (19:02)**: Yeah.

**Matt Coles (19:02)**: And, and well, I should say, that's the, that's the premise of this article. The, the, the idea really should have been was that developers with, and engineering teams when they're building software or products or systems or whatever, should have looked at security as one of the properties of that system. Like they do quality today.

**Izar Tarandach (19:25)**: Let's—

**Chris Romeo (19:25)**: Matt, before we go too deep into this, let's deal with that first paragraph because there is, there is some— they are basically pointing at this 100x number that we've thrown around.

**Matt Coles (19:40)**: Whose—

**Chris Romeo (19:40)**: the first time I found it published was in an IBM. I found it referenced in an IBM document from the late '70s, but they've even got it. They've got it further back to TRW. Yeah. And so they're, they're basically saying that the 100x metric doesn't apply today. But I guess the question I have for you guys, I'd love to get your take on this, is do we agree or disagree with this statement that it is cheaper to fix a problem during the design phase than it is in production? —when it makes its products. Yes, but not for the same reasons that may have started the 1:100 thing.

**Matt Coles (20:21)**: And yes, but with a big caveat. Okay. So let me go first with the caveat. Modern application development, as you guys have tried to hammer into my brain for the past X number of episodes, I'm still stuck in a we build hardware kind of world. But if you're using, you know, if your entire development is taking other people's code and configuring it, so you're actually not doing a lot of custom development, you're doing a lot of stitching of, you know, microservices and Kubernetes and, you know, web frameworks together to make stuff happen, then maybe that doesn't hold because what is your design phase? Your design phase is really which components do I put together and how do I stitch them together, not architecting one of those components from scratch, right? So I'll say that caveat is, are you throughout that maybe not in the way we are expecting or thinking?

**Izar Tarandach (21:26)**: 'Cause we have that model stuck in our heads that you create software, you publish software, you sell software, and then there's something that you have to fix and oh my God, now we have to run around and replace all existing versions and all that stuff. and that connects back to the hardware aspect of the thing. And that's damn expensive. Nowadays, when people claim to be, what is it, deploying thousands of times a day, Lord knows if I ever seen that. Like, I've seen people claim that. I mean, people doing that.

**Matt Coles (21:58)**: I think the last number I've seen was Amazon, every 8 seconds, they, in theory, could push to production. Right? Yeah.

**Chris Romeo (22:04)**: So, what they define as production is a very vast ocean.

**Izar Tarandach (22:09)**: Yeah, and what they define as deployment is also, but anyway, but today it's easy to bypass, like it's cheap to bypass that, oh my God, there's something wrong. Click a button, everybody gets the new version, everybody's happy. If they haven't used the software in 8 seconds, then they haven't even noticed it. But the point is that we are talking about design here. So design in my very much biased head means that the fix is probably not a 3-line patch. It's a lower-level kernel. And still fixing that, changing that, and I'm sorry, I meet that every single time that I speak with a startup and I go, oh, but you guys have SSL support, right? And they go, no, but it's going to be trivial to put it in. And then trivial turns into a battle. And are you guys really, really sure that you want this? Don't know. Our password management stuff is much better than any SSO. No, no, seriously, you don't need this. It's okay. Listen to me. It's okay, brother. Don't you trust us? You should, you should. You totally should. But my point is that real, real design flaws, are probably still way more expensive to fix than just deploying the newest patch, right? So 1 to 100, I have absolutely no numbers to support anything that I say like that. So that's why I don't use the 1 to 100.

**Matt Coles (23:48)**: It's more, it's more, but it's honestly a particular like number more.

**Chris Romeo (23:53)**: Well, you think it's— So hold on, wait, what does that mean? You just said that it is more than 1 to 100. It's relative.

**Matt Coles (23:59)**: It's relative. It is more than fixing, than patching in the, you know, patching a software bug. Yeah. If you have to change your architecture or you have to make a design change, but is it 100x?

**Chris Romeo (24:09)**: So it's exponential. You're saying it's an exponential change in terms of greater problem. No, no, no, no, no, no, no. It's greater.

**Izar Tarandach (24:15)**: Well, on a, on a comparative scale, it's probably more expensive to fix design flaws. Right. Probably.

**Matt Coles (24:23)**: And Almost certainly, almost certainly more expensive.

**Chris Romeo (24:26)**: We're way too wishy-washy here with this problem.

**Matt Coles (24:29)**: Well, okay. Let's, so first off, we agree, I think we generally agree it's more expensive to fix a design flaw than it is to fix a software flaw, right? To change a buffer versus changing your programming language, it is almost certainly more expensive to change programming language.

**Chris Romeo (24:46)**: Well, that's when you're kind of using an extreme example of one and a simple example of the other. You kind of got to, you're not, you're not, it's not an apples-oranges comparison what you just gave. It's an apples. I understand that. It's an apples and battleship comparison.

**Matt Coles (24:58)**: So give me an apples to apples comparison.

**Chris Romeo (25:00)**: I don't know.

**Izar Tarandach (25:01)**: It's cheaper to change our tire than to redesign your transmission. Okay. Okay? That's a given. It is. It is. The question is how many times you're going over the lifetime of a car, how many times are you going to have to change a tire versus how many times you're going to have to redesign the transmission?

**Chris Romeo (25:20)**: You could say rotate the tire would be an even better example because that's probably— I don't change a lot of tires these days.

**Matt Coles (25:25)**: But my tires are rotated every time I change my oil, right?

**Chris Romeo (25:28)**: That's because you have AAA as a third-party service.

**Izar Tarandach (25:31)**: But you know what?

**Matt Coles (25:32)**: Tire rotation is a preventative measure, not a reactive measure.

**Izar Tarandach (25:36)**: Ah, good point. Good point. Eventually, you're going to have a flat, especially if you drive in New York. My God. But my point is that in some cars, you won't ever have to redesign the transmission. But when you have one that does get a recall because the transmission is bad and it has to be redesigned, That's going to be pretty expensive.

**Chris Romeo (25:57)**: Or replace, right?

**Matt Coles (25:58)**: Replacing a transmission, right, is a pretty serious change to your vehicle.

**Chris Romeo (26:06)**: That's like a software— the transmission in this example has become a software component. If I'm replacing it, it's a software component. If I'm redesigning it, that's a whole—

**Izar Tarandach (26:15)**: that's an architecture flaw. That's a design flaw. That's a design flaw. Yeah. Right? You know, it's not something that you're going to do every Tuesday and Thursday. But when it happens, it hits you, right? So—

**Matt Coles (26:29)**: But is it definitively 100x? I mean, you can't put a number on it until it happens. And then you get— you start to build up data.

**Chris Romeo (26:37)**: We don't really need a number. That's what I'm saying. No, no, wait, wait, wait.

**Izar Tarandach (26:39)**: You know what? Redesigning a transmission is a bad example. Remember a couple of years ago that there was some car that was found that if it gets hit at a certain angle, the—

**Matt Coles (26:49)**: Pinto.

**Izar Tarandach (26:50)**: The gas tank. Would explode. That's exactly it, right?

**Matt Coles (26:54)**: Well, actually, you could even do a modern example of you have a big shiny box of a thing that has door angles and things that don't open or close properly and no handles when it's frozen and that sort of thing. I mean, yeah, you could go back to the Pinto or you can come up with something a little bit closer.

**Izar Tarandach (27:16)**: Yeah, that's a solution with other use case if I ever saw one. But anyway, so my point is, yeah, it's not going to be a cost that you are putting out of your pocket every week, but when it happens, it hits you hard. The point is, the question here is one versus the other, because the vulnerabilities, you probably are spending that money many times over the lifetime of a product. Does that cumulative cost compare to design flaws? And is that at all something that we should be doing? Because my point is, even if it's well-designed, those vulnerabilities will still keep coming up because of what Matt keeps telling us, that it's a quality thing. And just to go back to the developers, developers don't want to deal with this shit.

**Chris Romeo (28:06)**: So let's see if we can boil this down into a final nugget of truth. So it sounds like we are, we are all saying that it will be more expensive to fix any, any, any software, any, any issue you have with something that you're building, it will be more expensive to fix it in production than it would be to fix it earlier in the lifecycle. Well, we are, what we're saying is we're not ready. We're not going to attach 100x, just like the article said, not willing to put 100x number on it anymore because it might only be 2x. It might be 1.5x. But the point is, If somebody, and this is how I've always explained it to people, is you release something to production, 3 to 6 weeks go by, somebody finds a problem with it. As a developer, you get a bug ticket. Your brain has already moved on. You've grabbed new user stories. You've removed the history of that feature you built from your memory banks. You've loaded in a new one. You're in the middle of working on it, designing it, making it work. Now you have to unload all the things in your head now, reload what you did 6 weeks ago, and then that's rework, right? That's the whole principle of rework. That is going to be more expensive than if you had just done it correctly and not let the issue be created when you first built the feature.

**Izar Tarandach (29:33)**: You know, yes, but wait, 2 things. First of all, what you gave is just a great a great point for shift left, right? Well, what you just described—

**Chris Romeo (29:42)**: Whoa, whoa, then I take it back.

**Izar Tarandach (29:43)**: I forgot to correct you. What you just described is the shift left argument, the right one. And the second thing that I want to add is that we've been looking at economic costs. There's also another cost in there for the team, which is the pain of having to revisit the foundation of the thing that you built and say, guys, we should have thought it before.

**Chris Romeo (30:04)**: And that's an economic— it all boils down to economic costs though. Like, listen, I run businesses. It's deeper than that. But at the end of the day, to me, it comes down to my cost and my return on investment from my development engineering team.

**Izar Tarandach (30:19)**: It's a bunker boot— buster. Buster for team morale. Yeah. So I have to go back. Yeah.

**Matt Coles (30:27)**: No, wait, finish your thought because I'm actually, as I'm rereading this over and over again as you guys are talking, I think we're missing something too on top of that that does— finish your thought. Go ahead. Well, so just to continue your discussion, the thread you were on, what's the cost to the business if they release bugs, right? Bug over bug over bug or exploitable vulnerability over exploitable vulnerability or breach-capable software versus not. So there's some additional costs there which are not being factored in maybe. 100x is still a funny number. But as I reread this, there's something else that is interesting here. So, the very first paragraph about this talks about, is it better, is it cheaper to fix a software bug in design time versus fixing it in production? And you could take this and then it says, well, and, but this requires developers to buy very expensive tool you know, from vendors, right? So it's pushing a vendor-based or tool-based approach. But do they really mean design bugs? We've been arguing about design bugs, right? Or do they really mean fixing vulnerabilities in pre-production or during development? So, right.

**Chris Romeo (31:53)**: I think that's really more what has been, I mean, we've made, we've taken it to what we love, which is threat modeling, design, things like that, that we're kind of, we're putting our own security table colored glasses on and looking at it and how we like to think about it. But I think you're, I think you're 100% right. That's not how most people think about it. Right.

**Matt Coles (32:12)**: So if you're fixing, if you're talking about fixing a bug, I mean, so for decades, people have run static code analysis tools, right? And if you find 100,000, you know, string formatting errors, You're not going to fix them all, right? But you may buy a tool to say, okay, which ones of these do I have to fix? Because it's on my attack surface, I do some penetration testing or whatever, and these are exploitable. I do some fuzzing and these are the ones I'm going to worry about. You can prioritize them. Now you're adding tools to solve a, which needle, which of these needles in this haystack do I want to pull out before a customer steps on them? Okay. Right. and there's an argument to be made about who is responsible for security and/or quality of the system and who really cares.

**Izar Tarandach (33:02)**: Just to support that, a while ago I heard from someone, since the treaty, the person said that they run a, I don't remember what kind of tool, but they found so many bugs that they didn't fix any.

**Matt Coles (33:19)**: We gotta have a term for that. Look at Qualys, look at Fortify, look at any of the scanners or static analyzers that, I mean, hundreds of thousands of issues. And because of bad practice. It's a fine new overmold, but it's bad practice, right? So if you don't initialize a variable, which is a pretty common development practice, Do you have an exploitable flaw? Whether it's a vulnerability or not, I'm just talking, is it the quality or is it breaky?

**Chris Romeo (33:50)**: I never initialize vulnerabilities.

**Matt Coles (33:52)**: You can't know. You can't know. It's undefined behavior. I let them initialize themselves.

**Chris Romeo (33:56)**: You probably use languages that allow undefined behavior. Well, lucky you. I need a protective layer around my bad decisions when I code things. I need something that just assumes I meant to do the right thing and then adjusts.

**Matt Coles (34:12)**: Appropriate in the runtime. And that's an interesting thing because, well, if you do that, if you do that, right, then why are you buying so, why are you investing in a lot of these tools? Because the language insulates you from, or the technology choices insulate you from these issues.

**Izar Tarandach (34:29)**: But again, it's what we've been talking about for a long time. It's one more list, one more 10,000 things to do that doesn't tell me what to do. Lists, right?

**Matt Coles (34:39)**: But you have to buy another tool for that. That's why ESPMs and other things exist, right?

**Izar Tarandach (34:43)**: I mean, you have to buy the other single pane of glass.

**Chris Romeo (34:47)**: Well, I think I've had a bit of an epiphany here.

**Matt Coles (34:50)**: I think we have a tools problem in our industry.

**Izar Tarandach (34:54)**: You think? Where are the experts about this?

**Chris Romeo (35:01)**: This is part of the comedy segment, obviously.

**Izar Tarandach (35:04)**: Okay, wait, we have a segment?

**Matt Coles (35:06)**: This whole segment, the whole series is a comedy routine. There is more.

**Izar Tarandach (35:11)**: When you go down on the page, you get the beautiful heading, not the metrics you're looking for, right? So yeah, it's not about money. It's not about top 10 lists. It's not about anything. To me, again, the ship left thing, keeps coming back to the, yes, it makes perfect sense, but has anybody actually asked the people who are supposed to do those things if they want to do those things? And the answer that I keep going back to is no.

**Matt Coles (35:46)**: But again, whose responsibility is it? So in the article, there's a discussion here about security wants security fixes. Why does security want security fixes? Shouldn't developers want security fixes? Who does quality? Quality doesn't want quality fixes. Developers want quality fixes, right? That's my point.

**Izar Tarandach (36:06)**: Developers don't want to look at things as fixes. Developers want, from my experience meeting and working with developers, they want to do the right thing. They want to avoid.

**Matt Coles (36:18)**: They don't want to react.

**Izar Tarandach (36:19)**: They want to avoid. So the point is that we keep writing these amazing tools, supposedly shifting them left. Putting a lot, a lot, a lot of weight on these 4 developers that are under so much pressure already. And I'm not being cynical here. I fully stand behind saying this. We are just adding pressure to what's already a pressure cooker.

**Matt Coles (36:43)**: But you're fundamentally, but there's a fundamental problem to what you just said. Why do we need to push a tool into that environment? Exactly. It's because they're creating, it's because they're creating buggy software. It's a cause and effect.

**Chris Romeo (36:57)**: Exactly. The tool is— Yeah, I mean, then they're doing things.

**Izar Tarandach (37:00)**: And what's the solution? What's the solution that is mentioned in the article? We have to shift left, left. So start left. Just start left. Start left, but how do you start left? By teaching these people the right— By requirements, training. No, by training. By teaching these people what the right thing is.

**Matt Coles (37:16)**: Yeah. Right. And guardrails and paved roads. It's everything. I think that's what we've been highlighting.

**Chris Romeo (37:19)**: Yeah, it's the whole package.

**Izar Tarandach (37:20)**: That's the catch. But again, no, no, even before—

**Chris Romeo (37:22)**: Is it shifting anywhere?

**Izar Tarandach (37:23)**: Even before guardrails, look, if you teach it right, if it becomes part of the discipline, in theory, you don't need guardrails. I keep going to performance. We don't have performance guardrails, for God's sake. People just write performance code.

**Chris Romeo (37:39)**: I mean, the place I've landed on this and why this has pissed me off for the last couple of years, and I've just railed against this whole idea of shift left and made fun of it at every chance I get. Is because— It's a cynical view. It's really what we're talking about though is secure development lifecycle. You just need a secure— all you need is a secure development lifecycle, a solid approach to process, people, process, tools, governance. The people need to use the process. They can use the tools to support the process and you need a governance angle to run through it.

**Matt Coles (38:15)**: That is a shift left. And your lifecycle has to start with requirements, training, and technology choices.

**Chris Romeo (38:22)**: Yeah, 100%. We keep saying trust.

**Izar Tarandach (38:24)**: We solved it. We're done. No, no, no, we haven't. Wait, wait, wait, wait, wait. We keep saying trust but verify, right? And we keep putting a lot of money and time into building the tools that do the verify part. But we do nothing about the trust part. We don't go to these developers and say, okay, you know what? I will give you the training. I will give you the time. This will become part of your discipline. This is going to be incorporated in the way that you do things.

**Chris Romeo (38:50)**: And then I can verify. So do you want to have a trust score for each individual developer?

**Izar Tarandach (38:54)**: No, I just need to spend some time teaching them the right thing and hoping that a good thing comes out of the end.

**Chris Romeo (39:01)**: I mean, you know, I'm a big fan of training developers because that's what Security Journey, my first company did.

**Izar Tarandach (39:07)**: And you know that I paid a lot of money for Security Journey training because I liked the way that it was done.

**Chris Romeo (39:11)**: That's why we became friends. Yeah. And we got a lot of money and then I was like, we've got some, and we got some very good results. Right. No, no, it did it. Yeah. So I mean, training has to be one of the foundational pieces of this, of the process. Right.

**Matt Coles (39:26)**: And more importantly, that means developers are invested in making this work.

**Izar Tarandach (39:31)**: Exactly. Chris, even with Security Journey, which, which isn't a really good product and I have no qualms saying so. We still had to fight a budget thing uphill. We still had to ask developers to give us time. We still had to ask managers to give the developers time. My whole point is, when are people going to say, we have solutions, let's just go and use them the right way. Let's give these developers the time to learn the things that they need to learn.

**Chris Romeo (40:00)**: I think I have the answer to that one too. It's when the business drives those actions. The business is not forced to drive those actions today because they can get away with not doing them and still make money.

**Izar Tarandach (40:13)**: It's when CISA, instead of saying that the major flaw in your product development process is that you don't put CWEs and CVEs, says that the major flaw is you don't train your developers enough. You don't teach them what they actually need to know, not to create CVEs based on CWEs.

**Chris Romeo (40:30)**: And listen, like I've said before, and I'm going to say again in this talk I'm doing later this week, if CISA wants that what they're creating to have teeth, put a penalty clause in it that says if you don't get, if you don't improve by 10% in this category, let's say they said train their developers, that if you don't improve your developers' knowledge in a measurable way by 10%, you have to give away 10% of your gross profit to a technology firm security.

**Izar Tarandach (40:56)**: Now you're promoting a security team to security police. And I would love to kick some doors.

**Matt Coles (41:01)**: So let's not do that. Now you're basically saying just do what the EU is doing and make it by regulation.

**Chris Romeo (41:07)**: No, no, no, you don't have to. It's not mandatory that you sign up for this, but if you're going to make— the problem that I've had with this stupid pledge is it doesn't actually mean anything. So I can pledge, I could go pledge any company and I don't have to do anything and I can get my name on the website that says rah rah me, let's go secure by design, we all love it. My point is if I'm going to sign the pledge, I'm making a step. So there should be— so you wanna make it interesting, make some consequences that are measurable and published. And yeah, but how many—

**Matt Coles (41:36)**: what's the over and under on the spin? There's a Vegas thing going on here.

**Chris Romeo (41:39)**: Everybody bets everything these days. Bring in FanDuel, MGM Sports or something. Let them dick give us over.

**Izar Tarandach (41:45)**: How many businesses are going to voluntarily do that? Right? Well, that's the point.

**Chris Romeo (41:49)**: That's my point is it's just lip service. They're paying lip service and there's no teeth.

**Izar Tarandach (41:53)**: But on the other hand, A business that steps up.

**Chris Romeo (41:57)**: Getting me fired up here, Monday morning.

**Izar Tarandach (41:59)**: A team that steps up and says, you know what, we pledge in a year to have 90% of all our developers trained on the stack that they use. You know what, I'll even give it to you, you can use it as marketing as much as you want. But damn, you pledge to something that's actually achievable and something that's actually measurable. It's measurable. And something that actually will have an impact.

**Chris Romeo (42:25)**: I'd get behind that 100% because it's not just me paying lip service to say security is the most important thing to me. Now I'm going to go back to my regularly scheduled programming. Like that's something that you said, hey, we're going to do this. You can measure it. You can publish it. That's exciting. That my point is none of the Secure by Design pledge has any of that accountability associated with it.

**Izar Tarandach (42:46)**: I think that in a month, the Vici has to come out with a statement that 100% of your developers have been trained. There we go. By the way, I'd be surprised if they hadn't yet, but—

**Chris Romeo (42:58)**: My development team at DaVinci has been with me for a long time, so yeah, they—

**Izar Tarandach (43:03)**: But that's the kind of thing that I want to see coming from especially a security company. To go back to that episode that we had about the security utilities approach, if you are going to go teach someone, help someone to do their stuff securely, I'm expecting you to fully come forward, not you, but you, the business owner.

**Matt Coles (43:24)**: Your own dog food, yeah.

**Izar Tarandach (43:26)**: To come forward and say, hey, you know what? I took the time, I made the investment, my developers are all fully trained. Now I can trust them and now I can verify them.

**Chris Romeo (43:34)**: Yeah, no, I'm with you. I think we're out of time for today with this episode of Security Table. And now I'm like, it's Monday morning, 10 AM Eastern time. And I'm 100% fired up. I'm like, Let's go knock over some walls or something here. This is, this is good. Got me excited.

**Matt Coles (43:49)**: Need a vacuum sucker robot kind of thing going. That's too different. A virtual one.

**Chris Romeo (43:53)**: I don't want to accidentally hurt myself. All right. Well, folks, this has been another episode of Security Table. We started with Woodpeckers and IT Crowd and we landed. I think we made some progress against the challenges. I was able to express my challenges and you guys were able to explain why 100x isn't really the answer. It's much less. But hey, tune in next week for probably some other animal to be assaulting Matt's house.

**Izar Tarandach (44:23)**: And let's see the Matt Kaul's Animal Show International. That's right.

**Chris Romeo (44:28)**: Like on the Johnny Carson Show, that animal guy would show up and a monkey would be climbing on like Reese Witherspoon's head or something. You know, that was hilarious. Well, folks, thank you for tuning in. And to all our friends in Australia, and around the world, we're signing off.
