Why 100X Isn't the Answer
With Chris Romeo, Izar Tarandach, and Matt Coles
A good discussion today covering two different articles, the first covers CISA's list of product security "bad practices", questioning whether it provides real value or is just content marketing. Then the discussion moves onto an article about Shift Left.
About this episode
A good discussion today covering two different articles, the first covers CISA's list of product security "bad practices", questioning whether it provides real value or is just content marketing. Then the discussion moves onto an article about Shift Left. The group debates whether it is truly more expensive to fix design flaws versus implementation bugs, noting the difficulty of quantifying the cost difference. They argue that the focus should be on providing proper training and incentives for developers to build secure software, rather than just adding more security tools.
Articles discussed in the episode:
Product Security Bad Practices
Shift Left Pushback Triggers Security Soul Searching
Follow AI Security Table:
➜ Home: https://securitytable.ai/
➜ X: https://x.com/AISecurityTable
➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/
➜ YouTube: https://www.youtube.com/@AISecurityTable
Transcript · 45 min conversation
0:09Matt Coles Google newsletter. Trying to sign up for the Google newsletter with an ad email valid email address from Google and it doesn't let me with my own email address. I think this is insane.
0:20Izar Tarandach Did you try with two ads? Like your first one and then a second one at Gmail.com? and it drops the second.
0:28Matt Coles I tried the plus sign with a, with a tag so I can filter and it doesn't like them.
0:33Chris Romeo So welcome to an episode of Matt's Inbox.
0:36Matt Coles Oh yeah.
0:37Chris Romeo And setup process.
0:39Izar Tarandach We have an MLM for that.
0:40Matt Coles We do.
0:41Chris Romeo This is a segment we do once a, once a month where we, uh, fix Matt's IT problems.
0:50Matt Coles Fix. That's a, that's a funny word. You, I, You keep using that word. I don't think it means what you think it means.
0:56Chris Romeo Hello? Hello, IT? Did you turn it off and then back on again?
1:00Matt Coles Hello, computer? Computer?
1:02Izar Tarandach I'm sorry, but as somebody whose initials is IT, I think that's very—
1:09Chris Romeo That did mess me up one time when you joined a call and I was like, who, why is the IT department here? I'm like, IT, get the heck out of here. We don't need you. We fix our own computer problems and like—
1:21Izar Tarandach Oh, but that's, people keep telling me even if my initials were something else. Go the way you're built.
1:27Matt Coles If, uh, if nobody gets the reference to, uh, to that, to that quote, or computer, computer, it has to be in, it has to be in a Scottish accent. Uh, so I can't do a Scottish accent. I can only do a British accent. Is that from Her?
1:42Chris Romeo Is it Her? The movie Her? No. What is it from?
1:45Matt Coles It's The IT Crowd. The IT Crowd?
1:49Izar Tarandach No, but wait, wait, wait. Where did it come from before? Ah, ah, ah, ah, sorry, you're missing it.
1:55Matt Coles Uh-oh.
1:56Izar Tarandach It comes from Star Trek, from one of the movies.
1:59Chris Romeo Oh, really?
2:00Izar Tarandach Where they travel to the past and they are shown a computer. And the first thing that Scotty does is grab the mouse and go, computer.
2:09Matt Coles Hey, computer.
2:10Chris Romeo That's right, I remember that.
2:12Matt Coles Okay, it was funnier still on the crowd though.
2:14Izar Tarandach Oh yeah, that was better.
2:16Chris Romeo The IC crowd is something that the security table recommends that you check out, especially if you've ever worked at a help desk job.
2:28Izar Tarandach Especially if you want to understand why we are so cynical sometimes.
2:32Matt Coles Yes. And if you like British comedy, of course.
2:34Chris Romeo Oh, of course. It's brilliant. Brilliant. I laugh every time. I've probably watched it 5 times at this point.
2:41Matt Coles That was the first episode too, I think. The good one.
2:44Izar Tarandach I think so. But that's a reboot waiting to happen.
2:47Chris Romeo They tried. They were going to bring Moss to America. It was going to be an American version of it, and they did the pilot, which I saw somewhere. I watched the pilot and it wasn't good at all. They lost the magic. Like, what's that guy's name? The guy that plays Royke, first something, the Irish guy. Um, without him, he's like the, the, the match to Moss. Like, you need his over-giant personality with Moss's like lack of personality.
3:16Izar Tarandach But the boss, the boss, the boss stayed.
3:19Chris Romeo Which one?
3:19Izar Tarandach I mean, the big boss, the guy that later on went to what we do in the shadows.
3:25Chris Romeo Yeah, he's a big Broadway, uh, London theater performer. Keith Olsef. Yeah, I can't remember his name. It's Renom. Renom was the original boss, and then I don't remember what that What's that guy's name?
3:39Matt Coles Okay.
3:40Chris Romeo It'll come to me later while we're talking.
3:42Izar Tarandach Oh, what we going to do today?
3:45Chris Romeo We're not just going to analyze The IT Crowd? No. For the whole episode? Okay.
3:49Izar Tarandach And this is not going to turn into Pinky and the Brain.
3:52Chris Romeo That's true.
3:53Matt Coles All right.
3:53Chris Romeo Well, our first item on the docket is apparently the good folks at CISA produced a set of product security bad practices.
4:05Izar Tarandach Something we've been doing for thousands of years already. We have best practices everywhere. We produce them every day.
4:12Chris Romeo You've been building products for thousands of years?
4:15Matt Coles It feels like it.
4:15Izar Tarandach Accumulated, accumulated.
4:17Matt Coles Accumulated. Number of products, number of years per product. Yeah, yeah. Thousand years sounds about right.
4:22Izar Tarandach You know what? Tell you what, sometimes it feels like we have.
4:27Chris Romeo So I think that the challenge here with our product security best practices or bad practices, sorry, is it's a bit of a cop-out. And the reason I say that is because all they did was take their document and flip it around and put not, and now they produced a new piece of content. So do we, do we just see content marketing at work here? Or is there any actual new value to what they've come up with as their list of bad practices?
4:55Izar Tarandach It's basically formal proofing, right? They, they, they gave you like, one part and then they inverted it and said she's still valid so that thing works.
5:07Matt Coles Hold on a sec guys I have to go there's a woodpecker at my door look right back welcome to the medical zoo—
5:16Chris Romeo Matt let the record show Matt is literally leaving the recording room to tangle with a woodpecker! All of a sudden we hear Matt go, "Oh, stop, move." And then you hear this.
5:38Izar Tarandach Tune in next week as we see Matt chase a coyote.
5:44Matt Coles So normally I'm muting because the dogs are barking like crazy, but today it's a woodpecker that's in my, well, Wooden house and woodpeckers, you know, don't mix.
5:55Chris Romeo So what is his name? What have you named him?
5:57Matt Coles I don't care.
6:00Chris Romeo Poke?
6:01Matt Coles Poke.
6:01Izar Tarandach So just—
6:04Matt Coles So we don't get in trouble with the animal rights folks, I won't say dead woodpecker, but chased off woodpeckers.
6:13Chris Romeo Yes. No woodpeckers were harmed in the filming of this podcast.
6:16Matt Coles Right. All right, so bad practice. Some copper siding maybe. Bad practices, yeah. Shielding, clouding your house in copper siding.
6:27Chris Romeo So how could you use the product security bad practices to prevent woodpecker attacks?
6:32Izar Tarandach Exactly, you know, that's such a good analogy, like putting those strips in the corners of your house, hoping that's to drive the woodpeckers away. It's so much like some of the stuff here.
6:44Matt Coles Well, and of course, the bad practices paper doesn't call out that you're not doing threat modeling, of course.
6:54Chris Romeo Yeah. So that would be a bad practice to not do. So that's where this gets confusing is it's like all the negative.
7:00Matt Coles Yeah. But they're not, but they're not calling out that they're being very selective of what they're calling out. Right. So they're calling out lack of memory safety, right? They're on, on the memory safety train. We could dive into that. They did, they did actually expand that one. We could talk about that a bit more. But then they called out, you know, OS command injection and SQL injection, but not like cross-site scripting or other vulnerabilities. Why, why that one? Very selective. So in this inversion, it wasn't simply, you know, not having defenses against, you know, common vulnerabilities. It was specifically against a specific vulnerability.
7:46Izar Tarandach But it is, because later on they say presence of known exploited vulnerabilities.
7:51Matt Coles Oh, sorry. I should say weakness, not a vulnerability, right? So calling it an OS kernel injection specifically.
7:58Izar Tarandach It's like, do we really, are we really at the stage that we need somebody to pop up and say, guys, having known exploited vulnerabilities is a bad thing, okay? Let's not do it. By the way, can you sign here and pledge that you won't? It's, you know.
8:15Matt Coles Well, I mean, alternatively, alternatively, you could go the way the EU is going, right? So if you, if you're familiar with, say, the CRA, I think it's called, the Computer Resilient Act or something, I forget what CRA stands for. It calls out having no vulnerabilities when you ship something. I mean, so you have CISA saying you shouldn't have vulnerabilities, that's a bad practice, you know, voluntary sign up for doing something about it. And then you have the CRA saying, if you want to ship to Europe, no varnishes.
8:52Chris Romeo And it's just— That just shows that their legislation was created by people who have no idea how systems work, how software works at this stage. I would love for us to have a world where we could say this piece of software has no vulnerabilities, but it's just not possible. It's just not even, it's not even, it's just not even in the realm of possibility.
9:15Izar Tarandach I'm sorry, it goes even deeper than that. We do live in a world where people say this piece of software has no vulnerabilities with absolutely no base on truth. They just say it because it became a marketing thing. Yeah, we checked ourselves and we found that we are fine.
9:31Matt Coles Who we read as, you know, that doesn't work against the piece of technology that we use and therefore didn't find anything.
9:37Izar Tarandach So in this world of promises and pledges, you know, we got to a point where people can, they feel good enough to go and declare themselves safe and that apparently is enough for some people.
9:49Chris Romeo Yeah, but it makes you back it up. Right. But then, well, SolarWinds is a great example of this, who they would have told you they were safe. A month before their famed incident. And then they lost a lot of customers as a result of their supply chain or their build pipelines being—
10:08Izar Tarandach But to go back to our last two episodes, they still have a lot of customers because people are saying, okay, so they had a bad— fine, cool. They still need the functionality. Let's move on.
10:19Matt Coles Yeah, they had a bad day. Everybody has a bad day.
10:22Izar Tarandach But on the chapter of things written by people who really have possibly not a lot of experience in what they are supporting. When you go down and when you scroll down, you get to the organizational processes and policies. So they made two choices to put in there, failing to publish timely CVEs with CWEs and failing to publish a vulnerability disclosure and disclosure policy. Now, step back and remember, to me, 15 minutes ago, When were you involved in organizational processes and policies of publishing hopefully secure software? Seriously, of all the problems that are in there, these two were the ones that we should go to first?
11:06Matt Coles Well, so I have a theory about the CWE one, but the vulnerability disclosure practices is actually a good common sense one, I think, right? If you don't tell your customers how you treat vulnerabilities and you're not upfront about that, there's a lot of FUD that can happen, and having been in industries where vulnerability reporting was not really a thing, it led to a lot of consternation, like, hey, I found a bug in your software. I rely on this software for critical function XYZ. What am I going to do? Is this a problem for you? Right? You go to a support forum. You go to a support forum and you submit an article to a support forum.
11:44Chris Romeo Everybody's got that now. Though the bad thing again, I'm not saying that it's bad.
11:48Izar Tarandach I'm not even saying that everybody's got it. A lot of people don't. But if you are creating a foundational document, something that people are going to carry for years to come, are these the two hills you are choosing to die on? Is this the impact that you can make industry-wise?
12:04Matt Coles I have a theory about the CWE one. I have a feeling that it's really a, we don't, we, and I say we collectively, you know, CISA or somebody in the government or other places, we don't know how big the mountain really is. And so let's add data, let's get companies to add data that we can use to do better reporting later. So by, so today, if you do a vulnerability and you get a CVE, you provide a description, and that description has been historically pretty horrible. Right, has not provided a lot of details about what the vulnerability is. You get, maybe you get a CVSS vector that provides some information that you can pull, you know, does it, is it remotely exploitable and is it disastrous if it's exploited, that sort of thing. But the underlying root cause is never really brought out in those descriptions or very rarely brought out in those descriptions. And so by adding the CWE or CWEs that are involved, you now have root cause. And so if you have this approach where you want to reduce, say, injection and memory safety vulnerabilities, and there's no, but there's no weaknesses, but there's no way to know that you're actually getting memory safety as a root cause of a CVE, adding a CWE to a CVE is a good way to do that.
13:30Izar Tarandach Yes, but going back to the, to the documents, They say that for products used in service of critical infrastructure, NCFs, it is dangerous and significantly elevates risk to national security, national economic security, and national public health and safety for the software manufacturer to not issue CVEs in a timely manner. Nobody says what timely manner it means. For a minimum of critical and high-impact vulnerabilities, right? Additionally, it is dangerous and significantly elevates risk to national security, national economic security, and National Public Health and Safety to not include the CWE field in every CVE record. What I'm missing here is how.
14:14Chris Romeo Well, there's no teeth. That's been my whole challenge with everything that they've done here. There are no teeth in this thing, and the requirement is wishy-washy.
14:24Matt Coles I signed it first though.
14:25Izar Tarandach But wait, it's not even teeth. It's we should have data to support this assertion. Tell me, when was the last time that national security got impacted because somebody didn't put a CWE in a CVE?
14:38Matt Coles Okay, so here's the thing, right? If you have just— so if you have just a CVE with a description, and let's just say for a moment NIST is still bogged down in trying to get— doing CVE reviews and doesn't have a vector associated with that severity. So you have a severity, but you may not have a vector. You may not even have a severity, you may just have a description of a vulnerability. How do you prioritize that? And how do you prioritize that effectively? I'm not saying this is the best answer, but this is my theory, right? It adds data to a description which historically has been weak in providing information to a defender.
15:15Izar Tarandach I'm sorry, I watched all of 24. I watched all of the Tom Clancy movies.
15:21Chris Romeo Okay.
15:22Izar Tarandach I watched a bunch of Clint Eastwood, not Clint Eastwood, Harrison Ford ones.
15:27Chris Romeo What about Jack Ryan?
15:28Izar Tarandach Jack Ryan, that too, and that's like the newest one.
15:32Matt Coles Before or after Clancy was alive?
15:35Izar Tarandach I haven't heard one. Don't get me started there. I haven't heard one single line of dialogue that begins with, "Qui, what is the CWE? This is a matter of national security." I mean, they're saying here it is dangerous and significantly elevates risk to national security. So we probably have something to point back and say, oops, this happened if we only had the CWE.
16:00Chris Romeo Yeah, no, you've pointed that out. You've pointed that out. I didn't catch that the first time I read this. And so I am, as you continue to talk, I've been reading and rereading that statement. It's pretty darn funny that That basically the world is going to end if you don't put the C-W-E field in every C-V-E record. The world will end. Society as we know it will disappear. Zombies will roam the earth. Armageddon, apocalypse, whatever. Oh, he's just— the end is beginning now. He's not able to talk.
16:37Matt Coles We saw that.
16:38Chris Romeo I heard that. Nope. It's the beginning of the end.
16:51Matt Coles One minute he's there and the next minute Cees is listening and going, oh, I blocked him.
16:56Chris Romeo Look, I, I, I, what can I say?
16:59Izar Tarandach They love me at the NSA.
17:02Matt Coles Hoots. It's not the NSA you're worried about.
17:05Izar Tarandach And by the way, guys, I love you back. I love you both.
17:09Matt Coles Do you still start every Zoom call as, hello, whoever you are on the line? Yeah.
17:16Izar Tarandach Plus I have black helicopters over here all the time.
17:21Matt Coles Well, that's because you have a famous person running through the woods who you don't know what she's doing in there.
17:25Izar Tarandach No, it's because I'm the line between the Hudson and the Sikorsky factory. It go by one day.
17:32Chris Romeo All right, we've probably talked about this one enough. For the amount of new material.
17:37Matt Coles We've just gotten started, but, uh, Shift Left is something else.
17:42Chris Romeo All right, second item on the docket is the Stark Reading article, Shift Left Gets Pushback, Triggers Security Soul Searching. So I feel like we've been throwing some noise at Shift Left. I've been throwing noise at Shift Left for at least a year or two because I actually wrote a post about making fun of, I went and Googled Shift Left. Found a bunch of vendors that were using it in different ways and then made fun of each one of them individually. So I feel like that shows I was not, you know, I was one of the first people to turn on the whole shift left idea, but apparently there's a movement now. We've started something. We should get t-shirts.
18:21Matt Coles So, I mean, my mantra has always been start left, don't shift left, start left. But if you were breaking down the bear, if you're breaking down the concept that shift left is an invalid concept, then start left is invalid too, because it fundamentally goes to who's responsible for what. And then that's highlighted in this article, right? So this is an article by, mostly by Acquia, but written by Dark Reading. And, you know, the question is, so shift left is, of course, the notion that security needs something done and tells developers to go do it.
19:02Chris Romeo Yeah.
19:02Matt Coles And, and well, I should say, that's the, that's the premise of this article. The, the, the idea really should have been was that developers with, and engineering teams when they're building software or products or systems or whatever, should have looked at security as one of the properties of that system. Like they do quality today.
19:25Izar Tarandach Let's—
19:25Chris Romeo Matt, before we go too deep into this, let's deal with that first paragraph because there is, there is some— they are basically pointing at this 100x number that we've thrown around.
19:40Matt Coles Whose—
19:40Chris Romeo the first time I found it published was in an IBM. I found it referenced in an IBM document from the late '70s, but they've even got it. They've got it further back to TRW. Yeah. And so they're, they're basically saying that the 100x metric doesn't apply today. But I guess the question I have for you guys, I'd love to get your take on this, is do we agree or disagree with this statement that it is cheaper to fix a problem during the design phase than it is in production? —when it makes its products. Yes, but not for the same reasons that may have started the 1:100 thing.
20:21Matt Coles And yes, but with a big caveat. Okay. So let me go first with the caveat. Modern application development, as you guys have tried to hammer into my brain for the past X number of episodes, I'm still stuck in a we build hardware kind of world. But if you're using, you know, if your entire development is taking other people's code and configuring it, so you're actually not doing a lot of custom development, you're doing a lot of stitching of, you know, microservices and Kubernetes and, you know, web frameworks together to make stuff happen, then maybe that doesn't hold because what is your design phase? Your design phase is really which components do I put together and how do I stitch them together, not architecting one of those components from scratch, right? So I'll say that caveat is, are you throughout that maybe not in the way we are expecting or thinking?
21:26Izar Tarandach 'Cause we have that model stuck in our heads that you create software, you publish software, you sell software, and then there's something that you have to fix and oh my God, now we have to run around and replace all existing versions and all that stuff. and that connects back to the hardware aspect of the thing. And that's damn expensive. Nowadays, when people claim to be, what is it, deploying thousands of times a day, Lord knows if I ever seen that. Like, I've seen people claim that. I mean, people doing that.
21:58Matt Coles I think the last number I've seen was Amazon, every 8 seconds, they, in theory, could push to production. Right? Yeah.
22:04Chris Romeo So, what they define as production is a very vast ocean.
22:09Izar Tarandach Yeah, and what they define as deployment is also, but anyway, but today it's easy to bypass, like it's cheap to bypass that, oh my God, there's something wrong. Click a button, everybody gets the new version, everybody's happy. If they haven't used the software in 8 seconds, then they haven't even noticed it. But the point is that we are talking about design here. So design in my very much biased head means that the fix is probably not a 3-line patch. It's a lower-level kernel. And still fixing that, changing that, and I'm sorry, I meet that every single time that I speak with a startup and I go, oh, but you guys have SSL support, right? And they go, no, but it's going to be trivial to put it in. And then trivial turns into a battle. And are you guys really, really sure that you want this? Don't know. Our password management stuff is much better than any SSO. No, no, seriously, you don't need this. It's okay. Listen to me. It's okay, brother. Don't you trust us? You should, you should. You totally should. But my point is that real, real design flaws, are probably still way more expensive to fix than just deploying the newest patch, right? So 1 to 100, I have absolutely no numbers to support anything that I say like that. So that's why I don't use the 1 to 100.
23:48Matt Coles It's more, it's more, but it's honestly a particular like number more.
23:53Chris Romeo Well, you think it's— So hold on, wait, what does that mean? You just said that it is more than 1 to 100. It's relative.
23:59Matt Coles It's relative. It is more than fixing, than patching in the, you know, patching a software bug. Yeah. If you have to change your architecture or you have to make a design change, but is it 100x?
24:09Chris Romeo So it's exponential. You're saying it's an exponential change in terms of greater problem. No, no, no, no, no, no, no. It's greater.
24:15Izar Tarandach Well, on a, on a comparative scale, it's probably more expensive to fix design flaws. Right. Probably.
24:23Matt Coles And Almost certainly, almost certainly more expensive.
24:26Chris Romeo We're way too wishy-washy here with this problem.
24:29Matt Coles Well, okay. Let's, so first off, we agree, I think we generally agree it's more expensive to fix a design flaw than it is to fix a software flaw, right? To change a buffer versus changing your programming language, it is almost certainly more expensive to change programming language.
24:46Chris Romeo Well, that's when you're kind of using an extreme example of one and a simple example of the other. You kind of got to, you're not, you're not, it's not an apples-oranges comparison what you just gave. It's an apples. I understand that. It's an apples and battleship comparison.
24:58Matt Coles So give me an apples to apples comparison.
25:00Chris Romeo I don't know.
25:01Izar Tarandach It's cheaper to change our tire than to redesign your transmission. Okay. Okay? That's a given. It is. It is. The question is how many times you're going over the lifetime of a car, how many times are you going to have to change a tire versus how many times you're going to have to redesign the transmission?
25:20Chris Romeo You could say rotate the tire would be an even better example because that's probably— I don't change a lot of tires these days.
25:25Matt Coles But my tires are rotated every time I change my oil, right?
25:28Chris Romeo That's because you have AAA as a third-party service.
25:31Izar Tarandach But you know what?
25:32Matt Coles Tire rotation is a preventative measure, not a reactive measure.
25:36Izar Tarandach Ah, good point. Good point. Eventually, you're going to have a flat, especially if you drive in New York. My God. But my point is that in some cars, you won't ever have to redesign the transmission. But when you have one that does get a recall because the transmission is bad and it has to be redesigned, That's going to be pretty expensive.
25:57Chris Romeo Or replace, right?
25:58Matt Coles Replacing a transmission, right, is a pretty serious change to your vehicle.
26:06Chris Romeo That's like a software— the transmission in this example has become a software component. If I'm replacing it, it's a software component. If I'm redesigning it, that's a whole—
26:15Izar Tarandach that's an architecture flaw. That's a design flaw. That's a design flaw. Yeah. Right? You know, it's not something that you're going to do every Tuesday and Thursday. But when it happens, it hits you, right? So—
26:29Matt Coles But is it definitively 100x? I mean, you can't put a number on it until it happens. And then you get— you start to build up data.
26:37Chris Romeo We don't really need a number. That's what I'm saying. No, no, wait, wait, wait.
26:39Izar Tarandach You know what? Redesigning a transmission is a bad example. Remember a couple of years ago that there was some car that was found that if it gets hit at a certain angle, the—
26:49Matt Coles Pinto.
26:50Izar Tarandach The gas tank. Would explode. That's exactly it, right?
26:54Matt Coles Well, actually, you could even do a modern example of you have a big shiny box of a thing that has door angles and things that don't open or close properly and no handles when it's frozen and that sort of thing. I mean, yeah, you could go back to the Pinto or you can come up with something a little bit closer.
27:16Izar Tarandach Yeah, that's a solution with other use case if I ever saw one. But anyway, so my point is, yeah, it's not going to be a cost that you are putting out of your pocket every week, but when it happens, it hits you hard. The point is, the question here is one versus the other, because the vulnerabilities, you probably are spending that money many times over the lifetime of a product. Does that cumulative cost compare to design flaws? And is that at all something that we should be doing? Because my point is, even if it's well-designed, those vulnerabilities will still keep coming up because of what Matt keeps telling us, that it's a quality thing. And just to go back to the developers, developers don't want to deal with this shit.
28:06Chris Romeo So let's see if we can boil this down into a final nugget of truth. So it sounds like we are, we are all saying that it will be more expensive to fix any, any, any software, any, any issue you have with something that you're building, it will be more expensive to fix it in production than it would be to fix it earlier in the lifecycle. Well, we are, what we're saying is we're not ready. We're not going to attach 100x, just like the article said, not willing to put 100x number on it anymore because it might only be 2x. It might be 1.5x. But the point is, If somebody, and this is how I've always explained it to people, is you release something to production, 3 to 6 weeks go by, somebody finds a problem with it. As a developer, you get a bug ticket. Your brain has already moved on. You've grabbed new user stories. You've removed the history of that feature you built from your memory banks. You've loaded in a new one. You're in the middle of working on it, designing it, making it work. Now you have to unload all the things in your head now, reload what you did 6 weeks ago, and then that's rework, right? That's the whole principle of rework. That is going to be more expensive than if you had just done it correctly and not let the issue be created when you first built the feature.
29:33Izar Tarandach You know, yes, but wait, 2 things. First of all, what you gave is just a great a great point for shift left, right? Well, what you just described—
29:42Chris Romeo Whoa, whoa, then I take it back.
29:43Izar Tarandach I forgot to correct you. What you just described is the shift left argument, the right one. And the second thing that I want to add is that we've been looking at economic costs. There's also another cost in there for the team, which is the pain of having to revisit the foundation of the thing that you built and say, guys, we should have thought it before.
30:04Chris Romeo And that's an economic— it all boils down to economic costs though. Like, listen, I run businesses. It's deeper than that. But at the end of the day, to me, it comes down to my cost and my return on investment from my development engineering team.
30:19Izar Tarandach It's a bunker boot— buster. Buster for team morale. Yeah. So I have to go back. Yeah.
30:27Matt Coles No, wait, finish your thought because I'm actually, as I'm rereading this over and over again as you guys are talking, I think we're missing something too on top of that that does— finish your thought. Go ahead. Well, so just to continue your discussion, the thread you were on, what's the cost to the business if they release bugs, right? Bug over bug over bug or exploitable vulnerability over exploitable vulnerability or breach-capable software versus not. So there's some additional costs there which are not being factored in maybe. 100x is still a funny number. But as I reread this, there's something else that is interesting here. So, the very first paragraph about this talks about, is it better, is it cheaper to fix a software bug in design time versus fixing it in production? And you could take this and then it says, well, and, but this requires developers to buy very expensive tool you know, from vendors, right? So it's pushing a vendor-based or tool-based approach. But do they really mean design bugs? We've been arguing about design bugs, right? Or do they really mean fixing vulnerabilities in pre-production or during development? So, right.
31:53Chris Romeo I think that's really more what has been, I mean, we've made, we've taken it to what we love, which is threat modeling, design, things like that, that we're kind of, we're putting our own security table colored glasses on and looking at it and how we like to think about it. But I think you're, I think you're 100% right. That's not how most people think about it. Right.
32:12Matt Coles So if you're fixing, if you're talking about fixing a bug, I mean, so for decades, people have run static code analysis tools, right? And if you find 100,000, you know, string formatting errors, You're not going to fix them all, right? But you may buy a tool to say, okay, which ones of these do I have to fix? Because it's on my attack surface, I do some penetration testing or whatever, and these are exploitable. I do some fuzzing and these are the ones I'm going to worry about. You can prioritize them. Now you're adding tools to solve a, which needle, which of these needles in this haystack do I want to pull out before a customer steps on them? Okay. Right. and there's an argument to be made about who is responsible for security and/or quality of the system and who really cares.
33:02Izar Tarandach Just to support that, a while ago I heard from someone, since the treaty, the person said that they run a, I don't remember what kind of tool, but they found so many bugs that they didn't fix any.
33:19Matt Coles We gotta have a term for that. Look at Qualys, look at Fortify, look at any of the scanners or static analyzers that, I mean, hundreds of thousands of issues. And because of bad practice. It's a fine new overmold, but it's bad practice, right? So if you don't initialize a variable, which is a pretty common development practice, Do you have an exploitable flaw? Whether it's a vulnerability or not, I'm just talking, is it the quality or is it breaky?
33:50Chris Romeo I never initialize vulnerabilities.
33:52Matt Coles You can't know. You can't know. It's undefined behavior. I let them initialize themselves.
33:56Chris Romeo You probably use languages that allow undefined behavior. Well, lucky you. I need a protective layer around my bad decisions when I code things. I need something that just assumes I meant to do the right thing and then adjusts.
34:12Matt Coles Appropriate in the runtime. And that's an interesting thing because, well, if you do that, if you do that, right, then why are you buying so, why are you investing in a lot of these tools? Because the language insulates you from, or the technology choices insulate you from these issues.
34:29Izar Tarandach But again, it's what we've been talking about for a long time. It's one more list, one more 10,000 things to do that doesn't tell me what to do. Lists, right?
34:39Matt Coles But you have to buy another tool for that. That's why ESPMs and other things exist, right?
34:43Izar Tarandach I mean, you have to buy the other single pane of glass.
34:47Chris Romeo Well, I think I've had a bit of an epiphany here.
34:50Matt Coles I think we have a tools problem in our industry.
34:54Izar Tarandach You think? Where are the experts about this?
35:01Chris Romeo This is part of the comedy segment, obviously.
35:04Izar Tarandach Okay, wait, we have a segment?
35:06Matt Coles This whole segment, the whole series is a comedy routine. There is more.
35:11Izar Tarandach When you go down on the page, you get the beautiful heading, not the metrics you're looking for, right? So yeah, it's not about money. It's not about top 10 lists. It's not about anything. To me, again, the ship left thing, keeps coming back to the, yes, it makes perfect sense, but has anybody actually asked the people who are supposed to do those things if they want to do those things? And the answer that I keep going back to is no.
35:46Matt Coles But again, whose responsibility is it? So in the article, there's a discussion here about security wants security fixes. Why does security want security fixes? Shouldn't developers want security fixes? Who does quality? Quality doesn't want quality fixes. Developers want quality fixes, right? That's my point.
36:06Izar Tarandach Developers don't want to look at things as fixes. Developers want, from my experience meeting and working with developers, they want to do the right thing. They want to avoid.
36:18Matt Coles They don't want to react.
36:19Izar Tarandach They want to avoid. So the point is that we keep writing these amazing tools, supposedly shifting them left. Putting a lot, a lot, a lot of weight on these 4 developers that are under so much pressure already. And I'm not being cynical here. I fully stand behind saying this. We are just adding pressure to what's already a pressure cooker.
36:43Matt Coles But you're fundamentally, but there's a fundamental problem to what you just said. Why do we need to push a tool into that environment? Exactly. It's because they're creating, it's because they're creating buggy software. It's a cause and effect.
36:57Chris Romeo Exactly. The tool is— Yeah, I mean, then they're doing things.
37:00Izar Tarandach And what's the solution? What's the solution that is mentioned in the article? We have to shift left, left. So start left. Just start left. Start left, but how do you start left? By teaching these people the right— By requirements, training. No, by training. By teaching these people what the right thing is.
37:16Matt Coles Yeah. Right. And guardrails and paved roads. It's everything. I think that's what we've been highlighting.
37:19Chris Romeo Yeah, it's the whole package.
37:20Izar Tarandach That's the catch. But again, no, no, even before—
37:22Chris Romeo Is it shifting anywhere?
37:23Izar Tarandach Even before guardrails, look, if you teach it right, if it becomes part of the discipline, in theory, you don't need guardrails. I keep going to performance. We don't have performance guardrails, for God's sake. People just write performance code.
37:39Chris Romeo I mean, the place I've landed on this and why this has pissed me off for the last couple of years, and I've just railed against this whole idea of shift left and made fun of it at every chance I get. Is because— It's a cynical view. It's really what we're talking about though is secure development lifecycle. You just need a secure— all you need is a secure development lifecycle, a solid approach to process, people, process, tools, governance. The people need to use the process. They can use the tools to support the process and you need a governance angle to run through it.
38:15Matt Coles That is a shift left. And your lifecycle has to start with requirements, training, and technology choices.
38:22Chris Romeo Yeah, 100%. We keep saying trust.
38:24Izar Tarandach We solved it. We're done. No, no, no, we haven't. Wait, wait, wait, wait, wait. We keep saying trust but verify, right? And we keep putting a lot of money and time into building the tools that do the verify part. But we do nothing about the trust part. We don't go to these developers and say, okay, you know what? I will give you the training. I will give you the time. This will become part of your discipline. This is going to be incorporated in the way that you do things.
38:50Chris Romeo And then I can verify. So do you want to have a trust score for each individual developer?
38:54Izar Tarandach No, I just need to spend some time teaching them the right thing and hoping that a good thing comes out of the end.
39:01Chris Romeo I mean, you know, I'm a big fan of training developers because that's what Security Journey, my first company did.
39:07Izar Tarandach And you know that I paid a lot of money for Security Journey training because I liked the way that it was done.
39:11Chris Romeo That's why we became friends. Yeah. And we got a lot of money and then I was like, we've got some, and we got some very good results. Right. No, no, it did it. Yeah. So I mean, training has to be one of the foundational pieces of this, of the process. Right.
39:26Matt Coles And more importantly, that means developers are invested in making this work.
39:31Izar Tarandach Exactly. Chris, even with Security Journey, which, which isn't a really good product and I have no qualms saying so. We still had to fight a budget thing uphill. We still had to ask developers to give us time. We still had to ask managers to give the developers time. My whole point is, when are people going to say, we have solutions, let's just go and use them the right way. Let's give these developers the time to learn the things that they need to learn.
40:00Chris Romeo I think I have the answer to that one too. It's when the business drives those actions. The business is not forced to drive those actions today because they can get away with not doing them and still make money.
40:13Izar Tarandach It's when CISA, instead of saying that the major flaw in your product development process is that you don't put CWEs and CVEs, says that the major flaw is you don't train your developers enough. You don't teach them what they actually need to know, not to create CVEs based on CWEs.
40:30Chris Romeo And listen, like I've said before, and I'm going to say again in this talk I'm doing later this week, if CISA wants that what they're creating to have teeth, put a penalty clause in it that says if you don't get, if you don't improve by 10% in this category, let's say they said train their developers, that if you don't improve your developers' knowledge in a measurable way by 10%, you have to give away 10% of your gross profit to a technology firm security.
40:56Izar Tarandach Now you're promoting a security team to security police. And I would love to kick some doors.
41:01Matt Coles So let's not do that. Now you're basically saying just do what the EU is doing and make it by regulation.
41:07Chris Romeo No, no, no, you don't have to. It's not mandatory that you sign up for this, but if you're going to make— the problem that I've had with this stupid pledge is it doesn't actually mean anything. So I can pledge, I could go pledge any company and I don't have to do anything and I can get my name on the website that says rah rah me, let's go secure by design, we all love it. My point is if I'm going to sign the pledge, I'm making a step. So there should be— so you wanna make it interesting, make some consequences that are measurable and published. And yeah, but how many—
41:36Matt Coles what's the over and under on the spin? There's a Vegas thing going on here.
41:39Chris Romeo Everybody bets everything these days. Bring in FanDuel, MGM Sports or something. Let them dick give us over.
41:45Izar Tarandach How many businesses are going to voluntarily do that? Right? Well, that's the point.
41:49Chris Romeo That's my point is it's just lip service. They're paying lip service and there's no teeth.
41:53Izar Tarandach But on the other hand, A business that steps up.
41:57Chris Romeo Getting me fired up here, Monday morning.
41:59Izar Tarandach A team that steps up and says, you know what, we pledge in a year to have 90% of all our developers trained on the stack that they use. You know what, I'll even give it to you, you can use it as marketing as much as you want. But damn, you pledge to something that's actually achievable and something that's actually measurable. It's measurable. And something that actually will have an impact.
42:25Chris Romeo I'd get behind that 100% because it's not just me paying lip service to say security is the most important thing to me. Now I'm going to go back to my regularly scheduled programming. Like that's something that you said, hey, we're going to do this. You can measure it. You can publish it. That's exciting. That my point is none of the Secure by Design pledge has any of that accountability associated with it.
42:46Izar Tarandach I think that in a month, the Vici has to come out with a statement that 100% of your developers have been trained. There we go. By the way, I'd be surprised if they hadn't yet, but—
42:58Chris Romeo My development team at DaVinci has been with me for a long time, so yeah, they—
43:03Izar Tarandach But that's the kind of thing that I want to see coming from especially a security company. To go back to that episode that we had about the security utilities approach, if you are going to go teach someone, help someone to do their stuff securely, I'm expecting you to fully come forward, not you, but you, the business owner.
43:24Matt Coles Your own dog food, yeah.
43:26Izar Tarandach To come forward and say, hey, you know what? I took the time, I made the investment, my developers are all fully trained. Now I can trust them and now I can verify them.
43:34Chris Romeo Yeah, no, I'm with you. I think we're out of time for today with this episode of Security Table. And now I'm like, it's Monday morning, 10 AM Eastern time. And I'm 100% fired up. I'm like, Let's go knock over some walls or something here. This is, this is good. Got me excited.
43:49Matt Coles Need a vacuum sucker robot kind of thing going. That's too different. A virtual one.
43:53Chris Romeo I don't want to accidentally hurt myself. All right. Well, folks, this has been another episode of Security Table. We started with Woodpeckers and IT Crowd and we landed. I think we made some progress against the challenges. I was able to express my challenges and you guys were able to explain why 100x isn't really the answer. It's much less. But hey, tune in next week for probably some other animal to be assaulting Matt's house.
44:23Izar Tarandach And let's see the Matt Kaul's Animal Show International. That's right.
44:28Chris Romeo Like on the Johnny Carson Show, that animal guy would show up and a monkey would be climbing on like Reese Witherspoon's head or something. You know, that was hilarious. Well, folks, thank you for tuning in. And to all our friends in Australia, and around the world, we're signing off.
Transcript supplied by AssemblyAI. It may contain transcription errors.
Originally published as The Security Table. Part of the AI Security Table archive.
