# Why Developers Will Take Charge of Security, Tests in Prod

URL: https://securitytable.ai/why-developers-will-take-charge-of-security-tests-in-prod/
Published: 2024-05-21
Duration: 2890 seconds
Season: 2
Episode: 15
Hosts: Chris Romeo, Izar Tarandach, Matt Coles

## Show notes and chapters

The script delves into a multifaceted discussion encompassing critiques and praises of book-to-movie adaptations like 'Hitchhiker's Guide to the Galaxy', 'Good Omens', and 'The Chronicles of Narnia'. It then transitions to a serious examination of developers' evolving role in security, advocating for 'shift left' and DevSecOps approaches. The conversation navigates through challenges developers encounter in security practices, stressing the necessity of a DevSecOps framework, secure coding languages, and executive support for fostering a robust security culture within organizations.

Chris, Izar and Matt begin the episode with a lighthearted discussion about books turned into movies, including Hitchhiker's Guide to the Galaxy and The Chronicles of Narnia series. The main topic of conversation on today’s episode is an article titled "Why Developers Will Take Charge of Security, Tests in Production" by Lorraine Lawson, which interviews Larry Meshrom. The article suggests that developers should take on more responsibility for security, including testing in production environments, as security teams are often perceived as a blocker and don't understand the day-to-day work of developers. 

The guys question whether developers truly want to take on more security responsibilities, given the constantly evolving nature of security threats and the time it takes to stay up-to-date. They also discuss the role of product managers in driving security and privacy prioritization, and the need for executives to understand the business value of investing in security. The hosts argue that while mature organizations have governance processes in place to enforce security, smaller companies may lack such mechanisms. 

Ultimately, it is concluded that product managers are best positioned to communicate the business value of security to executives, as they are closest to understanding customer needs and revenue drivers. They propose that the industry should focus on educating and empowering product managers to prioritize security and privacy, and to make the case for investing in these areas to executives. This approach could help bridge the gap between security teams and developers, and drive a culture of security within organizations.

Link to article:  [https://thenewstack.io/why-developers-will-take-charge-of-security-tests-in-prod/](https://thenewstack.io/why-developers-will-take-charge-of-security-tests-in-prod/)

Follow AI Security Table:

➜ Home: [https://securitytable.ai/](https://securitytable.ai/)

➜ X: [https://x.com/AISecurityTable](https://x.com/AISecurityTable)

➜ LinkedIn: [https://www.linkedin.com/company/ai-security-table/](https://www.linkedin.com/company/ai-security-table/)

➜ YouTube: [https://www.youtube.com/@AISecurityTable](https://www.youtube.com/@AISecurityTable)

## Transcript

**Chris Romeo (0:09)**: 34 is not the answer to the meaning of life, the universe, and everything.

**Matt Coles (0:14)**: You're off by a bit. You're off by a bit, man.

**Chris Romeo (0:17)**: You know this from the book. Come on.

**Izar Tarandach (0:19)**: It's inflation.

**Matt Coles (0:21)**: It's deflation.

**Chris Romeo (0:24)**: Deflation. Deflation.

**Izar Tarandach (0:24)**: Deflation. Sorry. Yeah. Deflation.

**Chris Romeo (0:26)**: We're in more of a deflationary environment. So that's— Yes. Yes. That restaurant at the end of the universe, you know, that sort of thing. So yeah, you know, in case anybody doesn't know what we're talking about, Google it.

**Izar Tarandach (0:37)**: Google it.

**Chris Romeo (0:38)**: I'm not gonna tell you where it's from.

**Matt Coles (0:39)**: Or ChatGPT 4.0 it.

**Izar Tarandach (0:41)**: No, first you Google to find what it is and then you go to ChatGPT and you ask for, for a, a concise, uh, explanation of it.

**Chris Romeo (0:49)**: Yeah.

**Izar Tarandach (0:49)**: And you just cross your fingers that it doesn't hallucinate.

**Chris Romeo (0:52)**: Or what did you guys—

**Matt Coles (0:53)**: go by Cliff Notes.

**Chris Romeo (0:55)**: What did you think of the movie?

**Izar Tarandach (0:58)**: No, no, no, no.

**Matt Coles (1:00)**: Oh, come on. Alan Rickman was hilarious.

**Chris Romeo (1:01)**: All right. We're talking about, we're talking about Hitchhiker's Guide to the Galaxy. So for those that were too lazy to Google this, um, but so, so yeah. So, so Matt, sounds like you're maybe thumbs up and Ezar's thumbs down.

**Matt Coles (1:15)**: It was a, it was a, we're on camera. It was a sort of a, mm, yeah, mostly not, but sort of some were really good. And then the guy who lived, Alan Rickman was good, but You know, the, the, the, the, the Evil Rocks wasn't not so good.

**Izar Tarandach (1:29)**: No, the, the design of, uh, Marvin, like, no, it was too chirpy.

**Matt Coles (1:39)**: We'll have to watch it again. It was, it was, uh, all right. So there were some funny parts. It was pretty campy, but overall it wasn't bad.

**Izar Tarandach (1:46)**: Ah, we like campy. Come on. We like campy.

**Chris Romeo (1:49)**: I mean, it was consistent with the story though, right? They didn't, it was, it wasn't like Lord of the Rings where they like, who's that person? No, Hobbit. The Hobbit movie's like, who's that lady? I don't remember her in the book. The elf, there was an elf lady that they just introduced as a new character and like, she was a great character, but I don't remember. I went and looked in the book, she wasn't there.

**Izar Tarandach (2:08)**: Yeah. But you know, that, that, that's my biggest fear. There are some books, some authors that we keep reading. I keep reading again and again and again. And when I hear that there's going to be an adaptation for TV or whatever, I, I shudder. So like get, uh, Good Omens. Good Omens to me is one of the like most hilarious books I ever read. Terry Pratchett and Neil Gaiman. The Amazon adaptation, fair, good. I like, I like the actors, but took a lot from the book. But then they come out with a season 2. Come on. No, no, no, no. Now you're just doing it for the money.

**Matt Coles (2:42)**: Well, so Lion, the Witch and the Wardrobe and Voyage of the Dawn Treader in that series, you know, the CS Lewis.

**Izar Tarandach (2:47)**: Yeah.

**Matt Coles (2:48)**: C.S. Lewis series Chronicles of Narnia held to the book pretty well.

**Chris Romeo (2:53)**: Pretty well.

**Matt Coles (2:54)**: And with some evolution in graphics over, over the years that they built that, uh, did those—

**Chris Romeo (2:59)**: I recently read, reread that series. Come to find out there's a book before The Lion, the Witch and the Wardrobe.

**Matt Coles (3:07)**: Is there always?

**Chris Romeo (3:08)**: Yeah, it's called The Magician's Nephew and it—

**Izar Tarandach (3:11)**: oh yeah.

**Chris Romeo (3:11)**: Informs you so much about The Lion, the Witch and the Wardrobe. And I never knew that there was another book. I never read it.

**Matt Coles (3:18)**: But that's written, but that's not, that was written after. And so it's a prequel.

**Chris Romeo (3:22)**: Is it? Okay.

**Matt Coles (3:24)**: Yeah.

**Chris Romeo (3:25)**: Okay. All right.

**Izar Tarandach (3:26)**: So it was written after, but it's a prequel.

**Matt Coles (3:28)**: Yeah. Prequel. Kind of like episode 1 of Star Wars.

**Izar Tarandach (3:32)**: Oh, okay.

**Chris Romeo (3:35)**: Except no Jar Jar Binks.

**Izar Tarandach (3:37)**: So no.

**Matt Coles (3:37)**: Oh my God. No Jar Jar Binks.

**Izar Tarandach (3:39)**: Uh, no, let's, let's, let, let, let, no, no, no. Don't go there. Let, let's talk to something else.

**Chris Romeo (3:44)**: Okay. Well, that wraps up this segment of books that you could have read that turned into movies that, that you shouldn't have watched, but that you would have avoided, should have avoided completely. Uh, I guess we're actually here to talk about something else on the Security Table.

**Matt Coles (4:01)**: In theory. Yeah.

**Chris Romeo (4:02)**: So Matt, why don't you set the stage for us on this article, Why Developers Will Take Charge of Security Tests in Production?

**Matt Coles (4:10)**: Uh, There's a comma there.

**Izar Tarandach (4:13)**: The comma.

**Matt Coles (4:14)**: So the comma is kind of important. Uh, why would, why would developers will take charge of security, comma, and testing and production.

**Chris Romeo (4:24)**: Ah, okay.

**Matt Coles (4:26)**: So, uh, this was an article was written, um, just a couple of days ago about, um, I mean, in a nutshell, how traditional security functions are completely out of the loop and I'm misunderstanding or don't know how developers work and why can't developers do a better job? I'm paraphrasing. And, and really this goes to the shift left discussion, you know, and DevSecOps or, or whatever it was supposed to be versus what it became. In a nutshell, there's really, it's highlighting a couple things. First off, that You know, security is a blocking function or perceived as a blocking function in many cases. And, and can developers do a better job? Should they be given the responsibility to do a better job? And also that, and in part, security is a blocking function because it's based on, you know, an old lifecycle model. We've had this conversation multiple times on the, on the podcast. And, and also, you know, we're, we in security don't understand what developers do for a day on a day-to-day basis. So I think there's a few things we could touch upon. I'd like to really touch upon the work. Not all security people are created equal. We have different roles.

**Izar Tarandach (5:50)**: And thank God for that.

**Matt Coles (5:51)**: And, and there's a couple other topics I'm sure we could cover. I'm not sure we want to. How we want to talk about testing in production. That sounds bad, but, you know, I'm just a security person.

**Chris Romeo (6:04)**: So. You want to start there first and work our way backwards? The, you want to talk about the testing in production as a launchpad?

**Matt Coles (6:10)**: So, um, yeah, so testing in production.

**Izar Tarandach (6:14)**: What can go wrong? I love the characterization by Chris as a launchpad. What else happens on launchpads?

**Chris Romeo (6:22)**: I'm not that witty, but thank you for that.

**Matt Coles (6:25)**: If you haven't, if you haven't tested yet, uh, bad things happen.

**Izar Tarandach (6:28)**: If you haven't tested either, you put it on the launch pad, it goes up a bit, then bang.

**Chris Romeo (6:32)**: Yeah.

**Matt Coles (6:33)**: I think it goes up a bit.

**Chris Romeo (6:35)**: I think the argument that I've heard and one that I've made over the last number of years is, because I mean, as our audience knows, I'm old enough to have been around from the days where you would not ever test anything. You would not send test cases towards the production instance of this application. You could knock it over, cause it to stop working. Like that, I mean, I grew up in that era, right? And so that, but the thing that's always caught me is like, production means attackers don't follow the rules of we can't scan something in production, we can't test it, we can't attempt to knock it over. Attackers are going to do that anyway. And so this whole idea that like, you know, production is somehow some sacred place, it just, it doesn't, it never made sense to me, like, because attackers are out there pounding on production instances all day long.

**Izar Tarandach (7:32)**: You could see that production is all that attackers see.

**Matt Coles (7:37)**: Yeah. Well, yeah, I mean, if you, if they see your backend environment, you have a bigger problem.

**Izar Tarandach (7:43)**: Yeah, then you have bigger problems then. But on a day-to-day basis, production is basically where attackers live. It's the attack surface that they're banging their heads against.

**Chris Romeo (7:53)**: Right?

**Izar Tarandach (7:54)**: So my thing, my thing is actually not even with the testing. My thing is, okay, let's say that you run tests in production. Fine. You found something.

**Chris Romeo (8:05)**: Now what?

**Izar Tarandach (8:08)**: Like, we know the speed of solution, right? And you know that if you found something and you are trying to solve, then chances are that somebody's already banging their head against that. So what do you do now? You, you, you stop everything until you fix that. You go back to a, an old version. What, what, what's the, the protocol here?

**Matt Coles (8:29)**: That's a really interesting question. And now I do wanna, I wanna touch upon something actually, the article The article's a bit misleading in, I think, in this, and, and, and I will, I will admit some ignorance here, maybe proving the point of the author. I, I think I understand what a pull request is. And by the way, there was a, a quote in the article itself saying that, that security people don't know what pull requests are. I, I think that that's, that's a bunch of hogwash and we can talk about that. But, but I think I know what a pull request is, but the, Quit the notion of, of the author's contention is that, um, testing should be done in production through automated tools. And, and, and by that he means starting at the pull request. I don't think a pull request happens in production. Does it?

**Izar Tarandach (9:27)**: I think that at some point you can trace back a pull request to something that went into production. So you should be able to, but it doesn't start that.

**Chris Romeo (9:35)**: Yeah, because it's, I mean, think about it. Where's the pull request begin its life? It's on a developer's workstation. They've made changes to code. They're, they're making a commit to Git, right? And then those Gits are, or those commits are wrapped together when you create the pull request. So we're in the source code control system. We're nowhere near the cloud infrastructure that's delivering the application unless, unless he has some other meaning.

**Matt Coles (10:01)**: Well, unless a pull request means when I, when the pull request is approved, it kicks off an automation toolchain that results in a push to production. In which case, in effect, right, your, your movement to production starts with the pull request. Yeah.

**Izar Tarandach (10:16)**: As soon as you trigger a pull request, that's a chain of events.

**Matt Coles (10:18)**: As soon as you pull up, move, execute a pull request. The next logical step, if it doesn't get blocked by any of the various functions, is production.

**Izar Tarandach (10:30)**: And assuming the fact that you pass through a test environment.

**Chris Romeo (10:33)**: Yeah. Right.

**Matt Coles (10:33)**: And that was, that's the contention from the author is that testing in production means starting at the pull request, executing automated tools. And, uh, I, I will be careful here because that sounds a lot like DAST. It does.

**Izar Tarandach (10:50)**: It does. It does.

**Matt Coles (10:51)**: Because you're not running static code analysis, right? Because how do you stop? How do you stop that?

**Chris Romeo (10:57)**: But wait, hold on. Hold on. So you're not running, you're not advocating for running SAST in a pipeline? I'm not.

**Matt Coles (11:03)**: I'm, I'm, I would advocate for it as a security person.

**Izar Tarandach (11:06)**: No, wait, wait.

**Matt Coles (11:07)**: But is that really an automation pipeline kind of thing you can do pre-product, like at that stage of I'm ready to go to production? You should have done that well before, I think.

**Chris Romeo (11:15)**: Yeah, but you have to run it in the pipeline as a check and balance between, yes, I, I'm hoping that that got ran when the code was committed, and hopefully there was a process or policy that enforced that constraint, but I still gotta run the balance to the, if the code check-in is the check, the balance is running the tool in the pipeline to make sure that it doesn't have some critical flaw that was just not mitigated properly through the the code commit, the code review, and the PR process.

**Izar Tarandach (11:44)**: Wait, I have to throw a flag here.

**Chris Romeo (11:46)**: Please.

**Izar Tarandach (11:46)**: So in the article, when he says testing in production, it starts with, one way to get developers to take security more seriously is to move testing and fixing vulnerabilities to correspond to the pull request, he added. That will mean testing in production, which he contended is more effective and efficient anyway.

**Chris Romeo (12:08)**: I see.

**Izar Tarandach (12:09)**: I don't know. These two sentences sort of like go against each other, right? Is it just in my head or do you guys feel that too?

**Matt Coles (12:19)**: We've established that a PR can result in a production change.

**Izar Tarandach (12:23)**: Yeah, but how does he jump from— is he talking about a different pull request from what we are? Is he talking about a different production than we are? What's the deal here? Okay. So let's look at the slide here. The pull request is the ideal place to surface feedback. Fine. Fair. Okay. The pull request, best of my knowledge, I'm just a security person. I'm sorry. I don't know a lot about pull requests. It's a great vehicle to talk back and forth about this new feature that's coming in. So it says developers are proud of the code they wrote this morning. I hope so. They want nothing more than their teammates to say how wonderful it is by merging it into the next higher-level branch that afternoon. Yep. I, I would feel like that. They'll jump through hoops to make sure it looks as wonderful as it can. Okay. Here we are starting to disagree. Days later, they have already moved on to a new heartthrob, the code they wrote that morning. Totally agree. Totally agree. Now, from here to say testing production, I, I, I'm missing, there's a jump here that I'm missing.

**Matt Coles (13:42)**: Well, actually, I would say there's two jumps. Uh, so two, what responsibility? I guess one, one is how do you get from developers do a pull, a check-in and a pull request, and then there's. Quote unquote testing in production that occurs, but also if the argument is that developers should own security, shouldn't the developer who make that code change this morning own its security through its lifecycle to the time it makes it to production? But if they've moved on to other things, how does that actually happen?

**Chris Romeo (14:15)**: I mean, I think he's just, he's just covering the standard way that this normally happens. Introducing the idea that rework is going to be the the outcome of not properly going through this channel. The challenge I'm having with it is how do I metric?

**Izar Tarandach (14:32)**: Oh my goodness.

**Chris Romeo (14:33)**: How do, but how do I metric this? Like if the PR is the place where we keep all of the correspondence, I don't like ticketing systems. I don't love ticketing systems, but what they do is they do give me metrics and traceability. I don't know that I can extract that from a PR.

**Izar Tarandach (14:48)**: Guys, I think that we are misreading the article.

**Chris Romeo (14:51)**: That's probably not the first time it's happened. So do tell.

**Izar Tarandach (14:56)**: I think that what he's saying is we have to do all the things that we have been talking about since we started talking here about testing as early as we can, testing on the desktop, testing in the test environment. Otherwise, there will be testing in production. He's not saying testing in production is the right thing to do. He's saying, unless you do the Shit left. The next thing to happen is testing will happen in production because you didn't test before. Because notice he says, if an automated tool gives you a message that says this has this vulnerability problem, this has this linting problem, then you're going to fix it immediately because you definitely don't want the guy on your team or the girl on your team who's responsible for reviewing this to say, hey, you don't pass that automated test. I'm not going to approve your code to get merged in.

**Chris Romeo (15:43)**: I don't, I'm not seeing this in the same way you are. Let me read you a quote from the article. This is in the testing and production section that we were just operating in. And by the way, this article was written by Lorraine Lawson, uh, interviewing Larry Meshram. Um, it says, one way to get developers to take security more seriously is to move testing and fixing vulnerabilities to correspond to the pull request, he added. That will mean testing in production, which he contended is more effective and efficient anyway.

**Matt Coles (16:15)**: Yeah. So that sounds like the right, that sounds like the positive states that he wants, not pre-test before you get to the pull request.

**Chris Romeo (16:23)**: Yeah. I don't think that's arguing for staying the testing. I don't think he's talking about, he's talking about doing testing in production.

**Izar Tarandach (16:31)**: I think it got lost in translation in the interview. I say we invite Larry in and get to the bottom of this thing.

**Chris Romeo (16:38)**: I wish I had the way to some internet device that could immediately connect me with anyone on earth. Bring them into our room to this call to be like, hey, we were just talking about your article.

**Izar Tarandach (16:50)**: Startup.

**Chris Romeo (16:52)**: Another startup idea generated by the security table.

**Izar Tarandach (16:57)**: Hmm. 17328.

**Chris Romeo (17:02)**: All right. Well then, let's got it. Let's, let's leave. Let's, let's, let's put, uh, push that issue off to the side. What else is here?

**Izar Tarandach (17:08)**: But that's a Drupal request. Yeah.

**Chris Romeo (17:10)**: Yeah.

**Izar Tarandach (17:10)**: What else, what else is here? So now we have the Agile drives development to eat other function.

**Chris Romeo (17:16)**: Yes.

**Izar Tarandach (17:16)**: Which I am inclined to agree, but.

**Matt Coles (17:22)**: Do you say?

**Izar Tarandach (17:26)**: The long arc of history for software engineering has been that the development team itself takes over more and more responsibility for all of these things that used to be done by security, by specialists. I don't know how long the arc of history is. I'm in this game for 25 years and all I see is developers running away from taking responsibility for security more and more. There's probably a very, very long arc, right?

**Matt Coles (17:55)**: I mean, well, so what you're talking about, you're talking about not just developers have a role in, in the process. But what, what this article is highlighting is trying to say is developers should own the process.

**Izar Tarandach (18:09)**: And not only that, they want to own the process.

**Chris Romeo (18:13)**: That might be where we have some difference of opinion based on what we've seen in reality with our experiences. But I think his, I think his intent is correct here. To say that we want to, to acknowledge the fact that, you know, IT operations is no longer even a thing that developers live within. And also to acknowledge or to say that we want, we would love a world where security is led by developers. We've talked about that in past episodes and we've gone through thought experiments. I remember what would happen if security was gone. Like we just got rid of security and we just said developers are going to do it. So this is the state we want to head towards because we all know the scalability of what we do, right? There's never enough security people to line up with the number of developers and it's, nobody's ever going to solve this problem. So that's, so that, that's a goal. That's a futuristic thing we're looking towards, but it, it hasn't been a reality in, in the environments that I've seen.

**Matt Coles (19:15)**: And it comes down to resources. It comes down to, uh, accountability and, and getting features out the door because that's what drives revenue, right?

**Chris Romeo (19:27)**: Yeah.

**Matt Coles (19:28)**: Whereas security takes time, takes effort, takes money. So it's like quality. It's like the quality problem also, right? And we've talked about this too. Security is a lot like quality. If you're, if you're pressured to get things out the door, maybe quality takes a little bit of a backseat. Well, security would be the same way.

**Izar Tarandach (19:50)**: But the, the difference for me, and you know that I am a big proponent of security and quality go together. I, I think that's the, the discriminator for me is that quality improves over time, even if you don't do anything, because you're going to make mistakes and you're going to improve on them because those mistakes are like easier to put your finger in. Like, you do an off-by-one, you learn that, hey, you write differently so you don't have an off-by-one, and you don't write it next time. But the quality aspect of security sometimes is so opaque. It's so, it applies to this situation, but not to that situation. Right? That there's not like no, unless you go down to principles, if you're on the surface, if you're, if you're on the top and you're just fixing lines of code without thinking about applying actual security principles, it's not like doing more of it is going to make you better at it. It's just you're dealing with those things that are pure quality of the code, but I just don't see developers running to me saying, please, please, please teach us how to code right. I mean, yes, but no, not all of them. We still have the security champion. We still have those good teams that take time to go and learn from Jim. We have those organizations that go to something like Security Journey and get the whole package and learn and at least go through it.

**Chris Romeo (21:32)**: Right?

**Izar Tarandach (21:33)**: But I still don't see developers clamoring for make me a security wizard so that I can be a better programmer. And by the way, I don't need you next week because I already know everything I need to know about security. Plus, I think that code quality, and sorry that I'm going so long, but I think that code quality is like a constant, right? Your code has to be written in a certain way. To be considered quality code, but the security keeps changing. They take away one part of the stack, put another one, that thing is bringing different security problems with itself. And that dynamic, I think, would take way too much investment from a developer to keep on top of, unless we are talking, again, security principles. And I just don't see developers coming in and saying, hey, you know what? I already have a week with 120 hours. I might as well do 140 and use the next 20 just to learn security. Yeah, I just don't see it happening.

**Chris Romeo (22:32)**: Yeah.

**Matt Coles (22:32)**: So I, I think that this is, um, the, the, the fundamental issue I have with the way the article is written is that it suggests that security can go away completely, completely as a function. And I think that the answer is no, that's not true. We may want it to, but I, I, I think there's a role for this. So I think you, you brought it up, right? I think there's an evolution of security over time and somebody needs to keep, keep aware of that. How you implement that, how you operationalize that might be, uh, pushing that into development, whether that's through champions or training or just tool enhancements, right? So security as a tools team, right? The same way that quality is. To some extent, a tools team, right? Take institution— take institutional and, and cutting-edge knowledge and build it into the capabilities that developers use so that it becomes, uh, fairly transparent to them, right? Part of their day-to-day operations. I run and, and, you know, obviously with AI and Copilot and things like that, now you can potentially— the developers will have this knowledge directly at their fingertips. Um, make them responsible for the code that they write from that developer who writes something to this morning that they're really proud of to whatever gets integrated that's going to go to production. Take the human out of the loop and put the automation, right? Bake, bake the automate, bake the knowledge into the automation.

**Izar Tarandach (24:07)**: Steer.

**Chris Romeo (24:11)**: Advocating for the guardrails, paved roads, pipelines, all of these things.

**Matt Coles (24:19)**: Yes.

**Chris Romeo (24:19)**: Keeping the developers on the straight and narrow. Yeah. And not giving them room to swerve off the side of the mountain because there's a steel wall that prevents them.

**Matt Coles (24:30)**: Oh, I wouldn't go that far. I wouldn't go that far. I, I would say that those guardrails should be should be metal guardrails, not like, uh, if anyone's driven through Connecticut, you still, you still have steel cables as guardrails. Um, those are, those are probably too dangerous, but, but I wouldn't put up Jersey barriers as guardrails.

**Izar Tarandach (24:53)**: Look, a guardrail at, at the end of the day has to be flexible enough so that you stop people going down the cliff. But if they want to fly, they can, right? Because otherwise we are just stifling innovation. So paved roads, guardrails, secure by design, secure by default. That's where I think that security will always be there by creating those things. But creating a culture that consumes those things as part of development, That's where I think that the next challenge is because Lernals, it's very easy to write lots and lots and lots and lots of documentation. And I think that there's a reason why we don't actually go back and look at how many people actually read it, because it would be depressing. But this comes back like again and again and again in every team that I join. Hey, we have to create the documentation. Yes, we have. But now how do you get people to actually use it? That's the, the culture part that we haven't yet, we, we haven't broken that code yet.

**Matt Coles (26:05)**: And security has a role in, uh, the article highlights that security sort of has a role in accountability, right? There's a, there's a structure for accountability that has to be put in place so that you don't have, I mean, development organizations know how to manage risk today with quality.

**Izar Tarandach (26:27)**: They like to think so.

**Matt Coles (26:28)**: Yeah. With quality. But, but even then we know that there are, need to be checks and balances. You don't have the developer committing code and the developer com— approving the pull request also reporting on its quality. Well, maybe you do, but maybe you shouldn't. And there needs to be a structure so that the powers that be can make informed risk decisions. And I'll take care of all—

**Chris Romeo (26:50)**: I want to take it. I'm going to cover on the same topic that you were just describing here about the powers that be and whatnot. I'm going to, I'm going to disagree a little bit with something that's said here in the article, right in that first paragraph. No, no, second paragraph, which is a direct quote from Larry. The reason there's no consequences though is because the power dynamic is that you as developers have more power than the security folks. The CISO has been beat up in meetings with his boss and VP of engineering. Where they said security just threw this crap on us and we couldn't ship on time. And then pre— in the previous paragraph, it says there aren't consequences when development ignores security, but often there are consequences for the CISO when they try to play hardball. And that's a maturity problem inside of an organization. 'Cause like, even back in, before I left Cisco, back in 2014 or so, a governance model was put into place and there was, we used to jokingly refer to it as the hammer that John Stewart could drop the hammer on a product team. John Stewart was the SVP of, uh, of security, basically chief security officer. And, uh, but, but they introduced this concept of a stop ship where John Stewart could stop ship any product inside the company. And so It was, it was the mature, it was a governance and maturity move after spending years investing in CSDL, Cisco Security Development Lifecycle, building the processes, having all the things, the tools, the documentation, everything people needed for success. And the crown achievement coming out of that was there was a governance stop ship mechanism. That's back in 2014, this was happening. That's 10 years ago now. So my, so I'm, I'm gonna, I'm gonna take exception to that and say, I don't think that's the case in, in mature organizations, what, what Larry's describing here. Sure. In new start and small startups, maybe even mid-sized startups, there's gonna be a governance. You're not gonna have a governance function in ProdSec, AppSec, but in mature companies, you better have governance at this point. If you don't, put that in your K-1, you better disclose that thing in your K-1 so I can sell my stock if I have any in your company.

**Izar Tarandach (29:01)**: But you know what? I think that sometimes it's not really a, a matter of, uh, uh, maturity, I'll go back and say it's a matter of culture. Because, and especially, you know, with the market pressing down on people so much to put things out there, put flashy stuff out there as quick as they can, stopping something for a security person, not a developer, but for a security person to build a case that's strong enough to stop shipment That thing has to be ironclad and it's going to be like questioned by all levels of the business up to the person who actually comes down and says, yeah, let's stop it. And I don't think that security people, many security people nowadays feel strong enough to put such a case on the table. And many times—

**Matt Coles (29:54)**: You're also not encouraged to do so, by the way.

**Izar Tarandach (29:56)**: Yes. And many times we don't have the language of risk. To be able to translate the thing that's for us in our heads is just like so freaking clear. Something's going to explode here. But as you move the, the, the, the peg through the holes, that risk gets attenuated more and more and more and more until it gets to the function that actually decides stopping shipping. And they say, no, there's no risk here. Ship it.

**Chris Romeo (30:24)**: And this is something that I thought the market was gonna correct. For the last decade or two, I've been watching the world of security and how security plays out inside of companies that build things. And I just kept thinking to myself, the market is gonna correct this. We are gonna reach a flashpoint where consumers, meaning businesses and B2B environments, enterprises are gonna say, I'm not buying this crap if it's insecure. And we just haven't seen that happen yet. And I'm shocked that it's, it's taken this long and there hasn't been a flashpoint point. We had the data breach decades. We've had, you know, extreme malware Greek decade here in the last decade. Like, it's not like things are quiet and like, oh, maybe we don't need to do security. Like, it's, the risk is there in front of us. The threat landscape is larger than it was, but the market just never caught up.

**Matt Coles (31:22)**: And we see this, we see this coming in with, with now with, uh, government, especially, you know, the CISA secure attestation and the recent SEC action, you know, against the company for, you know, in their case for falsified statements on their disclosure forms, right? That they don't meet, they didn't meet their security posture and therefore they have unexplained risks. Um, but that's not what you're talking about here, right? That being governments have, governments that buy things have always had a rigorous assurance process.

**Chris Romeo (32:04)**: Yes.

**Matt Coles (32:05)**: In terms of deterrence.

**Chris Romeo (32:07)**: But governments, while they are the largest customer of most companies, they don't represent, like, I'm talking about the market, like, I'm talking about, I thought other enterprises were going to force security and stop buying insecure things, but I just haven't seen it happen. And that's, that's what I'm shocked about.

**Izar Tarandach (32:28)**: I think that it's because at some point we came to a, a feeling of learned helplessness. We, I know that a lot of people are at the point of breaches will happen. It's a fact of life. I'll just lock my credit report and hope for the best.

**Chris Romeo (32:47)**: And then cyber insurance had an impact on the institutional data breach because you could just buy insurance to a certain level to protect you. Now, it turns out collecting on that insurance isn't as easy as it sounds in the paperwork, but that there is some risk mitigation happening there.

**Izar Tarandach (33:07)**: And if it was mandated security insurance, I don't think it is. I, I, I'm not very current on the details, but I don't think that it is yet. I think that it's still optional. Then the powers of the market might influence that through the, okay, you have to buy insurance, so let's do something that will lower the risk. But, uh, I keep going back to that thing that at some point we used as security professionals, we used to scare the business into doing what we needed by saying, hey, imagine if there is a breach, the hits that you're going to get in your in your stock, right? But turns out that there is no hit in the stock. It's a little bump and then it goes back right to where it was. Again, showing help to, uh, learn the helplessness.

**Matt Coles (33:52)**: Yeah. Like that there's, that there's no, that there's a, what, so my question then is what, what does a business, what does a development organization, what does an engineering team or product team care about, like what drives the needle for them. So quality defects, you get customer complaints, they drive, you know, customer support calls. Ultimately, some portion of customers stop buying. That doesn't improve— does that improve quality enough? And does that, does that work for security?

**Izar Tarandach (34:31)**: So I'm going to throw something on the table and please feel free to disagree with me, but This is what I experienced. Life got way more complicated for developers when the center of gravity of development moved from the architect to the product manager. The product manager speaks a language of A/B testing and feature promises to customers. Drives the bottom line, because that's what you're selling. And then what the product manager puts on the line, or sorry, on the, on the table is what the developer has to pick up and somehow make work. And the race here is not to create the best possible, most high-quality product, which would be if a developer as an artist would be thinking about crafting the best thing that they can and putting it in front of the customer and enjoying the customer enjoying that. That's creation, but it becomes just this, this blob on the table with all sorts of shiny things without anything behind it that some product manager, for some reason that Lord knows what it is, decided that is what has to go in front of the customer, right? The, the, the wow factor became way more influential in what we get to experience as customers. Than quality and functionality?

**Chris Romeo (36:07)**: I mean, product managers are just doing what they have to do to serve the needs of the customer. The best product managers are listening to the customers, are listening to the market, and they're building things to meet those requirements. Specifically. So I'm not gonna put, and yeah, I get in the old days we used to have architects that would drive where the products were going, but this is a new age. This is a new era. Like it's, it's about features and it's about speed to market. It's about, I think, wow.

**Matt Coles (36:41)**: Yeah. I think that's the important part. I think that's the important part. It's, it's the product manager is defining features. The product manager is not defining the how should all these features come together and work and is there a unified vision and is it supportable and manageable and, and meet certain It hits me.

**Chris Romeo (36:57)**: I would argue that they do. I would argue that a product manager has a bigger— now, maybe in a giant company, in a giant product, there's 10 product managers that are working on one product. But I would argue that they do have a unified vision. The challenge has been they've had no motivation to push security or privacy from where they sit.

**Izar Tarandach (37:16)**: Exactly. Exactly. Because their motivation is in making that big customer that sits in the advisory board satisfied, right? More than creating a continuous line in the product development that makes sense. I'll give you this feature and then I'm going to amplify with this feature and then I'm going to amplify this feature. No, right now is I'm going to put this thing in front of you because somebody in the cab told me that they want it and they're paying us a lot of money. It's not going to work perfect. but eventually it'll grow into something. And then you end up with like the, a tree with partial branches. It's a huge tree. You look from a distance, oh my God, the thing is green and luscious and beautiful. And as you get close to it, each one of the branches is broken. And at the end of the day, the functionality is just not there, or it's just not giving you what you want.

**Matt Coles (38:12)**: Let's bring this back to the article a bit. So let me pose the question to you, uh, because, so I, I think I agree with you, but I'm, I'm gonna pose the question to you, Azar and, and Chris. Um, let's say for a moment the developers take on security. First off, does this solve the— does this help solve the problem that you've raised, that the product manager is working towards a particular agenda that developers who care about their code may, may not see? Uh, and alternatively, one of the challenges I think I've seen over the years is a, is a question of air cover. Developers want to do great things. That takes time, which means they're not working on, you know, velocity and, and feature development. They're shoring up the things that they built, not building more stuff. And so they need air cover to do that. So, okay. Uh, I guess based on what you're describing and what Larry is suggesting, how does this work? How do developers take on security? Given they're not getting direction or air cover to take on security.

**Izar Tarandach (39:20)**: That's the thing. They only—

**Chris Romeo (39:21)**: but it's automated. That's, I mean, his, he's, he's talking about a DevSecOps framework where security is, you're dealing with—

**Izar Tarandach (39:29)**: where security happens.

**Chris Romeo (39:30)**: Yeah. But you're not, you're not, you're not, developers are not now the security department. It's the security department has wired up the pipelines in such a way that the developers can then respond to security problems automatically generated and attached to their PR in the way that, that Larry was describing.

**Izar Tarandach (39:51)**: And that by definition is a snapshot operation because you give those tools to the stack that you're working with now. Tomorrow the stack changes. If you don't have the security people to change the tools and to change the, the, the processes, the developers are not going to do it.

**Matt Coles (40:09)**: It is a very reactive process, what, though, what you're describing. So let's talk about something that came up in the Secure by Design papers, you know, from, from CISA, for instance, around using memory-safe languages, right? That's not a, that's not an automated tool, you know, from a PR to an automated tool finding solution. That is a, I'm gonna refactor my code in a language that solves these problems for me.

**Chris Romeo (40:33)**: Yeah. And that's an architecture problem. Right? Like, it's a design decision that needs to be either done upfront if you're starting something brand new, or it needs to be retroactively applied now. It's going to be a challenge to change to a memory-safe language from an unsafe language if the application's already in flight. It's just, it's just the reality of it. Could you imagine telling a team, hey, we're going to refactor the entire application? We're gonna migrate from, uh, PHP to Go. And we're gonna, yeah, I mean, it's, it's not, I mean, it's a, at that point you might as well just build a new product. You might as well just end of life the thing that you were, you know, start working on a new product and just start from scratch and build something. You're not gonna convert that. So there are some things where the business case just isn't gonna be there. To what you were just describing as far as—

**Matt Coles (41:28)**: So when does that ever happen? When can that happen? If, if developers want it, you need the air cover to do it, or you need a team to help, right? Is that security's role?

**Chris Romeo (41:40)**: Security team doesn't solve that problem though. I mean, they, they could provide the stack, the current best of breed stack. If you're starting a new application, start with these pieces today, which that's important. That's something that's not always as easy for every developer perhaps to understand what are the right pieces I should snap together here. So that could be part of security's role. But on the air cover front, I mean, security doesn't really provide air cover for developers. So that's not, so the air cover, it really has to come from the product management because product management needs to be driving what developers are doing and prioritizing security and privacy. In a perfect world for me, that's something that product managers are doing, and then they're providing the air cover. They're stepping up to executives and saying, we're not shipping an insecure product, we're not shipping a product that violates privacy. And so if either of these core things are broken, I'm gonna, I'm gonna direct engineering to do the things that they need to do to fix that. And it just requires a, a new breed of product manager that's very security and privacy savvy.

**Matt Coles (42:48)**: Or it's a shift in our, in our focus as security people. We've been for the past decade working on developers. Maybe we actually did a good job with developers. Is our new focus product managers?

**Chris Romeo (43:01)**: I think so.

**Izar Tarandach (43:01)**: Or, or the function that we are forgetting here and the one that we're supposed to provide the air cover from the beginning is the CISO, which in here he says gets beaten around the head. Where is it? Uh, because the power dynamic is that you as developers have more power than the security folks, so that CISO has been beat up in meetings with his boss and the VP of engineering, where they said security just threw this crap on us and we couldn't ship on time. So there's a number of assumptions happening here that the CISO did give the arc over and the developers were asked to stop what they were doing and do the security thing. And that made things go slower. And now the VP of engineering and the boss are coming and screaming at the CISO, why did you do this? So the air cover was in place. The problem is that somebody brought in the anti-air batteries and is now trying to shoot down your, your, your, your air cover.

**Chris Romeo (43:59)**: He's describing an environment where executive management is not on board with prioritizing security and privacy. And that's the whole point I was making about, I thought the market was going to correct around that. I thought. Market, I thought business and enterprise consumers were going to start seeing that as a failure and stop buying from companies that had that philosophy. That's part of the challenge though, is that just people continue to buy from people that don't take security and privacy seriously.

**Izar Tarandach (44:29)**: And I think that one of the biggest problems here is that what's a big, uh, example that we have of some CEO that said, Stop the presses and, um, we are going to do security now.

**Matt Coles (44:46)**: Yeah, that would be my guess.

**Izar Tarandach (44:46)**: Have we heard of anybody lately?

**Chris Romeo (44:48)**: William Gates, 2004, trustworthy company.

**Matt Coles (44:50)**: And then, and, and then recently in 2024, not Bill. Okay.

**Izar Tarandach (44:55)**: Yeah. And if we get those exceptions, and then Satya Nadella, the currency.

**Chris Romeo (44:59)**: Yeah. You had another security moment. Okay.

**Matt Coles (45:01)**: Yep.

**Izar Tarandach (45:01)**: And now we go and we put this in front of a CEO expecting him to see the light and say, from now on, we have a security culture here. And they look back at you and say, what do you want from me? I'm not Microsoft.

**Chris Romeo (45:12)**: Well, no, they look back and say, what's the return on investment for this? If they're smart.

**Izar Tarandach (45:17)**: That, that's the thing. No, no, no. That's the thing. I can, I can come and say, listen, if Microsoft found it right to do it, then there must be some return of investment. I may not be able to give you numbers. Yeah, but there must be one because Microsoft is doing it.

**Chris Romeo (45:30)**: It doesn't, that, that, he's going to look at me. That doesn't convince me as an executive. I'm like, okay, they're doing it.

**Izar Tarandach (45:34)**: That's the thing. He's going to look at me and say, I'm not Microsoft.

**Chris Romeo (45:37)**: Take a walk and, and figure out what your return on investment proposition is for me, because executives deal in business value, in dollars saved, dollars gained, revenue increases. If you're, if you can't sell your product, your, your new security vision in such a way that it's gonna save me money, it's gonna get me more customers, it's gonna increase my revenue, I'm sorry, but it's not a priority for me.

**Izar Tarandach (46:03)**: And between developers, security people, and product managers, which one of these three tribes speaks that language?

**Chris Romeo (46:12)**: Product managers are the closest. Product managers.

**Izar Tarandach (46:16)**: QED. I rest my case, your honor.

**Chris Romeo (46:17)**: I think we've solved it.

**Izar Tarandach (46:18)**: We have solved something on the security table.

**Chris Romeo (46:20)**: We have a victory. Now industry, listen up. Here's what you're going to need to do. Get your product managers all together at an internal conference. Called—

**Izar Tarandach (46:30)**: Wait, wait, I need notes. Notes. I need to take notes.

**Chris Romeo (46:32)**: It's recorded. It's available. You can have AI summarize it for you. Well, I do think we made some progress here. We don't often make progress on the security table. Often we end up back where we started or flummoxed a bit at the end. But this time I feel like we've provided our industry with a new vision. So industry, take it forward. Be successful.

**Izar Tarandach (46:53)**: Do your thing.

**Chris Romeo (46:54)**: Send us small stipend check for every month of success that you have. And you can send that to the security table at PO Box 10011001100. And that's Worcester, Massachusetts, 2743124. I don't know, I just made that up. Don't send anything that doesn't exist. Guys, thanks again. This is, uh, was a fun conversation. and we look forward to doing it again in the future on the Security Today.
