Why Developers Will Take Charge of Security, Tests in Prod
With Chris Romeo, Izar Tarandach, and Matt Coles
The script delves into a multifaceted discussion encompassing critiques and praises of book-to-movie adaptations like 'Hitchhiker's Guide to the Galaxy', 'Good Omens', and 'The Chronicles of Narnia'. It then transitions to a serious examination of developers' evolving role in security, advocating for 'shift left' and DevSecOps approaches.
About this episode
The script delves into a multifaceted discussion encompassing critiques and praises of book-to-movie adaptations like 'Hitchhiker's Guide to the Galaxy', 'Good Omens', and 'The Chronicles of Narnia'. It then transitions to a serious examination of developers' evolving role in security, advocating for 'shift left' and DevSecOps approaches. The conversation navigates through challenges developers encounter in security practices, stressing the necessity of a DevSecOps framework, secure coding languages, and executive support for fostering a robust security culture within organizations.
Chris, Izar and Matt begin the episode with a lighthearted discussion about books turned into movies, including Hitchhiker's Guide to the Galaxy and The Chronicles of Narnia series. The main topic of conversation on today’s episode is an article titled "Why Developers Will Take Charge of Security, Tests in Production" by Lorraine Lawson, which interviews Larry Meshrom. The article suggests that developers should take on more responsibility for security, including testing in production environments, as security teams are often perceived as a blocker and don't understand the day-to-day work of developers.
The guys question whether developers truly want to take on more security responsibilities, given the constantly evolving nature of security threats and the time it takes to stay up-to-date. They also discuss the role of product managers in driving security and privacy prioritization, and the need for executives to understand the business value of investing in security. The hosts argue that while mature organizations have governance processes in place to enforce security, smaller companies may lack such mechanisms.
Ultimately, it is concluded that product managers are best positioned to communicate the business value of security to executives, as they are closest to understanding customer needs and revenue drivers. They propose that the industry should focus on educating and empowering product managers to prioritize security and privacy, and to make the case for investing in these areas to executives. This approach could help bridge the gap between security teams and developers, and drive a culture of security within organizations.
Link to article: https://thenewstack.io/why-developers-will-take-charge-of-security-tests-in-prod/
Follow AI Security Table:
➜ Home: https://securitytable.ai/
➜ X: https://x.com/AISecurityTable
➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/
➜ YouTube: https://www.youtube.com/@AISecurityTable
Transcript · 48 min conversation
0:09Chris Romeo 34 is not the answer to the meaning of life, the universe, and everything.
0:14Matt Coles You're off by a bit. You're off by a bit, man.
0:17Chris Romeo You know this from the book. Come on.
0:19Izar Tarandach It's inflation.
0:21Matt Coles It's deflation.
0:24Chris Romeo Deflation. Deflation.
0:24Izar Tarandach Deflation. Sorry. Yeah. Deflation.
0:26Chris Romeo We're in more of a deflationary environment. So that's— Yes. Yes. That restaurant at the end of the universe, you know, that sort of thing. So yeah, you know, in case anybody doesn't know what we're talking about, Google it.
0:37Izar Tarandach Google it.
0:38Chris Romeo I'm not gonna tell you where it's from.
0:39Matt Coles Or ChatGPT 4.0 it.
0:41Izar Tarandach No, first you Google to find what it is and then you go to ChatGPT and you ask for, for a, a concise, uh, explanation of it.
0:49Chris Romeo Yeah.
0:49Izar Tarandach And you just cross your fingers that it doesn't hallucinate.
0:52Chris Romeo Or what did you guys—
0:53Matt Coles go by Cliff Notes.
0:55Chris Romeo What did you think of the movie?
0:58Izar Tarandach No, no, no, no.
1:00Matt Coles Oh, come on. Alan Rickman was hilarious.
1:01Chris Romeo All right. We're talking about, we're talking about Hitchhiker's Guide to the Galaxy. So for those that were too lazy to Google this, um, but so, so yeah. So, so Matt, sounds like you're maybe thumbs up and Ezar's thumbs down.
1:15Matt Coles It was a, it was a, we're on camera. It was a sort of a, mm, yeah, mostly not, but sort of some were really good. And then the guy who lived, Alan Rickman was good, but You know, the, the, the, the, the Evil Rocks wasn't not so good.
1:29Izar Tarandach No, the, the design of, uh, Marvin, like, no, it was too chirpy.
1:39Matt Coles We'll have to watch it again. It was, it was, uh, all right. So there were some funny parts. It was pretty campy, but overall it wasn't bad.
1:46Izar Tarandach Ah, we like campy. Come on. We like campy.
1:49Chris Romeo I mean, it was consistent with the story though, right? They didn't, it was, it wasn't like Lord of the Rings where they like, who's that person? No, Hobbit. The Hobbit movie's like, who's that lady? I don't remember her in the book. The elf, there was an elf lady that they just introduced as a new character and like, she was a great character, but I don't remember. I went and looked in the book, she wasn't there.
2:08Izar Tarandach Yeah. But you know, that, that, that's my biggest fear. There are some books, some authors that we keep reading. I keep reading again and again and again. And when I hear that there's going to be an adaptation for TV or whatever, I, I shudder. So like get, uh, Good Omens. Good Omens to me is one of the like most hilarious books I ever read. Terry Pratchett and Neil Gaiman. The Amazon adaptation, fair, good. I like, I like the actors, but took a lot from the book. But then they come out with a season 2. Come on. No, no, no, no. Now you're just doing it for the money.
2:42Matt Coles Well, so Lion, the Witch and the Wardrobe and Voyage of the Dawn Treader in that series, you know, the CS Lewis.
2:47Izar Tarandach Yeah.
2:48Matt Coles C.S. Lewis series Chronicles of Narnia held to the book pretty well.
2:53Chris Romeo Pretty well.
2:54Matt Coles And with some evolution in graphics over, over the years that they built that, uh, did those—
2:59Chris Romeo I recently read, reread that series. Come to find out there's a book before The Lion, the Witch and the Wardrobe.
3:07Matt Coles Is there always?
3:08Chris Romeo Yeah, it's called The Magician's Nephew and it—
3:11Izar Tarandach oh yeah.
3:11Chris Romeo Informs you so much about The Lion, the Witch and the Wardrobe. And I never knew that there was another book. I never read it.
3:18Matt Coles But that's written, but that's not, that was written after. And so it's a prequel.
3:22Chris Romeo Is it? Okay.
3:24Matt Coles Yeah.
3:25Chris Romeo Okay. All right.
3:26Izar Tarandach So it was written after, but it's a prequel.
3:28Matt Coles Yeah. Prequel. Kind of like episode 1 of Star Wars.
3:32Izar Tarandach Oh, okay.
3:35Chris Romeo Except no Jar Jar Binks.
3:37Izar Tarandach So no.
3:37Matt Coles Oh my God. No Jar Jar Binks.
3:39Izar Tarandach Uh, no, let's, let's, let, let, let, no, no, no. Don't go there. Let, let's talk to something else.
3:44Chris Romeo Okay. Well, that wraps up this segment of books that you could have read that turned into movies that, that you shouldn't have watched, but that you would have avoided, should have avoided completely. Uh, I guess we're actually here to talk about something else on the Security Table.
4:01Matt Coles In theory. Yeah.
4:02Chris Romeo So Matt, why don't you set the stage for us on this article, Why Developers Will Take Charge of Security Tests in Production?
4:10Matt Coles Uh, There's a comma there.
4:13Izar Tarandach The comma.
4:14Matt Coles So the comma is kind of important. Uh, why would, why would developers will take charge of security, comma, and testing and production.
4:24Chris Romeo Ah, okay.
4:26Matt Coles So, uh, this was an article was written, um, just a couple of days ago about, um, I mean, in a nutshell, how traditional security functions are completely out of the loop and I'm misunderstanding or don't know how developers work and why can't developers do a better job? I'm paraphrasing. And, and really this goes to the shift left discussion, you know, and DevSecOps or, or whatever it was supposed to be versus what it became. In a nutshell, there's really, it's highlighting a couple things. First off, that You know, security is a blocking function or perceived as a blocking function in many cases. And, and can developers do a better job? Should they be given the responsibility to do a better job? And also that, and in part, security is a blocking function because it's based on, you know, an old lifecycle model. We've had this conversation multiple times on the, on the podcast. And, and also, you know, we're, we in security don't understand what developers do for a day on a day-to-day basis. So I think there's a few things we could touch upon. I'd like to really touch upon the work. Not all security people are created equal. We have different roles.
5:50Izar Tarandach And thank God for that.
5:51Matt Coles And, and there's a couple other topics I'm sure we could cover. I'm not sure we want to. How we want to talk about testing in production. That sounds bad, but, you know, I'm just a security person.
6:04Chris Romeo So. You want to start there first and work our way backwards? The, you want to talk about the testing in production as a launchpad?
6:10Matt Coles So, um, yeah, so testing in production.
6:14Izar Tarandach What can go wrong? I love the characterization by Chris as a launchpad. What else happens on launchpads?
6:22Chris Romeo I'm not that witty, but thank you for that.
6:25Matt Coles If you haven't, if you haven't tested yet, uh, bad things happen.
6:28Izar Tarandach If you haven't tested either, you put it on the launch pad, it goes up a bit, then bang.
6:32Chris Romeo Yeah.
6:33Matt Coles I think it goes up a bit.
6:35Chris Romeo I think the argument that I've heard and one that I've made over the last number of years is, because I mean, as our audience knows, I'm old enough to have been around from the days where you would not ever test anything. You would not send test cases towards the production instance of this application. You could knock it over, cause it to stop working. Like that, I mean, I grew up in that era, right? And so that, but the thing that's always caught me is like, production means attackers don't follow the rules of we can't scan something in production, we can't test it, we can't attempt to knock it over. Attackers are going to do that anyway. And so this whole idea that like, you know, production is somehow some sacred place, it just, it doesn't, it never made sense to me, like, because attackers are out there pounding on production instances all day long.
7:32Izar Tarandach You could see that production is all that attackers see.
7:37Matt Coles Yeah. Well, yeah, I mean, if you, if they see your backend environment, you have a bigger problem.
7:43Izar Tarandach Yeah, then you have bigger problems then. But on a day-to-day basis, production is basically where attackers live. It's the attack surface that they're banging their heads against.
7:53Chris Romeo Right?
7:54Izar Tarandach So my thing, my thing is actually not even with the testing. My thing is, okay, let's say that you run tests in production. Fine. You found something.
8:05Chris Romeo Now what?
8:08Izar Tarandach Like, we know the speed of solution, right? And you know that if you found something and you are trying to solve, then chances are that somebody's already banging their head against that. So what do you do now? You, you, you stop everything until you fix that. You go back to a, an old version. What, what, what's the, the protocol here?
8:29Matt Coles That's a really interesting question. And now I do wanna, I wanna touch upon something actually, the article The article's a bit misleading in, I think, in this, and, and, and I will, I will admit some ignorance here, maybe proving the point of the author. I, I think I understand what a pull request is. And by the way, there was a, a quote in the article itself saying that, that security people don't know what pull requests are. I, I think that that's, that's a bunch of hogwash and we can talk about that. But, but I think I know what a pull request is, but the, Quit the notion of, of the author's contention is that, um, testing should be done in production through automated tools. And, and, and by that he means starting at the pull request. I don't think a pull request happens in production. Does it?
9:27Izar Tarandach I think that at some point you can trace back a pull request to something that went into production. So you should be able to, but it doesn't start that.
9:35Chris Romeo Yeah, because it's, I mean, think about it. Where's the pull request begin its life? It's on a developer's workstation. They've made changes to code. They're, they're making a commit to Git, right? And then those Gits are, or those commits are wrapped together when you create the pull request. So we're in the source code control system. We're nowhere near the cloud infrastructure that's delivering the application unless, unless he has some other meaning.
10:01Matt Coles Well, unless a pull request means when I, when the pull request is approved, it kicks off an automation toolchain that results in a push to production. In which case, in effect, right, your, your movement to production starts with the pull request. Yeah.
10:16Izar Tarandach As soon as you trigger a pull request, that's a chain of events.
10:18Matt Coles As soon as you pull up, move, execute a pull request. The next logical step, if it doesn't get blocked by any of the various functions, is production.
10:30Izar Tarandach And assuming the fact that you pass through a test environment.
10:33Chris Romeo Yeah. Right.
10:33Matt Coles And that was, that's the contention from the author is that testing in production means starting at the pull request, executing automated tools. And, uh, I, I will be careful here because that sounds a lot like DAST. It does.
10:50Izar Tarandach It does. It does.
10:51Matt Coles Because you're not running static code analysis, right? Because how do you stop? How do you stop that?
10:57Chris Romeo But wait, hold on. Hold on. So you're not running, you're not advocating for running SAST in a pipeline? I'm not.
11:03Matt Coles I'm, I'm, I would advocate for it as a security person.
11:06Izar Tarandach No, wait, wait.
11:07Matt Coles But is that really an automation pipeline kind of thing you can do pre-product, like at that stage of I'm ready to go to production? You should have done that well before, I think.
11:15Chris Romeo Yeah, but you have to run it in the pipeline as a check and balance between, yes, I, I'm hoping that that got ran when the code was committed, and hopefully there was a process or policy that enforced that constraint, but I still gotta run the balance to the, if the code check-in is the check, the balance is running the tool in the pipeline to make sure that it doesn't have some critical flaw that was just not mitigated properly through the the code commit, the code review, and the PR process.
11:44Izar Tarandach Wait, I have to throw a flag here.
11:46Chris Romeo Please.
11:46Izar Tarandach So in the article, when he says testing in production, it starts with, one way to get developers to take security more seriously is to move testing and fixing vulnerabilities to correspond to the pull request, he added. That will mean testing in production, which he contended is more effective and efficient anyway.
12:08Chris Romeo I see.
12:09Izar Tarandach I don't know. These two sentences sort of like go against each other, right? Is it just in my head or do you guys feel that too?
12:19Matt Coles We've established that a PR can result in a production change.
12:23Izar Tarandach Yeah, but how does he jump from— is he talking about a different pull request from what we are? Is he talking about a different production than we are? What's the deal here? Okay. So let's look at the slide here. The pull request is the ideal place to surface feedback. Fine. Fair. Okay. The pull request, best of my knowledge, I'm just a security person. I'm sorry. I don't know a lot about pull requests. It's a great vehicle to talk back and forth about this new feature that's coming in. So it says developers are proud of the code they wrote this morning. I hope so. They want nothing more than their teammates to say how wonderful it is by merging it into the next higher-level branch that afternoon. Yep. I, I would feel like that. They'll jump through hoops to make sure it looks as wonderful as it can. Okay. Here we are starting to disagree. Days later, they have already moved on to a new heartthrob, the code they wrote that morning. Totally agree. Totally agree. Now, from here to say testing production, I, I, I'm missing, there's a jump here that I'm missing.
13:42Matt Coles Well, actually, I would say there's two jumps. Uh, so two, what responsibility? I guess one, one is how do you get from developers do a pull, a check-in and a pull request, and then there's. Quote unquote testing in production that occurs, but also if the argument is that developers should own security, shouldn't the developer who make that code change this morning own its security through its lifecycle to the time it makes it to production? But if they've moved on to other things, how does that actually happen?
14:15Chris Romeo I mean, I think he's just, he's just covering the standard way that this normally happens. Introducing the idea that rework is going to be the the outcome of not properly going through this channel. The challenge I'm having with it is how do I metric?
14:32Izar Tarandach Oh my goodness.
14:33Chris Romeo How do, but how do I metric this? Like if the PR is the place where we keep all of the correspondence, I don't like ticketing systems. I don't love ticketing systems, but what they do is they do give me metrics and traceability. I don't know that I can extract that from a PR.
14:48Izar Tarandach Guys, I think that we are misreading the article.
14:51Chris Romeo That's probably not the first time it's happened. So do tell.
14:56Izar Tarandach I think that what he's saying is we have to do all the things that we have been talking about since we started talking here about testing as early as we can, testing on the desktop, testing in the test environment. Otherwise, there will be testing in production. He's not saying testing in production is the right thing to do. He's saying, unless you do the Shit left. The next thing to happen is testing will happen in production because you didn't test before. Because notice he says, if an automated tool gives you a message that says this has this vulnerability problem, this has this linting problem, then you're going to fix it immediately because you definitely don't want the guy on your team or the girl on your team who's responsible for reviewing this to say, hey, you don't pass that automated test. I'm not going to approve your code to get merged in.
15:43Chris Romeo I don't, I'm not seeing this in the same way you are. Let me read you a quote from the article. This is in the testing and production section that we were just operating in. And by the way, this article was written by Lorraine Lawson, uh, interviewing Larry Meshram. Um, it says, one way to get developers to take security more seriously is to move testing and fixing vulnerabilities to correspond to the pull request, he added. That will mean testing in production, which he contended is more effective and efficient anyway.
16:15Matt Coles Yeah. So that sounds like the right, that sounds like the positive states that he wants, not pre-test before you get to the pull request.
16:23Chris Romeo Yeah. I don't think that's arguing for staying the testing. I don't think he's talking about, he's talking about doing testing in production.
16:31Izar Tarandach I think it got lost in translation in the interview. I say we invite Larry in and get to the bottom of this thing.
16:38Chris Romeo I wish I had the way to some internet device that could immediately connect me with anyone on earth. Bring them into our room to this call to be like, hey, we were just talking about your article.
16:50Izar Tarandach Startup.
16:52Chris Romeo Another startup idea generated by the security table.
16:57Izar Tarandach Hmm. 17328.
17:02Chris Romeo All right. Well then, let's got it. Let's, let's leave. Let's, let's, let's put, uh, push that issue off to the side. What else is here?
17:08Izar Tarandach But that's a Drupal request. Yeah.
17:10Chris Romeo Yeah.
17:10Izar Tarandach What else, what else is here? So now we have the Agile drives development to eat other function.
17:16Chris Romeo Yes.
17:16Izar Tarandach Which I am inclined to agree, but.
17:22Matt Coles Do you say?
17:26Izar Tarandach The long arc of history for software engineering has been that the development team itself takes over more and more responsibility for all of these things that used to be done by security, by specialists. I don't know how long the arc of history is. I'm in this game for 25 years and all I see is developers running away from taking responsibility for security more and more. There's probably a very, very long arc, right?
17:55Matt Coles I mean, well, so what you're talking about, you're talking about not just developers have a role in, in the process. But what, what this article is highlighting is trying to say is developers should own the process.
18:09Izar Tarandach And not only that, they want to own the process.
18:13Chris Romeo That might be where we have some difference of opinion based on what we've seen in reality with our experiences. But I think his, I think his intent is correct here. To say that we want to, to acknowledge the fact that, you know, IT operations is no longer even a thing that developers live within. And also to acknowledge or to say that we want, we would love a world where security is led by developers. We've talked about that in past episodes and we've gone through thought experiments. I remember what would happen if security was gone. Like we just got rid of security and we just said developers are going to do it. So this is the state we want to head towards because we all know the scalability of what we do, right? There's never enough security people to line up with the number of developers and it's, nobody's ever going to solve this problem. So that's, so that, that's a goal. That's a futuristic thing we're looking towards, but it, it hasn't been a reality in, in the environments that I've seen.
19:15Matt Coles And it comes down to resources. It comes down to, uh, accountability and, and getting features out the door because that's what drives revenue, right?
19:27Chris Romeo Yeah.
19:28Matt Coles Whereas security takes time, takes effort, takes money. So it's like quality. It's like the quality problem also, right? And we've talked about this too. Security is a lot like quality. If you're, if you're pressured to get things out the door, maybe quality takes a little bit of a backseat. Well, security would be the same way.
19:50Izar Tarandach But the, the difference for me, and you know that I am a big proponent of security and quality go together. I, I think that's the, the discriminator for me is that quality improves over time, even if you don't do anything, because you're going to make mistakes and you're going to improve on them because those mistakes are like easier to put your finger in. Like, you do an off-by-one, you learn that, hey, you write differently so you don't have an off-by-one, and you don't write it next time. But the quality aspect of security sometimes is so opaque. It's so, it applies to this situation, but not to that situation. Right? That there's not like no, unless you go down to principles, if you're on the surface, if you're, if you're on the top and you're just fixing lines of code without thinking about applying actual security principles, it's not like doing more of it is going to make you better at it. It's just you're dealing with those things that are pure quality of the code, but I just don't see developers running to me saying, please, please, please teach us how to code right. I mean, yes, but no, not all of them. We still have the security champion. We still have those good teams that take time to go and learn from Jim. We have those organizations that go to something like Security Journey and get the whole package and learn and at least go through it.
21:32Chris Romeo Right?
21:33Izar Tarandach But I still don't see developers clamoring for make me a security wizard so that I can be a better programmer. And by the way, I don't need you next week because I already know everything I need to know about security. Plus, I think that code quality, and sorry that I'm going so long, but I think that code quality is like a constant, right? Your code has to be written in a certain way. To be considered quality code, but the security keeps changing. They take away one part of the stack, put another one, that thing is bringing different security problems with itself. And that dynamic, I think, would take way too much investment from a developer to keep on top of, unless we are talking, again, security principles. And I just don't see developers coming in and saying, hey, you know what? I already have a week with 120 hours. I might as well do 140 and use the next 20 just to learn security. Yeah, I just don't see it happening.
22:32Chris Romeo Yeah.
22:32Matt Coles So I, I think that this is, um, the, the, the fundamental issue I have with the way the article is written is that it suggests that security can go away completely, completely as a function. And I think that the answer is no, that's not true. We may want it to, but I, I, I think there's a role for this. So I think you, you brought it up, right? I think there's an evolution of security over time and somebody needs to keep, keep aware of that. How you implement that, how you operationalize that might be, uh, pushing that into development, whether that's through champions or training or just tool enhancements, right? So security as a tools team, right? The same way that quality is. To some extent, a tools team, right? Take institution— take institutional and, and cutting-edge knowledge and build it into the capabilities that developers use so that it becomes, uh, fairly transparent to them, right? Part of their day-to-day operations. I run and, and, you know, obviously with AI and Copilot and things like that, now you can potentially— the developers will have this knowledge directly at their fingertips. Um, make them responsible for the code that they write from that developer who writes something to this morning that they're really proud of to whatever gets integrated that's going to go to production. Take the human out of the loop and put the automation, right? Bake, bake the automate, bake the knowledge into the automation.
24:07Izar Tarandach Steer.
24:11Chris Romeo Advocating for the guardrails, paved roads, pipelines, all of these things.
24:19Matt Coles Yes.
24:19Chris Romeo Keeping the developers on the straight and narrow. Yeah. And not giving them room to swerve off the side of the mountain because there's a steel wall that prevents them.
24:30Matt Coles Oh, I wouldn't go that far. I wouldn't go that far. I, I would say that those guardrails should be should be metal guardrails, not like, uh, if anyone's driven through Connecticut, you still, you still have steel cables as guardrails. Um, those are, those are probably too dangerous, but, but I wouldn't put up Jersey barriers as guardrails.
24:53Izar Tarandach Look, a guardrail at, at the end of the day has to be flexible enough so that you stop people going down the cliff. But if they want to fly, they can, right? Because otherwise we are just stifling innovation. So paved roads, guardrails, secure by design, secure by default. That's where I think that security will always be there by creating those things. But creating a culture that consumes those things as part of development, That's where I think that the next challenge is because Lernals, it's very easy to write lots and lots and lots and lots of documentation. And I think that there's a reason why we don't actually go back and look at how many people actually read it, because it would be depressing. But this comes back like again and again and again in every team that I join. Hey, we have to create the documentation. Yes, we have. But now how do you get people to actually use it? That's the, the culture part that we haven't yet, we, we haven't broken that code yet.
26:05Matt Coles And security has a role in, uh, the article highlights that security sort of has a role in accountability, right? There's a, there's a structure for accountability that has to be put in place so that you don't have, I mean, development organizations know how to manage risk today with quality.
26:27Izar Tarandach They like to think so.
26:28Matt Coles Yeah. With quality. But, but even then we know that there are, need to be checks and balances. You don't have the developer committing code and the developer com— approving the pull request also reporting on its quality. Well, maybe you do, but maybe you shouldn't. And there needs to be a structure so that the powers that be can make informed risk decisions. And I'll take care of all—
26:50Chris Romeo I want to take it. I'm going to cover on the same topic that you were just describing here about the powers that be and whatnot. I'm going to, I'm going to disagree a little bit with something that's said here in the article, right in that first paragraph. No, no, second paragraph, which is a direct quote from Larry. The reason there's no consequences though is because the power dynamic is that you as developers have more power than the security folks. The CISO has been beat up in meetings with his boss and VP of engineering. Where they said security just threw this crap on us and we couldn't ship on time. And then pre— in the previous paragraph, it says there aren't consequences when development ignores security, but often there are consequences for the CISO when they try to play hardball. And that's a maturity problem inside of an organization. 'Cause like, even back in, before I left Cisco, back in 2014 or so, a governance model was put into place and there was, we used to jokingly refer to it as the hammer that John Stewart could drop the hammer on a product team. John Stewart was the SVP of, uh, of security, basically chief security officer. And, uh, but, but they introduced this concept of a stop ship where John Stewart could stop ship any product inside the company. And so It was, it was the mature, it was a governance and maturity move after spending years investing in CSDL, Cisco Security Development Lifecycle, building the processes, having all the things, the tools, the documentation, everything people needed for success. And the crown achievement coming out of that was there was a governance stop ship mechanism. That's back in 2014, this was happening. That's 10 years ago now. So my, so I'm, I'm gonna, I'm gonna take exception to that and say, I don't think that's the case in, in mature organizations, what, what Larry's describing here. Sure. In new start and small startups, maybe even mid-sized startups, there's gonna be a governance. You're not gonna have a governance function in ProdSec, AppSec, but in mature companies, you better have governance at this point. If you don't, put that in your K-1, you better disclose that thing in your K-1 so I can sell my stock if I have any in your company.
29:01Izar Tarandach But you know what? I think that sometimes it's not really a, a matter of, uh, uh, maturity, I'll go back and say it's a matter of culture. Because, and especially, you know, with the market pressing down on people so much to put things out there, put flashy stuff out there as quick as they can, stopping something for a security person, not a developer, but for a security person to build a case that's strong enough to stop shipment That thing has to be ironclad and it's going to be like questioned by all levels of the business up to the person who actually comes down and says, yeah, let's stop it. And I don't think that security people, many security people nowadays feel strong enough to put such a case on the table. And many times—
29:54Matt Coles You're also not encouraged to do so, by the way.
29:56Izar Tarandach Yes. And many times we don't have the language of risk. To be able to translate the thing that's for us in our heads is just like so freaking clear. Something's going to explode here. But as you move the, the, the, the peg through the holes, that risk gets attenuated more and more and more and more until it gets to the function that actually decides stopping shipping. And they say, no, there's no risk here. Ship it.
30:24Chris Romeo And this is something that I thought the market was gonna correct. For the last decade or two, I've been watching the world of security and how security plays out inside of companies that build things. And I just kept thinking to myself, the market is gonna correct this. We are gonna reach a flashpoint where consumers, meaning businesses and B2B environments, enterprises are gonna say, I'm not buying this crap if it's insecure. And we just haven't seen that happen yet. And I'm shocked that it's, it's taken this long and there hasn't been a flashpoint point. We had the data breach decades. We've had, you know, extreme malware Greek decade here in the last decade. Like, it's not like things are quiet and like, oh, maybe we don't need to do security. Like, it's, the risk is there in front of us. The threat landscape is larger than it was, but the market just never caught up.
31:22Matt Coles And we see this, we see this coming in with, with now with, uh, government, especially, you know, the CISA secure attestation and the recent SEC action, you know, against the company for, you know, in their case for falsified statements on their disclosure forms, right? That they don't meet, they didn't meet their security posture and therefore they have unexplained risks. Um, but that's not what you're talking about here, right? That being governments have, governments that buy things have always had a rigorous assurance process.
32:04Chris Romeo Yes.
32:05Matt Coles In terms of deterrence.
32:07Chris Romeo But governments, while they are the largest customer of most companies, they don't represent, like, I'm talking about the market, like, I'm talking about, I thought other enterprises were going to force security and stop buying insecure things, but I just haven't seen it happen. And that's, that's what I'm shocked about.
32:28Izar Tarandach I think that it's because at some point we came to a, a feeling of learned helplessness. We, I know that a lot of people are at the point of breaches will happen. It's a fact of life. I'll just lock my credit report and hope for the best.
32:47Chris Romeo And then cyber insurance had an impact on the institutional data breach because you could just buy insurance to a certain level to protect you. Now, it turns out collecting on that insurance isn't as easy as it sounds in the paperwork, but that there is some risk mitigation happening there.
33:07Izar Tarandach And if it was mandated security insurance, I don't think it is. I, I, I'm not very current on the details, but I don't think that it is yet. I think that it's still optional. Then the powers of the market might influence that through the, okay, you have to buy insurance, so let's do something that will lower the risk. But, uh, I keep going back to that thing that at some point we used as security professionals, we used to scare the business into doing what we needed by saying, hey, imagine if there is a breach, the hits that you're going to get in your in your stock, right? But turns out that there is no hit in the stock. It's a little bump and then it goes back right to where it was. Again, showing help to, uh, learn the helplessness.
33:52Matt Coles Yeah. Like that there's, that there's no, that there's a, what, so my question then is what, what does a business, what does a development organization, what does an engineering team or product team care about, like what drives the needle for them. So quality defects, you get customer complaints, they drive, you know, customer support calls. Ultimately, some portion of customers stop buying. That doesn't improve— does that improve quality enough? And does that, does that work for security?
34:31Izar Tarandach So I'm going to throw something on the table and please feel free to disagree with me, but This is what I experienced. Life got way more complicated for developers when the center of gravity of development moved from the architect to the product manager. The product manager speaks a language of A/B testing and feature promises to customers. Drives the bottom line, because that's what you're selling. And then what the product manager puts on the line, or sorry, on the, on the table is what the developer has to pick up and somehow make work. And the race here is not to create the best possible, most high-quality product, which would be if a developer as an artist would be thinking about crafting the best thing that they can and putting it in front of the customer and enjoying the customer enjoying that. That's creation, but it becomes just this, this blob on the table with all sorts of shiny things without anything behind it that some product manager, for some reason that Lord knows what it is, decided that is what has to go in front of the customer, right? The, the, the wow factor became way more influential in what we get to experience as customers. Than quality and functionality?
36:07Chris Romeo I mean, product managers are just doing what they have to do to serve the needs of the customer. The best product managers are listening to the customers, are listening to the market, and they're building things to meet those requirements. Specifically. So I'm not gonna put, and yeah, I get in the old days we used to have architects that would drive where the products were going, but this is a new age. This is a new era. Like it's, it's about features and it's about speed to market. It's about, I think, wow.
36:41Matt Coles Yeah. I think that's the important part. I think that's the important part. It's, it's the product manager is defining features. The product manager is not defining the how should all these features come together and work and is there a unified vision and is it supportable and manageable and, and meet certain It hits me.
36:57Chris Romeo I would argue that they do. I would argue that a product manager has a bigger— now, maybe in a giant company, in a giant product, there's 10 product managers that are working on one product. But I would argue that they do have a unified vision. The challenge has been they've had no motivation to push security or privacy from where they sit.
37:16Izar Tarandach Exactly. Exactly. Because their motivation is in making that big customer that sits in the advisory board satisfied, right? More than creating a continuous line in the product development that makes sense. I'll give you this feature and then I'm going to amplify with this feature and then I'm going to amplify this feature. No, right now is I'm going to put this thing in front of you because somebody in the cab told me that they want it and they're paying us a lot of money. It's not going to work perfect. but eventually it'll grow into something. And then you end up with like the, a tree with partial branches. It's a huge tree. You look from a distance, oh my God, the thing is green and luscious and beautiful. And as you get close to it, each one of the branches is broken. And at the end of the day, the functionality is just not there, or it's just not giving you what you want.
38:12Matt Coles Let's bring this back to the article a bit. So let me pose the question to you, uh, because, so I, I think I agree with you, but I'm, I'm gonna pose the question to you, Azar and, and Chris. Um, let's say for a moment the developers take on security. First off, does this solve the— does this help solve the problem that you've raised, that the product manager is working towards a particular agenda that developers who care about their code may, may not see? Uh, and alternatively, one of the challenges I think I've seen over the years is a, is a question of air cover. Developers want to do great things. That takes time, which means they're not working on, you know, velocity and, and feature development. They're shoring up the things that they built, not building more stuff. And so they need air cover to do that. So, okay. Uh, I guess based on what you're describing and what Larry is suggesting, how does this work? How do developers take on security? Given they're not getting direction or air cover to take on security.
39:20Izar Tarandach That's the thing. They only—
39:21Chris Romeo but it's automated. That's, I mean, his, he's, he's talking about a DevSecOps framework where security is, you're dealing with—
39:29Izar Tarandach where security happens.
39:30Chris Romeo Yeah. But you're not, you're not, you're not, developers are not now the security department. It's the security department has wired up the pipelines in such a way that the developers can then respond to security problems automatically generated and attached to their PR in the way that, that Larry was describing.
39:51Izar Tarandach And that by definition is a snapshot operation because you give those tools to the stack that you're working with now. Tomorrow the stack changes. If you don't have the security people to change the tools and to change the, the, the processes, the developers are not going to do it.
40:09Matt Coles It is a very reactive process, what, though, what you're describing. So let's talk about something that came up in the Secure by Design papers, you know, from, from CISA, for instance, around using memory-safe languages, right? That's not a, that's not an automated tool, you know, from a PR to an automated tool finding solution. That is a, I'm gonna refactor my code in a language that solves these problems for me.
40:33Chris Romeo Yeah. And that's an architecture problem. Right? Like, it's a design decision that needs to be either done upfront if you're starting something brand new, or it needs to be retroactively applied now. It's going to be a challenge to change to a memory-safe language from an unsafe language if the application's already in flight. It's just, it's just the reality of it. Could you imagine telling a team, hey, we're going to refactor the entire application? We're gonna migrate from, uh, PHP to Go. And we're gonna, yeah, I mean, it's, it's not, I mean, it's a, at that point you might as well just build a new product. You might as well just end of life the thing that you were, you know, start working on a new product and just start from scratch and build something. You're not gonna convert that. So there are some things where the business case just isn't gonna be there. To what you were just describing as far as—
41:28Matt Coles So when does that ever happen? When can that happen? If, if developers want it, you need the air cover to do it, or you need a team to help, right? Is that security's role?
41:40Chris Romeo Security team doesn't solve that problem though. I mean, they, they could provide the stack, the current best of breed stack. If you're starting a new application, start with these pieces today, which that's important. That's something that's not always as easy for every developer perhaps to understand what are the right pieces I should snap together here. So that could be part of security's role. But on the air cover front, I mean, security doesn't really provide air cover for developers. So that's not, so the air cover, it really has to come from the product management because product management needs to be driving what developers are doing and prioritizing security and privacy. In a perfect world for me, that's something that product managers are doing, and then they're providing the air cover. They're stepping up to executives and saying, we're not shipping an insecure product, we're not shipping a product that violates privacy. And so if either of these core things are broken, I'm gonna, I'm gonna direct engineering to do the things that they need to do to fix that. And it just requires a, a new breed of product manager that's very security and privacy savvy.
42:48Matt Coles Or it's a shift in our, in our focus as security people. We've been for the past decade working on developers. Maybe we actually did a good job with developers. Is our new focus product managers?
43:01Chris Romeo I think so.
43:01Izar Tarandach Or, or the function that we are forgetting here and the one that we're supposed to provide the air cover from the beginning is the CISO, which in here he says gets beaten around the head. Where is it? Uh, because the power dynamic is that you as developers have more power than the security folks, so that CISO has been beat up in meetings with his boss and the VP of engineering, where they said security just threw this crap on us and we couldn't ship on time. So there's a number of assumptions happening here that the CISO did give the arc over and the developers were asked to stop what they were doing and do the security thing. And that made things go slower. And now the VP of engineering and the boss are coming and screaming at the CISO, why did you do this? So the air cover was in place. The problem is that somebody brought in the anti-air batteries and is now trying to shoot down your, your, your, your air cover.
43:59Chris Romeo He's describing an environment where executive management is not on board with prioritizing security and privacy. And that's the whole point I was making about, I thought the market was going to correct around that. I thought. Market, I thought business and enterprise consumers were going to start seeing that as a failure and stop buying from companies that had that philosophy. That's part of the challenge though, is that just people continue to buy from people that don't take security and privacy seriously.
44:29Izar Tarandach And I think that one of the biggest problems here is that what's a big, uh, example that we have of some CEO that said, Stop the presses and, um, we are going to do security now.
44:46Matt Coles Yeah, that would be my guess.
44:46Izar Tarandach Have we heard of anybody lately?
44:48Chris Romeo William Gates, 2004, trustworthy company.
44:50Matt Coles And then, and, and then recently in 2024, not Bill. Okay.
44:55Izar Tarandach Yeah. And if we get those exceptions, and then Satya Nadella, the currency.
44:59Chris Romeo Yeah. You had another security moment. Okay.
45:01Matt Coles Yep.
45:01Izar Tarandach And now we go and we put this in front of a CEO expecting him to see the light and say, from now on, we have a security culture here. And they look back at you and say, what do you want from me? I'm not Microsoft.
45:12Chris Romeo Well, no, they look back and say, what's the return on investment for this? If they're smart.
45:17Izar Tarandach That, that's the thing. No, no, no. That's the thing. I can, I can come and say, listen, if Microsoft found it right to do it, then there must be some return of investment. I may not be able to give you numbers. Yeah, but there must be one because Microsoft is doing it.
45:30Chris Romeo It doesn't, that, that, he's going to look at me. That doesn't convince me as an executive. I'm like, okay, they're doing it.
45:34Izar Tarandach That's the thing. He's going to look at me and say, I'm not Microsoft.
45:37Chris Romeo Take a walk and, and figure out what your return on investment proposition is for me, because executives deal in business value, in dollars saved, dollars gained, revenue increases. If you're, if you can't sell your product, your, your new security vision in such a way that it's gonna save me money, it's gonna get me more customers, it's gonna increase my revenue, I'm sorry, but it's not a priority for me.
46:03Izar Tarandach And between developers, security people, and product managers, which one of these three tribes speaks that language?
46:12Chris Romeo Product managers are the closest. Product managers.
46:16Izar Tarandach QED. I rest my case, your honor.
46:17Chris Romeo I think we've solved it.
46:18Izar Tarandach We have solved something on the security table.
46:20Chris Romeo We have a victory. Now industry, listen up. Here's what you're going to need to do. Get your product managers all together at an internal conference. Called—
46:30Izar Tarandach Wait, wait, I need notes. Notes. I need to take notes.
46:32Chris Romeo It's recorded. It's available. You can have AI summarize it for you. Well, I do think we made some progress here. We don't often make progress on the security table. Often we end up back where we started or flummoxed a bit at the end. But this time I feel like we've provided our industry with a new vision. So industry, take it forward. Be successful.
46:53Izar Tarandach Do your thing.
46:54Chris Romeo Send us small stipend check for every month of success that you have. And you can send that to the security table at PO Box 10011001100. And that's Worcester, Massachusetts, 2743124. I don't know, I just made that up. Don't send anything that doesn't exist. Guys, thanks again. This is, uh, was a fun conversation. and we look forward to doing it again in the future on the Security Today.
Transcript supplied by AssemblyAI. It may contain transcription errors.
Originally published as The Security Table. Part of the AI Security Table archive.
