# Is it Necessary? Not everything requires an LLM

URL: https://securitytable.ai/is-it-necessary-not-everything-requires-an-llm/
Published: 2024-12-10
Duration: 2559 seconds
Season: 2
Episode: 33
Hosts: Chris Romeo, Izar Tarandach, Matt Coles

## Show notes and chapters

We debate the necessity and efficiency of LLMs in finding code vulnerabilities in a C library compared to traditional static code analyzers and fuzzing techniques. The conversation explores broader topics in application security testing, including the evolving landscape of Dynamic Application Security Testing (DAST), fuzzing, and the potential of emerging technologies like Application Detection and Response (ADR).

Follow AI Security Table:

➜ Home: [https://securitytable.ai/](https://securitytable.ai/)

➜ X: [https://x.com/AISecurityTable](https://x.com/AISecurityTable)

➜ LinkedIn: [https://www.linkedin.com/company/ai-security-table/](https://www.linkedin.com/company/ai-security-table/)

➜ YouTube: [https://www.youtube.com/@AISecurityTable](https://www.youtube.com/@AISecurityTable)

## Transcript

**Chris Romeo (0:09)**: Hey folks, welcome to another episode of Security Table where Yezar has lost his microphone and, uh, both earbuds or just one there, Yezar? No, he's got the earbuds. He's lost the microphone. I don't know. You can't find it. It's missing. Which should make for—

**Izar Tarandach (0:26)**: Nobody moves. Nobody moves.

**Chris Romeo (0:28)**: Wait, we're all— we're not moving. We're stationary. But yet we live far away.

**Izar Tarandach (0:34)**: The two of you? Yeah, both.

**Chris Romeo (0:37)**: And we have— we do our own stunts and our own audio video here.

**Matt Coles (0:43)**: Oh.

**Izar Tarandach (0:44)**: When you—

**Chris Romeo (0:44)**: if you think this is scripted like the Mike Tyson-Jake Paul fight, it's not.

**Izar Tarandach (0:49)**: Don't.

**Matt Coles (0:49)**: Don't.

**Izar Tarandach (0:50)**: Why did you do this?

**Matt Coles (0:53)**: Okay, that's gonna drive, that could drive paychecks right off the bat because you're gonna put that in the—

**Izar Tarandach (0:57)**: 2 days trying to forget that. 2 days trying to forget.

**Chris Romeo (0:59)**: Did you watch it?

**Izar Tarandach (1:01)**: Of course, from where I watched it.

**Matt Coles (1:03)**: I didn't watch it. Did you?

**Izar Tarandach (1:05)**: No, I don't put money into things like that, but it was my one opportunity to finally say, you see, you see, a 50-something guy can do something.

**Chris Romeo (1:13)**: No, he can't.

**Izar Tarandach (1:14)**: No, even if it's Mike Tyson.

**Chris Romeo (1:16)**: So the, the internet trolls are claiming that it's It was scripted.

**Matt Coles (1:23)**: That it was staged?

**Chris Romeo (1:24)**: Yes, it was staged.

**Izar Tarandach (1:26)**: You know, if it was scripted in Bash, I wouldn't have a problem with, but it wasn't.

**Chris Romeo (1:32)**: Yeah, but how many regular expressions would they have in that bash script?

**Izar Tarandach (1:35)**: No, no, there was absolutely no bashing whatsoever.

**Matt Coles (1:37)**: I have a feeling it was Rusty, wasn't it?

**Chris Romeo (1:40)**: Yeah.

**Izar Tarandach (1:41)**: It was Rust.

**Chris Romeo (1:41)**: It was Rust. And they were using the biggest gloves possible. As well. There was some— well, there was controversy about that like a year ago because the bigger the glove, the less damage— it's like a more of a pillow— the less damage it does when you get hit by it.

**Matt Coles (1:58)**: Were these like those image— those generated big gloves like they have on like TikTok or something like that?

**Izar Tarandach (2:03)**: I'm telling you, I didn't know if it was— okay, first of all, I'm scared to know what they have on TikTok, but I didn't know if I was watching a fight or if it was my, uh, Nintendo Wii turned on or something. Because it was basically like, what's the name of that, uh, thing with the two little puppies that suck each other?

**Chris Romeo (2:26)**: Oh, Rock 'Em Sock 'Em Robots.

**Matt Coles (2:28)**: That's what I thought it was.

**Chris Romeo (2:29)**: Yeah. Now, I mean, the best boxing game of all time was Mike Tyson's Punch-Out!!

**Matt Coles (2:35)**: Yeah.

**Chris Romeo (2:35)**: On Nintendo.

**Izar Tarandach (2:36)**: Which platform was that? Sega?

**Chris Romeo (2:37)**: I think it was Nintendo.

**Matt Coles (2:38)**: Was it NES?

**Chris Romeo (2:40)**: Was it the original Nintendo? I think so.

**Matt Coles (2:43)**: If I remember correctly.

**Izar Tarandach (2:44)**: I don't know, I'm an Atari boy. I dream in blocks.

**Matt Coles (2:50)**: There's nothing like playing Asteroids in the ray trace, uh, you know, if it's simple geometry.

**Chris Romeo (2:56)**: Did you have Pong?

**Izar Tarandach (2:57)**: Did you have a 2600?

**Matt Coles (2:58)**: I had a 2600, but I also had Pong before that.

**Chris Romeo (3:01)**: Pong was a separate thing.

**Matt Coles (3:05)**: Pong was, well, with sliders originally. It had sliders, not even— Yeah.

**Izar Tarandach (3:09)**: The Philips Palm, yeah.

**Matt Coles (3:10)**: All the Play was Palm.

**Chris Romeo (3:12)**: Okay.

**Matt Coles (3:12)**: It had two controllers that were wired and they were sliders and you would slide the thing up and down. I was on the—

**Izar Tarandach (3:19)**: that was a different era. And it was great because if you had to play by yourself, because Lord knows some kids might think it wasn't cool, you could like drop a switch and then both paddles would move at the same time. So you would have like just one slider to move both of them.

**Matt Coles (3:34)**: Oh, I didn't know about that switch. I just played with two hands.

**Izar Tarandach (3:39)**: That explains a lot.

**Matt Coles (3:44)**: I wish it made me ambidextrous as a result, but it didn't.

**Chris Romeo (3:46)**: So, you know, you could only pitch left-handed.

**Matt Coles (3:49)**: One side was always better than the other. So I couldn't, you know, I always lost to myself.

**Chris Romeo (3:56)**: That's really sad that you lost to yourself.

**Matt Coles (3:59)**: Well, I was a sheltered kid as a child.

**Izar Tarandach (4:01)**: But on the other hand, on the other hand, he won as well. So not that bad.

**Matt Coles (4:05)**: I had a 50-50 win. Yeah.

**Chris Romeo (4:08)**: I mean, it's tough to bet on that because it's, you know, 50-50.

**Izar Tarandach (4:11)**: As long as you don't go Jacko and hide on it.

**Chris Romeo (4:13)**: Oh, I lost. I lost.

**Izar Tarandach (4:16)**: So you're fine. You're fine.

**Chris Romeo (4:18)**: Created a split personality scenario. So there's got to be something in the world of cybersecurity that we could talk about other than the Mike Tyson fight and Matt's childhood and Izar dropping a microphone.

**Izar Tarandach (4:34)**: Let's talk about lists.

**Matt Coles (4:36)**: No. Okay. Let's talk about what is necessary in this world.

**Chris Romeo (4:43)**: What is necessary?

**Matt Coles (4:44)**: What is necessary?

**Chris Romeo (4:45)**: What are we getting philosophical?

**Matt Coles (4:46)**: We are getting philosophical.

**Izar Tarandach (4:48)**: I thought about what's necessary.

**Chris Romeo (4:51)**: Necessary.

**Matt Coles (4:53)**: Wait, is that like Is that like Neil Patrick Harris saying something like, it's necessary?

**Izar Tarandach (4:57)**: That's legendary.

**Chris Romeo (4:59)**: Wait for it. Sorry. Wait for it. And I'm lactose intolerant.

**Matt Coles (5:03)**: Dairy. Okay. Excuse me, bro.

**Chris Romeo (5:09)**: That is a classic. If you think about like across all the different TV shows where somebody pulls the same bit over and over again, it's gotta be the best one because it was always witty.

**Matt Coles (5:23)**: Honestly, it is most definitely the best one. There are tons out there, you know, Fire, Fire, or Deadman's Genius, but, uh, but definitely Legend. Wait for it.

**Izar Tarandach (5:35)**: Wait for it.

**Chris Romeo (5:36)**: But there was always a little, an additional set, half a sentence that was original that made you laugh about it. So yeah, Barney Stinson.

**Matt Coles (5:44)**: I bet it was all ad-libbed. It was all improv.

**Chris Romeo (5:47)**: Thanks, sir.

**Matt Coles (5:49)**: I imagine so. He's that kind of actor.

**Chris Romeo (5:51)**: He's pretty creative. He's a creative guy when it comes to, you know, creating lines, just like us on the Security Table. I mean, none of this is scripted, folks. This is all ad-lib from top to bottom, if you can't tell. Okay, I'm getting—

**Matt Coles (6:05)**: So what I meant by— what I meant, actually, I was going to tee up our topic for tonight with what is necessary. So we were pointed to this article, what is it called? "Naptime to Big Sleep: An Analysis of Code Vulnerabilities Using Large Language Models." And it was pointed against, in this example, it was pointed against a C, a library written in C. Do you really need an LLM to find coding problems like a buffer overflow. Is it really over— is that overkill? Or should there have been other ways to find that vulnerability? And could that vulnerability have been found using a more tried and true method, like, say, a reasonable static code analyzer that knows how to look at memory calls? Or a memory profiler that they've been running for 30 or plus years, or should have been running for 30 plus years. Now, granted, this is an open source project. We don't know. I know I don't know personally how well-funded it is. And this was research done under Google's Project Zero. But seriously, do you need an LLM to do this? It seems like overkill.

**Chris Romeo (7:26)**: I did see some interesting write-ups, how Google's using LLMs for fuzzing now to create derivatives and whatnot in the world of fuzzing, right? Because your fuzzer's as good as the, amount you can exercise the interface to reach close to 100% as possible of all possible input, right, that could go into it, or permutations of input, not necessarily every, you know, every—

**Matt Coles (7:55)**: And that makes a lot of sense, by the way. That, I would say, is a valid use for an LLM, right? Because a fuzzer is supposed to be testing a bunch of inputs and you want it to test, you know, a bunch, meaning billions, thousands, thousands, hundreds, hundreds of thousands or millions or billions of inputs. And you probably need a good generator, an LLM, to come up with some interesting unique cases that a human or a procedure can't. So that may make sense. But a buffer overflow?

**Izar Tarandach (8:28)**: So here's the thing. The thing with fuzzing is that at the end of the day, it's Do the gorilla test, right? Banging on the keyboard and throwing garbage at something and expecting bad things to happen. But when you're talking about something like the target that they have here is 2 Lite, and it's an interpreter expecting SQL, could just throw garbage at it. It's probably going to be dropped or refused way before it hit the place where it could do something. So where an LLM could help is in building those specific SQL valid inputs that will pass the parser and reach the deeper parts of the target here, right? So my point, I guess, is, are we using the LLM instead of the fuzzer? Are we marrying the LLM with the fuzzer? Are we using this new technology to make the old technology work better? I guess that that's the, as always, the golden middle to reach, right? The point of, how do we have to put LLMs in just everything now? It's exhausting.

**Matt Coles (9:43)**: Well, in fact, just looking at the, so the Project Zero write-up here has an excellent write-up about where they found the vulnerability, where the vulnerability exists and what, how it presents itself. And one might wonder as you're looking through it, why does this even exist? Why? Ignore for a moment, do you need an LLM to find it? There's just tricks of the language, not tricks of the language, but the specification and what SQLite looked like they were doing should never have been. Somebody should have thought better about this before. So design review. Code review, code analysis, fuzzing, everything before, before an LLM came along. Why was this never discovered before? Why does this code construct exist? Do you really need an LLM to tell us about a bad design or a bad programming construct? Could a coding standard have fixed this? Could a good design standard have fixed this? Could guardrails have fixed this?

**Izar Tarandach (10:51)**: Could a developer having 10 more minutes to think about what they're writing have fixed this? I think that at the end of the day, it's— And there is a reason why we have a cycle of tools and procedures and processes. We want one thing to cover the next. It's defensive depth.

**Matt Coles (11:09)**: And it's interesting too. There's a particular note in here, right? It's sort of buried in the write-ups. That there's an assertion looking for the error condition, which of course gets compiled out during runtime, during production, right, for release build. They knew they were looking for something, which meant they knew there could be a problem.

**Izar Tarandach (11:32)**: Wait, go, let's go back to that for a sec. So the assertion is around the bit of code that actually was found to be vulnerable.

**Matt Coles (11:42)**: Uh, let's see. So there was an assertion at, so just looking at the code, there's an assertion at line 706 looking for a column. So the issue is a column identifier could be negative resulting in, um, in a buffer stack buffer overrun or under, I guess it would be underflow.

**Izar Tarandach (12:02)**: Underflow, yeah.

**Matt Coles (12:03)**: Uh, and so there's an assertion looking for making sure that the indexes that are used are positive. Meaning non-negative, so 0 or more. And so it's when it's a negative 1, apparently that's when the issue occurs. And the issue, and the negative 1 is a valid, is a valid value for another use case, apparently. So, so it's a design, so there's a design fault here at the very end. But they were looking for, specifically looking for when the index was negative 1. And it would've asserted, which means there was no test case or there was a faulty test case. If that, that's one possibility that could've found this well before having the LLM look at. But I guess then that this brings the other question, is this code so complex? So this func, this particular function starts on line 619 and ends on 720. So it's 100 lines long. Is that too much to do a manual code review?

**Izar Tarandach (13:09)**: This is probably what we would call Brito code. It may be, it may make perfect sense when you're writing it, when you're deep into it. When you take that step back and you look at it, you start seeing the cracks and the places where things could could break. Now, from a quick look here, the line before that, the one that's actually creating the variable that can go bad, it's a pointer value minus something that's all in caps. So I'm assuming that it's a constant.

**Matt Coles (13:46)**: Top. Yeah.

**Izar Tarandach (13:48)**: So, I mean, would it hurt too much to check if the constant is bigger than the thing that it's going to be taken out of? But again, it's Monday morning. We are literally Monday morning quarterbacking here. Yeah.

**Chris Romeo (14:07)**: And that's a static analysis play, right? What, that is like checking if the constant is somehow bigger than the data that it's going into is a static analysis move, not really a fuzzing move. And so I guess is the question, can LLMs exceed and replace static analysis? I would say the, regardless of what the particular article is we're looking at, I've seen a couple of different articles and a conference talk on this that the answer is no, they're not there yet to be able to detect at the same rate. One of the guys from Carnegie Mellon did a talk on, he used ChatGPT and he used the CWE examples. So, you know, all those little snippets of code that are in the CWE and ChatGPT was terrible at finding, no, 'cause he took the known vulnerable code and fed that into ChatGPT and it was terrible at finding those events. And those tend to be pretty simple. They're examples, right? They're right in front of your eyes. It's not like a complicated, scenario that created it.

**Matt Coles (15:18)**: And by the way, that variable is 1. It's pound sign defined to 1. So just so you know.

**Chris Romeo (15:24)**: What?

**Izar Tarandach (15:24)**: Series column sort is 1?

**Matt Coles (15:26)**: It's 1, yeah.

**Izar Tarandach (15:26)**: Oh gosh. But the 2 points that I would like to make here, one of them is about the, oh, the LLM can't do that. Why are we expecting the LLM to be able to do that in the first place? There is still this, anthropomorphism of the LLM where you expect it to behave like a person just because you can talk to it like a person.

**Matt Coles (15:52)**: Anthropomorphism.

**Izar Tarandach (15:53)**: It doesn't happen.

**Matt Coles (15:54)**: Anthropomorphism. Again, I like USA.

**Chris Romeo (15:56)**: You see?

**Izar Tarandach (15:56)**: Not even you can do it, yeah?

**Chris Romeo (15:58)**: Anthropomorphization. Oh. No, I missed something. Anthropomorphism. For crying out loud.

**Izar Tarandach (16:07)**: I am, I am. So the point is we seem to have been conditioned to think that these things can do a lot of— sorry, not to think, to expect these things to be able to do a lot of stuff just because of the fact that we relate to them as humans. Now, LLMs, most of them, the way they are, the way that we interact with them as chatbots and all that, have limited, if any, memory of what they're doing, limited or any, and understanding of the context that they are living in when we are talking to them. And yet we expect them to, and that's when they hallucinate and we are ready to receive that because it makes sense in that context. And us not thinking, hey, you know, this thing's just fitting in the blanks. It's just a big ad-lib, right? And we lose sense that we are relating to basically a random word generator with a bunch of weights added in and we expect it to make sense and then it doesn't make sense and then we get surprised. How come the LLM can't do this? How come it can't count the number of Rs in the word strawberry? That's so surprising. It's supposed to do everything, right? And then there's the other thing, which is we said, oh, but a fuzzer didn't get this and I was expecting the LLM to. Or the static analysis too. And I'm not, okay, this is going to sort of contradict something that I said on LinkedIn a couple of days ago. I'm not going to say that fuzzing is dead because it's not, it's not, it's not, it's not. I'm not saying that it's not.

**Chris Romeo (17:47)**: Did you really say that on LinkedIn?

**Izar Tarandach (17:48)**: I did not. I did not. No, I said that people should take the pulse of something before they declare it dead. It dead. But the point is that we keep banging on DAST, saying that DAST is dead, right?

**Matt Coles (18:04)**: Why?

**Izar Tarandach (18:04)**: Because DAST, because it's doing a bunch of things at the end of the, and let's not go left or right or anything political like that. It's just doing it after the thing has been written, right? And fuzzing to me is—

**Matt Coles (18:17)**: It's reactive. Let's start using those words. It's reactive versus proactive.

**Izar Tarandach (18:20)**: Exactly. And fuzzing to me is like, it's close to that, right? Because now I am throwing something at the end of the chain and I'm budgeting it a certain amount of CPU hours to work. And after that budget is expended, I'm saying I fuzzed it and my thing survived. My thing is good.

**Matt Coles (18:42)**: I think you're 100%.

**Izar Tarandach (18:43)**: And we don't know if the 1 hour plus 1 is the one who will trigger the case.

**Chris Romeo (18:46)**: I think you're 100% wrong.

**Matt Coles (18:48)**: I think he's actually 98% right, but I have a different flavor.

**Chris Romeo (18:54)**: Okay, but the reason I say you're wrong is that the DAST is dead. So DAST versus fuzzing provides a very different outcome and return on investment today. Fuzzing is still, yes, you could say theoretical unless we went into a laboratory and tested two different things and had a control group and everything. But what I find in modern, at least in tools that are called DAST, the problem I have with them is they're scanning for things from 10 years ago. Now you could tell me that DAST is now an API scanner and it's moved forward and maybe we could pull that thread. But when I think of fuzzing, fuzzing is still a raw type of tool like Codenomicon's Protocol Fuzzers were some of the coolest things on earth, right? Was the first? Yeah, yeah, Defense. I don't even know if it exists anymore. It's Synopsys too. Black Duck.

**Matt Coles (19:47)**: I think it does. It's still not, well, actually, that's a good question. It was Synopsys, whether it is still.

**Chris Romeo (19:52)**: It's Black, they renamed it Black Duck now. But my point is like, theoretically, fuzzing can provide a meaningful outcome. It can, you can test something based on some profile of input and get some type of feedback as to whether there is a vulnerability. The challenge I've continued to have with DAST is I don't feel like for most, a lot of DASTs have been left behind in that they're still in that 10-year-ago cycle of what they're looking for.

**Matt Coles (20:22)**: So, yeah, so let me, let me, let me just, I agree with you.

**Chris Romeo (20:26)**: I agree with you 100%.

**Matt Coles (20:28)**: I agree with you 100%. Let me, let me drop a, drop a couple of thoughts here. DAST has always been an API scanner, right? A web application scanner. Looks at web APIs. We never called them web APIs before, although they were API calls. They were HTTP API calls.

**Chris Romeo (20:47)**: Well, every API—

**Matt Coles (20:48)**: every HTTP call wasn't—

**Chris Romeo (20:50)**: Doesn't API— I mean, it's a small point, but doesn't API exist after the web application was broken into two? Remember, web apps used to be front and backend all in one package.

**Matt Coles (21:04)**: That is a— That is an overuse of the, of the term API. Calling a library function is an API. Calling a web app— web interface is an API. An application programmer interface is a point, a point to where a machine talks to a machine or process talks to a process. Or a human using a web browser, the web browser talks to that process. So every HTTP object, every HTTP resource is an API. Every call to a library is an API. I guess if that's the— But now we call this other thing an API too.

**Chris Romeo (21:40)**: The, I mean, if you could, yes, the textbook definition, I'm talking about how people used it. Like I didn't, we didn't refer to API in the early days of web applications. It was just requests and responses. You ran the front end and back end was all cooking together and you were getting—

**Matt Coles (21:54)**: So it takes on, it takes on new meaning now, right? What an API is, that it's for us an architecture and whatnot, but they technically were all APIs to begin with.

**Chris Romeo (22:03)**: Okay.

**Matt Coles (22:04)**: And on top of that, I agree with you because DAST, when it's, when DAST is a dumb rule-based analyzer, it can have a tendency to get out of date, right? It can be stuck in the past with old constructs. Like, this is what's, this is what cross-site scripting looked like 10 years ago and it's never changed since. So, we're still looking for that, right? So, yeah, it can get stale, but, so let me ask you the other question. Is fuzzing not a DAST tool? It is a dynamic application security testing tool. Well, I mean, fuzzing is not dead, therefore DAST can't be dead.

**Chris Romeo (22:40)**: You could, you could say anything's a dynamic application security testing tool.

**Matt Coles (22:44)**: No, no, no. Static analysis is not a dynamic SCA.

**Chris Romeo (22:47)**: SCA is technically a dynamic application security testing tool. It's very dynamic.

**Matt Coles (22:53)**: No, it's not. No, it is dynamic. Software composition analysis.

**Chris Romeo (22:56)**: What is dynamic? If you want to break down to what the rules mean, what does dynamic mean?

**Matt Coles (23:01)**: Dynamic means it has a way, it has the ability to adapt to the application runtime state.

**Izar Tarandach (23:06)**: And it's exercising the code.

**Chris Romeo (23:08)**: Not that SCA adapts to the runtime state. It just happens to not be running when it scans.

**Matt Coles (23:13)**: Nope.

**Izar Tarandach (23:13)**: Nope.

**Chris Romeo (23:13)**: That's the runtime state. It's not running.

**Matt Coles (23:15)**: SCA is entirely a static analysis tool.

**Chris Romeo (23:17)**: Everything's DAST.

**Izar Tarandach (23:19)**: But the thing to me that, the thing to me that, that, that means—

**Chris Romeo (23:24)**: I'm just, I'm just being difficult now. This is purely for entertainment purposes.

**Izar Tarandach (23:29)**: Actually, what you're saying is everything is dead or soon will be. But on a less—

**Chris Romeo (23:33)**: Well, that is true because I mean everything—

**Izar Tarandach (23:37)**: On a less depressing note, sort of, my thing with fuzzing and DAST is that perhaps on a subjective thing, after you finish running them, you feel fuzzy inside because you say, I spent this many hours running these things. But you don't really have a promise that something was achieved. You know that for all these number of cases that were tested, nothing bad happened, but you don't know if the n+1 case is the one that's going to explode the whole thing.

**Matt Coles (24:12)**: Sure.

**Izar Tarandach (24:12)**: Static code analysis, on the other hand, you have this expectation or illusion that every single possible code path was explored.

**Chris Romeo (24:24)**: Yeah, but it's all grep. It's all grep-based searches.

**Izar Tarandach (24:28)**: No, there is grep, but there are others who could do other, I don't, I won't say smarter, but I will say more powerful, more thoroughly, thank you, approaches that do offer a bit more of a measure of yes, every single possible code path was explored.

**Matt Coles (24:44)**: Right. But those, those get lost in the shuffle, especially in a, in a modern DevOps CI/CD pipeline because you want the pipeline to finish quickly. —and a lot of that analysis takes a lot of time. Yeah.

**Izar Tarandach (24:56)**: And on the other hand, if you do that PR time, you lose the whole context of the thing. So you are just checking atomically and you say, oh, this new piece of code is fine, but you miss the whole context. Yeah. So people go and say, okay, let's do the local and then every once in a while, let's do the big one.

**Chris Romeo (25:15)**: So let me address Matt's question. This is DAS fuzzing because I've been pondering that. Is fuzzing DAST? I would say, or is fuzzing DAST? According to the textbook definition, yes. But from an industry perspective, we don't consider a fuzzing tool to be, we don't call a DAST, we don't call a fuzzing tool a DAST. They are unique categories in the modern way that we talk about it, whether the definition lines up or not.

**Izar Tarandach (25:45)**: They live in different quadrants.

**Chris Romeo (25:47)**: Is what he's trying to say. Ah, yes. Yeah. 100%.

**Matt Coles (25:51)**: Let's be careful of that. There's the technical definition. It is a DAST tool, but it is not in the Gartner quadrant for DAST. Yes.

**Chris Romeo (26:02)**: And so therefore, I don't think there's a fuzzing— if there ever was a Gartner magic quadrant for fuzzing, it doesn't exist anymore. Yeah, I don't think so.

**Izar Tarandach (26:11)**: Give it 5 minutes. Nothing. It just created it.

**Matt Coles (26:16)**: So, yeah, so, yeah, so, I mean, time is the enemy of completeness or thoroughness or validity, you know, this sort of, and even fuzzing to that, to some extent, right? I mean, and on top of that, where do you get the best value out of fuzzing? You get value out of fuzzing using the pre-production, you know, non-release code, which has the assertions and other things built in. And all the instrumentation, or does it benefit the production code more? And one might argue, you need the instrumentation to get the better value out of the fuzzer, but you really want to be testing the production code because all of your guardrails are missing.

**Chris Romeo (26:57)**: Yeah. Another philosophical dilemma. There's not a, there's not a, I mean, you can argue both sides of it. I could argue both of them depending on, you could just tell me to choose one and then I can, I would make an argument for why it's better to have the assertions, have the traceability that any instrumentation might provide, à la IAST, maybe IAST and fuzzing together.

**Matt Coles (27:19)**: I was going to bring up IAST-Nest to see how you felt about that, but is that a—

**Chris Romeo (27:23)**: No, I think, I mean, I think I've been a big fan of IAST and RASP since it came out. I mean, and I think I probably told this story before, but I don't care. I tell the same stories all over and over again. Like I was a, um, one of the very first customers of Screen when we were at Security Journey because I saw it and I was like, I was looking at another vendor on the market and they were like $50 grand or something. I'm like, I ain't paying $50 grand for a RASP license. And so I found Screen and it was like $200 a month. That was like $100 a month to start. So I was like, turns out I didn't know it, but I was like their third or fourth customer. And I had no idea. They didn't tell me. And so I was on this, like, as their pricing went up, I was on this, like, super discounted $100 a month plan. And finally, the salesperson set up a call with me 2 years later and said, I'm begging you, can you please upgrade off this plan? You're the only person on it in our entire company. But it was brilliant because they had a number of different libraries for web frameworks, and then they just happened to support Ruby on Rails, which is what we were using. We plugged Screen in, SQL injection, cross-site scripting. You ran tools at it and it just dropped it. It was, it didn't get to the, it didn't get to the point where it executed. So I've been a big fan of those tools since the beginning, just because I don't think they're the only thing. I think the biggest fault with them is people are going to get a false sense of security and say, I've got a RASP. I don't need to worry about secure coding principles. I can just write crap and then the RASP will, will tell me when it hits. But other than that, I had really good success and really good results.

**Matt Coles (28:57)**: Doesn't RASP introduce performance problems?

**Chris Romeo (29:00)**: That's the thing. Very minimal.

**Izar Tarandach (29:03)**: Very minimal for us. So the screen team is absolutely brilliant. I had a chance of working with them and what's left of them in Datadog and they are amazing people. But then the problem comes when as a security person, you come to a developer and you say, well, here's the library. I want you to compile your code now with this library or run it on a on the environment and painful things are going to happen. Yeah. And developers love that kind of thing, right? Because then they come and say, what? That thing that I wrote last week is not working. I didn't change anything. Ooh, wait, the security people asked me to run something. It's default. Yeah. Right?

**Chris Romeo (29:43)**: Now in the RASP context though, like Screen's approach was very, at least with Ruby on Rails, I don't have experience across other web frameworks, but Ruby on Rails was very non-invasive in that you added the package, you added a single configuration file to load the library in the way Ruby on Rails loads packages, and it just worked. It didn't—

**Matt Coles (30:07)**: Doesn't Rails also require introspection? Like you have to have a language that supports introspection in order for this to work?

**Chris Romeo (30:13)**: What do you mean by introspection? Hold on.

**Izar Tarandach (30:15)**: You can use overloading as well, I would think.

**Chris Romeo (30:17)**: What do you mean by introspection? I'm not a computer science person.

**Matt Coles (30:21)**: Like, like Java, or you can look into the objects at runtime. The objects can look at themselves.

**Izar Tarandach (30:28)**: The runtime can query itself for the objects.

**Chris Romeo (30:32)**: Right. No, I mean, I think, I don't think you have to, I don't think RASP requires that. What RASP does is, at least on the Ruby on Rails version, is it plugged in, it plugged itself into the request-response stack inside of Ruby on Rails. And it just started looking at requests and responses in as they came through. So it was like a, it was like a shim almost in how it works in this, in the processing of the packet processing.

**Matt Coles (31:01)**: Oh, what the hell's the name of that programming paradigm? Aspect-oriented programming, right? Where you had, where you had basically a layer between, between the APIs that can intercept calls. Yeah.

**Chris Romeo (31:17)**: Now, I think the interesting thing, if we're going to continue on from RASP and IAST, is application detection and response is the future. Because it's the next step beyond. So the challenge, I would say the challenge with RASP and IAST is it doesn't talk to anything operational. It's its own thing. And so you have to send a feed from it into the SOC. You have to have security people deal with the results of it. What you're going to see, or what, what the ADR market is, the application detection and response market is creating, are tools that plug into the operational pipelines of companies. So instead of having security get the results of this, the operations team is getting the results and is monitoring the application security, performance, status, and potential vulnerabilities of the production fleet. Are we talking about telemetry?

**Matt Coles (32:14)**: Like traditionally what we would call Prometheus and whatnot telemetry feeds? I think it's part of the package.

**Chris Romeo (32:21)**: Some of it comes into that. It's RASP and IOT. It's RASP with telemetry in a feed that makes it consumable by the operations team and not the security team.

**Matt Coles (32:32)**: But the, I mean, the operations team could just have run Splunk themselves. Why, what, I mean, the fact that you send to the SOC is just an organizational construct, not a technical construct, right?

**Chris Romeo (32:41)**: For whatever reason. Yeah.

**Izar Tarandach (32:43)**: It seems to me that by adding those things together, you are building a context that is transforming the SOC into something that's more security operational rather than just looking for— Agree. Looking for intrude signatures. Now they are looking for intrusion behaviors, intrusion, which could be signatures as well, of course, but they differ more from one case to another. And you just have more of what we have always been asking for, contextual information of what's happening to help you declare this is really an incident or just something normally going wrong.

**Matt Coles (33:29)**: Yeah. Right. Well, so I guess I'm still hung up on the, so you send it to the operations team and not the SOC. It's just a different team you're sending it to.

**Izar Tarandach (33:41)**: Well, assuming that you have an operations team.

**Chris Romeo (33:43)**: Yeah. And I think it's bringing additional context from what I understand that make it more unified with what operations is looking at across the board.

**Matt Coles (33:53)**: So is this, but is this another example of like DevSecOps where you're giving developers responsibility without capability?

**Chris Romeo (34:01)**: No, I mean, the thing that, and I was having this conversation with Jeff Williams on the Application Security Podcast, that's what's kind of driven some of my opinion here. Really, the challenge is DevSecOps forgot the ops. Stop. Think about it.

**Matt Coles (34:22)**: It was DevSec. It was DevSec.

**Chris Romeo (34:23)**: Think about it. Think about it. Think about the—

**Izar Tarandach (34:25)**: DevOps or DevSec, it's okay.

**Chris Romeo (34:26)**: Think about the programs that you've seen happening. Where is the ops team in the conversation? They only appear in The Phoenix Project. That's it. And that's fiction. It's not real. Right. Right. Most of the time it's DevSec is where the partnership is happening and where we've seen movement. And we've seen some positive steps forward. In DevSec, the DevSec partnership, but we called it DevSecOps, but nobody brought the ops team into the actual equation.

**Matt Coles (34:54)**: Did DevOps run the ops team?

**Chris Romeo (34:57)**: In the very early days, I would say, but I think DevOps kind of became— Dev. Dev. But DevOps became fancy way to build software and it lost some of that cultural move.

**Izar Tarandach (35:11)**: That's the paradox of silence, right? If Dev and Sec do everything right, then Ops is just going to look at things as they churn along and they won't have something to do in the book calls. So that makes the Ops silent in DevSecOps, which is an expected thing at the end of the day. No, as we call it.

**Chris Romeo (35:32)**: They're still monitoring health and they're looking at things from a business perspective.

**Izar Tarandach (35:39)**: But they're monitoring from a point of view of if everything was done right before it got to us, then it will continue to run right as it goes.

**Chris Romeo (35:46)**: But we know the chances of everything being done right is—

**Izar Tarandach (35:49)**: Oh yeah, zero. But the thing that's coming up to me is that not to put an LLM into the loop, but I think that we are in a position to say from now on what we're going to start seeing is self-resilience coming up. Because this context that's being built with the operational information, with the runtime information and everything, is going to start letting environments decide by themselves how things die and how they start again. When containers came up, when Cadence came up, the line was, well, if there's a problem in the execution, just let the container die and it's going to restart itself and things are going to be routed to other containers that are working and you won't even notice, right?

**Chris Romeo (36:42)**: It was brilliant. Brilliant. It was wonderful.

**Matt Coles (36:44)**: Yep. Awesome.

**Izar Tarandach (36:46)**: I think that right now we are starting to have enough contextual information so that these containers can say, rather than dying, let me look if I'm having an incident in here or basically to say if my heartburn is a heart attack or not. I can just shut up and start again, or I can start screaming and saying I'm having a heart attack.

**Matt Coles (37:06)**: So what you're really bringing up then is sort of this idea that's been around for ages, but maybe not implemented very well, was self-resilient, self-healing systems. Self-healing systems.

**Izar Tarandach (37:21)**: Yep.

**Matt Coles (37:21)**: Well, not just self-healing. I guess the resilience part, right? That they can be resilient, but they can also be detecting of their own issues and then And then attempt to recover on the road.

**Chris Romeo (37:30)**: He threw it, he sprinkled LLM magic dust on top of it though.

**Matt Coles (37:34)**: Yeah, but do you need an LLM to do it? Do you need an LLM to do it?

**Chris Romeo (37:39)**: No. No. Who's done it? I mean, that idea came out 10+ years ago. Oh, decades ago.

**Matt Coles (37:45)**: Decades ago. Decades.

**Chris Romeo (37:47)**: But whoever implemented it without an LLM.

**Izar Tarandach (37:49)**: Not that I'm aging myself, but yeah.

**Matt Coles (37:51)**: This is stuff that came out of DARPA and, you know, IBM and others, I'm sure.

**Chris Romeo (37:56)**: But I never saw anybody actually implement it. And they're not heard?

**Matt Coles (38:00)**: Not in the commercial space because it's not com— I believe, I will tend to believe that it is not commercially viable in those systems, right? Maybe those systems are things like, you know, critical infrastructure or military systems or academic research systems, high-performance computing environments that have a need to maximize their CPU performance and CPU usage and make sure systems are always running because they're doing, you know, taking track of like satellites and stuff like that and not commercially viable systems.

**Izar Tarandach (38:34)**: I'm 99% sure that I'm talking out of my output, but I think that the Voyager series had some sort of self-healing capabilities. Yeah. And we've seen that in cases where the modems stopped working and somehow the machine corrected itself. Yeah, it was very, very basic, very, very, uh, almost hardware-based and, uh, not what today by modern standards we would call self-healing. Well, and, and, and those systems have redundancies, right?

**Matt Coles (39:06)**: They have built-in extra hardware, extra software code. You wouldn't do that today. You, you really can't do that today in commercially viable systems because that's a cost, that's a cost either either power or material or maintenance or—

**Chris Romeo (39:23)**: And it's 2x, right? Or more. Think about airline computers on airliners, right? There is at least 2 of everything so that if one of those fails, it can fail over and you never know as a passenger. The number of times that components have failed when we— You never know as a passenger if they fail too. No. No, you don't, you don't know if— well, I guess, yeah, you don't. Yeah, as an observer on the ground, you know, but as a passenger, you don't. But as a passenger, you don't really know what— if a system, if a redundant system failed over in the midst of a flight. That's true, you don't. And I don't want to know. I don't want the captain to come on, uh, excuse me, ladies and gentlemen, just want to let you know that our engines have failed, but don't worry, the backup's kicked in.

**Matt Coles (40:08)**: Well, okay, so would it surprise you that— would you— would it surprise you to know that a lot of your consumer electronics also have redundant systems in place. I'll give you a good example. Headphones or earbuds, right? Not if you wash them. Many of them, many of them do. He just did.

**Chris Romeo (40:29)**: I did my, that's—

**Matt Coles (40:30)**: Many of them have 8, many of them have A/B memory slots for firmware, right? So you can, you're running one piece of firmware and you can write to the other one and if it boots, The other one is then the old version, old copy. Gotcha. The new one doesn't boot, you always have a fail, a fallback, right? Because these are devices that while you don't want, they're too expensive to throw away, you, but they need to be reasonably functioning and it's relatively cheap to have two, two firmware locations. Yeah. But that's the extent of the, of the redundancy.

**Chris Romeo (40:59)**: Well, 'cause a lot of that's software, right? It's just, it's firmware. It's just making a, I mean, memory is cheap these days as far as storage.

**Matt Coles (41:07)**: You'd be surprised because that's not that type of memory we're talking about here. It's not RAM. We're not talking about RAM here. We're talking about something else. My first hard drive was 10 megabytes.

**Chris Romeo (41:15)**: Yeah. And it cost a lot of money. So like I'm talking historically, storage is cheap compared and comparatively over history. Right. Right.

**Matt Coles (41:25)**: A lot of these, a lot of these AP firmware slots are in memory, just not at that, not the RAM, not that type of memory. Yeah. But that's the extent of the redundancy. It's not like they have redundant circuits for managing, you know, the audio because it's not that expensive. Yeah. All right, well, we gotta land the plane here.

**Chris Romeo (41:45)**: So any conclusions? We need more Philandrites.

**Izar Tarandach (41:49)**: Oh no, wait.

**Matt Coles (41:50)**: Be, be care— don't, don't, I guess don't rush into everything being, requiring an LLM. Use the LLM for what it's good at. What it's good at and not what it's not good at.

**Chris Romeo (42:00)**: Yeah, that makes sense.

**Izar Tarandach (42:01)**: And keep reminding yourself what it is actually good at.

**Chris Romeo (42:04)**: Yeah, it's true. I think that's a good place to end right there. I mean, I wish we could add more words, but Matt, as normal, lands the plane with like 7 words.

**Matt Coles (42:17)**: As opposed to the observer on the ground noticing if it all went well. Yeah.

**Chris Romeo (42:20)**: Wait, is that the door? Is there a propeller? Is that a propeller flying by? Is that one of those bolts? Folks, thank you for listening to the Security Table and have a great rest of your week.
