42 minSeason 4, episode 13

Mostly Dead or Mostly Back: The Zombie Resurrection of DAST in an AI World

With Chris Romeo, Izar Tarandach, and Matt Coles

Security testing & vulnerability management

Is DAST disappearing, or is AI penetration testing giving its underlying techniques a new market? Chris Romeo, Izar Tarandach, and Matt Coles trace dynamic application security testing from network scanners and open source tools to commercial products, then debate where scanning ends and adaptive testing begins.

Part of: AI in security testing and bug bounty

Episode title image: Mostly Dead or Mostly Back: The Zombie Resurrection of DAST in an AI World

Listen to the conversation

Episode chapters · 14 chapters
  1. Intro and movie detoursAudio
  2. The Final Countdown and the return of DASTAudio
  3. What do we mean by AI penetration testing?Audio
  4. The case for DAST becoming strongerAudio
  5. From open source scanners to a marketAudio
  6. Nmap, misconfiguration, and testing goalsAudio
  7. Where does DAST end and AI testing begin?Audio
  8. DAST technology versus penetration testingAudio
  9. Finding many issues versus serious exploit pathsAudio
  10. Definitions and adaptivenessAudio
  11. Compounding scanner errors and AI uncertaintyAudio
  12. Gray-box context and finding zero daysAudio
  13. Can AI find something nobody has seen?Audio
  14. Separating the technology from its marketAudio

About this episode

Is DAST disappearing, or is AI penetration testing giving its underlying techniques a new market? Chris Romeo, Izar Tarandach, and Matt Coles trace dynamic application security testing from network scanners and open source tools to commercial products, then debate where scanning ends and adaptive testing begins. They question whether a tool that finds many issues serves the same purpose as a tester who follows an exploit chain. AI-generated findings raise another problem: what happens when scanner errors combine with model uncertainty? The discussion considers gray-box context, zero-day discovery, and the difference between a technology's usefulness and the business built around it. Along the way, cats perform an unforgettable version of The Final Countdown.

Mentioned in this Episode:
➜ Nmap
➜ OWASP Benchmark

Follow AI Security Table:

➜ Home: https://securitytable.ai/

➜ X: https://x.com/AISecurityTable

➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/

➜ YouTube: https://www.youtube.com/@AISecurityTable

Transcript · 42 min conversation

0:10Chris Romeo Hey folks, welcome to Ferris Bu— no, no, Security Table. Uh, much ado about nothing, but we, we are big fans of Ferris Bueller's Day Off, and we do hope to maybe experience that day at some point in the future.

0:27Matt Coles Just not when he, not when he shoots the, the car off the back of the—

0:31Chris Romeo Yeah, but we do wanna borrow—

0:32Izar Tarandach I cried.

0:34Chris Romeo Matt's Ferrari and drive around the city of Chicago.

0:37Matt Coles It's a little Matchbox Ferrari, so it's, you know, it might not get very far.

0:41Chris Romeo So we're gonna need a shrink ray. Dang it. Have to invent some new technology.

0:46Matt Coles Honey, I Shrunk the Threat Modelers.

0:49Chris Romeo Yeah, that would be a terrible movie. I wonder if Rick Moranis is available though. Oh, he'd be awesome as the fourth. Person there.

0:56Izar Tarandach He, he's preparing for Ludicrous Speed.

0:59Chris Romeo Yeah, true. That was his other, uh, he was known for a couple of different things. Plus Ghostbusters, wasn't he? Ghostbusters.

1:07Izar Tarandach Yeah, yeah, yeah, yeah.

1:09Matt Coles I heard he had a, he had a little bit of a partner, kind of important role, kind of tied the whole thing together.

1:16Chris Romeo You know, spoiler alert, sorry for those that haven't seen a 35-year-old movie. Just have AI reproduce it.

1:23Matt Coles And hey, I, I had an old movie weekend I watched The Final Countdown.

1:31Izar Tarandach The Philadelphia Experiment.

1:33Chris Romeo The song by Europe?

1:34Izar Tarandach No, no, no, no. The movie.

1:35Chris Romeo The Final Countdown. Oh, there's a different one.

1:41Izar Tarandach Wait, I'm shocked.

1:43Matt Coles What, he can sing?

1:45Chris Romeo Oh no, I can't.

1:46Izar Tarandach I just fake it.

1:47Chris Romeo And well. I just fake it. That was a complete fake.

1:49Matt Coles All right.

1:49Chris Romeo Final Countdown.

1:51Izar Tarandach This is the one where the Nimitz goes back into time. To December 7th, 1941. And then you have the whole, uh, do we or don't we, which is actually a very, very nice thing to entertain. Would you or wouldn't you?

2:06Matt Coles Okay. Because you brought up, because you, because you brought up, uh, Europe, you don't have to play it now, but I'm going to put it in chat.

2:19Chris Romeo Is it The Final Countdown?

2:20Matt Coles It is, but it's not. But it is, but it isn't.

2:23Chris Romeo Okay.

2:24Izar Tarandach Okay, now I have to. Oh my God, no.

2:31Matt Coles It's the ultimate cat video.

2:34Izar Tarandach No, no, no, just no, no.

2:36Chris Romeo I'm not even gonna be tortured by a cat video. Only Matt would send a cat video on a podcast.

2:42Matt Coles No, no, no. For those who are listening at home, you have to go check out Rock and Roar.

2:49Izar Tarandach That's it from Matt. Your cat's the final meow down. It's Monday, it's Monday, 10 before 8, and it can only get better now because that's it.

2:59Chris Romeo Is our find your happy place, find your happy place.

3:02Matt Coles Damn.

3:02Chris Romeo Uh, can I have a picture of a dog please? Yeah, something other than cats meowing the final countdown.

3:10Matt Coles That's just the cats don't meow, but that's okay. Okay.

3:14Izar Tarandach I, I, I think that that's a, a perfect segue because there is only one thing that we can talk about after, uh, cats playing Europe.

3:24Chris Romeo That would be—

3:24Izar Tarandach Final meow. Oh my God.

3:26Matt Coles Final meow.

3:30Izar Tarandach Let's talk about DAST.

3:32Chris Romeo Let's talk about AI pentesting and DAST.

3:36Matt Coles Can DAST be deader?

3:39Chris Romeo Oh yeah, DAST is— we said on a previous episode that DAST was mostly dead.

3:44Matt Coles Mostly dead.

3:46Chris Romeo But the zombie of Dast is Dast, a zombie that's making its way back into the zeitgeist rather than, than zombie.

3:56Izar Tarandach I feel like somebody dug it up, beat it, and I was putting it in the ground again.

4:03Chris Romeo So like, it's like, it's like Weekend at Bernie's. Weekend at Bernie's. If Bernie was dead for like 2 weeks, sorry, something like that. Then they dug him up and tried to put the ropes on his hand to make him wave and everything.

4:14Izar Tarandach Oh my God. And then the hand comes off and the arms fall.

4:18Chris Romeo That's funny.

4:19Matt Coles It's Night of the Living Dead.

4:21Izar Tarandach It is.

4:22Chris Romeo And it's—

4:23Matt Coles Oh no, no, it's Army of Darkness death.

4:27Chris Romeo I mean, you wanted a chainsaw on your hand or—

4:32Izar Tarandach Always.

4:33Chris Romeo No, man.

4:34Izar Tarandach But it could be worse. It could be Hot Fuzz and then we'll be talking about fuzzing as well.

4:38Chris Romeo That's true. All right, well, getting serious now.

4:42Matt Coles Wait, what? What?

4:45Izar Tarandach I didn't sign up for that.

4:46Chris Romeo I just tried to make it, just tried to do a transition here to, uh, you know, a serious topic. Get into the role.

4:51Matt Coles Get into the role.

4:52Chris Romeo So just if you guys could both slap yourselves a couple of times like they do on TV, that'd be great.

4:56Matt Coles Thanks. It'd be really cool if you had an effect.

4:59Chris Romeo You could go like, sure, AI can add it in post. Uh, let's see. So, so AI pen testing. When we're, I mean, when we say AI pen testing, we talking about Mythos? Are we talking about Claude doing some pen testing. Like, what are we—

5:16Matt Coles just think of what pen testing is, right? It's investigating a system, trying a bunch of stuff, and then finding bugs. So if AI has the ability to do things like interface with network services and test ports, right? That's what— that's just runtime, runtime testing as opposed to static testing, right? And so it's dynamic. If humans can do it, why can't they?

5:45Izar Tarandach No, no, no, wait, wait, wait. Big difference here. Dynamic and runtime.

5:50Chris Romeo Yeah.

5:51Izar Tarandach Let's not mix both. DAST is dynamic. Runtime.

5:54Matt Coles Okay. Sorry. So you're right. It's, it's, it is dynamic in that traditionally it's human, human executed so that the human can perform a test, look at the output, and then adapt the test to attempt to find a path through the system to exploitation.

6:15Izar Tarandach That's how it's done, but it's called dynamic in opposition to static. So you have static code analysis and then you have dynamic testing. People want it to be smart.

6:26Matt Coles So this is static time, right?

6:28Izar Tarandach Right. No, test happens when you have something running.

6:31Chris Romeo Right.

6:32Izar Tarandach Right. So the people wanted to be very discerning between the two disciplines and said, okay, static code analysis, it's static. The code just sits there. You grab, you read. It's fun. Awesome for the whole family. It's fun.

6:48Chris Romeo It's not fun.

6:50Matt Coles It's not fun. That's why we let the AI do it because it's not fun.

6:53Izar Tarandach Wait, let me go to the other side of the scale. Now you have dynamic testing, which is Dynamic. Ooh. Aerodynamic.

7:04Chris Romeo You know what that, you know, you know what that means? That means it'll take at least a week to finish running. It's, that's how dynamic it is.

7:11Izar Tarandach And you still can't say if it covered the whole place.

7:15Matt Coles And, and, and mind you, it only runs a week because you can't run it longer cuz you run outta tokens.

7:20Chris Romeo Right.

7:20Izar Tarandach No, no, wait, wait, wait. We haven't got into the tokens yet. We, we are still DAST tokens.

7:25Chris Romeo I don't, I have, I don't think DAST is being driven by AI at this point.

7:29Izar Tarandach No.

7:29Chris Romeo At least I haven't seen anybody say that, you know, this is the AI DAST.

7:34Izar Tarandach So, no, actually, I've seen it. I've seen it. I think that my question is, is it still DAST if it's AI-driven? Are we still talking about the same thing?

7:45Matt Coles It's still dynamic application security testing.

7:48Izar Tarandach Right, but—

7:49Matt Coles Just AI-driven, not human-driven.

7:52Izar Tarandach Right, but then—

7:53Matt Coles Or tool-driven, for that matter. I mean— Maybe, maybe the AI wrote a specific tool to target a particular interface and then executed it.

8:02Izar Tarandach So wait, let's walk it back and put things into that word again, context. We are addressing the post by Amanda Alvarez, right? A DevSecOps enthusiast that begins with the hot take, AI pentesting is going to make the case for DAST stronger, not weaker. Let's start from the bottom line. Anything will make the case stronger because it can't be weaker. Well, if it's mostly dead, it could be weaker, but it's mostly dead because people still have a fondness to selling it. It's dead, Jim. It's dead, Jim.

8:40Matt Coles Okay. Tis but a scratch. It's like the knight in Monty Python.

8:44Izar Tarandach No, it said that it's yellow when it was blue and it was tossed into the valley. So forget about it. It's not—

8:53Matt Coles forget about it.

8:54Izar Tarandach Forget about it. So the idea behind the post is that AI accelerates everything. The attack surface grows faster than ever. Lord knows how. And the pressure calls for better, smarter testing and a clearer understanding of where each tool actually fits in the stack. Again, I will go out in the rain and say, if it calls for better, smarter testing, then we are not talking about DAST. DAST was an exhaustion thing. DAST was enumerate endpoints and hit each one of them with this battery of PHP, do you have PHP in the server calls, right?

9:39Chris Romeo Yeah.

9:39Matt Coles What you're raising is a really important point, Ezra, I think. DAST was a reaction to a concept or an idea that QA-based security testing was faulty because QA doesn't know how to do security testing. And so a plethora of tools were invented.

10:01Chris Romeo DAST, I just, I just had this epiphany. You guys can tell me if I'm right or wrong here, but DAST is the, the commercial market of DAST tools is the result of an evolution of open source tools. Called scanning tools that people used to use, but they never cost money. Like, I remember Nessus in the early days, right? Nessus was a free open-source tool that everybody was using and nobody was paying for. And then Nessus got the idea to turn it into a company, and all of a sudden the world of DAST was born.

10:39Matt Coles You mean somebody monetized open source?

10:42Chris Romeo Yes, but my, my, but I've just, I've just realized I never had this, this brainwave before that DAST actually wasn't invented. It was just morphed from, from scanning tools. And in those days it still kind of ran the same way. It still would take you like a week to scan something with a lot of those old scanning tools.

11:02Matt Coles Well, I mean, part of that was there was a host of reasons for that, right? Nessus had, had whether Early days, I don't think Nessus parallelized scanning.

11:11Chris Romeo Oh no, it was like single-threaded, right?

11:13Matt Coles It was single-threaded and, and, you know, systems were getting more and more complex, so they would take some time and then have to adapt and they would crash or they would have like, you know, these insane timeouts because you, you know, you wanted to wait forever for results.

11:25Chris Romeo In those days you would crash the actual unit under test.

11:29Matt Coles Yes. With the scanning tool without, without any sort of way of like, oh, can you restart my VM? Oh, let me call IT for that.

11:37Chris Romeo Yeah, it was, there was no VM in those days. It was like, hey, can you go restart that server down in the data center, please?

11:43Matt Coles I had to, I gotta go, I gotta fly to the data center to go turn it off and back on again.

11:49Izar Tarandach I'm going to agree with you guys, but also go back a bit more and say that it wasn't because of the open source tools. I think that that came about because there was Nmap and people would run Nmap and then look at the list of ports that came out and say, okay, What now? And a bunch of people said, oh, wouldn't it be cool if you found a port 80 that you had like this script that would try for this known issues that everybody keeps falling into and just see if they are there. And that's what started the whole thing. But, but notice, you guys put it to the QA place, to the QA. Let's see if things are working well. Let's, I pulled it to the, let's see if it's misconfigured or if I can do something with it. So DAST had always this thing between QA and between being a very powerful tool for offensive objectives that we shouldn't publish and we shouldn't let fall into the wrong hands.

12:51Chris Romeo Whoa.

12:53Izar Tarandach So that's where I think the DAST got its bad name from.

12:58Matt Coles Well, so interesting. So a couple of things here.

13:03Izar Tarandach QA.

13:04Matt Coles So first off, QA testing should always be doing positive and negative testing, right? There should always be this. You want to validate the functionality, but you also want to look for what can go wrong in QA. That's what QA traditionally would have been checking for. Security tools were always looked at as special because they were security tools and QA Didn't have security experience for the most part, unless you found security people or you, and, or you had security teams using those tools. So I wonder if it may be a combination of, did DES appear because there was always a, there's always a shadow IT problem. Ooh, scary shadow IT. And you had these hacker tools that were getting used by people to, to discover things and Well, and of course they're using open source, so it comes from somebody in their, who's coding in their basement. Let's build a market segment around packaging these capabilities to give to engineering teams so they can do their own security testing and monetize that. Is that where the market of tools and the capability of tools came from?

14:21Izar Tarandach That's exactly what happened. And I can say so because I was there. Like, no, no, seriously. I mean, back in the dark ages—

14:29Matt Coles QA or security or the marketing monstrosity?

14:32Izar Tarandach Back in the dark ages, '97, '98, I think it was. And that's actually the first time that I worked with Adam, right? We were working at a place called NetAct that later got bought by BindView. And we were writing this first vulnerability scanner. Of sorts. It was called Drum— Drums, it was called HackerShield. Actually, HackerShield 2.

15:01Matt Coles Oh, it's 2.

15:02Chris Romeo Not just the first one, it's the second one.

15:04Izar Tarandach Yeah, it was the second one. It was the improved version. But good times, good times. Amazing people. Seriously, amazing people. People that I deeply respect to this day. But the fun thing is that it blurred the, it blurred the division between testing to pen testing to the name didn't exist back then and QA tool. So you could point it at yourself, you could point it at others. It was a whole fun thing. But I think that the place where I'm starting to get something with, oh, so first of all, the post doesn't exist in a vacuum. There is going to be the June 10 discussion at the League of Cyber Defense Champions, which is actually a very interesting group.

15:58Matt Coles You know them?

15:59Izar Tarandach Yeah, yeah, yeah, I do. It's the evolution of the Security Champions community that existed by Dustin.

16:09Matt Coles That's a layers effort.

16:13Izar Tarandach Yes. And it's going to be a discussion themed, will AI pentesting make DAST obsolete? And what I keep coming back to the text here is that I don't think that intentionally, but I think that it creates a very interesting question, at least in my head. When you read something like, where should DAST end and AI pentesting begin? Forget the end of the thing, but when I see that, I'm immediately thinking, wait, wait, wait, wait, we're talking about DAST and AI pentesting as pentesting of AI or AI pentesting as AI doing the pentesting.

17:02Matt Coles That's not where I thought you were going to go.

17:05Izar Tarandach Where did you just go?

17:07Matt Coles Is, is DAST and pentesting two different activities?

17:11Chris Romeo Oh, 100%. Of course.

17:13Izar Tarandach Yes.

17:13Matt Coles Yes. So does DAST have a place for, does DAST have a place versus pentesting or should pentesting replace DAST as a thing?

17:26Chris Romeo I think you use some of the DAST techniques to Izar's point earlier about the, about an AI potentially writing a script. To replicate some type of function that a DAST would've done, the AI pen testing process. So yeah, I think, I think you use some of the principles of DAST in pen testing. I think you have to, you know, you're not reinventing port scanning, for example.

17:50Matt Coles Well, the market of DAST, the market of DAST was for, as we talked about, it was a set of tools that did automated, automated analysis, interrogation, and, and interface testing primarily, but its market was automation, taking human out of the process, right?

18:10Chris Romeo I can even tell you where it started before that. Back in the dark ages, in the years that I was working, I started in security the same time Eazar was talking about, he and Adam at this other startup. People used to pay us to do what was called a vulnerability scan.

18:26Izar Tarandach Exactly.

18:27Chris Romeo And because they didn't know, they didn't know what they did. Nobody knew how to do a vulnerability scan. So they would pay us a lot of money to run a set of tools and then write a report for them that showed them what their vulnerabilities were. Now, it was not a pen test, right? It was a vulnerability scan, right? And it used to suck because it was really boring to execute those things.

18:52Izar Tarandach And building on top of that, let's be honest, that today is pentesting that you bought on Temu. Okay.

19:02Chris Romeo It is. It is.

19:03Izar Tarandach Sometimes you order a pentest, you get this nice report and you start reading it and you go, I know exactly which tool generated this stuff. Right. So unfortunately there are still some less reputable outfits. That work like that. Oh, we're just going to run our customized tool and work off the results. If it finds something strange, then we're going to look for more stuff.

19:30Matt Coles But that's the big point, right? DAST was mostly automation to generate a set of— provide a set of information to the customer, right? So they get— you run the tool, The tool would try to be dynamic, meaning it would try to adapt to problems that it showed. Like, so if it was going to do a port scan, that's going to run through a series of ports. And if it runs into a problem, or if it says, oh, I find this port and that's maybe a redirect, it may or may not have intelligence to do the redirect. A pen test is human-generated traditionally, where they take that output and then try the next thing. They try to do the pivot, they try to do the exploit, they try to do the whatever, right? To figure out, okay, this, this happens, so therefore I should do this. Right. Well, so is DAST as a technology versus pen testing as a, as a function, are they mutually exclusive? The market for DAST may be dead. Look again, but the technologies are still, are still there. And pen testing rely and pen testing as you've described relies on them.

20:48Izar Tarandach Let's, I, I, I get everything that you said. I agree with 99% of it.

20:54Matt Coles 9%.

20:56Izar Tarandach 99. Okay. The, the, the, my, my point, I agree with 9%, but just 9%.

21:02Matt Coles But my, my, not even 9.9.

21:05Izar Tarandach My, my issue here, my issue here is to say I think that you put it like the DAS technology. There is no DAS technology because you went there and you, in like one sentence, you described the whole what is DAS thing. You POC mapped the thing. You found something. You test it with a bunch of stuff. If I found this, then I do that, right? So it's a set of rules. You just throw them on, pop, pop, pop, pop, pop. Based on what you get from the port, based on the headers that you get, based on the banners, whatever.

21:40Matt Coles But solely on those, on that port.

21:41Izar Tarandach Yeah, but no, but no. In theory, if you take that creativity thing that you pointed out, that that's the difference between the pentesting and DAST. If you were to take any any, I'm not saying anything specific, but any coding agent today and say, here's an IP, pentest it for me. That thing would be able to write the port scanner itself. Assuming it has privileges, it's going to be able to open the ports. It's going to be able to do the scan. It's going to be able to interpret the results. It's going to be able to, in very, very big air quotes, have the creativity to figure out the next step.

22:33Matt Coles And we should— okay, go. So finish your thought. I, I'll wait.

22:37Izar Tarandach Some people contend that it's going to even have the huge air quotes intelligence to figure out stuff that other people haven't figured out yet, to infer new ways of attacking what it finds.

22:51Chris Romeo Right?

22:51Izar Tarandach Why? Because it's going to be able to go into the network, into the net and GitHub and find the source code of the, of the daemon that he just found that it's running. Blah, blah, blah.

23:02Matt Coles And that's actually the really important part, I think, that we're missing in here, or need to— we're not missing, we need to bring into this conversation. The goal of DAST versus the goal of pentesting and the methods that pentesters use traditionally, Right. What is DAST? DAST is, I need to be thorough to find as many vulnerabilities as possible. And thorough and find as many vulnerabilities as possible may have a mismatch. We've seen this with static code analysis, right? There are methods for static code analysis, which are basically grep the code, look for everything, high false positive rate, or try to be smart about it and miss a bunch of stuff and Now you question whether that tool is actually valuable. DAST hit the same problem, right? You can be very, very thorough and verbose, take 8, you know, take 8 days to run and produce tens of thousands of issues that nobody can look at, or take an hour to run and produce nothing and then wonder why I bought this tool, right? Whereas, but its goal, its goal is to find as many problems as possible.

24:11Chris Romeo Your dog agrees.

24:13Matt Coles I know. Uh, but pentests is meant to find the, the ways in, the vulnerabilities.

24:23Izar Tarandach It has to, it, pentesting.

24:30Matt Coles So DAST is meant to throw it, find as many vulnerabilities as possible. Pentesting is meant to find the most serious things.

24:37Chris Romeo So just, just an FYI for our audience, Matt is actually at our Northeast studio today, which just happens to be next door to a veterinarian. So that's why you constantly hear some barking.

24:49Matt Coles Slash, slash, uh, ducky daycare. Yes.

24:51Izar Tarandach Yes.

24:51Chris Romeo It's also a ducky daycare.

24:53Izar Tarandach I think I'm going to disagree there.

24:55Matt Coles Mm-hmm.

24:55Izar Tarandach Because I think that DAST, it's not looking for trollnets. DAST is, is looking to go over the catalog of things that it knows about. There's this, this hard set of rules that—

25:06Chris Romeo not inventing anything new. That's the method.

25:08Matt Coles I think you're, I think you're highlighting the method versus the goal of—

25:12Izar Tarandach right, right, right, right. But that's the thing, because it's bound, because the method itself is bounded, we can't say that it's trying to aim for finding the maximum. It's going to try to identify all the things that it knows. And once it runs through the catalog, it's going to raise its hand and say, hey, I don't, I don't know anything else to do here.

25:30Matt Coles That's right.

25:31Izar Tarandach If there was theoretically one more port with one more daemon running in it and it's not in my catalog, I'm fucked.

25:38Matt Coles Yeah, certainly. It may have— it almost certainly will have been implemented with a fixed set of rules.

25:45Izar Tarandach But on the other hand, AI is supposed to be this algorithmically flexible thing that's going to figure out from what's coming from the scan, what are the possible implications of it and figure out how to best attack it. And we have this specifically trained models that are more sensitive to the possibilities and this and that and the other one. At the end of the day, they are trying to achieve the same result. One of them is much newer and has much more technology applied to it and much more research. And all the things that we learned in all these years that DAST has been dead are being applied to AI pentesting. Right.

26:36Chris Romeo So I actually don't think DAST is dead anymore.

26:41Matt Coles Oh, you heard it here, folks.

26:43Chris Romeo I think we should start a DAST company. And build a new tool. I was doing a little bit of research here 'cause I thought, I, I, I asked Claude, are people still buying DAS tools? Oh yeah. And Claude pointed out that the DAS market was valued at roughly $3.8 billion in 2025, expanding to $4.5 billion in 2026, with projections to reach $12.7 billion by 2032.

27:13Izar Tarandach But Chris, of course they are.

27:15Chris Romeo So it's self-assurance. It is. And regulatory pressure is high and, and people driving a lot of this. Yeah. Yeah. Which, that's a great business idea. So that's my next business idea. I'm actually, after, after throwing all this, saying all these mean things for the last couple of years, I'm joining the cup. I'm joining the ice. No, I'm just kidding.

27:39Matt Coles Let's just call it Lazarus.

27:41Izar Tarandach You couldn't even say it.

27:43Chris Romeo But it does seem like it's, I mean, I was not expecting that. I was expecting to see a decline, like the market was shrinking.

27:49Izar Tarandach No, no, no, no, no. Exactly because of what Matt was saying. No, it's fire and forget, right? You, you just press the button, walk away, come back a couple of hours later. It didn't, it didn't find anything. We are perfect. We're good. We're done.

28:02Chris Romeo Yeah. Shift.

28:03Matt Coles So, so interestingly, so I did, I was also doing some research to see All right. To remember what the actual definition of DAST versus pen test is. And here's one resource I found. I won't tell you who said it. Dynamic application security testing is the practice of using software tools to simulate malicious interactions with the system to perform and to perform actions to find vulnerabilities. Penetration testing is the practice of attempting to find and exploit security vulnerabilities in a live environment. They're the same thing.

28:35Chris Romeo Yeah, that's not, I wouldn't describe that as a very good definition.

28:39Matt Coles So, but the important part is adaptiveness. Pen testing is human controlled and can take action based on the output of the tool to do the next thing, maybe refocus the tool to do something else. Whereas the DAST tool is not adaptive and like, this is method versus function.

29:02Chris Romeo What I learned in my cloud research is that most of the DAST providers have, are using what they call AI-powered validation engines. And they've, they claim to, the leading vendors claim to have dropped the false positive rate from 82%, which is what the OWASP benchmark data showed us a number of years ago that, uh, I think Dave Wickers did. Uh, to, in 2024, drop the rates to 5 to 8% through this use of AI to validate the findings and confirm they're actually real. So at least it's not as crappy as it used to be. Still crappy at 5 to 8% error rate, but—

29:46Izar Tarandach Which is amazing marketing nowadays, right? Yeah.

29:49Matt Coles But, but humans— Buy us!

29:50Izar Tarandach We're not as crappy as we used to be.

29:52Matt Coles Human pentesting may have hit a false positive rate as well. But they weeded out those false positives because they wanted it because of the value that you get out of the human act, human interaction from pen testing. Now, if AI can do both, is there an— so is DAST the automation of human testing?

30:15Izar Tarandach Not now.

30:17Matt Coles Is that what DAST, the market segment, or DAST technology is?

30:22Chris Romeo It's where it came from because we were running these things manually and now we're going back to that.

30:26Matt Coles So now AI replaces the human. Is it automation or is it not automation?

30:31Chris Romeo So you're saying that AI replaces DAST? AI pen testing is replacing DAST?

30:36Matt Coles No, I'm saying that AI is making DAST as capable as pen testing and you may not need pen testing anymore.

30:42Izar Tarandach No, no, no. What I say is that—

30:45Chris Romeo I like controversy, so I'm just trying to hype it up.

30:47Izar Tarandach You had— you had, you had your false positives and false negatives in DAST for one set of reasons, for, for the rigidity of the rules, for the different environments that you could set it up. And I'm assuming non-comparative DAST, right? Where we point things, where you point your scanner at something that you don't control and basically you're doing pen testing with it. And now on top of that, we are putting in the false positives and false negatives that come from the uncertainty of using AI. So we are compounding two sets of false stuff to create a new reality that we are saying it's better than the old one.

31:30Matt Coles Mm-hmm.

31:31Izar Tarandach I'm going to go, go out on a, on a limb and I'm not a statistician, but nice use of the word though. Right? It sounds—

31:41Matt Coles You used it properly in a sentence. There you go.

31:44Izar Tarandach Right? And it's only Monday morning.

31:46Matt Coles Now can you spell it?

31:48Izar Tarandach Yeah. S-T-A-S-T-I-C-I-N. Statistician. Statistician. Okay, fifth language. So anyway.

32:03Matt Coles After C.

32:05Chris Romeo That's his first language. He's literally growing up as a baby and he's just got a C book and he doesn't even He doesn't even speak any other language other than English.

32:15Izar Tarandach It's not that far from the truth, unfortunately, but yeah.

32:17Matt Coles int i= 2.

32:22Izar Tarandach I was born with an asterisk in my hand. But, uh, the, where was I going with this? So yeah, so we have false times false and you get to even more false, right? So I think that in the mirror illusion hall of this whole thing, it just comes as, again, what Chris said before, we suck a bit less, but we still suck. So I think that if I had to get a budget and decide where am I going to do my testing and where am I going to get my assurance. I think that there's a bunch of other things that I would be doing before I rely on DAST or AI-assisted DAST. And that definitely includes people because people will definitely look at you and say, uh, yeah, we should be testing in a different way and not like this. So to this question that's in the post again, will AI pentesting tools augment human pentesters or eventually replace them? I could see the eventually replacing them, sure.

33:41Chris Romeo Yeah.

33:41Izar Tarandach But I think that the augmentation is already happening and there isn't just a huge case being done about it. But the thing is, it is focused on people, not on that. So you said—

33:56Chris Romeo AI's got to come up with an original thought though, before it could ever replace human pentesting.

34:02Matt Coles Does it really? If it knows all the code paths, if it can identify all the code paths.

34:08Chris Romeo But can it find this? Can it find something that it's never, that nobody's ever found before?

34:15Izar Tarandach But now you're hitting the white box, gray box, black box problem, right? So Matt is saying if it knows all the code paths, so it's white, white box. Chris is saying, can you find something that nobody has found? Before. So it's probably black box. I'm looking at the results of a port scan and figuring out something that people haven't yet. As always, probably the answer is in the gray box, where you're going to have some access to a threat model, some access to source code, some runtime stuff to see the way that you are poking, what it's actually doing to the box. So probably that's where the value resides at the end of the day.

34:58Matt Coles So we already know, we already know that frontier AI models can find zero days in open source products, right? Copy fail, a host of Linux vulnerabilities, the one curl vulnerability, I think that Mythos found. Um, it's, we know that open source, even with full white box techniques, You can still find zero days.

35:23Izar Tarandach Mm-hmm.

35:25Matt Coles Can a human, a human could have found it or AI could have found it.

35:28Izar Tarandach Mm-hmm.

35:29Matt Coles Mm-hmm. But are there some things that AI today cannot find that humans must be able to?

35:37Chris Romeo That's what I'm, that's my premise here is that I have not seen anything to tell us that we've reached AGI.

35:47Matt Coles I'm not suggesting AGI. I'm just saying that in the course of, of execution, that it stumbles upon something that has never been seen before. Is it possible?

35:57Chris Romeo But will it know that it's stumbled upon something if it's never seen it? It's if nobody's ever seen it before, meaning no human on earth has ever seen it. Also, no AI, no computer has ever seen it. Like, can AI as it exists today connect the dots on an original thought of this is something that's nobody's ever seen before. That's what I don't honestly know the answer. I don't know that anybody knows the answer.

36:22Matt Coles In other words, you're saying that it will find some, it may find something and just, oh, it's a finding without the understanding of, oh wait, that's a, that's a critical zero day.

36:32Chris Romeo Yeah.

36:32Izar Tarandach So he, here's the thing. Apparently AI has already proven mathematical theorems that have not been proven by people. I read somewhere that it's happened and now comes the question, is that, uh, uh, creativity or exhaustion of a solution space that people just didn't have the time or, or inclination or whatever to go look for? And for us, it ends up being the same thing. I think that's where things get, uh, more relevant and something that we have actually pointed out a number of times in the context of threat modeling. And I think that it's exactly that same context that comes back here is the garbage in, garbage out thing. AI will gladly work on what it knows about something, will even more gladly invent stuff that it thinks that it needs. If it doesn't know, so it's going to fill in the blanks, play Mad Libs with whatever it needs, and we have to deal with the consequence.

37:38Matt Coles Yeah.

37:39Izar Tarandach But at the end of the day, it will only act on the things that it knows certainly to do, whatever degree of certainty that it knows. So I think that the limitation here is what does it know? What are the base facts that it's acting out of, right? So there's the whole, uh, theory of, again, the, the, the Chinese room. If you're giving a system symbols and that system has rules to translate those symbols into other symbols and you are getting symbols back, is that intelligence? Some people say yes, some people say no. My point here is that for us, for our problem, for security, The important thing is, are you giving all the right symbols so that you can trust that whatever comes back is something that you can use? So I don't think that it's an AGI. I don't think that it's a creativity problem. I think is, are we painting the right initial picture here so that an AI can actually be, uh, uh, helpful. Generating the symbols that we have to consume and make decisions on, right?

38:54Chris Romeo And do we have the right checkpoints being done between different AIs to confirm whether that something anomalous has occurred?

39:12Matt Coles Anomalous means being able to detect that there's a change that is meaningful, that is, that is a change that is unexpected or meaningful, right?

39:21Chris Romeo Yeah. So that was my thought.

39:23Matt Coles So that goes back to the, does it, can it recognize when it found something that's not supposed to be there?

39:30Izar Tarandach But that's the thing. Is it going to find, because it's looking at that specific discrete event and say, based on what I've been trained on, this event is security notable. Or is it going to have access to the whole knowledge of the SOC to be able to go and look at what's in there and compare with what it found now and say, oh, these two things are different. Sorry, this thing and the average of the other things is different enough that now this is a security notable event.

40:00Chris Romeo I can make an argument for either option, for giving it access to all of it or not giving it access to all of it, depending on how conspiracy theory laden.

40:10Matt Coles I want to leave you guys with a parting thought here.

40:14Chris Romeo Okay.

40:16Matt Coles If separate the technology of DAST from the market of DAST, if the technology of DAST continued as open source and was run by people who saw value in the use of the tool for the functions they were doing and it wasn't monetized, And then throw a kit and then processed into the throw the kitchen sink at this problem to find as many, many things as possible, irregardless of the false positive rate. Is that, if you think about what happened there, and maybe those tools might've been useful as Swiss Army knives and not as kitchen sinks. And now think about AI and pen testing. Are we gonna run into the same problem of monetization trying to one-upmanship versus it's a tool that people can use to find things?

41:15Chris Romeo I think that's a good place to leave us with that. We'll let the audience ponder that while we ponder it and are unable to sleep for probably the next week.

41:28Matt Coles And also the nightmare of Chris going out and writing a DAST tool. That's right.

41:30Chris Romeo I've already got Cloud working on it. It's already in design.

41:35Izar Tarandach I'm sorry.

41:36Chris Romeo I'm still stuck. We'll be shipping. I'll be shipping this by lunchtime.

41:39Izar Tarandach So I'm still stuck to the final meal. I haven't recovered yet.

41:45Chris Romeo Well, that's why I didn't click on the, I know better than to click on a link somebody sends me. You just got Rickrolled by the cats of the Final Countdown. All right. Well, folks, thanks for tuning in to this episode of the Security Table and Just remember that it's the final meow down.

Transcript supplied by AssemblyAI. It may contain transcription errors.

Originally published as The Security Table. Part of the AI Security Table archive.