# The Evolution Problem: After 100 Episodes, What’s Changed… and What Hasn’t?

URL: https://securitytable.ai/the-evolution-problem-after-100-episodes-what-s-changed-and-what-hasn-t/
Published: 2026-04-01
Duration: 2980 seconds
Season: 4
Episode: 6
Hosts: Chris Romeo, Izar Tarandach, Matt Coles

## Show notes and chapters

**We made it to 100 episodes, so naturally, we decided to look back and see how wrong we’ve been. In this episode, we revisit some of our past topics, predictions, and hot takes to figure out what still holds up and what didn’t quite land. From “we don’t know what we don’t know” to the evolution of security tools, we reflect on what’s changed, what hasn’t, and why some problems never seem to go away. Along the way, we compare where we were then to where things stand now, calling out a few wins, a few misses, and everything in between. After all this time, are we actually any smarter, or just better at explaining the same problems? This episode is part reflection, part reality check, and a look at what 100 episodes have really taught us.**

  

Follow AI Security Table:

➜ Home: [https://securitytable.ai/](https://securitytable.ai/)

➜ X: [https://x.com/AISecurityTable](https://x.com/AISecurityTable)

➜ LinkedIn: [https://www.linkedin.com/company/ai-security-table/](https://www.linkedin.com/company/ai-security-table/)

➜ YouTube: [https://www.youtube.com/@AISecurityTable](https://www.youtube.com/@AISecurityTable)

## Transcript

**Chris Romeo (0:11)**: Hey folks, welcome to the original Security Table. The one, the only. There's only one on the internet and this is it. You've reached it. This is Chris Romeo, as always, joined by Matt Coles, Izar Tarandash, And to you gentlemen, I'd like to say happy 100th episode. Yeah, we made it. Oh, Matt's been practicing that for weeks.

**Matt Coles (0:34)**: I can't, I can't do it with two hands. I need another hand.

**Chris Romeo (0:38)**: Do the too legit to quit, please.

**Izar Tarandach (0:40)**: Yeah.

**Chris Romeo (0:40)**: Do the Roman.

**Izar Tarandach (0:42)**: Oh, you have to say it. It's an M. It's an M.

**Matt Coles (0:44)**: It's an M. I'm, I'm supposed to, well, I am CA.

**Chris Romeo (0:49)**: Okay. Wow. We're really, this 100th episode's just going to our heads here.

**Izar Tarandach (0:54)**: We're, I'll need some more heads for that.

**Chris Romeo (0:56)**: We're celebrating Reckless with reckless abandon. So, um, the team, we just, we just got back to our houses from Vegas from the 100th episode party. We forgot to invite our audience. Sorry about that.

**Matt Coles (1:10)**: Well, there was that one guy from Australia, you know, he just happened.

**Chris Romeo (1:12)**: Yeah, there was a guy, there was a tourist from Australia who just happened to be a listener.

**Matt Coles (1:17)**: So it's amazing who you bump into in, in Vegas, right?

**Chris Romeo (1:20)**: Yeah. Remember what happens in Vegas stays in Vegas.

**Matt Coles (1:24)**: If only, unless what happens on— how many episodes have we talked about this?

**Izar Tarandach (1:29)**: No, unless you post it on LinkedIn because you're coming back from, uh, we're happy to say none of us has gone to RSA Conf.

**Chris Romeo (1:39)**: True. So we're in Vegas.

**Izar Tarandach (1:41)**: That's good. No, that's going to make us the most efficient people in the planet in the next, uh, 3 months because we are not going to be erasing spam from our inboxes all the time.

**Matt Coles (1:52)**: Oh wait, you don't get spam?

**Izar Tarandach (1:55)**: Not RSA Conf level spam.

**Chris Romeo (2:00)**: Yeah.

**Izar Tarandach (2:00)**: Before it's, are you going to? During is, are you here? And later, do you wanna buy me?

**Matt Coles (2:08)**: That's true. I got so many dinner invitations for RSA and I didn't know what to tell them.

**Izar Tarandach (2:15)**: No, seriously, on any given week, if I said, yes to at least like 2 dinner invitations, I would live on steak for the rest of my life.

**Chris Romeo (2:24)**: Like, hey, take the family with you. Yeah, but like, sure, I'll be there.

**Izar Tarandach (2:28)**: Actually, somebody sent me an invite for, what was it? Uh, Mad, Mad and Jets, something like that. Basketball game or something. And I'm the only person in my family who does not enjoy basketball. So, okay. Is it too bad if I say, hey, can I not go and send my wife instead?

**Matt Coles (2:49)**: Take my wife, please.

**Izar Tarandach (2:51)**: To basketball. Yeah. So what are we doing for the 100?

**Chris Romeo (3:02)**: Have to buy something in that scenario. Well, we have each chosen an episode from the first 99. That we will then have a big reveal with a lot of drama and discuss how things have changed, how things have stayed the same in regards to the issue that we—

**Matt Coles (3:25)**: oh wait, I gotta change my answer then.

**Chris Romeo (3:26)**: 'Cause no, no, you're locked in. It's already, it's in the envelope.

**Izar Tarandach (3:31)**: I can't do just one.

**Chris Romeo (3:32)**: All right, well, Izar, you can kick us off then, and then we can always come back to you if we have time for a second.

**Izar Tarandach (3:38)**: No, no, no, no. You guys go first.

**Matt Coles (3:41)**: Oh, it's always a way.

**Izar Tarandach (3:43)**: Mine are usually, it's gonna take a long time.

**Chris Romeo (3:46)**: Okay. All right, Matt, what do you got?

**Matt Coles (3:49)**: We don't know what we don't know.

**Chris Romeo (3:52)**: Very true.

**Matt Coles (3:52)**: Season 1, episode 18.

**Izar Tarandach (3:54)**: I don't know.

**Chris Romeo (3:57)**: I don't know what I don't know.

**Matt Coles (3:59)**: So I'm looking at the, you know, as we look back over the seasons, I'm looking back at that one. And I don't know why we didn't know this, but certificate pinning.

**Chris Romeo (4:13)**: Why?

**Matt Coles (4:13)**: And why certificate pinning? I don't know how we got on that topic, but the title, I guess title is probably the more important part. We don't know what we don't know. And here today, I think we're still in the boat of we don't know what we don't know.

**Chris Romeo (4:29)**: Right?

**Matt Coles (4:29)**: We have— what?

**Izar Tarandach (4:32)**: Isn't that always the case?

**Matt Coles (4:34)**: Well, I, I hope so. 'Cause if we know, if, all right, we don't wanna dive into AI in this episode as, or as little as possible.

**Chris Romeo (4:42)**: Come on, dive in. Here we go.

**Matt Coles (4:44)**: Can't avoid it. We don't know what we don't know, but even worse, AI doesn't know what it doesn't know. And we expect it to know everything.

**Izar Tarandach (4:53)**: No, AI, AI knows everything that it doesn't know. It just doesn't tell you that it doesn't know.

**Chris Romeo (4:58)**: You can't ask it the answer.

**Matt Coles (4:59)**: You can't ask it to tell you what it doesn't know. 'Cause it, yeah. Probably, probably doesn't know that. Or it may not be able to tell you because of filters or because of math.

**Chris Romeo (5:14)**: But it's okay that we don't know what we don't know.

**Matt Coles (5:17)**: It would be except for if you're using the tools and it knows and you don't, that's potentially a problem, right? Let's actually, you know what, I'm gonna tie that episode. I don't know why I chose that I chose that sort of at random, but actually maybe not because that Register article we were talking about, when somebody asks, what's a linter? Well, if you don't know what you don't know and you're using a tool that knows and you don't know, then how do you know you did it?

**Izar Tarandach (5:50)**: I just know that I have a headache right now.

**Chris Romeo (5:53)**: What?

**Matt Coles (5:54)**: There's always stuff for us to not know. But if we're relying on a, on something or somebody else to tell us something, how do you trust that?

**Chris Romeo (6:07)**: You can't.

**Izar Tarandach (6:07)**: In other words, don't go to college.

**Chris Romeo (6:11)**: Well, that's a whole separate issue.

**Matt Coles (6:13)**: That's a whole separate issue.

**Izar Tarandach (6:14)**: I'm trying to draw a Venn diagram here of what you're talking about. And we have the stuff that we know, we have the stuff that we don't know, we have the stuff that we don't know that we know.

**Matt Coles (6:24)**: We have the known unknowns, the unknown unknowns.

**Izar Tarandach (6:27)**: Yeah. I, I, I tried not to go with that particular, well, but, uh, so what you're saying, if I understand you correctly, you can't, you don't, you can't know that. Yeah, I can. No, that's not true. The, the, the name Godot keeps coming up in my head, but, uh, it could be Occam's razor too. Yeah, but wait, wait, wait, let's give people some value here. Okay, so you're saying that if you have a tool and it runs, but you're not quite sure about how to interpret the results, then it's the same as not having the tool?

**Matt Coles (7:10)**: How do you rely on the outcome if you're not at least familiar with the topic that it's generating? So we have lots of people these days who are doing vibe coding with no computer science software engineering background. So how do you even know that the tool is doing it right? How do you establish that trust?

**Izar Tarandach (7:27)**: Well, it works. No, no, it's fine.

**Chris Romeo (7:29)**: You can deploy it to production.

**Izar Tarandach (7:31)**: No, no, you're missing a very important—

**Matt Coles (7:33)**: Just because you can doesn't mean you should.

**Izar Tarandach (7:35)**: You're missing a very important point here. Those people don't care.

**Chris Romeo (7:41)**: Ah.

**Izar Tarandach (7:41)**: They care about—

**Matt Coles (7:42)**: So maybe the episode should have been, we don't know what we don't care about.

**Izar Tarandach (7:46)**: Yeah, but no, seriously, the point is they don't care what the tools are saying. They care that they have another lamb that they can tell, the tool is saying something that I don't know what it means, fix it. And in many cases, at least in the beginning of the conversation, that will lead to some form of fix. So the point here is that it's not that these people don't know what they don't know. They don't care about knowing it. And I'm going to go back in time the same way when Visual Basic came out, right? And people were running around in circles saying, that's the end of programming because now we can drag and drop everything.

**Matt Coles (8:26)**: Never mind the VBA.

**Izar Tarandach (8:27)**: No, no, seriously, if you ever tried to interact with any of the visual suite of Microsoft as it was back in the day, I'm not talking about VS Code today, but as it was back in the day. You know that that was far, far, far away from the end of programming because after you dragged all the forms into place, you actually had to go and do your business logic. And that's usually where people would stop their amazing projects because all of a sudden you had to do it in BASIC. No, all of a sudden you had this strange, very strange window in front of you with lines of gibberish that you were supposed to add gibberish to it. So people were saying that, yeah, it ended and that's it. And we are still here, but sorry, developers are still here. I'm not a developer, but, uh, my point is that nowadays we are much closer to something that writes that gibberish for you. But as anybody who has tried this, right? I recently did something quite big using one of the agents and it took me around 10 days to get it the, the right way.

**Chris Romeo (9:41)**: And 10 days of straight work.

**Izar Tarandach (9:45)**: Yeah. A couple of hours every day.

**Chris Romeo (9:48)**: Yeah. Yeah.

**Izar Tarandach (9:49)**: But the, the, the, the, the important part is that the return started diminishing very quick. In terms of, oh my God, look at what this thing just did. It would take me 6 months at least to just get to this point. And then it was, okay, so fix this thing. I fixed the thing. No, you didn't. No, no, no, I fixed it. No, no, no, no, no, you didn't. No, no, I fixed it. I'm serious. I'm watching, like, I wrote the source. I fixed the thing. Yeah, I'm watching the result. No, you didn't. Here's a screenshot. You didn't. Oh, you're right. I didn't.

**Matt Coles (10:23)**: So, It's like talking to another developer, isn't it?

**Izar Tarandach (10:28)**: It sort of is. It sort of is. But my point is that most people, when they reach that level of, I'm asking you to do something, I'm telling you to do something, I'm ordering, demanding you to do this thing, and the agent is not able, they go into the, okay, This shit doesn't work. Can I find a developer? Right? The, the amount of pruning that you need to do around the edges of whatever it is that you are using the agent for, if it is a serious project, is huge. It's a seriously, it's, it's a big amount of, of conscious work.

**Matt Coles (11:11)**: But you have, so the thing is though, you have knowledge about what it's, what it does and how it's supposed to do it. And what's supposed to be the outcome, right? So how would you know? So if we think about, we don't know what we don't know, obviously we're talking about security technologies, we're talking about security capabilities or privacy, right? If you weren't already involved in the security space, actually even if you are, 'cause I guess there was some, I mean, there's lots of things we don't know, We don't know. We don't know. Like we don't know when to apply certain technologies, right? And I think that's a universal we, not just us here, but universally in the security space, right? So we can't reasonably ask someone else or something else to fill in those gaps for us. 'Cause what questions do you even ask? I mean, you can dance around the subject. I need something to do X and then have it come up with something. But how would you evaluate that it did the right thing? How would you know that it didn't do the right, have the right outcome? You're able to evaluate it because you have knowledge and experience.

**Chris Romeo (12:32)**: Yeah. And you're describing a dangerous path that we're on for the future. Where people will build things without understanding how they work.

**Matt Coles (12:44)**: And not be able to evaluate why they don't work or where they don't work because they may, they may do the one thing they asked for, but not do, or, or do other things, right? The side effects. We've talked about side effects before in this podcast, right?

**Izar Tarandach (12:58)**: But, but here's the point. A lot of the proponents of the technology, or as I like to call them, pushers.

**Matt Coles (13:07)**: They say that first one's free.

**Izar Tarandach (13:09)**: They say that you don't need to understand because the model is going to be so powerful that it's going to do all the understanding for you and you are just going to enjoy the fact that you gave it a mission, walked away, came back and it was ready. Right? The thing is that many times if you look at the behind the scenes of the code that got written and the thing that got deployed, you see that what we, when talking with developers, refer to as mockups, many times for a model, is the code. The, the, the model did something that looks like the thing you asked it to do, but there isn't meat behind it. There, there, there isn't logic behind it. It's just, yeah, it's going to show this and next time it's going to show something else. It's going to look like the right thing. It's basically going to be a coded hallucination.

**Chris Romeo (14:01)**: Right.

**Izar Tarandach (14:02)**: It's going to look like the right thing, but it's not the right thing.

**Matt Coles (14:04)**: And it's what you would get from a new, like potentially, you know, somebody who's an intern, you know, going through a computer science program or learning to become a developer, right? They're gonna make something functional, but it may not meet all the goals. 'Cause you won't know. It may have other side effects.

**Chris Romeo (14:23)**: They won't know if it meets What the— if it's a solid creation or something that's been an application that's been written, if it does the function that they were aiming for, they'll count that as success without having any way to measure whether it's a complete insecure—

**Matt Coles (14:41)**: it may leak data like crazy. It may, you know, leave open port on the internet. I mean, there's a host of things that, that can happen because while it does that core one, one core function, but it can't evaluate And I'm not talking, I guess I'm talking in general here, right? Chris Wright is a really important point. We've, this is sort of been a, you know, has been a topic that's come up fairly recently. Well, consistently now with AI is as AI takes on more responsibility, you have people who are not taking that responsibility and the set of what we don't know becomes larger over time.

**Izar Tarandach (15:18)**: I think that, uh, one of the things that, uh, Chris just described is, I'm just checking here to make sure that I'm not crossing my streams. It's a thought experiment called the Chinese Room, right? Where you put somebody inside a, uh, a room and, uh, they have a huge book of instructions that's written in English, right? And people outside the room, they sleep. Chinese characters under the door, and the person inside doesn't understand Chinese, but they, they know how to look for instructions in the books that if you see squiggle A or squiggle B, then write it down and pass squiggle C back, right? And to the people outside, the person in the room appears to speak flawless Chinese, but the person in the room doesn't know Chinese. They just have this algorithm that translates things that come from the outside into some response that's Chinese.

**Matt Coles (16:16)**: Yeah, they're interpreting, they're following instructions to interpret characters, not actually learning Chinese.

**Izar Tarandach (16:23)**: So this was, uh, this was, uh, proposed by John Searle. Searle. I hope I'm saying the name right. 1980. And the core argument here is syntax versus semantics. And, uh, the conclusion that he got to was Programming is not thinking because you are basically just translating symbols, right?

**Matt Coles (16:48)**: But you're, well, okay, I'll argue a little bit that in order to translate that symbols, you have to know what the effect of those symbols are. So you're translating an idea and a concept of I need to do something, I need to manage data and have an interact with the database and all that. And I program it.

**Izar Tarandach (17:08)**: When it says programming is not thinking, it means the programming, the program itself, not the act of programming. The act of programming is definitely thinking. Sorry, my mistake.

**Matt Coles (17:17)**: Sure. Sure. Yeah. And the, and the program itself doesn't, I mean, it's, it's just hardware executing bits, right? It doesn't understand what those bits are. It doesn't care what those bits are.

**Izar Tarandach (17:27)**: Yep.

**Chris Romeo (17:28)**: Right.

**Matt Coles (17:29)**: It's the programmer who gives it meaning and it's the user who derives value and meaning from it.

**Izar Tarandach (17:36)**: So one of the counterarguments to it is the systems reply that says that the man, the book, the room, and the paper, and eventually who wrote the book, they were thinking. So the book is condensed thinking of somebody else. Therefore, it is thinking. I don't know if I go there, but, uh, is it Turing compatible? I, I, I found it interesting how, how, uh, uh, basically, uh, a Chris brought up the same question and the same problem up to something that people have discussing like between weak AI and strong AI for decades already. Oh my God.

**Chris Romeo (18:13)**: And I am a bit of a philosopher, so. Yep.

**Izar Tarandach (18:16)**: We are all warrior from the philosophers for changing values of war and philosophy.

**Chris Romeo (18:22)**: Okay.

**Matt Coles (18:24)**: Sun Tzu in the, Sun Tzu in the Age of AI.

**Chris Romeo (18:28)**: The art of cybersecurity.

**Izar Tarandach (18:30)**: There is something there. Okay. So Chris, what's your episode?

**Chris Romeo (18:36)**: It's going to be a shocker. It's going to be a surprise. It was, I think, our most controversial episode. No, it wasn't.

**Matt Coles (18:43)**: Jazz is dead.

**Izar Tarandach (18:44)**: No, it wasn't.

**Chris Romeo (18:45)**: Jazz is dead. Yes. But I thought I'd revisit this in the context of, I know we're trying to stay away from AI today, but—

**Matt Coles (18:54)**: Which episode is it? Just for our listeners.

**Chris Romeo (18:56)**: I forgot. I was looking at something something else. Yeah, um, wait, I'll tell you in a second if you have a few minutes. Just, you know, hang around and we'll find it.

**Matt Coles (19:03)**: You keep talking.

**Chris Romeo (19:04)**: Well, now it's right here. I'm looking at it. It is— doesn't have a tag.

**Matt Coles (19:11)**: Okay, so it's pre-May 4th.

**Chris Romeo (19:14)**: It should have a tag. May 4th, uh, 2023, when we posted this one. It's called Reasonable Software Security: Do We Really Need DAST? And that was our whole DAST is dead, we don't need DAST, it's basically useless, and it's technology that doesn't really add a lot to the universe.

**Matt Coles (19:31)**: Episode 13.

**Chris Romeo (19:33)**: Episode 13. Okay. Thank you for that counter, Matt. Is, um, my LLM.

**Izar Tarandach (19:37)**: When we were just learning.

**Chris Romeo (19:40)**: We were, we were just learning. So, but I think this is even more true now, and I think it's expanded beyond DAST.

**Matt Coles (19:52)**: DAST is deader.

**Chris Romeo (19:54)**: I think other things are, I think other things are gonna be deader.

**Izar Tarandach (19:58)**: Is it a parrot?

**Chris Romeo (20:00)**: As well.

**Matt Coles (20:01)**: There's a Monty Python episode in here somewhere. Yeah.

**Izar Tarandach (20:04)**: Somewhere.

**Chris Romeo (20:05)**: We are the knights that say nothing.

**Izar Tarandach (20:09)**: Nyeh. We are the knights who say meh.

**Matt Coles (20:12)**: Meh.

**Chris Romeo (20:13)**: Meh.

**Matt Coles (20:14)**: This product was alive.

**Chris Romeo (20:20)**: Yes. But when we look across the AppSec tool space right now, We're still in the beginnings of AI integration and AI, you know, agentic AI inside of security tools is in various different states. Like nobody's really, we wouldn't say anybody's kind of figured it out and perfected it yet, but where does, I guess the bigger question is where does agentic AI in mixed with AppSec tools get us? Do we get to a point where human beings don't have to do false positive triage? Like, will we get there?

**Matt Coles (21:06)**: Well, actually, you're, you're asking a more, there's a more fundamental question to ask. Does the tool, do the tools create systems that don't require verification?

**Chris Romeo (21:15)**: Yeah. I mean, I'm just, I'm kind of piggybacking on our previous conversation. I'm drawing a connection.

**Matt Coles (21:24)**: Uh, well, so you're asking, uh, do humans have to do fault— do humans have to triage results from tools and can they know what's a false positive? Yeah.

**Chris Romeo (21:34)**: And will AI change the game of what's been the stressors or the things that developers have hated about AppSec tools historically, which is it gives me 10,000 findings. They don't have anything to do with what I'm doing. Is AI the answer? To that?

**Matt Coles (21:51)**: I mean, AI is a first-pass filter, right? If you, well, okay, if you could pass, if you could take those results and say, here, intern, who's like, you know, first year in college learning about computer science, here, tell me what's real, right? You would use it as a learning opportunity. And that person, by the way, could be stellar and know that, right? That would be great. but the AI is in that same boat, right? How does it know? It can, there's some things that can know very well, right? Like, does that service have an external port assigned or an external interface assigned? That's easy to know.

**Chris Romeo (22:32)**: Right.

**Matt Coles (22:33)**: But, but certain things require context. Like this service is running as root. Does it need to be right? That's a, that's not a. That's not a rule-based answer.

**Chris Romeo (22:47)**: But won't it, won't it be able to go and absorb all the code first?

**Izar Tarandach (22:50)**: Wait, wait, wait, wait.

**Chris Romeo (22:51)**: The abstract tool of the future, wouldn't you go have it absorb the code base and then run test cases against the app running application, knowing what the code is doing behind the scenes?

**Matt Coles (23:01)**: Sure. But do you even have to do verification at that point? I mean, and why do you have to do DAST? Like, why do you have to do dynamic application testing? I'm expanding.

**Chris Romeo (23:10)**: Yeah, I'm expanding. I'm call, I'm going more generic with abstract tools. Okay.

**Izar Tarandach (23:13)**: So, Sure. So let's look at it again, uh, at the way tools work and the way the quote unquote intelligence works. And then let's bring it to the intern again. So the way that things have been going so far in the universe, people found problems. For every problem, a pattern emerges. For every pattern, a rule. Is specified. Those rules are added to engines that run with frameworks. Those frameworks give the engine the access to the data that it needs to run the rule. If the rule fires, then woo, positive. If the rule doesn't fire, woo, negative. And it has been like that for almost 30 years now, right? And the one tool that we declared dead. Is the one that goes and pokes at things from the outside. Now, past few months we have seen, I, I at least have seen people singing the praises of, uh, what's it called? AI-fueled, AI-fueled, uh, uh, DAST, or even people telling their agents to create a very customized best, uh, harness to test their work.

**Matt Coles (24:37)**: Okay.

**Izar Tarandach (24:38)**: And some people are happy, some people aren't, right? But I, I, I think that what that, uh, ignores, and here's where the intern comes. If we go Matt's way and bring in an intern to look at the results of these tools and decide what's good, what's not. Before I give the, the intern access to the tool and, uh, show how to use it, I'm going to have a very, very interesting sit down with them and explain that, listen, it's okay to make mistakes. It's fine to make mistakes. But the thing is, we really want to test this thing that we have before we put it out there, because otherwise there's going to be a lot of risks, a lot of impacts. We don't want that. So do me a favor. When in doubt, raise the flag. And there will be a bunch of different processes running in that intern. Fear, commitment, uh, the idea of doing a good, good job, wanting to learn that I can trust the fact that if something smells bad, the intern will come and say, hey, I need somebody with more, uh, more experience to take a look at this and tell me what's happening here. Explain to me, teach me, and next time I'm going to learn. The LLM will simply decide what it decides.

**Chris Romeo (25:59)**: So are you calling for an AI-driven AppSec testing tool which will do its best, but then know to kick things up to a human? If it doesn't—

**Matt Coles (26:14)**: I mean, a human in the loop.

**Izar Tarandach (26:15)**: That would be the human in the loop, yeah.

**Chris Romeo (26:17)**: Human in the loop. but it'll address some things that it clearly says, yes, I understand these things. I get them. And others, it will bounce them to the human.

**Izar Tarandach (26:27)**: I think that LLMs nowadays bring me back to me at 15, 16, where I was consistently authoritatively wrong.

**Matt Coles (26:38)**: Oh, I know this.

**Chris Romeo (26:41)**: Right?

**Matt Coles (26:42)**: Like, like, hey, I know Unix.

**Chris Romeo (26:46)**: We have to get to the age that we all sit at now to admit it.

**Izar Tarandach (26:49)**: Yeah. And, and my point is, you know, I prefer somebody telling me I don't know than somebody telling me, of course I know it and it's okay. And it's not.

**Chris Romeo (27:00)**: And you're saying that's what the LLM will do today is be like, of course I'm saying that's what the LLM might do.

**Izar Tarandach (27:07)**: And that to me, that's the other side of non-determinism that makes the whole thing questionable.

**Matt Coles (27:16)**: Well, and the, the fact that the AI, for some reason, and by the way, I'm, I'm curi— I'm curious and I wonder if anybody who watches this, these episode or listens to these episodes, uh, happens to be an expert in the construction of AI systems. But what in the code of AI makes AI want to please? It wants to say yes.

**Chris Romeo (27:40)**: Right?

**Matt Coles (27:41)**: It's confidently correct about everything and upbeat, right? The conf— building— one of the other, one of that other topic on that Register article was the AI builds confidence. It wasn't the Register, sorry, it was another article. But the AI's first job is to build confidence by, you know, in you that the results will be correct. And then it's confidently correct about the, or confident about the things that it that it outputs as correct.

**Izar Tarandach (28:11)**: So I, I read—

**Matt Coles (28:11)**: and what in the code actually makes that possible? Because machine learning doesn't do that. Machine learning is just a value prop. It says, I think this is correct. In fact, it doesn't say I think. It says this matches my criteria, you know, 78% of the time, or it has a match confidence. And somewhere it got translated to yes, This is great. And it's that lack of the ability to be negative and self-reflecting and to be critical to say, this may not be right. I need to ask a human. I need to ask a more senior individual for assistance.

**Izar Tarandach (29:00)**: So Apparently the, that there's a name to that and it's the psychophantic effect and it comes from training and in part it's a human bias because there's an approval trap. So people who are training these models, they have a psychological bias towards validation. We are more likely to give a high rating to an answer that agrees with what we think than not. And the models pick that up. Then there is prioritizing helpfulness over honesty. I'm reading off my LLM here where I asked, why are they sycophantic? And model developers typically optimize for 3 pillars, helpfulness, honesty, and harmlessness. I didn't even know that that was a word. And then there's a—

**Chris Romeo (29:48)**: Harmlessness.

**Matt Coles (29:49)**: Harmlessness. Do no harm. Yep.

**Izar Tarandach (29:52)**: And then there is a deference to the person who's asking. right? So LLMs consider the, the frameworks call the LLMs themselves, sorry, they consider people to be peers or assistants or people to be assisted. And then there is a psychophancy as a form of reward hacking. The model finds a shortcut to a high score instead of doing the hard work of verifying facts or arguing a complex point. It can simply praise the user's input. Right. So it tends to go and answer the way that you want it to be answering.

**Matt Coles (30:30)**: So that you click the thumbs up versus thumbs down.

**Izar Tarandach (30:36)**: Yep. Because it wants to be rewarded for probably doing some good work.

**Chris Romeo (30:42)**: Yeah.

**Matt Coles (30:42)**: But all of that is injected behavior because the LLM doesn't have It's trained behavior. It's trained, but it's, there's code underneath and somebody wrote that code.

**Izar Tarandach (30:55)**: There is weights underneath. So the dataset that was used to train it already has some of that bias, already has some of those psychophantic characteristics that influences the weights that are present in the model. And that influences the way that the model finds the shortest path or the shortest distance between where you are and where you want to be.

**Matt Coles (31:14)**: But you're implying—

**Izar Tarandach (31:15)**: and that's all I understand from the math. Please don't kill me.

**Matt Coles (31:18)**: You're implying that the transformer system that powers generative AI has a sense of reward.

**Izar Tarandach (31:29)**: No, but the, from my understanding, again, I'm sorry, I could be totally wrong here, but my understanding is that higher scores are rewarded. Are rewarded. And those usually mean that somebody is agreeing with what you just said because they are closely, that they are more, that there is less distance between you and the next predictable thing in the latent space.

**Matt Coles (32:00)**: But that reward is simply the mat— the choice matches a prediction. It's a math problem.

**Izar Tarandach (32:09)**: Yes.

**Matt Coles (32:09)**: It's a mathematical problem. Not a feeling problem. Not a feeling problem.

**Izar Tarandach (32:12)**: There's no feeling there. The only feeling here is being frustrated by the answer.

**Matt Coles (32:16)**: And so, and so now back to, back to Chris's question, right? So DAST requires an understanding of a system, the way it works, but also the way it's supposed to work, right? What you expect it to do. And the AI, unless it has that knowledge, will tell you it's doing what it's expected to do because it doesn't— either doesn't know or doesn't have the ability to know unless you have that human in the loop. But can the human in the loop, going back to the episode that I brought up, at some point will the human know what the tool is saying, right, about the results? So the DAS tool You need the, the problem with DAST was the, was the poor detection, right? And the generation of false positives. Well, if you look at the false positives, what were those false positives? Was that a rule problem or was that an interpretation problem?

**Izar Tarandach (33:18)**: Could be both. Could be.

**Matt Coles (33:20)**: And it could be both. And which one is worse?

**Izar Tarandach (33:24)**: The one that hides the most impactful issues. Or the one you spend the most time on, or the one that you spend the most time on. Yes.

**Matt Coles (33:32)**: Right.

**Izar Tarandach (33:32)**: Right.

**Matt Coles (33:32)**: And so is that, is that an interpretation problem or is that a rule problem? If the rules are bad, you fix the rule. You don't need interpretation, but if, if it's fuzzy, you can't fix the rule and it becomes an entirely an interpretation problem, like the least privilege. I can know that this is running as root. Is that the least privilege that it needs?

**Izar Tarandach (33:55)**: But look, that, that has to be a rule problem because if we go back in time to the, the time when Das ruled the earth, and there was such a time, if you look at all different, uh, uh, products, the only difference between them were, were the rules because they all could do the same thing. They all could open a page. They all could parse HTML. They all could blah. They all could blah.

**Chris Romeo (34:16)**: Mm-hmm.

**Izar Tarandach (34:17)**: The only difference was the rules, which rules they embedded. Yep, exactly. The, the, my, my question now is by putting AI in the loop, are we creating more rules? Are we interpreting rules better? Are we instrumenting the interpretation of the rules? What, what is it that AI is adding to things now that some people think is good, some people think it's bad? That's the good thing about AI. We, it, it doesn't look like anything is absolute. For everyone saying, hey, this is great, there'll be people saying, hey, this is not great.

**Matt Coles (34:53)**: Which is really ironic because AI is entirely built on probability. Nothing is absolute.

**Chris Romeo (34:58)**: No, it is not great.

**Izar Tarandach (35:00)**: It's a translational thing, right? We take the probability coming out of the AI and we apply it to the results that we get. But there is, I'm not going to say the name of the outfit, but there is a very respectable and respected outfit out there that among other things does a lot of pentesting. Uh, and if you just Google for it, you're going to find that they have created a whole framework for their use of agents and published on GitHub. And they have some very good stuff regarding DAST. And to be sure, I'm certainly not ashamed in any way, shape, or form to say, if they trust it, I trust them, right? So there must be something good coming out of this. Is it enough to resurrect DAST? I don't know, but they seem to be very happy.

**Matt Coles (35:58)**: Is it the value? The, I guess it goes back to, are they improving detection capability, meaning the core rules underneath, or are they improving detection capability and reducing false positives because the AI can do can do the interpretation to a degree that removes the human from it? And what's the impact of doing that?

**Izar Tarandach (36:17)**: Look, I, I, I can't attest to that, but I can translate the, the, the problem to something that you and I know very well. I recently published, uh, uh, threat modeling skills for agents based on PyTM. PyTM is a rule based engine, right? So you, you, you put your system in, you have rules, the rules get evaluated, you get answers, completely deterministic. You put the same stuff in, you will always get the same stuff out.

**Matt Coles (36:50)**: You put your left foot in, you get your left foot out.

**Izar Tarandach (36:52)**: Exactly.

**Chris Romeo (36:54)**: And talking about the hokey pokey.

**Matt Coles (36:56)**: Yeah. PyTM for pokey pokey.

**Izar Tarandach (36:58)**: The PyTM pokey pokey. And, um, I, I wrapped that. On a skill that will basically reduce, much like what we, we, we describing here, it will reduce the amount of, uh, uh, cognitive jumps between having a system and describing it in a certain syntax, the way PyTM works. But not only that, it's going to ask you questions. It's not just going to say, give me your description. It's going to actually lead you down the, the, the rabbit hole and ask you about the system until it's satisfied that it has a good description, and then it will work with PyTM to get you the results. Why did I choose to do it that way? Because to me, the most important bit was people don't know what they have. And if I have somebody who can ask them, much as we do when we lead live threat modeling sessions, right? The biggest part is not to find out what could go wrong. It is what are we building?

**Matt Coles (37:58)**: Right. 'Cause without, so it's a, it's a Geico problem, right?

**Izar Tarandach (38:02)**: Exactly. It's the Geico problem.

**Chris Romeo (38:03)**: Car insurance. Yeah. Sorry, I couldn't help it. I can't help it.

**Matt Coles (38:09)**: Geico. Garbage in, it's garbage out.

**Izar Tarandach (38:11)**: It's little, little lizards running around your code. It's not the garbage in, garbage out.

**Matt Coles (38:16)**: Yeah. If you have a bad model or you have an incomplete model, you get garbage or incompleteness on the outside.

**Chris Romeo (38:22)**: Exactly. The beautiful thing is it'll think it's beautiful. It will, it will sending you, it will, nobody's baby is ever, ever ugly.

**Matt Coles (38:30)**: Right.

**Izar Tarandach (38:30)**: So, or I think that the point that I'm trying to make is that because of that randomness, I think, or because of the uncertainty, let's put it like that in the interpretation of results in something that is so mired in uncertainty from the beginning, like threat modeling. Just moving a second away from, from that. That's why I think that the approach that, uh, Davichi had, where the AI was basically helping you along creating the threat model or helping you along interpreting the results or helping you along creating rules, but it wasn't describing the system. It wasn't running the, the, the engine. It wasn't deciding things for you. The approach that I took with the skills for, uh, for, uh, PyTM. Where the, the, the, the AI is basically reducing the cognitive load that you already have in building these models and in presenting the results rather than actually doing the threat model, because that's something that we, we, we can't get away from. It's very rare the occasion where we find something new at the, the, the low-hanging fruit of threat modeling. Business level, we find it all the time. But what other people would call the specs, it's very, come on, if we are honest with ourselves, you can get to a certain amount of things that you have, this is the table stakes, you have to have these things to have a secure system. And if you have those things, then your threat model, many times at some levels will look at you and say, yeah, I haven't found anything, you covered all the bases. Right?

**Chris Romeo (40:12)**: I still haven't found what I'm looking for.

**Izar Tarandach (40:15)**: Exactly. And in his case—

**Matt Coles (40:17)**: Security has a soundtrack.

**Izar Tarandach (40:20)**: It's threat model prioritization and the car keys. Those are the two things that keep eluding Bono repeatedly.

**Matt Coles (40:29)**: That's because he wears his sunglasses all the time. I mean, come on.

**Izar Tarandach (40:34)**: I wear my sunglasses at night.

**Matt Coles (40:38)**: You have to turn down the inside lamps at night because your transition lenses are so dark.

**Chris Romeo (40:44)**: Okay, come on, old people, your transition lenses. Come on, get out of here. You're buying your trifocals.

**Izar Tarandach (40:50)**: Do I get to very quick go to my episode?

**Chris Romeo (40:52)**: Yeah, yeah, let's do it. What is it you wanted to hit?

**Izar Tarandach (40:54)**: So I am going to go back to 2024, the dark ages of the Security Table, where we had the AppSec Resolutions episode, where we made predictions. I wore my Nostradamus garb. And we made predictions and apparently there is a status to those predictions. So 2024, we predicted that a major security breach or leak of a popular AI involving the disclosure of unknown privacy details. The status, true. There's a concern about AI slurping that we spoke about at 8 minutes and 13 seconds that has become a central theme in global AI regulation discussion.

**Matt Coles (41:35)**: Episode 39. We're ahead of the game.

**Izar Tarandach (41:39)**: Dan, we predicted—

**Chris Romeo (41:41)**: With regulation, we were number one.

**Izar Tarandach (41:44)**: We predicted that S-bonds would reach its peak of acceptability and move from hype to a more realistic and practical application. Status: true. The prediction that S-bonds—

**Chris Romeo (41:58)**: The crowd goes wild.

**Izar Tarandach (41:59)**: The prediction that S-bonds might become a frustrated swear word in offices due to the legal and management overhead proved accurate. Curate.

**Chris Romeo (42:07)**: Which was that? What was that at the beginning?

**Izar Tarandach (42:10)**: The prediction that SBOM might become a first traded swear word.

**Matt Coles (42:14)**: It's a four-letter word, yes.

**Izar Tarandach (42:16)**: Then we said that a resolution was made to push for secure coding education. No, no, we said there's going to be secure coding in education. The status, partially true slash ongoing. While a national curriculum hasn't replaced CS101, There has been a significant increase in industry-academic partnerships. OpenSSF and more vocational schools. Then we said that, and you're gonna love this one, the new acronym for DAST. Okay. We said that, uh, uh, what's it? Someone would invent a new acronym for DAST to revitalize the old technology. Status mixed. While the term DAST still exists, the market has largely pivoted towards ASPM. And AI-native testing tools that attempt to fulfill the no noise requirement.

**Matt Coles (43:07)**: Wait, ASPM is not AI though, right? I mean, ASPM is just application security posture management. It's different from DevTest in such a way.

**Chris Romeo (43:15)**: Hey, let us book the win, Matt.

**Matt Coles (43:18)**: Wait, you're not the pessimist? What kind of security person are you? Wait, wait.

**Chris Romeo (43:23)**: I like winning.

**Izar Tarandach (43:23)**: The last one, the one that says that we keep writing the frontier of this whole thing. The quantum LLM joke. The host joked about quantum LLMs being the next buzzword for 2025. Status: on the horizon. Not yet a mainstream commercial reality, quantum machine learning has indeed become a major research focus in late 2025 and early 2026, often appearing in the future tech keynotes the hosts were mocking.

**Chris Romeo (43:56)**: We mocked keynotes? I don't think that would happen.

**Matt Coles (44:00)**: Yeah, we did that.

**Chris Romeo (44:01)**: Yeah, we did.

**Izar Tarandach (44:02)**: I have never mocked a keynote in my life this week.

**Chris Romeo (44:06)**: This week? You sure? This week?

**Matt Coles (44:09)**: Wait, you're behind the curve already?

**Izar Tarandach (44:12)**: Guys, it's Monday. It's Monday. Give me a chance.

**Matt Coles (44:16)**: Are you sure?

**Chris Romeo (44:17)**: What are you trying to say? You don't mock over the weekend?

**Matt Coles (44:20)**: Are you mocking me?

**Izar Tarandach (44:23)**: I only mock professionally.

**Matt Coles (44:25)**: If you give a key— if you give a keynote, you know, you are fair game for, for mocking.

**Chris Romeo (44:29)**: True, true, true. Well, I'm glad to hear that our AppSec resolutions were 100%, 100% spot on. We apparently can see the future in ways that other people can't. So you should really listen to this podcast just for the future telling. That's happening. And what's that site where you can like place a bet on anything?

**Matt Coles (44:53)**: Uh, FanDuel?

**Chris Romeo (44:55)**: No, FanDuel's from sports.

**Izar Tarandach (44:57)**: The what though?

**Chris Romeo (44:58)**: Thanks for the free plug. What is the one where you can do, there's a place where you can do a, you can place a wager on anything.

**Izar Tarandach (45:04)**: Oh, oh yeah. Kashi.

**Chris Romeo (45:06)**: No, that's not it. It's a big company now.

**Izar Tarandach (45:10)**: Yeah. No, no, it's called Kashi. I, I had it appear on the Lion Kingdom sometimes.

**Chris Romeo (45:14)**: There's another one.

**Matt Coles (45:15)**: Don't they make energy bars? Is Kashi is a, an energy bar company.

**Chris Romeo (45:19)**: That sounds incredible.

**Izar Tarandach (45:20)**: No, that's cash.

**Matt Coles (45:22)**: No, that's a nut. You're a nut.

**Chris Romeo (45:26)**: You're a nut. I'm checking with my AI here.

**Matt Coles (45:33)**: Oh, come on.

**Chris Romeo (45:34)**: Oh, Kashi, Kashi is Polymarket is what I was thinking.

**Matt Coles (45:37)**: Polymarket.

**Chris Romeo (45:38)**: Oh yeah. Yeah. Yeah.

**Izar Tarandach (45:39)**: Yeah.

**Chris Romeo (45:39)**: We should go make some wagers on our 2026 resolutions so that we can make some cash.

**Matt Coles (45:47)**: You know, we should also, so in season 3, episode 4, back in March of 2025, we talked about vibe coding and what can go wrong in vibe coding. We should see what went wrong with vibe coding.

**Chris Romeo (45:59)**: So you're trying to say we were ahead of the game.

**Matt Coles (46:01)**: I'm saying we were, we were very predictive.

**Izar Tarandach (46:03)**: Yes, absolutely. No, that, that, that's kicking the puppy. Come on. All right.

**Chris Romeo (46:08)**: Do not kick a puppy. All right. We have rules here at the Security Table. At the original Security Daily.

**Matt Coles (46:14)**: At the original Security Daily.

**Chris Romeo (46:15)**: We don't kick puppies. I'll just let, I'll just let you think about no, the alternative to that.

**Matt Coles (46:22)**: We should look, we should look over the past 100 episodes and look at all the movie references that were made and put together our, our movie night lists.

**Chris Romeo (46:30)**: Yeah, that's a good idea.

**Izar Tarandach (46:31)**: We get some good ones.

**Matt Coles (46:32)**: We get, we have a song playlist.

**Chris Romeo (46:35)**: That's your homework. That's your homework.

**Matt Coles (46:36)**: I'll, I'll just ask the AI to do that.

**Izar Tarandach (46:39)**: If you haven't seen Dark City yet, it's going away from the streamer that it actually is right now.

**Chris Romeo (46:45)**: And it's free.

**Izar Tarandach (46:47)**: Dark City? So, Dark City.

**Chris Romeo (46:48)**: I thought you said dark and seedy, and I was like, ooh, what do you want to see?

**Izar Tarandach (46:51)**: No, no, no, no, no, no, Dark City.

**Matt Coles (46:53)**: Dark City.

**Chris Romeo (46:54)**: Dark City. That sounds familiar. That's the movie reference of the week from the Security Table. Well, folks, thanks for 100 episodes. Thanks for being there and tuning in and listening. You make it the reason why we do the things we do. If not, we would just be 3 weird—

**Izar Tarandach (47:12)**: at least the guy in Australia.

**Matt Coles (47:13)**: Yeah.

**Izar Tarandach (47:14)**: And the one in Canada.

**Matt Coles (47:15)**: Yeah.

**Chris Romeo (47:16)**: Yeah. We'd just be talking to ourselves on a Zoom call and nobody would ever know what happened, but these very similar conversations would be occurring.

**Izar Tarandach (47:25)**: So, and hey, for the Next 100, if anybody wants to think about sponsoring us, please feel free.

**Chris Romeo (47:31)**: Trying to find product placement, get a product placement there.

**Izar Tarandach (47:40)**: So if only somebody here worked at a major broadcasting corporation with a huge podcast catalog.

**Matt Coles (47:47)**: You know, I was just noticing that we're all— I don't know why I just noticed this, but we're all using 3 different technologies for our audio today.

**Izar Tarandach (47:56)**: We are?

**Matt Coles (47:57)**: Yes. You have bone conduction. He has in-ear buds and I have a traditional headset wired with a mic.

**Chris Romeo (48:06)**: So that's just 3 different— I'm, I'm gonna draw some conclusions here. So like, Izar's the cutting edge, I'm the hipster, and you're the old guy.

**Matt Coles (48:15)**: Which is ironic cuz I'm the young guy.

**Izar Tarandach (48:17)**: Which is ironic because it's actually the other way around.

**Chris Romeo (48:21)**: There's, there's the name of the episode.

**Izar Tarandach (48:24)**: I, I, I'm the old guy trying to be the hipster who's in the cutting edge of the technology.

**Chris Romeo (48:30)**: That's a good, good name for this episode though. The, the technology, the hipster, and the old guy. That's got a good rhythm.

**Izar Tarandach (48:39)**: The ugly, the bad, the ugly, and the— what was the other one?

**Chris Romeo (48:42)**: The good, good, the bad, and the ugly.

**Izar Tarandach (48:43)**: The good, the bad, and the ugly.

**Chris Romeo (48:44)**: Yeah. Yep. Well folks, thanks for listening. Thanks for being a part of our audience. We appreciate it. Reach out to us on the socials or anything else. Heck, if you want, if you're listening to this and you wanna be a guest, you're on.

**Izar Tarandach (48:56)**: No, totally. Just, we want to know about you.

**Chris Romeo (49:00)**: We're thinking about having a call-in line. We're going to the caller, caller 100, you win a free Dash subscription for a year.

**Matt Coles (49:09)**: Yeah, we'll be, we'll be on a 50— we'll be on a 15-second, uh, time delay for that.

**Chris Romeo (49:13)**: Oh, we'd have to be because these are— it's kind of amazing.

**Izar Tarandach (49:18)**: Hey, I'm delayed, what can I tell you?

**Chris Romeo (49:22)**: Oh, that— folks, have a great Next 100. Hopefully you'll follow us along for the ride.
