44 minSeason 3, episode 9

Vibe Coding: Can You Put Your Trust in the Machine?

With Chris Romeo, Izar Tarandach, and Matt Coles

Secure software development

We’re discussing vibe coding again and how AI-generated code is reshaping software development. We discuss the trustworthiness and maintainability of AI-generated code, examining the challenges of reviewing and integrating automated changes at scale.

Part of: AI-written code and secure software development

Episode title image: Vibe Coding: Can You Put Your Trust in the Machine?

Listen to the conversation

About this episode

We’re discussing vibe coding again and how AI-generated code is reshaping software development. We discuss the trustworthiness and maintainability of AI-generated code, examining the challenges of reviewing and integrating automated changes at scale. The conversation spans from practical concerns about code quality to broader implications for open-source projects in an AI-augmented world. We talk about identifying telltale patterns in AI-generated code and why context and traceability are becoming essential for trusting automated systems.

Follow AI Security Table:

➜ Home: https://securitytable.ai/

➜ X: https://x.com/AISecurityTable

➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/

➜ YouTube: https://www.youtube.com/@AISecurityTable

Transcript · 44 min conversation

0:12Chris Romeo Hey folks, welcome to another episode of the Security Table. This is Chris Romeo joined by Matt Coles and Izar Terandash. And who knows where we'll go today. It's almost like Maybe we have a plan, maybe we don't, but it will have some huge impact.

0:29Izar Tarandach So you had a completely different plan.

0:32Chris Romeo Yeah. Now for something completely different, which is a famous line from something, can't remember. But Vibe Coding, we're back on this topic again. And I maybe will start with this Hacker News, not Y Combinator Hacker News, the Hacker News. Have to be specific. It's important in this world. So there was an article published on secure coding in a Vibe coding. Vibe, maybe something like that.

1:03Matt Coles I don't know. It was Vibe. Yeah, definitely.

1:05Chris Romeo Secure Vibe coding. How the kids are, how the young people are talking now. I'm a little behind the times.

1:12Izar Tarandach How them youngsters.

1:15Chris Romeo All right. So. This is revolutionary, right? This document. It's just gonna change the world. It's gonna make the AI follow, pay attention and be security conscious.

1:29Matt Coles Actually, if you did, if you did say, uh, when you're vibe coding, pay attention to this oracle when, when you're, when you're, when you're creating whatever it is we're creating, maybe. Somebody should try that, actually.

1:46Chris Romeo Some of it is, is generated by the, like, you know, multi-stepped prompting is a user or human interaction step, right? It's not a something that the machine can do. I guess unless you have one AI talking to another AI.

2:04Matt Coles Well, if you have the— yeah, if you build agents and whatnot, or if you bring in multiple AI, like, hey, when you're if you're talking to Claude to do vibe coding and you want, you know, bring in Gemini, like, Claude, go figure out what Gemini would do and then sort of merge.

2:20Chris Romeo So I'm still back on this vibe coding thing is not for production use. We argued about this on a previous episode, but since somebody brought up vibe coding again as something to talk about, like, this isn't for production.

2:36Matt Coles People are using it for production.

2:37Izar Tarandach Depends on what you call production.

2:40Chris Romeo Production is everything and production is nothing. But that's just, they doesn't, that's a nonsensical statement.

2:50Izar Tarandach No, but seriously, I mean, I mean, depends, depends on whose standards are. I thought that, I thought that Matt was troubling me there.

3:01Matt Coles Sorry.

3:01Chris Romeo He's covered his camera.

3:03Matt Coles I'm figuring out a better camera position here so I'm not like looking sideways.

3:07Izar Tarandach Anyway.

3:09Matt Coles This is my better profile.

3:11Izar Tarandach But no, really. So the question, the question, one of the questions here is what happens to production nowadays, right? So, it's true. I've seen a lot of people who get to a working MVP and say that this is good enough, this is what's going. And then they wait for some kind of events like, oh, manifest, actually put the everything around. Yeah, a lot of people have had some good success with code that is less production hardened than we usually would think to go into production. Hey, it's not for us to judge. People do what people will do.

3:54Chris Romeo So is that code—

3:55Matt Coles We can judge.

3:57Izar Tarandach Oh yeah, true.

3:58Chris Romeo We're security people. That's what we do.

4:01Izar Tarandach Yeah.

4:02Chris Romeo But is that code maintainable that comes out of the Vibe Coding process right now? Like, let's say, I'll give you a scenario. You use Vibe Coding to create your MVP while you're waiting for some financial event, like an investment. And then you get the investment. Can you then hire 10 developers and they can just pick up that code? Or are they going to have to start from scratch and rebuild?

4:26Izar Tarandach So here comes the question. If you're going to hire 10 But why did you call it in the beginning?

4:31Chris Romeo Because you didn't have the investment yet. You didn't have the investment, the money to pay for 10. That's right.

4:39Izar Tarandach The thing is, I think that we all agree that most chances are that the first one that's coming out is not going to be the final version that's going to do everything that you want.

4:51Chris Romeo Sure.

4:51Izar Tarandach So ideally it has to be. —make them code. And I've been having a lot of fun lately looking at things that people say that they've coded and looking at the different inputs that they gave to the system and figuring out what kind of impact guardrails put in the, at the vibe time had on the coding time. And you do get a lot of stuff that's— let's— I think, I think, I think my personal opinion is that there is a lot of value in this type coding thing, but not in the ways that most, that a lot of people seem to be thinking. I'm being very careful with my generalizations. Why?

5:48Chris Romeo Why now? Why are you being careful now?

5:50Izar Tarandach What, what's changed today that, that it's, it's early in the morning on a Sunday. I like, oh, on a Monday. I just don't want to piss off the whole world yet.

6:02Matt Coles So why is that any difference?

6:04Chris Romeo Why are we, why are we completely changing our dynamic of the situation? Wait, wait.

6:08Izar Tarandach I, I was going somewhere with this. Oh yeah. So I, I think that there is some value in, in, in the thing. But for the boilerplate code, right? The stuff that you are more than done writing again and again and again.

6:25Matt Coles What's boilerplate in a wide living world?

6:28Izar Tarandach You know, the stuff that you would go to GitHub and find the ultimate boilerplate for FastAPI. And that becomes like the the source of your, your own thing, because at the end of the day, you know, it already has the login stuff in there. It already has the recovery email in there. It already has the database stuff in there and all that.

6:53Matt Coles Well, so we need to bring in, we need to bring in a new topic. So build versus buy. Oh yeah. And the world of open source, right? So what you're talking about is Well, what's the value of open source in a world of AI and vibe coding? So if in the past you would've used open source as a way to bootstrap your development effort, 'cause, and you would've bought, well, bought, acquired, integrated components, but now why not just let the AI reinvent the wheel every time?

7:31Izar Tarandach So that too goes to maintainability, right?

7:33Matt Coles Well, it does, but what's the decision of the developer who doesn't know that, right? The developer who's using vibe coding, unless they know, oh, uh, make sure that you make use of open source whenever possible so that you don't reinvent the wheel each time. So that's a really good point.

7:49Izar Tarandach And during the weekend, I actually saw a GitHub repository. I think it's called Awesome Windsor Files or something. So Cursor and WinSERP and all that good stuff, you can actually put a series of rules files in place before you start doing your web crawling thing that they sort of like direct the thing around, right? They become part of the context sent in and that influences what you get back. And yeah, some people are having some very good results and some good ideas into what to put on those files. So that it shapes the, the final product. So for example, you could go in there and say, you are always going to use Python and your base framework is going to be FastAPI and you are going to use these libraries and you are going to use Pydantic and all that good stuff.

8:41Matt Coles So they're making architectural choices basically for the, for the LLM before, before the live coding session even starts.

8:49Izar Tarandach Yeah. And what's in the back of my head is we're seeing here an evolution from best practices and what I would do if it was me writing this thing from scratch. Away from that really, really funny video on YouTube about vibe coding that it's talking to a computer. Man, I laughed so hard at that one. But, uh, If it's moving away from the simple vibe code that we all run around headless chickens a month or so ago saying, oh no, my God, the market is going to be inundated by bad code, right?

9:38Matt Coles But, uh, in other words, cooler, cooler, cooler heads prevail and you have proper architecture leading in.

9:48Izar Tarandach I don't know if we can say that for sure, because at the end of the day, nobody's out there saying, here's how you vibe code, right? But it does seem to me that there is a movement towards, let's not start from zero, let's start from good foundations. But I don't know how popular that is. I don't know how that reaches the people who are vibe coding their prototypes and then putting them on production and whatnot. No. So I don't know. I guess that it's coming from good people out there who know what good code looks like and are trying to do the right thing. And thank God that we have those. But another thing that I was thinking about and I wanted to run through with you guys is the thing that I'm struggling with writing down. I have a theory that this whole thing is going to, if it hasn't already, and that's where I need you guys' help, that it's going to lead us to parallel of what you see in the SOC or in people's desktops. And that is click-through and alert fatigue. I mean, because of the people that we run with, I can certainly say yes, I know people who read through an EULA before clicking. But most people out there, I would venture, they don't. They just see a box with an OK button, mash on it, and go play Sonic on their Windows box.

11:33Matt Coles You haven't given up. On reading the same EULA over and over and over again.

11:41Izar Tarandach That's who we are. But even so, even so, I have seen EULAs that have the okay button. But anyway, that's a different discussion. But beyond EULAs, if you have seen or worked with people in a SOC, you do know the amount of alerts they, they get. And sometimes, unfortunately, what can you do? You mash through alerts and mash through messages because your brain already does the pattern recognition and you see that it's a false positive or no, you've seen this thing before and it's not for now, or you do your, your prioritization and that's not something that you need to jump at now. And taking an idea from that model, from those two things, and then putting together the fact that, and I read this article on mental models while developing that I found super interesting, and it was basically saying that Reading code is harder than writing code because when you are reading code, you have to build a model of the system in your head as you go. While when you write code, you have that model in your head and you are translating it into code. So you are bringing it to fruition.

13:14Matt Coles Developers have a model in their head of what they're working on. Basically. And— Beyond their, beyond their, beyond their single file? Maybe beyond the module that they're working on?

13:28Izar Tarandach Beyond something. They have some kind of structure and they are translating that structure into reality, into code. While when you're reading, if it's not in the context of that thing that you are building, then you have to build those structures in your head and put everything into context. And my God, I'm overloading the word context so much. You have to build that thing as you read to put that code into the right place, into that right structure to decide if this is good code or not. So my contention is that over time, people are going to use agents, live coding and whatever we call it to create code. As Chris pointed at earlier, That code is going to be corrected as it's written. So hopefully it comes out good. And by good, I mean performant, secure, and actually does the thing that it's supposed to do. But I think that as these systems evolve and people probably don't have the money yet to get those 10 developers. Did we talk about the 10 developers here or was it before we started? The recording.

14:39Chris Romeo That was, that was about 5 minutes ago.

14:43Izar Tarandach Fog, fog. Wait.

14:48Chris Romeo This moment brought to you by Yzhar's favorite drink.

14:51Matt Coles His Exploding Monster.

14:53Chris Romeo Oh, hey, you're not paying us any money. I didn't use the brand.

14:56Izar Tarandach Wait, wait, we have to get some. Sorry. Well, good. Monster, I suppose. So, um, monster.com. As I was saying, so people where we use like agents and whatnot instead of those 10 developers to fix the code that was initially written. And when, what they're going to get if they don't just set it for automatic and go home, what they're going to get is a set of PRs. And I think that over time we're going to end up with something like PR fatigue, and people are going to click through PRs assuming that the agent knows what it's doing, which was—

15:37Chris Romeo I think that's the scary part, what you just described. And just to summarize, so you said it's more difficult from this point forward to read code than it is to write code. The challenge is we haven't introduced the word trust. Into this conversation yet. And if you can't read the code, if you don't, if you can't process and read the code that's coming out of the machine, then we're in big trouble because you have no idea. You're just, you're just putting your, your trust into the machine and saying, okay, it's, I, I believe that ChatGPT's Codex that I was playing with here while we were talking, or, you know, they all have one. Everybody's got an agent that does this stuff now, right? Yeah. You're inherently trusting that this thing is creating the code in the way you described it— performant, secure. I'll even add focused on privacy of the end user, compliant with laws and rules and regulations of the world. We've got the EU CRA that's dropping now. It's about to flip the whole, the whole, our whole world upside down. Um, but how do you trust, like, how can you trust something that you can't understand what it's giving you? That's the scary thing.

16:57Matt Coles Especially as a vibe coder, you don't have a computer science background. You don't have a developer experience, most likely, right? If you're, if you've got, if we have successfully democratized development to a point that anybody can do it, those anybodies may have no experience in actually doing software engineering. Or system design or development. And so how do you trust? Like, what's your basis for trust? Because you don't know how to read the code you're writing in the first place. Like, you couldn't do it by hand, right? It's taking this to an extreme. I know most of the divide coding is probably done by developers who are working on a project, you know, either for an open source project or for a commercial organization. And are using vibe coding because they can be more efficient or more effective.

17:47Chris Romeo But I don't know that it's, I don't know that it is that I've heard of, like, it's, it, it, uh, in the market, I was reading something. I don't remember where it was that the marketing teams are some of the earliest vibe coders because they got tired of waiting for, and one of you guys might've said this in the previous episode. That sounds scary. Yeah. They got tired of waiting for developers' priority to catch that for their, their features to be caught up and to cut, to go to the top of the stack. So they started vibe coding their own solutions to their problems that they had.

18:21Matt Coles And there, and there, and there's, so let's talk, think about that. Just think about that scenario for a moment, right? So you have a, you have a mark, somebody in marketing who wants, needs something, needs something done because of their customer is, you know, pushing for it.

18:34Chris Romeo They have a sale on the line or whatever.

18:37Matt Coles So they write code. They don't know what they're getting out, right? They have to rely on others potentially to help them understand or to rely on the agents to be omniscient and be able to produce good quality code, whatever good quality code in their context. But if they get the PRs, they don't know what to do with the PRs.

19:01Izar Tarandach Wait, wait, wait, wait, wait. I, I, I have to ask for clarification here. So these marketing people, they, they are starting from zero or are they adding features to an existing system?

19:12Chris Romeo I thought what I, what I was reading is they were trying to solve their own business problems that required some development. So they weren't coding features for an, for a product. They were like, hey, I really need this data munged into this format and nobody in engineering or IT is available to help me. So I'm going to use Vibe Coding to write a script. To munge my data from format A to format B.

19:35Izar Tarandach Right, because those are two pretty different problems. Yeah, they are. But, uh, just to pull back to my thing, because basically what I'm looking for is feedback before I write it down, and I didn't feel like sending a Google Doc for it to you guys.

19:54Matt Coles Well, and you didn't want to ask, you know, what are your favorite LLMs?

19:58Izar Tarandach Oh yeah. So, uh, so, uh, so then my question is, do you guys think that this PR fatigue is going to be a thing or not based on what— and do you think the fact that we have tools coming out that declare that they are going to look at my coded code and fix it with PRs, automatic PRs, which may or may not be accepted automatically. Question. Wait, wait, wait, wait, wait, wait. Okay, finish that. Do you think that these tools using different models from the one that actually started with the coding Is that enough to start getting the trust that Chris mentioned? Or are we just making things more complicated than they actually need?

21:03Matt Coles Before we answer that, because I think the answer is B, the, how does, what does vibe-coded code look like? How do you know it's Fybe-coded code?

21:20Izar Tarandach It has a lot of em dashes. It uses the word Delve as the name of a variable a lot.

21:27Matt Coles It has, it has too, it has too many R's. It has too many R's for the length.

21:31Izar Tarandach It begins with import in an era of—

21:38Matt Coles I'm a little teapot, short and stout.

21:43Chris Romeo That's a good question though. Does it have a pattern? I haven't played around with it enough. I'm guessing it does. I mean, does it have, is it, it would seem that it would be more bloated. It probably would.

21:56Matt Coles And I guess this goes back to the use versus buy versus build. If you're, if you have, if you don't, I guess there's common tricks and techniques and constructs for a way to write things like loops.

22:09Izar Tarandach I'm going to apply the scape hatch of 9 among 10 people who deal with this kind of thing and say, it depends on how the model was trained, man.

22:23Chris Romeo You need to put that on a t-shirt. That's a t-shirt.

22:26Izar Tarandach I do have a similar one.

22:28Matt Coles Even basic things like loop optimization, right? There are ways to optimize loops. Right. And, and there's choices to be made on loop, you know, in the past, at least there used to be choices about loop optimization, which, do you make it more, more verbose so that you could do less, um, you know, compiler optimization, or do you make it more compact and let the compiler figure out the loop? But those sort of structures you're not gonna be able to rely on, I think. Right. There's no, there's no, nothing in, in that will tell you whether it's bytecode or Or not. He'd be relying, you'd be relying on something else to indicate that it's a live feature.

23:12Izar Tarandach Um, and I'm sure that there will be telltale. Okay. Because again, because of the way that these things work, right? At the end of the day, it's a bunch of monkeys throwing coins in the air and statistically telling you what the next token is. That's true. There will be patterns. There will be stuff coming out. And again, it depends on how you were trained, right?

23:33Matt Coles What patterns do you think? What patterns do you think you would be looking for?

23:37Izar Tarandach Just if you could— Very optimized idle loops.

23:43Chris Romeo Uh, so I just asked AI this question because why not? We should, of course. Oh, I didn't know that AI was listening.

23:49Izar Tarandach Oh, sorry, sorry. We didn't mean it.

23:52Chris Romeo It admits very Vibe code reflects what the AI infers from natural language prompts. It may create unconventional structures, inline logic, or variable names that feel right but aren't standardized.

24:05Izar Tarandach So if you don't see foo, bar, or baz, it was vibed code.

24:09Chris Romeo That's true. We should write a— we should, we should vibe code a tiny script that checks to see if something's been vibe code.

24:19Izar Tarandach I don't know, ever since I started drinking Monster years and years and years ago, I always felt like I was vibing like Goldie all the time.

24:27Matt Coles But, well, actually, so actually there is a way probably to do this. If you think about, because vibecoding systems are, well, LLMs are trained on preexisting content. If you utilize, and this is, I think what they're doing for um, like college courses and things, you know, papers, if you use an anti-plagiarism system, right, where you're looking for plagiarism, you're looking for pre-constructed pieces of snippets of code. If you see pre-construct— now the problem is if you see pre-constructed code, it could come from Stack Overflow or it could come from GitHub also. So how do you know it's been vibecoded versus developers copying from online sources?

25:13Izar Tarandach If it comes from Stack Overflow, the first line always says, you're wrong and you're an idiot.

25:18Chris Romeo True. Even on the best possible answer that really focuses on security.

25:23Izar Tarandach It's walk away from the computer.

25:27Chris Romeo So coming back to your PR fatigue question, I do see that we could enter a world where there is PR fatigue if we've got all these tools. That are vibe coding features for us, and then we've got a separate set of tools that are finding bugs and generating PRs to fix them. We could reach a world where there's hundreds of PRs being generated a day and people just start approving them. They, they, I think it all comes back to that trust problem. What's going to push people to hit the accept button instead of doing a tight code review of it is the trust. And if people trust the output of the AI, or as the trust meter goes up, people are going to click accept PR without looking at it more and more.

26:22Matt Coles And— I'm actually, I'm going to take the reverse approach. Accept is the easy answer. And so whether or not you trust the AI, you're going to click accept because as a white coder, And I guess as the bulk of web coders, maybe not developers who are doing web coding, the bulk of them may just, may just trust. There may be implicit trust. And as you learn more about how to make a good informed trust decision, you may not do the click-through because you'll start to, because you'll start to gain the experience to be able to say, oh, that one looks fishy.

27:02Chris Romeo Right.

27:03Izar Tarandach So let's throw a stone in the pond here. Sure, why not? Who is responsible for identifying the problems that get solved by a model-generated PR? In other words, if we say you need a man, you need a person in the loop to accept the PR. Some of the problems that started the generation of that PR will come from a person that identified a problem and said, this needs solving. Others will come from the tool itself that read the code and said, there's a security problem here. I have to solve this. So assuming that person in the loop is actually a value and, and people will stand for that and do the work. Is it inherently easier to read a PR that solves a problem that a person identified or that a machine identified? How do you know?

28:09Matt Coles Or it makes no difference at all. How do you know that a, that a issue, so now we're going back to the issue, not the PR. So how do you know the issue is identified by a human versus not? True. What information do you have that says Bill found this versus LLM Bill?

28:31Izar Tarandach Oh, you get that on the PR author. It's going to identify itself as either a person.

28:38Matt Coles No. You're really stretching this.

28:41Izar Tarandach On the issue. On the issue.

28:42Matt Coles On the issue. On the issue.

28:43Izar Tarandach Yeah, sure.

28:45Matt Coles The report may be a form. It may be a form that gets filled out, right? That gets automatically generated by a service agent. Or by a service. It's just, I mean, it could be a traditional service or it could be an agent.

29:00Izar Tarandach It could be a step in the CI/CD, right? Right. Oh, but that's the point. No, no, no, wait. If it's a step in the CI/CD, then yeah, it's going to come from something automatic versus something that is a person who will have an actual user there identifying the issue. Oh, okay. I see.

29:18Matt Coles So, although, although AI user, AI agents who are acting as users would be users, right?

29:27Izar Tarandach But they would have their own identities. And now we get to the whole non-human identity discussion. That's a great episode for the future. But anyway, so, uh, uh, In my head, it goes again to the mental models, right? If it's somebody that identifies something that needs fixing, chances are that in the course of identifying that, they already built that initial picture in their head of what's wrong, why is it wrong, what should we do about it. And if it's an agent, then of course there isn't, or he won't get it as kind of the PR. So is there a chance here for an advanced fixing agent that not only keeps the PR, but in the comments explain what FPR, what they, I don't want to use the words think or imagine, why they built the PR that way, like what's, chain of thought that led to the PR being that way? Would that solve the problem of, uh, hold on, Dr.

30:42Chris Romeo Evil's just entered the chat room.

30:55Matt Coles There we go. We got our cat on the internet fixed.

30:59Chris Romeo Completely derailed the episode right there.

31:06Izar Tarandach So what we're talking about is— I was asking—

31:10Matt Coles I think you're doomed one way or another. There's so much— Well, I think there's so much problems in this, right? As agents get better, they're going to become more of a human-like identity. And individuals who are doing vibe coding are not in a position or probably not in a position to understand the differences or appreciate that— not understand, appreciate the differences between human and non-human generated issues and human and non-human generated code changes, PRs. To make a meaningful difference. It depends on what their goal is. If their goal is get this out the door, make a prototype so I can get funding, or so I go off and do something else, or so my friends can play, you know, play some game or something, uh, or whatever, you know, they may or may not care, and they may just say it sucks. Versus those who are doing this for a living and understand how systems work, they're going to take the time regardless of its source. And, and yes, I understand, you know, the SOC example is probably a bad analogy because when you're dealing with a SOC, those are, you may have thousands of alerts streaming at you, right? They're going to be small events, right? They're not an incident declaration. They're events, you know, this file read occurred or this, you know, this port was accessed with this data. A PR is a block of stuff with intent and meaning. And, you know, if you get 1,000 of them, by the way, if I was working on a software project, and let's just say I was working on a mid-sized software project, so I have, there's a few developers on the project, If 1,000 PRs come in, in a, in a group, or even if they come in like one a day for like a regular time period, that may trigger a different response than just simply looking at PRs, right? Because now you have a flood of regular incoming content. What happens?

33:31Chris Romeo So we just invented a new category for startups. It's a tool that does PR summarization at scale, and then an agent, the agent decides whether to approve or reject.

33:47Matt Coles Can you fine code that?

33:49Izar Tarandach I already did it. Somebody already—

33:51Chris Romeo It's already available for purchase.

33:53Izar Tarandach No, but, but really, uh, I mean, I wouldn't be surprised if one of the, the many correctors, let's call it like that. Out there doesn't already have the functionality, right? But I'm thinking about the three of us have a lot of experience, not only doing code reviews, but watching people do code reviews. How many times did you see someone look at code and say, that's good, move on? When you were like, wait. That's a bunch of loops doing arithmetic on pointers. Perhaps we should sit down and really read that code line by line. How many times have you seen that? And isn't that a kind of fatigue? No. Am I identifying something that already exists?

34:48Matt Coles I don't, I don't know if I would classify that as fatigue. I would classify that as hubris.

34:55Izar Tarandach Interesting.

34:56Chris Romeo That's a good way to describe it.

34:58Izar Tarandach Yeah. Interesting. Cockiness.

35:01Chris Romeo Oh, I just processed that loop and I understand exactly what it does. Move on. Next. Yep. Yeah.

35:11Izar Tarandach Or the same loop. And that's a different developer.

35:14Matt Coles That's a different developer at fault, right? Yeah.

35:18Chris Romeo I want to come back to something you said, Izar, when you were talking about this advanced agent. Like perhaps we needed an agent that was advanced beyond the other things.

35:27Izar Tarandach There we go. He's starting another startup.

35:29Chris Romeo Yeah, here we go. But the trace of, like, I would, I would, or I expect traceability of any coding agent that exists today to be a core basic feature, meaning I should be able to understand how it got to the, why it did what it did, and I should have an audit to be able to go back and and say, you're laughing because it doesn't exist, but this is what I demand. No, no, no, no, no, no, no.

35:54Izar Tarandach I'm laughing because I think that's adorable that you think it should exist.

35:59Chris Romeo It should exist. Of course. You should be able to, you should be able, if we're gonna, if we're gonna rely upon vibe-coded and code, I shouldn't, we should, we should extend the definition to agent-generated code. 'Cause there's a lot more people that are using AI coding tools now that are developers using them, right? That aren't bots.

36:21Izar Tarandach I think that the word that they use is 10x.

36:23Chris Romeo So 10x accelerated code. There you go. It's, yeah, negative 10x, but that's a different, different conclusion. But look, for this thing to really work, to be able to trust the code that comes out of it, you have to have traceability to be able to understand like what was the prompt that caused it to generate that particular code in the way it did. And if you don't, if you don't have that, then you have, then you have no security architecture of something building with these tools.

36:55Izar Tarandach Well, some agents you can point to some piece of code and say, explain to me what's happening here, right? And you are going to get a pretty good explanation to the point that some of it you could cut and paste into a a documentation doc.

37:10Chris Romeo But I want the intent of the agent. That's what I want to be able to, as a security person, I want to be able to understand why it broke, why it screwed it up.

37:21Izar Tarandach From the little that I understand, and I could be wrong, I'm probably wrong, I don't think that the model even remembers how it got there. It's building that model in its head and forgetting at the same time, building and forgetting, building and forgetting.

37:39Chris Romeo No, it's, they're storing everything now.

37:41Matt Coles The reasoning and thinking models have memory. Now, how far that memory goes, I mean, you have, you do have problems, I think, with the number of tokens you can store in memory, so you might lose some of that context. But, so actually, Chris, are you talking about Are you talking about what the developer originally asked for? Yes. When coding? Yes. Well, so let's, uh, maybe take a brief moment to talk about that. If you're developing something for a commercial organization, you have a PR, you have a, not an issue, but you have a work, work request, work item. A Jira ticket or something like that, which is why you're doing the vibe coding in the first place as a developer. But otherwise, I don't think you ever get this context, right? If, like, if you're in, if you're looking at open source code, the open source developer had an idea in mind, a concept in mind, and out came code. Regardless of whether anybody bytecode it or not. So even still, you're, you're still, you're still required to, when you're doing the code review, like if you're reviewing open source code and you have an open source project, that project may be thinly described as to what its intent was or what its purpose is, what its purpose is, not necessarily what its intent was, what its purpose is. Right. And, and so you're sort of, stuck with deriving or understanding that meaning from the code and, oh, this looks like it's gonna be used in, I don't know, it's supposed to be implemented in hardware or it's supposed to be for IoT only and not for, you know, commercial applications. I mean, there's a host of things you have to do there to figure that out, right?

39:36Chris Romeo What I'm realizing is that the argument that I've been building that I didn't really understand until this moment, until you crystallized it for me, is using an open source project as an example, open source of the future is bigger than just the code. It's the, to be able to trust it, I need to understand, yes, the code will speak for itself, but if the code is so complicated because it was created by AI agents that I can't, that there's things hidden because it used some crazy logic that it thought was optimized to go faster, but it's really hard to read and understand as a human. The context, coming back to Izar's word he's used 100 times in this episode, the context of the, how the code was created becomes part of the open source project and my ability to trust it.

40:28Izar Tarandach I don't think so. Just look at what's happening today, right? Developers just download the first thing that Apparently does the thing that they need to happen and happily move on. It could be written in, in, in Fortran with, uh, brainfuck and, and, and whatever. The important thing is that you put in the thing that you have and you get back the thing that you want. I don't think that all of a sudden they're going to start treating it differently because it was written by AI. If it, if at most they're going to think, oh, it's probably better because it was written by AI.

41:04Matt Coles Well, that's an inverted, that's an interesting trust model.

41:07Chris Romeo Until we have the first named vulnerability as a result of AI agent making, committing a PR to an open source tool.

41:18Matt Coles We may have had that already. We don't know that, but that's—

41:20Chris Romeo It hasn't been publicized as such. You think it would be, if it happened and somebody thought it, they would have sounded the trumpet because it would have gotten a lot of attention to be—

41:31Izar Tarandach No, I'm thinking about that Echo, Echo whatever. Uh, you guys read about that one from last week? No. It was something related to MCP and at the end of the day it was pretty silly but— And it did use prompt injection and Echo something.

41:52Matt Coles So maybe this is a call to action to our are dozens of listeners, at least, that if, if you are aware of an AI-committed PR to an open source project that has resulted in a vulnerability appearing somewhere, put the, put it, put the CVE in the comments.

42:13Izar Tarandach EchoLeak.

42:15Chris Romeo Oh yeah, EchoLeak, I saw that one.

42:17Izar Tarandach CVE-2025-32711.

42:21Matt Coles All right, we've just—

42:22Izar Tarandach CVSS score of 9.3, everybody panic, everybody panic!

42:26Chris Romeo Run, run! Wait, come back, come back!

42:29Matt Coles 9.3, so network accessible, privileges required, high, high, high.

42:37Izar Tarandach What you just saw was Matt's LLM that he has behind his ear, takes the CVSS number and explodes the vector in real time in front of everybody. As you can see, it can be done. Is it useful? No. Inquiring voices want to know. All right.

43:00Chris Romeo Well, I think we've had enough fun. This is enough fun for one day. We can, uh, I'm sure vibe coding will be back again within the next couple of weeks. Keeps kind of circling into our view. But for Matt and Isar, I'm Chris Romeo, and thank you for tuning into the Security Table.

Transcript supplied by AssemblyAI. It may contain transcription errors.

Originally published as The Security Table. Part of the AI Security Table archive.