49 minSeason 3, episode 13

Decoding Mastro: AI Threat Modeling

With Chris Romeo, Izar Tarandach, and Matt Coles

Threat modeling & security architecture

We’re discussing the article, “Agentic AI Threat Modeling Framework: Maestro published back in February of this year on the Cloud Security Alliance blog. We discuss the various layers, patterns, and threats outlined in the framework, comparing it to existing methodologies like STRIDE and PASTA, and evaluate Maestro's structure, its potential complexity for developers, and its overall practicality and usefulness in the threat modeling arena.

Episode title image: Decoding Mastro: AI Threat Modeling

Listen to the conversation

About this episode

We’re discussing the article, “Agentic AI Threat Modeling Framework: Maestro published back in February of this year on the Cloud Security Alliance blog. We discuss the various layers, patterns, and threats outlined in the framework, comparing it to existing methodologies like STRIDE and PASTA, and evaluate Maestro's structure, its potential complexity for developers, and its overall practicality and usefulness in the threat modeling arena. Listen along as we unravel the intricacies of the framework and share our candid thoughts on its strengths and weaknesses.

Agentic AI Threat Modeling Framework Maestro



Follow AI Security Table:

➜ Home: https://securitytable.ai/

➜ X: https://x.com/AISecurityTable

➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/

➜ YouTube: https://www.youtube.com/@AISecurityTable

Transcript · 49 min conversation

0:08Chris Romeo Hey folks, welcome to another episode of the Security Table. This is Chris Romeo joined by Izar Tarendash and Matt Coles, the keepers of the Security Table. Not sure what that really entails, but sounds mythical and like it was derived from a fantasy. Novel somewhere.

0:26Izar Tarandach It, it, it's in the back room right here and we just roll it out once a week.

0:31Chris Romeo That's how we keep it.

0:32Matt Coles Where, where, where's our sword? Where's our swords and shields?

0:36Chris Romeo Uh, I thought we were, yeah, I, it's in my contract that I will specifically be issued a sword that is sharp.

0:42Izar Tarandach We Knights of Security Table, we dance whenever they boo with the routines.

0:47Chris Romeo We have a theme song.

0:49Izar Tarandach All right.

0:50Chris Romeo We're just gonna need, can you record that?

0:52Izar Tarandach Actually, we stole that one, but yeah.

0:54Chris Romeo Yeah. Could you just record a version of that? Or I can just, I can just generate one with my AI voice.

1:00Izar Tarandach Did you guys see the, the, the OWASP songs in the Leaders channel generated by that, uh, AI engine?

1:06Matt Coles I don't wanna know.

1:07Izar Tarandach There is some pretty good stuff there. Really? Really. Did they let it analyze the Leaders list so that it could find the equivalent of, uh, of, uh, listening to Eye of the Tiger on the way to the gym? Like you really wanna get that thing secure. No, really, some good stuff there.

1:26Matt Coles By reflection, there's also some really bad stuff in there.

1:32Izar Tarandach I'm not going to say it's a zero-sum game, but it's—

1:37Chris Romeo Are we talking about the OWASP News channel or the songs?

1:41Matt Coles I was talking about the songs.

1:44Izar Tarandach Yeah, the songs. Talk to me about the songs. Totally.

1:46Chris Romeo Totally. Yeah.

1:47Izar Tarandach Yeah.

1:48Chris Romeo Yeah.

1:48Izar Tarandach Not a word. Hey, which reminds me, it's, uh, OWASP, uh, board of directors election time again.

1:55Chris Romeo It is.

1:56Matt Coles Yeah.

1:57Izar Tarandach Matt, did you?

1:59Matt Coles No. Matt, did you?

2:02Izar Tarandach Well, now you can't anymore because it's over.

2:05Matt Coles Oh, it closed?

2:06Izar Tarandach It's locked in.

2:07Chris Romeo No.

2:08Izar Tarandach But Matt, did you?

2:09Matt Coles No.

2:10Izar Tarandach Matt, did you?

2:12Chris Romeo This is getting weird.

2:15Matt Coles And if, if somebody wants, if somebody wants to, uh, put my name in with, I'm looking for my mic. I can't, I can't, I, I do, I like you too much for that, but I'm sure it will be a great slate of candidates this year.

2:33Chris Romeo Yeah, me too.

2:36Matt Coles We sound so committed.

2:37Chris Romeo Well, it's early in the morning here. The eastern coast of the United States.

2:43Izar Tarandach Okay.

2:44Chris Romeo We get, we get, we should dive into this because this is probably gonna take a little while.

2:49Matt Coles Oh yeah.

2:50Chris Romeo The topic of the day, and we'll put this into the show notes, is Agentic AI Threat Modeling Framework, Maestro. And this was published on February 6th, 2025. I cannot, see this name without thinking about the character from Seinfeld, the Maestro.

3:12Matt Coles Yep. Or Sorcerer's Apprentice.

3:16Chris Romeo That's even better. That's, that's even more unique of a pull there, right? Okay, so Maestro, Maestro. It's, it's a, uh, AI. Yeah, AI. Well, I mean, every time we have another framework, and I'm gonna save my, I did a little bit of digging into this, and I have a conclusion that I think I can draw, but I'm gonna kind of let it be more of a crescendo, trying to work in Maestro-related terms. I don't have a lot of them to work with, but crescendo is one of them. Okay, so starts out with a comparison of existing threat modeling frameworks. And every time I see this list, everybody uses it. I'm like, do you know anyone that uses Octave, Trike, or Vast? Anyone?

4:08Matt Coles I know people who use Vast.

4:10Izar Tarandach There's a vast community that got on their trike and just Octaveled out of here.

4:16Chris Romeo Okay. So, all right. Octave and Trike. Do you know of, I don't, I do not know of anyone who's using these.

4:22Matt Coles Octave is an old ancient system that is probably used in some regulated environments and and, uh, particular industries, probably just because they haven't changed in X number of years.

4:34Izar Tarandach That, that sounds like a promo for Stargate. It's an ancient system that's used only on Kulos.

4:41Matt Coles Well, remember, OCTAVE is a giant acronym, right? But it's not a threat modeling acronym.

4:47Chris Romeo Yeah. It says organizational risk.

4:50Matt Coles Yep.

4:50Chris Romeo That seems like something that's very painful.

4:53Izar Tarandach But, but, but let's be positive here. It does not say dread and it does not say attack trees.

5:01Matt Coles Or CVSS.

5:02Izar Tarandach Or CVSS. So let's be positive here.

5:05Chris Romeo True. CVSS, why does that always show up as a threat modeling framework?

5:10Matt Coles Just for completeness, Octave, there's operationally critical threat asset and vulnerability evaluation.

5:17Izar Tarandach Yeah, sure, why not?

5:19Matt Coles Which is interesting because that sounds a lot like The acronym for pasta.

5:24Chris Romeo Well, so Stride, Pasta, Linden are the first 3 on the list, which I was just going to let those be, um, as you know, it's, it's worth when you're building something to compare it to what exists in the, in the, uh, known universe. Um, what you would— I think, I think what you would expect for strengths and weaknesses for Stride, Pasta, and Linden against what has been created here is Maestro, you know, basically saying Stride's great, but it's too simple. I'll skip over the PaSTA one. And Linden is about privacy, which privacy has connective, connective, uh, pieces into agentic AI, but Linden wasn't created for AI. It was created for a kind of base layer of, of privacy, right?

6:10Matt Coles Yeah, but more importantly, Linden and, and Stride especially, and to some extent PaSTA as well. Are general purpose threat modeling approaches, right? They can be applied, they can be applied to any type of threat modeling, whether you're doing, whether you're doing process, you're doing systems, you're doing privacy, you're doing security, you want to do both at the same time, right? You can apply these in a general purpose way. Uh, if you're doing space threat modeling, you have space systems, you're doing aeronautics, you're doing trains, ports, whatever, you can apply these approaches. They work better for some things than others, obviously, because of the awful literature that's available, but the, the approach of threat modeling that they follow, and really STRIDE and Lindem are not really methodologies, more so they're threat categories, right? PASTA is actually a methodology and not a threat category, set of threat categories, right? So, but these are general purpose systems that are methodologies that can be used. You know, pair them with, say, STRIDE for element or STRIDE for interaction or the 4-question framework as a methodology for applying the threats, they're general purpose. I think what CSA, well, what CSA did, not what I think, but what they did was they created a domain-specific threat modeling approach, kind of like a domain-specific language.

7:31Izar Tarandach So here's the thing. We, we, we, we keep moving between those terms, and, and I think that you addressed it perfectly. If you look at Stride, Stride is a categorization of threads. If you look at Stride per LM, sorry, Stride per element.

7:47Matt Coles Not per LLM.

7:49Izar Tarandach Let's not go there. No, actually, let's in a bit. But Stride per element, then it's the application of Stride in a certain way over elements of a system. So one is talking about a categorization. The other is talking about how to apply that categorization. So we are jumping from one thing to another thing. You have a catalog of things and you have how to apply the catalog of things. And somehow we ended up calling both of those things methodologies, which I believe is wrong.

8:27Matt Coles Yeah.

8:28Izar Tarandach Now we come to Maestro. And I look at my astro and I'm asking myself, what is it? Because it confuses me.

8:40Matt Coles Well, so—

8:41Izar Tarandach Is it a catalog? Is it a methodology? Is it both? Is it none?

8:45Matt Coles It's kind of both.

8:45Izar Tarandach Does it come in green?

8:47Matt Coles It's kind of both. And actually, so then they, if you read the description, I guess we could put the link in the, into the show notes. They describe a set of threats and a methodology for applying those threats or identifying those threats across an agentic system. I do want to call out something of interest. I don't know if anyone's noticed this, but Maestro is an agentic AI threat modeling methodology and, and set of threats. There's no general GenAI or LLM threat modeling specific methodology.

9:23Izar Tarandach So there's the thing. Maestro comes as a layer reference, right?

9:30Matt Coles Yep.

9:31Izar Tarandach And one of those layers is the foundation models.

9:36Matt Coles Yes. Layer 1, in fact.

9:38Izar Tarandach Yeah. Now, let me step that back for a second. We grew up in a world where you said 7 layers, you immediately thought about one thing. And I'm not thinking—

9:52Chris Romeo It was dip. It was like a dip, 7-layer dip. Yeah, yeah. You put your chips in, boom.

9:58Izar Tarandach And I'm not thinking ground meat, marshmallow, and Monica. What I'm thinking is we will think about the TCP/IP layer thing, right? So you have the frame, and inside the frame you have the TCP/IP, and then you have the physical layer, right? With the physical layer, yeah. Right? So things got encapsulated and you couldn't close your eyes and see that layer thing actually developing and understand from that point of view. I don't get the layers of mastery.

10:38Matt Coles Well, so let's go through them, right? So I'm looking at the chart here. Layer 1 is foundation models. Right? We know what those are, right? We think we know what those are. Layer 2 is data operations. So, right off the bat, I think what you're highlighting is OSI was a, a layer, it was a layered approach because it was a structure and a set of encapsulations, as you mentioned, right? Mm-hmm. The physical layer was a communication layer. Now on top of that, there was a, there were frames and there were packets and there was a, this sort of grouping of things that that ended up with the application at the top having its own understanding of the bits. Whereas it seems that Maestro is building sort of a collection of services and how they relate to each other.

11:23Izar Tarandach Or structures, even structures.

11:25Matt Coles But I guess if you look at it as like a, as a high-level diagram or, or as a, a functional model, right? You have, you have a foundation model as a, grouping of, of things, and then you have data operations, which is a function of those models. And then there's agent frameworks, which are functions of data operations and models, right?

11:44Chris Romeo I don't think that was their— I don't think that was the intent here. No, it's, it's not an OSI encapsulation thing where they all stack and they're, they're fit inside of the box of the previous one in more detail, right?

11:55Izar Tarandach And, and these are not layers because they, they don't base themselves on the previous one. Okay, you can have an agent framework without any concern whatsoever for foundational models. Most agent frameworks, by the way, you just say which model we want to use. You don't have to go thinking about all the things that could go wrong with a foundational model.

12:16Chris Romeo But I mean, our solid architecture, solid architecture is a layered approach, right? From top to bottom where we do understand, or at least have the ability to understand you know, if, if the three of us decided we were going to build an agentic AI solution, we would 100% look at the foundational models and, and consider what are the potential threats that exist in us building something on top of that. So my one thought on this is I, I kind of, I had an initial reaction like you did, Isar, and then I kind of stared at it for a while and I read some of the descriptions and I realized it does kind of build on top of the previous layers towards that agent being available, basically, in the ecosystem. The only one that I don't like is layer 6 security and compliance. I was just looking at that.

13:08Matt Coles Like, why is that so far up?

13:10Chris Romeo Yeah, why do I— I don't care about security till I get to layer 6. Like, you got to pull security and compliance off, turn it sideways, and let it ride across all of the layers.

13:18Izar Tarandach Exactly, because if you're waiting until layer 6 to do security and compliance, What's happening on 1 to 5?

13:24Chris Romeo Yeah.

13:25Matt Coles Well, by that same measure though, evaluation and observability, and even to some extent deployment and infrastructure is sort of this cross-cutting thing as well, right? So really your foundation models are things, your operations are things, your frameworks are things, your agent ecosystem are things. And then those 4, 5, and 6 are really cross-cutting across all of those. You deploy foundation models, you evaluate data operations, and you need security compliance for all of those. Mm-hmm. Right? So it's really a box.

13:59Izar Tarandach Yeah, but, but I don't, I don't, I'm sorry. I, I keep looking at the layers as if it's just tasty lasagna. What actually, what you get is an amazing spaghetti.

14:10Matt Coles Oh, remember Spaghetti Code? Remember when we loved Spaghetti Code?

14:14Chris Romeo Oh, but, but don't write code like that.

14:16Izar Tarandach What are you talking about? No, no, no, no, no, no. We vibe through that. So don't worry. The agents are going to fix our spaghetti code.

14:21Chris Romeo The agent creates the—

14:22Matt Coles I was gonna say, do agents create spaghetti code? Can you ask them to?

14:27Chris Romeo I bet you that's what they do by default.

14:28Izar Tarandach It depends on the foundational model.

14:32Chris Romeo Nice fake Italian accent there. Okay.

14:35Matt Coles So for hate mail, send it to—

14:37Chris Romeo send it to Isar.

14:39Izar Tarandach Hey, hey, hey, hey, hey, hey. I hate, I hate when people say that, that making an accent is offensive because it's not, it's just showing that you love that. And I love that. So there's a lot of love in Italian food. I do. I do. Okay.

14:57Chris Romeo So let me bounce a big picture idea off you that hit me every time I look at a new framework like this, my immediate reaction is. Everything is STRIDE. No.

15:12Izar Tarandach Yes.

15:13Chris Romeo But I had, I had an, I had a, I had ChatGPT help me to map which STRIDE categories apply to each of the threats in Maestro. And so one of the, so one of the challenges in the way he stacked these things up is the threats duplicate across the layers.

15:29Matt Coles Yeah, they do.

15:30Chris Romeo They step. All of that to say, like, what I love about STRIDE is it's simple. I can explain it to somebody in 5 minutes. I couldn't explain this to somebody in 5 minutes.

15:41Izar Tarandach I couldn't even really explain it. Well, if they understand it in 5 minutes, it's a completely different discussion. But yeah, you can explain to somebody.

15:46Matt Coles Sure.

15:46Chris Romeo But you can explain it in 5 minutes. And so when I've start— when I started going through this, I'm like, it's just compli— it's, it's, it's not simple. Like simple is what makes things possible for people to use.

16:00Izar Tarandach It's too complex. It's not that it's not simple. I feel like it's shoehorned. Like, we have this imperfect solution. Uh-oh. Not solution. We have this imperfect thing that we can use, that we already know that has been tested over the years, that has a clear approach. To look at those things that we have been saying for a long time, these are not new things.

16:27Chris Romeo These are just— So, I can—

16:28Izar Tarandach new flavors of old things.

16:30Matt Coles So let me tell you, this is, this, so I'm, I'm, I'm totally with both of you. And Stride is so easy to use because it's a general purpose thing again. You know what, you know what Maestro really is? Maestro is really a threat library.

16:46Izar Tarandach Yeah, exactly. It's another catalog.

16:49Matt Coles It's a catalog of specific threats per layer, right? So specific threats per domain, per thing, right? And, and that will, like, right, those things are encoded in layers. And, and so you're going to look for this set of threats when you do foundational models and only this set of threats, right? Whereas Stride is, you're going to look across your, your acronym, right, of what can go wrong in that layer. So why couldn't you have applied Stride to the, like, take the late, take the layers out. Take, pull the layers out and make them its own thing and use that as a methodology for applying Stride or Linten directly.

17:31Chris Romeo Yep. Or both.

17:32Matt Coles It would have worked.

17:35Chris Romeo Yeah. And that's what my ChatGPT analysis determined.

17:40Matt Coles I'm better than ChatGPT. I mean, sorry. My thing is that equivalent to ChatGPT.

17:44Izar Tarandach Then at some point Maestro jumps into cross-layer threats, right? And what's cross-layer threats? It's supply chain attacks. It's lateral movement. It's privilege escalation, data leakage, and goal misalignment cascades, which it's a, it's the one that I look at and say, you know what, cool. That's, that's something that might be new, but the other 4, I'm sorry. Why? Why, why, why complicate things that can at this moment in time be made simpler? Right?

18:16Matt Coles I, I have a feeling that's because of interactions. So one of the things about the layers is the layers are, the layers are described as these self-contained things. And then the cross, which is, which is interesting in itself. 'Cause you look at the foundation models, for instance, it's, if you look at the foundation model as a, as a layer that's self-contained, like Flatworld, you can't go above and below it. There's stuff that happens at that layer in that 2D layer, ignoring the 3D part. The cross-cutting is basically recognizing that there is a third dimension to the, to this, to this set of things. And there's a bleed of, a bleed across these layers.

18:58Izar Tarandach If you need to create a separate class of things that are cross-layer to show that there is a justification to the layers, then I would offer that the problem is not being explained right.

19:12Chris Romeo Okay.

19:12Izar Tarandach Because take prompt injection, for example, that we love to bang on. Foundational models, they are what they are, right? They have their guardrails. They come with their stuff. Not everybody who's going to write an agent right now, for whatever that means, is going to look at a foundational model and say, you have to retrain this thing to be much more robust against prompt injection. No. They're going to take the, the, the, the Kellogg's box off the shelf, use it as it is, and slap some stuff like Llama Firewall on top of it and say, okay, I took care of prompt injection. Or I said, ignore anything that looks like ignore your previous instructions. It's not like they're going to go down into the layer of the foundational model and say, what are the things that I have to worry about here? And what are my mitigations? Because they're not going to affect those mitigations. So why focus there?

20:09Matt Coles So, so I've got something else to throw at me for you to, to, to noodle over and, and, and bring that fire, Isaur. I, I think I can, I'm, I'm gonna inflame the, the rage that you have here.

20:20Izar Tarandach I'm not raging. Yeah.

20:21Matt Coles 3, 3 things I, 3 things I just noticed as I reread this. First, to Chris's point, look at layer 6 in the full description if you're reading the article.

20:34Izar Tarandach I am looking at it.

20:35Matt Coles Layer 6, security and complaints, parentheses, something.

20:40Chris Romeo Vertical layer.

20:41Matt Coles Vertical layer. So—

20:43Izar Tarandach Cuts across all of the layers. So this one, this one is the drunken layer. Everybody's like, no, this one goes 90 degrees.

20:53Matt Coles And then in Layer 3, agent frameworks, take a look at bullet, sub-bullet number 4.

21:00Izar Tarandach You're making me count early in the morning.

21:02Matt Coles I am, yes.

21:03Izar Tarandach Supply chain attacks targeting the AI framework's dependencies, compromising software before delivery and distribution, resulting in compromised AI.

21:11Matt Coles So the layers— So the layers already have the cross-cutting attacks built into them. So why have a separate category of cross-cutting attacks? All right. Last, last point. I dare you find prompt injection in here?

21:30Izar Tarandach I'm sure that it's there. I'm sure that it's there but with some fancy name unconstrained conversational autonomy okay a conversational AI agent that can process and respond to a wide range of inputs without tight constraints so what is this wait—

21:50Chris Romeo Wait!

21:50Matt Coles What now?!

21:50Izar Tarandach So let me ask: "Wait!" Okay...

21:52Chris Romeo right.

21:52Izar Tarandach Right.

21:52Chris Romeo Right. Right.

21:53Izar Tarandach Right. Right. Oh no, no, actually, go.

21:55Matt Coles What is this? What is unconstrained conservational autonomy? Where did this come from? Uh, this is a pattern. So this maestro, this maestro also includes patterns in addition to the layer, in addition to the layers. So now it's even more confusing.

22:15Izar Tarandach Yeah.

22:15Matt Coles Where do the patterns fit into the layers?

22:17Izar Tarandach And now that we are looking at patterns, I'm, I'm going to go with something That really, to me, was very vexing. When we get to the mitigations, okay, so the mitigation for unconstrained conversational autonomy is robust input validation and safety filters designed specifically for the conversational AI use case. I'm a developer. I just read that sentence. Either I am going to get up and go breathe some fresh air, or I'm going to put this into ChatGPT and say, how do I do this? Because it's not practical.

23:04Matt Coles It's jumbled is what it is. It's not practical because you, so how would you apply it? I guess that maybe that's the, that's the question we need to ask is how do you apply Maestro? So they have the layers, and then they have the patterns, and then they have other things. And some of those layers describe threats that are crosscutting, and some, and there's a separate category across those layers of crosscutting things. So how do you apply this?

23:35Izar Tarandach I, I, I think that at this level, at this time in, in, in the life of the, the whole thing, and by whole thing, I say threat modeling AI. Not Maestro specifically. It might be argued that there is value by itself in pointing at the problem simply because we don't have solution answers for every single problem, right? I'm not aware of something that works 100% for prompt injection. And I have said 1,000 times, if I have said once, I don't think that prompt injection is a security problem. I think that The things that prompt injection enable are a security problem. But prompt injection itself, I think that is just inherent to the fact that the way that these things work, right? But I, as a threat modeler, as someone who tried to develop a methodology, as someone who tries to be as much as I can educational to developers, to just throw a sentence, a generic sentence. It feels like somebody coming in and saying, there is something broken in the car. And I go back to them and say, well, I fixed something in the car. You're not giving me a defined problem and I'm not giving you a defined solution, but both of us are happy because you told me that there's a problem and I told you that I solved something. So it, it, it's not the actionable part is flimsy.

25:13Matt Coles I, I want to object to that a little bit, not because I don't think it's flimsy, but not for that reason. At least my show sort of brings together multiple things and really doesn't do a great, well, probably doesn't do a great job about doing all of those things well, if they had separated threats to agents and the mitigations for those threats as one group and how to look at agents and their frameworks and models to perform threat modeling and done that very well, this might have been a different conversation. But I think what you're highlighting is that they jumbled a lot of things together and, and didn't do all of it very well. Right? There's, there's a lot of information here. I mean, the, the threats per layer, those on our own are probably valuable.

26:13Izar Tarandach Yeah.

26:13Matt Coles The mitigations to those threats maybe needs work.

26:18Izar Tarandach Indeed.

26:20Chris Romeo Right.

26:20Matt Coles Now, I guess one One thing that's probably important that hasn't, isn't described here, isn't discussed here, at least as far as I can see, uh, is the nature of agents as human equivalent processes.

26:39Chris Romeo Isn't there a human in the loop mentioned here in the patterns?

26:42Matt Coles There, there might be, but I'm not talking about human in the loop. I'm talking about agents are human.

26:47Izar Tarandach Yes.

26:48Matt Coles Effectively, from a purpose of a system, an agent is effectively like a human. When a human calls a function, when a human goes to a website and makes a function call and it, you know, through an interface, an agent can do the exact same thing. And it does it.

27:04Izar Tarandach Identity with properties and— Right.

27:08Matt Coles So anything that applies to humans applies to agents in terms of their function and their operation. How they're, how they're built, how they're deployed, how they're, how they're called. Maybe you don't call a human the same way that you call an agent, but so there's some very specific things around agents that are not human equivalents, but everything that's human equivalent in an agent and its operation or function should be able to use any of the human equivalent models that we already have.

27:43Izar Tarandach I may be oversimplifying things, but for me, it's more like you go into a restaurant, you have a menu, and you have a chef, right? The chef is capable of doing a lot more. And the menu is just saying, well, in this restaurant, we are limiting the chef to doing these specific things.

28:01Matt Coles That's what Short Library does, right? That's it on artificial limits.

28:06Izar Tarandach I mean, for me, an agent would be the application of the menu on top of the chef.

28:10Chris Romeo Oh, sure.

28:10Izar Tarandach So an agent would be limiting and channeling an LLM to do these specific things, or, or, sorry, limiting and channeling the, the ability of an LLM to interpret things and create some seemingly appropriate output based on that input to these specific things, right? So one of the things that really calls out to me here on the, on Maestro and the layers and the mitigations and all that good stuff. Is that, and this may be a failure of mine, not of Maestro, but it's how I read the thing and the red light that comes on for me, is that in the middle of all the mitigations, in the middle of the generic approach to the mitigations and all that, that there's a jumble of personas in here that to me is very damaging to the whole effort. Not everybody who's going to apply this to an adjacent system is going to be able to, A, understand, B, evaluate, and C, perform the mitigations that are suggested at all levels. So this is going to generate a list of findings that will lead people to say, this is not relevant to me, I have no idea what this is, and I am buying this canned, so somebody else's responsibility here. And I'm not sure how, in its present form, without calling out that specific fact that not everybody is able to do the same thing at all the layers. I don't know if it's, if that's good, if that's bad, if it's something that should be better addressed, if it's Do we need Maestro for people who create foundational models? Do we need Maestro for people who create agents? Do we need to channel these things, separate these things? You understand my discomfort here?

30:17Chris Romeo When you said personas, it's missing personas. Did you mean like personas of who people are going to use it or personas that are part of the threat library?

30:30Izar Tarandach Personas of who's going to use it, because my understanding is that most of the time, the person who's just using framework like Lang-whatever, LangChain, Lang, the many Langs out there to build their client, right?

30:49Chris Romeo Yeah.

30:49Izar Tarandach It's not the same person who's training the foundational model. You have a data scientist doing the model and you have a developer doing the agent. And now you get this list of findings and this cross stuff and all that good stuff. And either each one of them is going to look at things in their appropriate way, or each one of them is going to say, this 90% of this thing doesn't touch me. So what's the, what's the value of this?

31:22Matt Coles Let's go back to the Threat Modeling Manifesto. Do you do this by yourself or do you do this with a group?

31:28Chris Romeo Yes, but. No, no.

31:29Izar Tarandach Yes, but.

31:31Matt Coles Yes, but.

31:32Izar Tarandach Yes, but. My point is that most of the times, the people who can look at all these things in one organization are just not there.

31:42Matt Coles Well, that's true. You're not buying the foundational model from yourself, right? Exactly. You get it from somewhere else.

31:48Chris Romeo I'm laughing at what Izar just said. The people who could evaluate all these things are not there.

31:53Izar Tarandach Yeah.

31:53Chris Romeo They're not there. How are they the people that could evaluate all the things?

31:56Matt Coles There are people that exist.

31:57Chris Romeo It's a killer logic.

31:59Izar Tarandach There are no, there are people who could be there, but they are not because, because there are other places.

32:04Matt Coles There are other places. There are other places doing other things.

32:07Izar Tarandach They're elsewhere. They exist, but not here. They are in a different plane of existence.

32:13Matt Coles We think that they exist or they're all in his head.

32:16Chris Romeo Schrödinger's people who could be doing AI agentic threat modeling.

32:21Izar Tarandach No, it's, it's, it's, it's chess, chess higher. You, you only see their smiles.

32:26Matt Coles Or, or it's, or it's, or it's 4D, it's 4D chess.

32:31Izar Tarandach Yeah.

32:31Matt Coles Uh, 4D laser chess. Uh, so I have a, I have a problem. I, I, I, I think I'm, I think I'm starting to understand the, uh, the mess, the spaghetti that is in his accent from earlier.

32:44Chris Romeo That's very dangerous.

32:44Izar Tarandach Explain to me then, please.

32:46Matt Coles Please tell me why there's patterns with threats and there are layers with threats, but they're not the same threats. There's no connection between the threats on the patterns and the threats in the layers.

33:04Izar Tarandach Wait, that came from my hand?

33:11Matt Coles Uh, if you look at the threats that are in the patterns and you look for those threats in the layers, they don't match in many cases. In most cases, they don't match.

33:21Chris Romeo I see what you're saying.

33:22Izar Tarandach I heard you like threats. I heard you like threats, so I put threats into your patterns.

33:26Matt Coles Well, so, as an example, prompt injection is in unconstrained conversational autonomy. It's not in any of the layers. Jailbreaking is also there. It's not in any of the layers. Unless it's redescribed as something, unrecognizable. It's not there. If you look at communication channel attacks on multi-pattern agents, or you look at compromise of a higher-level agent to control subordinates for the hierarchical agent pattern, where is that threat in the layer?

33:57Izar Tarandach Well, to, to, to be, to be fair, in the page that we are looking at, it says that this section provides some examples of agentic architecture patterns and the risks associated with them. So, It's not exhaustive.

34:08Matt Coles Sure. But why are they not in the layers? But why are they not in the layers? Because the layers are exhaustive.

34:15Izar Tarandach Yes.

34:16Chris Romeo So this is just an organizational problem that this document could have been organized to make it more simple to understand, I think is what you're ultimately trying to say.

34:25Matt Coles Yeah. Either they choose the layers or they choose the patterns because they're building a threat library there, but they're building two distinct threat libraries.

34:35Chris Romeo Right now. I see. Yeah.

34:36Izar Tarandach So if we look at this thing—

34:37Chris Romeo And there's duplication. And amongst the layers, there's duplication. So that, that makes the, it makes it even more challenging because the same threat appears in multiple spots.

34:46Matt Coles Well, and then—

34:47Chris Romeo Slightly different wording.

34:48Matt Coles And then they have these cross-functional threats, which are actually also built into the layers separately. And you have a layer that's vertical.

34:55Chris Romeo Yeah.

34:56Matt Coles Bleh.

34:56Izar Tarandach What? No, if you look at the step-by-step approach, right? It lists 6 steps. System decomposition.

35:04Chris Romeo D, it's under D, using Maestro.

35:07Izar Tarandach So system decomposition, layer-specific threat modeling, cross-layer threat identification, risk assessment, mitigation planning, implementation and monitoring.

35:20Matt Coles Sure.

35:20Izar Tarandach So I'm sorry.

35:23Matt Coles What are we working on? What could go wrong? What are we going to do about it? And did we do a good enough job?

35:31Izar Tarandach No, no, wait, wait, wait, wait, wait, wait. Let's be fair. Let's be fair. That is the root of everything, right? So kudos to Adam. He found the seed. But what I'm trying to point at here is that if you collapse layer-specific threat modeling with cross-layer threat identification, which for the love, for the life of me, I can't see why they have to be separate, right?

35:57Matt Coles You shouldn't, because if you're going to do the layer cake, you're going to have communication between layers. Otherwise, why?

36:04Izar Tarandach Anyway, I mean, it could be a matter of scoping. Let's look at this thing before we look at that thing. And that makes it easier to like focus on something at a time rather than just go crazy as sometimes we do.

36:17Chris Romeo Yeah, I mean, I can actually justify the separation of the two from just trying to make it more simple to understand. If you're, if you got a room full of people who are newer to threat modeling, instead of just saying, throwing everything out on the table, okay, we're going to start with layer 1 foundation models. Here's the 5 things. Is there anything here? Let's go to layer 2, layer 3. Okay, now let's talk about the cross-layer.

36:40Izar Tarandach Okay, so let's try that for a second.

36:41Matt Coles Sure.

36:42Izar Tarandach We are a team developing an agent. Okay. Just developing an agent.

36:48Matt Coles Like I said, we use— is that layer— is that layer 5 or layer 7?

36:52Izar Tarandach Yes. So we are just developing an agent here. We walk into the room. Everybody's in there. Okay, people, we are going to talk foundational models. Uh, we use ChatGPT, Claude, Gemini, Barkle. I don't know. Okay. No problems with foundational models.

37:13Chris Romeo Move on.

37:14Izar Tarandach Data operations. We buy it from, I don't know who. We just hold everything in the same place that we hold everything else. Okay, let's look at this one. There must be something in there. And then we get to layer 6, security and compliance, and everybody looks around and says, what have we been doing for the past 5 layers here? And because that's the vertical one, right?

37:34Matt Coles Actually, I have a more concerning part to Chris's point earlier. If you're building an agent, that stops at layer 3, which is agent frameworks. And actually the agent itself, the agent itself is actually not part of these layers.

37:48Izar Tarandach But wait, I'm sorry. It doesn't. It doesn't because it gets revisited in agents in layer 7, agent ecosystem.

37:56Matt Coles Right. So is an agent of, is a, is a singular agent part of an ecosystem?

38:00Izar Tarandach Yes.

38:04Chris Romeo Sure.

38:04Izar Tarandach Because it believes in an environment. It interacts with stuff.

38:10Matt Coles Okay, so what you're saying is, if you have an agent, all 7 layers apply.

38:19Izar Tarandach No, what I'm saying is, depending on how you're building that agent, depending on the choices that you are making or the choices that are given to you, that are impinged upon you. Some of the layers will make absolutely no sense to you because you can't do anything at that layer. And to tell me that you can solve lower level problems at a higher level, I, I, I, I will say no. Defense in depth, I mean, you need all the ducks aligned. You can't magically solve lower stuff up there.

38:51Matt Coles Point of order. You may not be able to do anything about it other than make a decision about whether you want to do something about it.

38:58Izar Tarandach Right? Yes.

39:00Matt Coles No, that's understandable, right? If you have, if you're building on top of a foundation model and you identify that there's a chance of, if you, if you think about a foundation model and you say, oh, that foundation model has, is known to have data poisoning, I can make a decision to change models. I can't do anything about the data poisoning. Other than transfer the risk or change the risk.

39:27Izar Tarandach That's right. But I'm going to be a bit cheeky here and offer that if you are doing your threat modeling of which model you should be using, at the moment that you are deciding to threat model your agent, you are very late in the game because this should really be different threat modeling sessions.

39:49Matt Coles There's ideal. And then there's, there's actual in-practice reality.

39:54Izar Tarandach And, and, and my point is the actual reprise in, in practice, we are pointing people to spend cycles thinking about things that, fine, we know that these problems exist. We can look at a catalog and say, these problems exist. And here's how each one of the foundational models says that they approach them. But at the level of I'm writing an agent. A, most probably you don't have the expertise to address those things. And B, most probably you're going to say the choice was made for me. There's nothing I can do.

40:29Matt Coles But those are still decisions you can make. I mean, you can, you may not be able to address the threats that come out of it. You may not even know what all the threats are, although you can see the symptoms of it and you can make, I'm going to, I'll use the phrase, I didn't know it's the wrong phrase. An informed risk decision to do something about it.

40:48Izar Tarandach A very smart person that I value very much said once that there is the practical and there is the theoretical. In theory, right? In practice, most people will look at that and say, again, I can't do anything about it. Why am I spending cycles on this?

41:07Chris Romeo Sure.

41:08Izar Tarandach But when you—

41:10Matt Coles Go ahead. No, you finish your thought.

41:11Izar Tarandach No, no, no, you go. No, no, no, you go.

41:14Matt Coles But that in and of itself is a problem. Why am I spending cycles on something I can't change? Because you have the ability, you have to think about it in order to make something, an informed decision.

41:26Izar Tarandach When I know how to answer that, my therapist is out of a job. So I can't say.

41:33Matt Coles What I mean is, if you can't do anything about, if you Even though you can't do something— well, sorry. You can do something about it, but your choices are limited. Ignoring the problem because your choices are limited is a problem.

41:48Izar Tarandach But that's the thing. You're not ignoring. You are looking at a— it's like any other, like, third-party component, right? It's bringing its threat modeling baggage with it.

41:59Matt Coles That's right. And that may be opaque to you or not, right?

42:02Izar Tarandach Yeah, yeah, yeah. Some of them, you can come and say, okay, I can do something with my defense in depth in here, put some bumps in the road. Some of them, you're going to say, hey, you know what, the decision was made. This is a base component of my design. I'm going to assume the risk. And both are valid. And both include looking at the problem and saying the problem exists. And there is very little I can do about it. My whole point here is that we are in a time in this technology where lumping things together for people that can't do anything about the stuff, and what they can do is very limited, is counterproductive at this time. If you are going to propose a framework to deal with agentic whatevers, and I'm just saying whatevers because I forgot the work, the word. It's not a whatevers. It's not a disrespect for whatever. It's just that I forgot the word. But if you're going to do a framework about agents, do the framework about agents. Create the framework in a way that people can actually respond to threats to agents, right? Don't try to boil the ocean. And go back to training models and whatever, because the mass, the 99% of people out there writing agents are not training models. There's enough complexity in here to address in a simple way each real layer of the complexity in ways that are actionable to that layer. And the spaghetti in my head says, you are juggling everything together. You, you, you are dropping a cognitive load on top of me of things that I am going to feel completely unempowered to, to solve. How exactly did you help me?

44:03Chris Romeo Interesting. Hard-hitting.

44:05Matt Coles That's deep. I just need some deep learning for that.

44:09Izar Tarandach I, I, I blame it on Monster. I blame it on Monster.

44:12Chris Romeo All right, well, we're, we're just about out of time here, so. I mean, when I, I, after diving into Maestro here, kind of kicking it around a little bit, maybe a little bit too much, I mean, I think there's still quite a bit of value in this, in this research and this work that's gone on. I think we're, we're, we're kind of looking at it for, we're trying to optimize it for enterprise use to some degree by saying it, it's needs to be refactored some to make it simpler and help people, personas understand how to use it. But all in all, I think there's some good stuff in here. And there's good threats that if this thing just gets repurposed as a threat library, I like that idea, Yasar. If it's a threat library, it's one of the best ones out there having to do with AI.

45:01Izar Tarandach Look, I would say it's definitely a very, very worthy addition to the tools that we have. And from reading it, I already learned a bunch, right? And he won't let me lie about it.

45:18Matt Coles I was waiting for the voice of approval over there.

45:21Izar Tarandach Right? This is a, yeah, it's a Doug and a, how do you say?

45:32Matt Coles Yeah, that.

45:32Izar Tarandach There you go. Okay, he said everything. But basically, just from reading it, I learned a bunch. I learned stuff that I did not take into consideration before. So definitely a lot of value in there, a lot of great work in there. Goes together with a lot of great work that's been done in the OWASP GenAI project. Amazing stuff. I really think that, and actually just saying it in my head, It jumps, hey, if it bothers you so much, why don't you go and do it? Who knows? There is a level of translation that needs to happen here into turning this actionable to developers. And the three of us, our whole career, we are about making things actionable for developers. So that's probably what bothers us here so much. I know that that's what bothered me. But definitely a lot of good stuff in here. Yeah, I'm going to make use of it.

46:31Chris Romeo Nice. All right. Well, Matt, any final thoughts on Maestro?

46:37Matt Coles I'm going to try and get a word in edgewise with the barking dog in the background.

46:41Izar Tarandach I would have loved—

46:43Matt Coles It's not going to happen, but I wonder if—

46:47Chris Romeo Folks, just so you know, Matt has no dogs. This is totally done in post-production just to try and—

46:52Matt Coles She's just nice on board.

46:53Chris Romeo To make him seem more human.

46:56Matt Coles Yes, yes, that's what it is. Uh, you know, I was gonna drop a bombshell. I would really love if, uh, I would really have loved if this layer approach— that's them, and hopefully my voice is going to come through as opposed to— yeah, we hear it.

47:12Chris Romeo Okay, we got you.

47:13Matt Coles Uh, the, um, I would have loved if the layer approach was something we did wide I wish we could have used the layered approach. It's something we could apply to SAS things. Yeah, yeah, yeah, yeah. All right. I'm not going to do anything here.

47:35Chris Romeo The layered approach. Matt wishes we could have used the layered approach for something.

47:40Matt Coles For systems. For systems.

47:42Izar Tarandach Hamburger.

47:44Matt Coles Hamburger.

47:45Izar Tarandach Hamburger.

47:45Matt Coles The layered approach. The layered approach in my, I, as I read through it, the layered approach probably makes a lot of sense if we look at infrastructure processing data, right? As layers.

47:56Chris Romeo Yeah.

47:57Matt Coles In a more generalized way. And then take the threat library or the threat categories separately or the patterns separately. But altogether, um, I'll go back to Maestro is a is a reinvention of the wheel when the wheel was perfectly— was almost perfectly fine.

48:18Chris Romeo Yeah.

48:20Matt Coles How you use the wheel might, might have been the important bits.

48:24Chris Romeo Yeah. And the wheel's had some improvements over time too. It has not made out of stone anymore or anything.

48:30Matt Coles And now they don't even require air.

48:32Chris Romeo True. Run forever. All right, folks. Well, thanks for tuning in to the Security Table. And hearing us discuss Maestro once again. We're not haters. We're just security people for a long time. So it just makes us how we are.

48:50Matt Coles Cynical. Skeptic.

48:54Izar Tarandach What? I don't believe you.

Transcript supplied by AssemblyAI. It may contain transcription errors.

Originally published as The Security Table. Part of the AI Security Table archive.