24 minSeason 2, episode 21

The Stages of Grief in Incident Response

With Chris Romeo, Izar Tarandach, and Matt Coles

Infrastructure, identity & resilience

Join Chris, Izar, and Matt as they sit around the Security Table to dissect and discuss the different stages of dealing with security incidents. In this episode, they explore the developer's stages of grief during an incident, and discuss a recent large-scale IT incident.

Episode title image: The Stages of Grief in Incident Response

Listen to the conversation

About this episode

Join Chris, Izar, and Matt as they sit around the Security Table to dissect and discuss the different stages of dealing with security incidents. In this episode, they explore the developer's stages of grief during an incident, and discuss a recent large-scale IT incident. They share insights from their multi-decade experience in security, analyze the fragility of current systems, and discuss the role of luck and probability in security failures. 

Follow AI Security Table:

➜ Home: https://securitytable.ai/

➜ X: https://x.com/AISecurityTable

➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/

➜ YouTube: https://www.youtube.com/@AISecurityTable

Transcript · 24 min conversation

0:09Chris Romeo Good morning, good afternoon, and good evening wherever you find yourself on this fine day. This is a new introduction to the Security Table. I'm Chris Romeo, joined by Izar Terrindosh and Matt Coles. We are the people that sit around the security table. And on the Security Table, we talk about stuff and things that happen in the world of security. And we analyze it, we dissect it, we make fun of it sometimes. But all in all, this is an educational podcast, hoping to learn from things that happen in the world and apply our multi-decades of experience to it. And so let's talk about stages of an incident first. I know, Izar, you were talking before we started recording, you mentioned this idea of the kind of the stages of grief in the developer's lifecycle of, and so I'm curious if you could maybe describe that a little bit more as a way to kick this off.

1:14Izar Tarandach I just got to the first one, which is denial, which is represented by it works on my machine.

1:20Matt Coles Okay.

1:21Izar Tarandach But let's look at the other one.

1:24Chris Romeo The other stages of grief to see how they connect into a technological world.

1:30Izar Tarandach We have, oh, the first one is shock.

1:33Chris Romeo Okay.

1:34Izar Tarandach What do you mean it doesn't work? What do you mean it's broken?

1:36Matt Coles Shocked off. Shocked off.

1:39Chris Romeo Okay.

1:40Izar Tarandach Then comes denial.

1:42Chris Romeo Okay. Couldn't happen. It couldn't have been my fault. There's no way.

1:45Izar Tarandach It works on my machine.

1:47Matt Coles It's not happening.

1:48Chris Romeo Yeah.

1:48Matt Coles It's not real.

1:50Izar Tarandach Then you have anger.

1:51Chris Romeo Okay.

1:52Izar Tarandach You must be doing something wrong. You must be doing something wrong.

1:55Chris Romeo Is this when you're banging on the keyboard too? Like at a height, like, you know, you're kind of hitting the keyboard at this point?

2:00Izar Tarandach No, it's when you're screaming at people that they are holding the mouse wrong.

2:03Chris Romeo Ah, okay. Okay.

2:06Izar Tarandach And then you have bargaining.

2:09Chris Romeo Bargaining.

2:10Izar Tarandach Bargaining, which should be, I'll stop beating you if you go away quietly.

2:16Chris Romeo Yeah. I mean, well, it's gotta be like, it's, I guess it's bargaining on how much time it's going to take you to fix it. Perhaps like, ah, you know, this is going to take—

2:22Matt Coles can you make my machine? Can you get my, get my machine up and running first? I'll do, I'll buy you coffee.

2:27Chris Romeo Yeah, there you go. Or it's going to, I'm bargaining in that it's going to take 2 years for us to fix this.

2:32Izar Tarandach If you're going for coffee, you're doing it wrong. Uh, and then comes depression.

2:38Chris Romeo Okay.

2:38Izar Tarandach Oh my, I'm, life is ruined.

2:41Chris Romeo I can't believe I wrote this code.

2:43Izar Tarandach It's when you, no, it's when you look at the ASPM and you see that you have only 64K vulnerabilities. And that's only because the counter only holds 64K.

2:53Chris Romeo They should have used a bigger data structure.

2:57Izar Tarandach And then you have testing.

3:01Matt Coles Testing?

3:01Izar Tarandach I heard of that. I heard of it. I heard of it. No, I know.

3:04Chris Romeo Like quality assurance?

3:06Izar Tarandach I remember this one. Wait, wait. People used to do this.

3:11Matt Coles Something's lost in translation here. It must be.

3:13Izar Tarandach Damn it. I get the concept. I haven't ever seen in the last 2 years somebody actually do it. And then finally, there is acceptance, which is when you actually boot Linux.

3:32Chris Romeo You move on to the Linux platform. So that's, so that kind of takes us, that takes us through the, the stages of grief in an incident, I guess.

3:42Izar Tarandach But the real fun is when you see people doing it in, uh, loops.

3:45Matt Coles You reconstruction and work through is the, is the 6th one.

3:51Izar Tarandach Oh, I called that postmortem.

3:56Chris Romeo He just, he optimized the list.

3:58Matt Coles Yeah.

3:58Chris Romeo As he was reading it, Joe, almost like an LLM.

4:02Matt Coles You had an AI do that, didn't you?

4:04Izar Tarandach No, I actually went to the old school and I Googled 7th Ages of the Crypt. That's how I roll, can I tell you.

4:12Chris Romeo That's good stuff.

4:16Izar Tarandach But the thing is, I think that in the space of, what, a weekend, we saw the whole loop going around with good and bad. Beautiful times. I will not say that it was fun to look because We have to agree that a lot of people got impacted by this and seriously like impacted by it. And, uh, but still it's, it, I, I think that is a dress rehearsal for an actual cyber attack. This weekend had a lot of value.

4:46Matt Coles Oh my God. It, it highlighted, it highlighted the, the ubiquity, ubiquity of, of the Windows platform.

4:55Izar Tarandach Right.

4:56Matt Coles I mean, that was not shocking, but shocking.

5:02Chris Romeo Yeah, I mean, it brings me back to the days of old when I was getting started in incident response and NIMDA, Code Red, SQL Slammer, all of these massive exploits. Blaster. Blaster, yeah, were being used. This is, you know, this, what's been happening in the last week here has been what was a software bug that was then accidentally deployed, which caused a bigger problem. But in those days, it was an active exploit and it was a worm-based exploit. So it was even better. So as it compromised machines and picked up speed, you had more outbound active scanning occurring, more compromising. And then it was a ripple effect across the whole thing. And so that's, you know, was the time where I realized the impact Windows had in those days in the data center and on the desktop, that Windows was everywhere. And one of those worms would shut everything down for a day or two. And it happened in a 3-year span, it happened 5 times.

6:10Izar Tarandach One lesson that I took away from this was that you can have the most modern, most well-defended, most well-patched data center. If you can't get to your data, then it's not worth much.

6:23Chris Romeo It's true. The access layer is crucial.

6:28Izar Tarandach So, yeah, it's, again, it's not schadenfreude and very, very far from it. I really wouldn't want to be in those people's shoes or anything. But it does remind me of a lot of stuff that we've been talking about here in the podcast. We talked about security utilities. We talked about the responsibility of software that's like base software for the internet, for the usability that we need. And I think that we even managed to talk a bit about fragility. And one thing that, one lesson that I'm taking away from this weekend is that, you know what, yeah, products will have bugs. That's just the way it is. It will always be like this. Things will happen.

7:11Matt Coles But, and by the way, we should highlight that the, from based on the information that's available publicly for this, for this issue, um, this was not a security vulnerability, right? This was a, this was a bad patch, right? This was a bad update that went out and, and it ended up causing a denial of service on a grand scale, but it's, it was not a, it wasn't a, um, uh, an exploitable It wasn't a problem that became exploited by an attack.

7:43Izar Tarandach That's why I call it a dress rehearsal to an actual incident, but without taking it, the value of it as an IT incident. But my point about Fragility was that if I understood it correctly, and believe you me, I'm far from knowing any kind of details or anything, it was a product-level thing that even the products running elevated impacted the system itself in such a way that it became an operating system error?

8:12Matt Coles Well, actually, I can tell you this, right? I can tell you that, you know, based on the information that we have available, right, this was a blue screen of death on Windows, right? And particularly because it was accessing invalid memory. So it was a pointer dereference, right? This is a common problem in C programming. We don't, obviously we don't know what language was used here, but you know, this happens often in drivers. So if you have a driver that's running at a very low level, right? And that driver has a bug where it, you know, references a null pointer or references a bad pointer, then you get a blue screen of death. There's, and this is, by the way, this is Windows being, this is Windows being intelligent about, Sorry, be careful. This is Windows being defensive, as it should, because when you have an exploitable vulnerability, when you have a, say, a buffer overflow in code, an RCE, right? If you had a remote code execution vulnerability in your source, in your code and in the driver, right? A crash is the right approach.

9:23Chris Romeo Why is this?

9:24Izar Tarandach No, no, no, no.

9:26Chris Romeo I got a question to ask then. Why is this still a thing? That's a modern operating system. Because like Linux doesn't have Linux.

9:34Matt Coles You can, I mean, you can still cause a kernel panic.

9:38Chris Romeo You can cause a kernel panic, but man, it doesn't happen very much anymore. I mean, in the old days, kernel panic would happen once a day. You're like, why'd my system shut down? Oh, look, uh, kernel panic because I ran out of memory.

9:50Izar Tarandach I think it's a couple of things, but just to relate to what Matt said, that's not Windows being defensive. That's Windows being Cartman. It's taking the ball and going home, right? So, you know, it's like in this day and age to say, I give up. I just saw no reference. I give up. I can't, I can't, I can't with you guys. I can't even. Okay. You can't do that thing anymore.

10:12Chris Romeo Yeah.

10:13Matt Coles But when you're operating at such a low level, how do you, how do you avoid that?

10:17Izar Tarandach Right.

10:17Matt Coles I mean, this is—

10:17Chris Romeo you gotta have isolation in the kernel. Like. The year is 2024. If the year was 2005, I would still believe, because I watched like those various worms and stuff caused blue screens of death ultimately. That was eventually the code was bad. It would just, it would just kill the machine because there was bugs in the, in the worm code itself. Right. But like, it seems like in 2024, we've had 20 years to build an operating system kernel that can provide isolation for individual drivers and not let one driver trounce the whole thing. It seems like they should have done that.

10:54Izar Tarandach That whole hypervisor virtual machine thingy stacky that puts everything separated and, you know, what was it, microkernels and personalities in the kernel and all that stuff. And you want to tell me that one single null reference and pop goes the weasel? That's not being, that's not doing the right thing.

11:18Chris Romeo That's not being resilient. Like we've been talking about resiliency for the last couple of years in cybersecurity as an umbrella level topic. And that's not a resilient system that you could rely upon to fly an airplane, for example, if it's got that level of—

11:33Matt Coles Hopefully you're not relying on Windows to fly your airplane, but—

11:36Chris Romeo They talked about it. They talked about it 20 years ago. It was on the table. There was some, I remember seeing some research going into it, but ultimately it wasn't.

11:47Matt Coles Hey, Windows IoT Core, maybe, but you know, it's Raspberry Pi or nothing.

11:52Izar Tarandach But now here's the funny connection. A couple of episodes ago, we spoke about the CISA and their advisories, alerts. I don't remember what it's called. And it came out with the use memory safe languages.

12:08Chris Romeo Yep.

12:09Izar Tarandach Well, I guess that the, the egg is on the other foot now.

12:14Chris Romeo Because, uh, how does an egg get on the other foot? Hold on.

12:19Izar Tarandach I waited to see how long that meme would trigger. But, uh, no, really, I mean, we lasted it. But here we are, humbled by a single null reference over a weekend.

12:33Chris Romeo And some people are going to feel this for a long time. Isn't Microsoft rewriting lots of Windows stuff in Rust though?

12:40Matt Coles They are.

12:40Chris Romeo I thought they were making a move towards Rust.

12:41Matt Coles Linux is too, right?

12:42Izar Tarandach Linux is.

12:42Matt Coles Linux has. Yeah.

12:44Chris Romeo But I thought Microsoft specifically was as well going towards Rust as their dominant programming language.

12:49Izar Tarandach Yeah.

12:49Matt Coles Their new, uh, the new SDL, the new, um, Microsoft SDL and trusted computing, whatever they're doing, was, was intending to use languages like Rust. I don't know if they, yeah.

13:01Izar Tarandach I, for one, I can't wait for Visual Rust++ hash and .rust and the whole amazing ecosystem of all things that look like Rust, but they're not Rust that Microsoft is going to come up with.

13:17Chris Romeo Don't forget Visual Rust.

13:19Izar Tarandach Yeah. Visual Rust++ hash.

13:21Matt Coles Now, is it true that this, this didn't affect all versions of Windows? This was only 10.11 that was impacted.

13:27Chris Romeo Yeah. Well, there was a funny tweet. I don't know if you saw this from Southwest Airlines that said, uh, haha, joke's on you, we're still flying today because we use Windows 3.1. Which, I mean, there, there, there's, there is an argument in security that we don't make enough for protection by running things that are so ancient that they don't have the problem that everybody's talking about because they were written so long ago and from a different code base.

13:57Matt Coles That's the, that's the FAA way, isn't it? They're running mainframes still for tracking flights.

14:04Chris Romeo Yeah.

14:04Matt Coles Because they just work.

14:06Chris Romeo But I mean, just using, using antiquated, and I remember, um, this situation coming up with, uh, Heartbleed. People were running some, some, I remember going through the process at Cisco where we were looking at all of the products trying to determine what products were vulnerable to Heartbleed. And there was a subset of products that were running an OpenSSL that was so old, didn't have the Heartbleed problem in it. And you know, like, yep, those are good. Take that out of the advisory. That doesn't need to be in there.

14:36Matt Coles You know, you know that there's, there's always going to be that one team is like, well, see, we didn't do HTTPS, so we're not impacted.

14:42Izar Tarandach Yep.

14:43Chris Romeo We just did plain text for everything and it's good.

14:47Izar Tarandach See, can't pitch a bug that's not there.

14:52Chris Romeo I don't know. That's not a security by— what would this be? Security by laziness?

14:57Matt Coles There is a, there is a CWE, there is a CWE for reliance on, on outdated and obsolete software or components.

15:05Chris Romeo Well, it sounds like that should be a positive though, not a negative.

15:08Izar Tarandach No, it's called security, security by learned helplessness.

15:12Chris Romeo Security by learned helplessness.

15:13Matt Coles It's called putting your head in the sand. It's, it's putting your head in the, I'm recommending this forward.

15:20Chris Romeo Take old components that are at least 20 years old and try to cobble them together into a solution. Yeah.

15:25Izar Tarandach Things that you know are not going to be patched anymore.

15:28Matt Coles Actually, you know, this is, this has a real, this has a real world analogy. You know, people who are all, there are, there are a group of people, there are a collection or population of people who, um, look for vehicles that are prior to some time period, 2000 or something like that, because they aren't reliant on, they don't have so many microprocessors. They don't have computer code. They don't have telemetry. They don't have software updates for their entertainment systems. And so you have less, if you're a conspiracy theorist and interested in that sort of thing, then, you know, somebody taking over the vehicles like they did with OnStar and whatnot in the Jeep situation a few years ago, right? So, There is a—

16:10Izar Tarandach because, because you can't just stop in some crossing in any random city and get your car taken over by somebody with a handgun. Well, it's a robbery, geez, step off.

16:24Chris Romeo But now there is, I mean, that is, that is a good analogy or a good connection there, Matt, that, that could—

16:29Izar Tarandach it is.

16:29Chris Romeo There are cars that are non-computer-based.

16:33Izar Tarandach And you, you know what? It's not that far because I, I have told already a bunch of people. You are not going to see me in a car with an autopilot anytime soon because I know the people who write those things.

16:50Chris Romeo Well, what are you trying to say?

16:53Izar Tarandach That, uh, I don't like the way that roads are built, that paved roads perhaps are not, that not every guardrail works well.

17:04Chris Romeo I like where you're kind of You're kind of circling this issue.

17:09Matt Coles You might, you might be, you might be driving one right now. How do you know?

17:13Chris Romeo Yeah. It's a, it's a trust issue then. You're saying, so you're saying you don't trust—

17:16Izar Tarandach that's a very good point.

17:17Chris Romeo You don't trust the technology behind those cars.

17:20Izar Tarandach Oh, I trust the technology. I don't trust the implementation.

17:24Chris Romeo Isn't it one and the same?

17:25Izar Tarandach Nope.

17:26Matt Coles So, so are these friends of yours? 'Cause they're, they're not gonna be friends of yours.

17:30Izar Tarandach Not for long. Not for long. But it's okay. They drive with the autopilot.

17:39Chris Romeo So what do we, what do we do with this?

17:41Matt Coles Like, what is, well, actually, so there's one, there's one actually really important concept. I think that, that we need to tease out here is if by some sheer luck you were running Mac and Linux, you weren't impacted by this. Now, having said that, Linux of course has the regression vulnerability and, and I read an article this morning that there's yet another regression-related vulnerability. Of course there is. Of course there is.

18:07Izar Tarandach Of course there is.

18:09Matt Coles So, so you're, you're, at what point, at what point do we, I guess, is there a point where people who make technology decisions, whether that's user, users at home or, or enterprises making decisions for their, for their, you know, their collection of users. Then it doesn't become luck as to what is secure or safe or randomness in my— this was completely random luck, right? If, if the, the, uh, a chain of events had to occur for this issue to, to, to be as, as prevalent as it was, right? It happened that this, you know, that the organization that put it, put out the patch is used almost everywhere apparently.

18:57Izar Tarandach Okay.

18:58Matt Coles It got very successful very quickly. They put out a patch that broke and then caused the ripple effect, chain effect, right? I mean, you know, on over the weekend you had, or last week you had gamers who couldn't game on Xbox Live 'cause Xbox Live was down. By the way, this is talking about cloud services, not just Windows servers, right? This took out—

19:18Izar Tarandach Yeah.

19:19Matt Coles It took out Xbox Live. It took out, um, uh, how many, how many airlines lost their ability to do, um, uh, reservations and, and bookings and things like that?

19:29Chris Romeo I think it's more the other way around, which airlines were actually able to operate. It was very, very short list.

19:33Izar Tarandach Right.

19:34Chris Romeo Well, no one. Yeah.

19:35Matt Coles And then we had, and then we had hospitals that had to, uh, you know, because hospitals use, you know, Windows machines to run MRI machines and, and CAT scans and other diagnostic equipment, right? So you had this cascade effect, right? But if they, if somebody had made a decision decades ago to stick with System V Unix, right? Or, or a Linux variant, right? I mean, Europeans were, were big on running Linux at some point. Did, was, was Europe taken out the way that America was? Um, you know, so at what point did technology decisions stop? Can we get away from the luck factor? Like, this is accidental, right? This isn't intentional, but, uh, it was pure luck that, that it happened.

20:24Izar Tarandach I think let's not forget that what some people call luck, other people call probability. And in the long tail of time, the probability of an incident is 1.

20:35Chris Romeo Yeah.

20:35Izar Tarandach So, It was about to happen, right? At some point it would. We've been talking forever about the castle of cards that this whole thing runs on. And at some point, you know, something is going to get knocked out. I just go back to the points that we are, and yeah, I'm going to agree with you. We were lucky that this was actually a bug and not an active incident, right? Complications of one of those might have been way bigger. But it just goes to show how vulnerable we are and how, you know what, even though now we have LLMs running everywhere and looking over our shoulders while we write code and whatnot.

21:21Matt Coles By the way, LLMs run on Linux.

21:23Izar Tarandach So, yeah, no, no, no. But what I mean is, what I mean is people are adding a level of trust now. We use that word in connection with LLMs all the time because I think it's like the root of the problem. But we are adding now another level of trust in the whole castle of cards because now we have this magical LLMs writing code for us. And I don't think that people realize that it's not very far away from an autopilot in the middle of the street, you know. Sometimes you just can't trust what that thing is doing.

21:55Chris Romeo Let me—

21:55Matt Coles so are you saying that we need to, um, we need to reinforce Development process.

22:03Izar Tarandach No, I think, yeah, I think that what I'm saying is that, you know, this is going to sound like job security, but yeah, we are still going to have to trust more into very talented developers than into very talented security people. Technology is coming in big, but we are not going anywhere anytime soon.

22:21Chris Romeo That's true for at least the next couple of decades. But let me put this in perspective because I had an exchange with a friend of mine who's a non-technology person. He asked me, he said, so, you know, what do you think about this, this thing that's happened here? You know, were you surprised by it? And I said, funny thing is with what I know about technology in lots and lots of big companies, I'm surprised it doesn't happen once a week. With all of the, think about all of the chances and coming back to probability, like what, with all of the chances that this could happen, multiplied by the range of a particular product as far as like how many computers it's installed in around the world. Like if we took the top 10 most installed things, I'm just amazed that it does— this doesn't happen more often.

23:13Matt Coles Yeah. I, I mean, and this is not the first time we've seen something like this happen, right? We had, um, I think it was Symantec a number of years ago had a, had a .dat file update And it, you know, took out their user base, right?

23:24Chris Romeo Yep.

23:25Matt Coles And so it's, it is not, it is not without precedent. It's just the scale.

23:33Chris Romeo Yep. And so I think that's, uh, a good place to wrap up for today based on a little bit of commentary. So, uh, we'll be back again next week to unpack something else. Who knows what it'll be? Only you can know.

23:47Matt Coles We never know.

23:49Izar Tarandach I have no idea. Even now, I have no idea.

23:53Chris Romeo All right. Thanks for listening to Security Table.

Transcript supplied by AssemblyAI. It may contain transcription errors.

Originally published as The Security Table. Part of the AI Security Table archive.