# The Stages of Grief in Incident Response

URL: https://securitytable.ai/the-stages-of-grief-in-incident-response/
Published: 2024-07-24
Duration: 1445 seconds
Season: 2
Episode: 21
Hosts: Chris Romeo, Izar Tarandach, Matt Coles

## Show notes and chapters

Join Chris, Izar, and Matt as they sit around the Security Table to dissect and discuss the different stages of dealing with security incidents. In this episode, they explore the developer's stages of grief during an incident, and discuss a recent large-scale IT incident. They share insights from their multi-decade experience in security, analyze the fragility of current systems, and discuss the role of luck and probability in security failures. 

Follow AI Security Table:

➜ Home: [https://securitytable.ai/](https://securitytable.ai/)

➜ X: [https://x.com/AISecurityTable](https://x.com/AISecurityTable)

➜ LinkedIn: [https://www.linkedin.com/company/ai-security-table/](https://www.linkedin.com/company/ai-security-table/)

➜ YouTube: [https://www.youtube.com/@AISecurityTable](https://www.youtube.com/@AISecurityTable)

## Transcript

**Chris Romeo (0:09)**: Good morning, good afternoon, and good evening wherever you find yourself on this fine day. This is a new introduction to the Security Table. I'm Chris Romeo, joined by Izar Terrindosh and Matt Coles. We are the people that sit around the security table. And on the Security Table, we talk about stuff and things that happen in the world of security. And we analyze it, we dissect it, we make fun of it sometimes. But all in all, this is an educational podcast, hoping to learn from things that happen in the world and apply our multi-decades of experience to it. And so let's talk about stages of an incident first. I know, Izar, you were talking before we started recording, you mentioned this idea of the kind of the stages of grief in the developer's lifecycle of, and so I'm curious if you could maybe describe that a little bit more as a way to kick this off.

**Izar Tarandach (1:14)**: I just got to the first one, which is denial, which is represented by it works on my machine.

**Matt Coles (1:20)**: Okay.

**Izar Tarandach (1:21)**: But let's look at the other one.

**Chris Romeo (1:24)**: The other stages of grief to see how they connect into a technological world.

**Izar Tarandach (1:30)**: We have, oh, the first one is shock.

**Chris Romeo (1:33)**: Okay.

**Izar Tarandach (1:34)**: What do you mean it doesn't work? What do you mean it's broken?

**Matt Coles (1:36)**: Shocked off. Shocked off.

**Chris Romeo (1:39)**: Okay.

**Izar Tarandach (1:40)**: Then comes denial.

**Chris Romeo (1:42)**: Okay. Couldn't happen. It couldn't have been my fault. There's no way.

**Izar Tarandach (1:45)**: It works on my machine.

**Matt Coles (1:47)**: It's not happening.

**Chris Romeo (1:48)**: Yeah.

**Matt Coles (1:48)**: It's not real.

**Izar Tarandach (1:50)**: Then you have anger.

**Chris Romeo (1:51)**: Okay.

**Izar Tarandach (1:52)**: You must be doing something wrong. You must be doing something wrong.

**Chris Romeo (1:55)**: Is this when you're banging on the keyboard too? Like at a height, like, you know, you're kind of hitting the keyboard at this point?

**Izar Tarandach (2:00)**: No, it's when you're screaming at people that they are holding the mouse wrong.

**Chris Romeo (2:03)**: Ah, okay. Okay.

**Izar Tarandach (2:06)**: And then you have bargaining.

**Chris Romeo (2:09)**: Bargaining.

**Izar Tarandach (2:10)**: Bargaining, which should be, I'll stop beating you if you go away quietly.

**Chris Romeo (2:16)**: Yeah. I mean, well, it's gotta be like, it's, I guess it's bargaining on how much time it's going to take you to fix it. Perhaps like, ah, you know, this is going to take—

**Matt Coles (2:22)**: can you make my machine? Can you get my, get my machine up and running first? I'll do, I'll buy you coffee.

**Chris Romeo (2:27)**: Yeah, there you go. Or it's going to, I'm bargaining in that it's going to take 2 years for us to fix this.

**Izar Tarandach (2:32)**: If you're going for coffee, you're doing it wrong. Uh, and then comes depression.

**Chris Romeo (2:38)**: Okay.

**Izar Tarandach (2:38)**: Oh my, I'm, life is ruined.

**Chris Romeo (2:41)**: I can't believe I wrote this code.

**Izar Tarandach (2:43)**: It's when you, no, it's when you look at the ASPM and you see that you have only 64K vulnerabilities. And that's only because the counter only holds 64K.

**Chris Romeo (2:53)**: They should have used a bigger data structure.

**Izar Tarandach (2:57)**: And then you have testing.

**Matt Coles (3:01)**: Testing?

**Izar Tarandach (3:01)**: I heard of that. I heard of it. I heard of it. No, I know.

**Chris Romeo (3:04)**: Like quality assurance?

**Izar Tarandach (3:06)**: I remember this one. Wait, wait. People used to do this.

**Matt Coles (3:11)**: Something's lost in translation here. It must be.

**Izar Tarandach (3:13)**: Damn it. I get the concept. I haven't ever seen in the last 2 years somebody actually do it. And then finally, there is acceptance, which is when you actually boot Linux.

**Chris Romeo (3:32)**: You move on to the Linux platform. So that's, so that kind of takes us, that takes us through the, the stages of grief in an incident, I guess.

**Izar Tarandach (3:42)**: But the real fun is when you see people doing it in, uh, loops.

**Matt Coles (3:45)**: You reconstruction and work through is the, is the 6th one.

**Izar Tarandach (3:51)**: Oh, I called that postmortem.

**Chris Romeo (3:56)**: He just, he optimized the list.

**Matt Coles (3:58)**: Yeah.

**Chris Romeo (3:58)**: As he was reading it, Joe, almost like an LLM.

**Matt Coles (4:02)**: You had an AI do that, didn't you?

**Izar Tarandach (4:04)**: No, I actually went to the old school and I Googled 7th Ages of the Crypt. That's how I roll, can I tell you.

**Chris Romeo (4:12)**: That's good stuff.

**Izar Tarandach (4:16)**: But the thing is, I think that in the space of, what, a weekend, we saw the whole loop going around with good and bad. Beautiful times. I will not say that it was fun to look because We have to agree that a lot of people got impacted by this and seriously like impacted by it. And, uh, but still it's, it, I, I think that is a dress rehearsal for an actual cyber attack. This weekend had a lot of value.

**Matt Coles (4:46)**: Oh my God. It, it highlighted, it highlighted the, the ubiquity, ubiquity of, of the Windows platform.

**Izar Tarandach (4:55)**: Right.

**Matt Coles (4:56)**: I mean, that was not shocking, but shocking.

**Chris Romeo (5:02)**: Yeah, I mean, it brings me back to the days of old when I was getting started in incident response and NIMDA, Code Red, SQL Slammer, all of these massive exploits. Blaster. Blaster, yeah, were being used. This is, you know, this, what's been happening in the last week here has been what was a software bug that was then accidentally deployed, which caused a bigger problem. But in those days, it was an active exploit and it was a worm-based exploit. So it was even better. So as it compromised machines and picked up speed, you had more outbound active scanning occurring, more compromising. And then it was a ripple effect across the whole thing. And so that's, you know, was the time where I realized the impact Windows had in those days in the data center and on the desktop, that Windows was everywhere. And one of those worms would shut everything down for a day or two. And it happened in a 3-year span, it happened 5 times.

**Izar Tarandach (6:10)**: One lesson that I took away from this was that you can have the most modern, most well-defended, most well-patched data center. If you can't get to your data, then it's not worth much.

**Chris Romeo (6:23)**: It's true. The access layer is crucial.

**Izar Tarandach (6:28)**: So, yeah, it's, again, it's not schadenfreude and very, very far from it. I really wouldn't want to be in those people's shoes or anything. But it does remind me of a lot of stuff that we've been talking about here in the podcast. We talked about security utilities. We talked about the responsibility of software that's like base software for the internet, for the usability that we need. And I think that we even managed to talk a bit about fragility. And one thing that, one lesson that I'm taking away from this weekend is that, you know what, yeah, products will have bugs. That's just the way it is. It will always be like this. Things will happen.

**Matt Coles (7:11)**: But, and by the way, we should highlight that the, from based on the information that's available publicly for this, for this issue, um, this was not a security vulnerability, right? This was a, this was a bad patch, right? This was a bad update that went out and, and it ended up causing a denial of service on a grand scale, but it's, it was not a, it wasn't a, um, uh, an exploitable It wasn't a problem that became exploited by an attack.

**Izar Tarandach (7:43)**: That's why I call it a dress rehearsal to an actual incident, but without taking it, the value of it as an IT incident. But my point about Fragility was that if I understood it correctly, and believe you me, I'm far from knowing any kind of details or anything, it was a product-level thing that even the products running elevated impacted the system itself in such a way that it became an operating system error?

**Matt Coles (8:12)**: Well, actually, I can tell you this, right? I can tell you that, you know, based on the information that we have available, right, this was a blue screen of death on Windows, right? And particularly because it was accessing invalid memory. So it was a pointer dereference, right? This is a common problem in C programming. We don't, obviously we don't know what language was used here, but you know, this happens often in drivers. So if you have a driver that's running at a very low level, right? And that driver has a bug where it, you know, references a null pointer or references a bad pointer, then you get a blue screen of death. There's, and this is, by the way, this is Windows being, this is Windows being intelligent about, Sorry, be careful. This is Windows being defensive, as it should, because when you have an exploitable vulnerability, when you have a, say, a buffer overflow in code, an RCE, right? If you had a remote code execution vulnerability in your source, in your code and in the driver, right? A crash is the right approach.

**Chris Romeo (9:23)**: Why is this?

**Izar Tarandach (9:24)**: No, no, no, no.

**Chris Romeo (9:26)**: I got a question to ask then. Why is this still a thing? That's a modern operating system. Because like Linux doesn't have Linux.

**Matt Coles (9:34)**: You can, I mean, you can still cause a kernel panic.

**Chris Romeo (9:38)**: You can cause a kernel panic, but man, it doesn't happen very much anymore. I mean, in the old days, kernel panic would happen once a day. You're like, why'd my system shut down? Oh, look, uh, kernel panic because I ran out of memory.

**Izar Tarandach (9:50)**: I think it's a couple of things, but just to relate to what Matt said, that's not Windows being defensive. That's Windows being Cartman. It's taking the ball and going home, right? So, you know, it's like in this day and age to say, I give up. I just saw no reference. I give up. I can't, I can't, I can't with you guys. I can't even. Okay. You can't do that thing anymore.

**Chris Romeo (10:12)**: Yeah.

**Matt Coles (10:13)**: But when you're operating at such a low level, how do you, how do you avoid that?

**Izar Tarandach (10:17)**: Right.

**Matt Coles (10:17)**: I mean, this is—

**Chris Romeo (10:17)**: you gotta have isolation in the kernel. Like. The year is 2024. If the year was 2005, I would still believe, because I watched like those various worms and stuff caused blue screens of death ultimately. That was eventually the code was bad. It would just, it would just kill the machine because there was bugs in the, in the worm code itself. Right. But like, it seems like in 2024, we've had 20 years to build an operating system kernel that can provide isolation for individual drivers and not let one driver trounce the whole thing. It seems like they should have done that.

**Izar Tarandach (10:54)**: That whole hypervisor virtual machine thingy stacky that puts everything separated and, you know, what was it, microkernels and personalities in the kernel and all that stuff. And you want to tell me that one single null reference and pop goes the weasel? That's not being, that's not doing the right thing.

**Chris Romeo (11:18)**: That's not being resilient. Like we've been talking about resiliency for the last couple of years in cybersecurity as an umbrella level topic. And that's not a resilient system that you could rely upon to fly an airplane, for example, if it's got that level of—

**Matt Coles (11:33)**: Hopefully you're not relying on Windows to fly your airplane, but—

**Chris Romeo (11:36)**: They talked about it. They talked about it 20 years ago. It was on the table. There was some, I remember seeing some research going into it, but ultimately it wasn't.

**Matt Coles (11:47)**: Hey, Windows IoT Core, maybe, but you know, it's Raspberry Pi or nothing.

**Izar Tarandach (11:52)**: But now here's the funny connection. A couple of episodes ago, we spoke about the CISA and their advisories, alerts. I don't remember what it's called. And it came out with the use memory safe languages.

**Chris Romeo (12:08)**: Yep.

**Izar Tarandach (12:09)**: Well, I guess that the, the egg is on the other foot now.

**Chris Romeo (12:14)**: Because, uh, how does an egg get on the other foot? Hold on.

**Izar Tarandach (12:19)**: I waited to see how long that meme would trigger. But, uh, no, really, I mean, we lasted it. But here we are, humbled by a single null reference over a weekend.

**Chris Romeo (12:33)**: And some people are going to feel this for a long time. Isn't Microsoft rewriting lots of Windows stuff in Rust though?

**Matt Coles (12:40)**: They are.

**Chris Romeo (12:40)**: I thought they were making a move towards Rust.

**Matt Coles (12:41)**: Linux is too, right?

**Izar Tarandach (12:42)**: Linux is.

**Matt Coles (12:42)**: Linux has. Yeah.

**Chris Romeo (12:44)**: But I thought Microsoft specifically was as well going towards Rust as their dominant programming language.

**Izar Tarandach (12:49)**: Yeah.

**Matt Coles (12:49)**: Their new, uh, the new SDL, the new, um, Microsoft SDL and trusted computing, whatever they're doing, was, was intending to use languages like Rust. I don't know if they, yeah.

**Izar Tarandach (13:01)**: I, for one, I can't wait for Visual Rust++ hash and .rust and the whole amazing ecosystem of all things that look like Rust, but they're not Rust that Microsoft is going to come up with.

**Chris Romeo (13:17)**: Don't forget Visual Rust.

**Izar Tarandach (13:19)**: Yeah. Visual Rust++ hash.

**Matt Coles (13:21)**: Now, is it true that this, this didn't affect all versions of Windows? This was only 10.11 that was impacted.

**Chris Romeo (13:27)**: Yeah. Well, there was a funny tweet. I don't know if you saw this from Southwest Airlines that said, uh, haha, joke's on you, we're still flying today because we use Windows 3.1. Which, I mean, there, there, there's, there is an argument in security that we don't make enough for protection by running things that are so ancient that they don't have the problem that everybody's talking about because they were written so long ago and from a different code base.

**Matt Coles (13:57)**: That's the, that's the FAA way, isn't it? They're running mainframes still for tracking flights.

**Chris Romeo (14:04)**: Yeah.

**Matt Coles (14:04)**: Because they just work.

**Chris Romeo (14:06)**: But I mean, just using, using antiquated, and I remember, um, this situation coming up with, uh, Heartbleed. People were running some, some, I remember going through the process at Cisco where we were looking at all of the products trying to determine what products were vulnerable to Heartbleed. And there was a subset of products that were running an OpenSSL that was so old, didn't have the Heartbleed problem in it. And you know, like, yep, those are good. Take that out of the advisory. That doesn't need to be in there.

**Matt Coles (14:36)**: You know, you know that there's, there's always going to be that one team is like, well, see, we didn't do HTTPS, so we're not impacted.

**Izar Tarandach (14:42)**: Yep.

**Chris Romeo (14:43)**: We just did plain text for everything and it's good.

**Izar Tarandach (14:47)**: See, can't pitch a bug that's not there.

**Chris Romeo (14:52)**: I don't know. That's not a security by— what would this be? Security by laziness?

**Matt Coles (14:57)**: There is a, there is a CWE, there is a CWE for reliance on, on outdated and obsolete software or components.

**Chris Romeo (15:05)**: Well, it sounds like that should be a positive though, not a negative.

**Izar Tarandach (15:08)**: No, it's called security, security by learned helplessness.

**Chris Romeo (15:12)**: Security by learned helplessness.

**Matt Coles (15:13)**: It's called putting your head in the sand. It's, it's putting your head in the, I'm recommending this forward.

**Chris Romeo (15:20)**: Take old components that are at least 20 years old and try to cobble them together into a solution. Yeah.

**Izar Tarandach (15:25)**: Things that you know are not going to be patched anymore.

**Matt Coles (15:28)**: Actually, you know, this is, this has a real, this has a real world analogy. You know, people who are all, there are, there are a group of people, there are a collection or population of people who, um, look for vehicles that are prior to some time period, 2000 or something like that, because they aren't reliant on, they don't have so many microprocessors. They don't have computer code. They don't have telemetry. They don't have software updates for their entertainment systems. And so you have less, if you're a conspiracy theorist and interested in that sort of thing, then, you know, somebody taking over the vehicles like they did with OnStar and whatnot in the Jeep situation a few years ago, right? So, There is a—

**Izar Tarandach (16:10)**: because, because you can't just stop in some crossing in any random city and get your car taken over by somebody with a handgun. Well, it's a robbery, geez, step off.

**Chris Romeo (16:24)**: But now there is, I mean, that is, that is a good analogy or a good connection there, Matt, that, that could—

**Izar Tarandach (16:29)**: it is.

**Chris Romeo (16:29)**: There are cars that are non-computer-based.

**Izar Tarandach (16:33)**: And you, you know what? It's not that far because I, I have told already a bunch of people. You are not going to see me in a car with an autopilot anytime soon because I know the people who write those things.

**Chris Romeo (16:50)**: Well, what are you trying to say?

**Izar Tarandach (16:53)**: That, uh, I don't like the way that roads are built, that paved roads perhaps are not, that not every guardrail works well.

**Chris Romeo (17:04)**: I like where you're kind of You're kind of circling this issue.

**Matt Coles (17:09)**: You might, you might be, you might be driving one right now. How do you know?

**Chris Romeo (17:13)**: Yeah. It's a, it's a trust issue then. You're saying, so you're saying you don't trust—

**Izar Tarandach (17:16)**: that's a very good point.

**Chris Romeo (17:17)**: You don't trust the technology behind those cars.

**Izar Tarandach (17:20)**: Oh, I trust the technology. I don't trust the implementation.

**Chris Romeo (17:24)**: Isn't it one and the same?

**Izar Tarandach (17:25)**: Nope.

**Matt Coles (17:26)**: So, so are these friends of yours? 'Cause they're, they're not gonna be friends of yours.

**Izar Tarandach (17:30)**: Not for long. Not for long. But it's okay. They drive with the autopilot.

**Chris Romeo (17:39)**: So what do we, what do we do with this?

**Matt Coles (17:41)**: Like, what is, well, actually, so there's one, there's one actually really important concept. I think that, that we need to tease out here is if by some sheer luck you were running Mac and Linux, you weren't impacted by this. Now, having said that, Linux of course has the regression vulnerability and, and I read an article this morning that there's yet another regression-related vulnerability. Of course there is. Of course there is.

**Izar Tarandach (18:07)**: Of course there is.

**Matt Coles (18:09)**: So, so you're, you're, at what point, at what point do we, I guess, is there a point where people who make technology decisions, whether that's user, users at home or, or enterprises making decisions for their, for their, you know, their collection of users. Then it doesn't become luck as to what is secure or safe or randomness in my— this was completely random luck, right? If, if the, the, uh, a chain of events had to occur for this issue to, to, to be as, as prevalent as it was, right? It happened that this, you know, that the organization that put it, put out the patch is used almost everywhere apparently.

**Izar Tarandach (18:57)**: Okay.

**Matt Coles (18:58)**: It got very successful very quickly. They put out a patch that broke and then caused the ripple effect, chain effect, right? I mean, you know, on over the weekend you had, or last week you had gamers who couldn't game on Xbox Live 'cause Xbox Live was down. By the way, this is talking about cloud services, not just Windows servers, right? This took out—

**Izar Tarandach (19:18)**: Yeah.

**Matt Coles (19:19)**: It took out Xbox Live. It took out, um, uh, how many, how many airlines lost their ability to do, um, uh, reservations and, and bookings and things like that?

**Chris Romeo (19:29)**: I think it's more the other way around, which airlines were actually able to operate. It was very, very short list.

**Izar Tarandach (19:33)**: Right.

**Chris Romeo (19:34)**: Well, no one. Yeah.

**Matt Coles (19:35)**: And then we had, and then we had hospitals that had to, uh, you know, because hospitals use, you know, Windows machines to run MRI machines and, and CAT scans and other diagnostic equipment, right? So you had this cascade effect, right? But if they, if somebody had made a decision decades ago to stick with System V Unix, right? Or, or a Linux variant, right? I mean, Europeans were, were big on running Linux at some point. Did, was, was Europe taken out the way that America was? Um, you know, so at what point did technology decisions stop? Can we get away from the luck factor? Like, this is accidental, right? This isn't intentional, but, uh, it was pure luck that, that it happened.

**Izar Tarandach (20:24)**: I think let's not forget that what some people call luck, other people call probability. And in the long tail of time, the probability of an incident is 1.

**Chris Romeo (20:35)**: Yeah.

**Izar Tarandach (20:35)**: So, It was about to happen, right? At some point it would. We've been talking forever about the castle of cards that this whole thing runs on. And at some point, you know, something is going to get knocked out. I just go back to the points that we are, and yeah, I'm going to agree with you. We were lucky that this was actually a bug and not an active incident, right? Complications of one of those might have been way bigger. But it just goes to show how vulnerable we are and how, you know what, even though now we have LLMs running everywhere and looking over our shoulders while we write code and whatnot.

**Matt Coles (21:21)**: By the way, LLMs run on Linux.

**Izar Tarandach (21:23)**: So, yeah, no, no, no. But what I mean is, what I mean is people are adding a level of trust now. We use that word in connection with LLMs all the time because I think it's like the root of the problem. But we are adding now another level of trust in the whole castle of cards because now we have this magical LLMs writing code for us. And I don't think that people realize that it's not very far away from an autopilot in the middle of the street, you know. Sometimes you just can't trust what that thing is doing.

**Chris Romeo (21:55)**: Let me—

**Matt Coles (21:55)**: so are you saying that we need to, um, we need to reinforce Development process.

**Izar Tarandach (22:03)**: No, I think, yeah, I think that what I'm saying is that, you know, this is going to sound like job security, but yeah, we are still going to have to trust more into very talented developers than into very talented security people. Technology is coming in big, but we are not going anywhere anytime soon.

**Chris Romeo (22:21)**: That's true for at least the next couple of decades. But let me put this in perspective because I had an exchange with a friend of mine who's a non-technology person. He asked me, he said, so, you know, what do you think about this, this thing that's happened here? You know, were you surprised by it? And I said, funny thing is with what I know about technology in lots and lots of big companies, I'm surprised it doesn't happen once a week. With all of the, think about all of the chances and coming back to probability, like what, with all of the chances that this could happen, multiplied by the range of a particular product as far as like how many computers it's installed in around the world. Like if we took the top 10 most installed things, I'm just amazed that it does— this doesn't happen more often.

**Matt Coles (23:13)**: Yeah. I, I mean, and this is not the first time we've seen something like this happen, right? We had, um, I think it was Symantec a number of years ago had a, had a .dat file update And it, you know, took out their user base, right?

**Chris Romeo (23:24)**: Yep.

**Matt Coles (23:25)**: And so it's, it is not, it is not without precedent. It's just the scale.

**Chris Romeo (23:33)**: Yep. And so I think that's, uh, a good place to wrap up for today based on a little bit of commentary. So, uh, we'll be back again next week to unpack something else. Who knows what it'll be? Only you can know.

**Matt Coles (23:47)**: We never know.

**Izar Tarandach (23:49)**: I have no idea. Even now, I have no idea.

**Chris Romeo (23:53)**: All right. Thanks for listening to Security Table.
