We Don't Know What We Don't Know
With Chris Romeo, Izar Tarandach, and Matt Coles
Certificate pinning is a security measure used in computer networking and something Chris candidly admits to his lack of understanding.
About this episode
Certificate pinning is a security measure used in computer networking and something Chris candidly admits to his lack of understanding.
Matt and Izar explain certificate pinning, a client-side operation that adds an extra layer of security to the Transport Layer Security (TLS) protocol and ensures that the client application checks the server's certificate against a known copy of that certificate.
The discussion leads to a reflection on the vast amount of knowledge required in cybersecurity, emphasizing the importance of continuous learning and the willingness to admit and fill gaps in one's understanding. Engage in further research and discussion, and realize cybersecurity is a "never stop learning" discipline.
Follow AI Security Table:
➜ Home: https://securitytable.ai/
➜ X: https://x.com/AISecurityTable
➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/
➜ YouTube: https://www.youtube.com/@AISecurityTable
Transcript · 23 min conversation
0:13Chris Romeo Can anybody really explain certificate pinning to me?
0:16Matt Coles Yes.
0:16Izar Tarandach No, first I feel like I need somebody to tell me how to say it. If it's pinning or pining.
0:22Chris Romeo Pinning.
0:22Izar Tarandach Pinning, because it's pin. Put a pin in there.
0:24Matt Coles It is most definitely pinning. It's P-I-N-I-N-G.
0:28Chris Romeo So it's two N's in a row. If it was pining, you'd be complaining about it, which is what I hear most people do. You don't have to do with it.
0:34Matt Coles Yes. If you're going to pine for a certificate, then you're lacking certificates, in which case you have a bigger problem.
0:42Chris Romeo All right. Well, I'm going to go ahead and start us right off, even though we're already recording, and say welcome to everybody on the Security Table because you just entered a conversation that the three of us were having. It was supposed to be the preemptory conversation. We weren't planning for it, but we started talking about certificate pinning. Now I want to pull on this thread.
0:58Matt Coles And actually, Chris, if I may, before we all came together, Isar and I were having a good debate about the newly released CVSS version 4.0.
1:09Chris Romeo Yay!
1:10Matt Coles Which is now in public preview, I should say. Not fully released as in GA, but in public preview.
1:14Izar Tarandach And everybody should read it, and everybody should comment if they feel the need, because it's a great piece of work.
1:21Chris Romeo Absolutely. Okay, it gets the ISAR seal of approval. That doesn't come very easily, folks. That's something that, uh, he only hands out once a decade. So I mean, that's great.
1:30Izar Tarandach When did I say the security journey was good?
1:34Matt Coles Uh, all right, so, so, so retract everything you just said about CDSS. Is that what you're saying?
1:41Chris Romeo No, well, he can use two. I mean, if he really wants to, he can use two. It's fine.
1:45Izar Tarandach It's okay. Security journey was really good, so It counts as 2 levels.
1:51Chris Romeo 2 levels. There we go.
1:53Matt Coles So cert pinning.
1:54Chris Romeo Yeah, let's talk about this because I'm— so let me just give a kind of an intro to this because I consider myself to be mildly intelligent and I don't know, I'm gonna just say, Gab, thank you for your vote of confidence. I'm gonna say like cert pinning I mean, this is kind of like a— this is a dangerous thing. I'm like, I'm having a confessional moment here. I don't know that I've ever really understood what it is, and I think I've— I can explain how browser certificates work and certificate authorities and all the pieces that fit together, but certificate pinning, I don't think I've ever managed to get a solid view of it in my head. Well, so if you could enlighten me, I will. This will be— this will be a great time.
2:40Matt Coles I'll, I'll start, uh, if I— if that's okay. Um, so first off, I'll say the, the, the go-to reference that I use whenever I'm talking with a team about, well, you know, you should look at using cert pinning, is there's an OWASP article that really does a good job of outlining what certificate pinning is, what does it mean, what are some of the implications. And so cert pinning is an approach, a mechanism for addressing trust of certificates between one system to another. In a way that the— I guess, the PKI or the notion of a certificate is issued by somebody else. And so you can trust them, so you can trust the cert. There's some challenges with that in certain environments. So cert pinning really is based on certain pieces of information in a certificate, either in a parent to a child, to a child or to a leaf cert or to the leaf cert itself. And this can also work if that leaf cert is a self-signed certificate because obviously it has no parent or no recognizable parent that you can decide to trust the certificate or not simply because it's something you know about and you can make that explicit decision. So if you're looking at a certificate that's coming from a web server, you're going to want, you're going to look at the information in that cert. Traditionally, you're going to look at that information, that's sort of what your browser does, right? And says, oh, this comes from a trusted signature source, right? The chain I can trust. And I can, I think that this is associated with this host because it has the O and the OU, et cetera. And that's great if that's not a self-signed cert. If it's a self-signed cert, you don't really have a way of trusting it. There's no third-party attestation of that trust by the signature chain. And so cert pinning provides a stopgap measure of being able to say, given these specific parameters that I can validate in that certificate, and that may use— that will usually include things like this, the fingerprint of the cert or the issuer of the cert or the OU, in case of the issuer, will be the same there for self-signed cert, information on that certificate to say, I trust this explicitly. So cert pinning is a way of explicit trust of a certificate, sort of independent of which chain it came from, or potentially including the chain, simply because you may get multiple— a browser may look and may have a cert store that will trust multiple certificates, and you really want to trust one, not any of, if that makes sense.
5:24Chris Romeo One and only one.
5:25Matt Coles One and only one based on this criteria, right? You may, you may pin the one, or you may pin to a an intermediate as the issuer of, so any of the ones that are issued by, but as opposed to all.
5:38Izar Tarandach Which is where things start getting complicated for the developer, because then you have, preferably you have to preload that into the app so that when you publish the app, it goes out with already the data.
5:50Matt Coles You have to know the knowledge ahead of time.
5:52Izar Tarandach There are some schemes that let you do it at the first time that you see it. But then again, you might, depending on the environment that you are in, you might be shooting yourself in the foot because the first thing that you're going to see is basically the one that's trying to intercept you. So you—
6:10Matt Coles Right. And this is where if you had a, if you, if you didn't have a PIN already established, you may, you know, for instance, have a call home mechanism. Hey, should I trust this? Or give me my PIN list.
6:19Chris Romeo Right.
6:20Izar Tarandach But then you start getting more and more complex and complexity introduces Bricks and bricks invalidate the whole scheme.
6:28Matt Coles Yeah, what do you do if you can't? What happens if you can't call home? Are you a brick, you know, at that point?
6:33Chris Romeo So, so this reminds me of an episode of The Office, as things tend to do, where Michael Scott is asking Oscar the accountant to explain something to him, and after a while he looks at him and he says, can you explain this to me like I'm 5? And then, and then Oscar uses this illustration of, well, your mommy and daddy give you $5 to start a lemonade stand. And then, um, so I feel like I'm still at that lack of clarity. I don't know that I need you to explain it to me like I'm 5 because that might be too much. But so is certificate pinning a client-side operation or a server-side operation?
7:10Izar Tarandach Client-side operation.
7:11Chris Romeo Or both?
7:12Izar Tarandach And the thing is that normally when you're verifying certificates, you are using the normal mechanisms of PKI against some fields and against a chain of trust. You are trusting that whomever gave you that key is somebody who's trusted by somebody else and it's trust all the way down, right? Well, now, once you introduce pinning, you're putting on top of the trust, you are saying, I'm expecting to see a certain certificate that has a certain characteristic to it. If it's a fingerprint, if it's a specific public key, if it's a, that there's a number of, a small number of expected good fields in the X.509 tree that you can use for that.
8:01Matt Coles Generally immutable fields that you wouldn't expect to be able to be tampered with easily.
8:07Izar Tarandach And then basically you're saying, after we did all the crypto validation, after we did all the chain of trust validation, I'm going to look at this one thing to see if it's the specific certificate that I'm looking for. And for example, one of the things that this breaks is, again, if you are in an enterprise environment and they have interception of encrypted content for DLP, for whatever reason it is, that's when you're going to see this thing break.
8:39Chris Romeo So if I use Let's Encrypt, which, which has a 90-day lifecycle, by default on certificates. Do I have to refresh my certificate PIN information every 90 days when it refreshes and gives me a new cert for my application?
8:55Izar Tarandach To the best of my understanding, and I could be wrong here, you do have the— you can play games with the valid, not valid before field and things like that so that even if the certificate is expired, you can still do the validation. But somebody who wants to be actually strict about that kind of thing, if, for example, they are writing a mobile application, they will have to reissue the whole app when the certificate that they're pinning changes to have the new value.
9:33Matt Coles Alternatively, what you pin to is the parents or the issuer.
9:39Izar Tarandach True.
9:40Matt Coles Such that when I do cert validation, if the issuer of the presented server cert is not the pinned certificate, I'm going to ignore it because it's not good, right, from my perspective. So keep in mind, a server can present whatever certificate it wants. The browser is going to accept whatever is in the cert store and, or, you know, validated through the cert store. And in theory, that certificate is associated— the leaf cert or the presented cert is is associated with the host that you're talking to, and if you do O and OU validation, right? But it may be issued by somebody else than you expect, right? So the pinning is really not, did I get a cert that has a trust chain that is correct? It's, did I get a certificate that meets my criteria for the one I want specifically?
10:35Chris Romeo All right, so now I'm gonna ask a question that you guys are just gonna hate having to answer. But that's okay. I'm going to do it anyway.
10:42Matt Coles What else?
10:43Chris Romeo What are the— what are the— what are the— what are the threats? This is a complete joke because you're going to love this question. What are the threats that I'm trying to counter here? Like, if I do— let's, let's threat model. Like, let's threat model a, a system or an application, let's say, that doesn't have certificate pinning. What are the threats where, where threat or certificate pinning is the mitigation? To counter the threat?
11:08Matt Coles So the, so one of them, an easy one here, just I'll bring it back to the, did I get the one I expect? The threat is subversion through another trust chain, right? So if a certificate could be issued for the same host against two parents, then, uh, or, or by two parents, uh, and you want to know you're connecting to one versus the other, um, That's, that's the, the threat here is that somebody could, somebody who owns a root, a, the ability to sign from a root certificate but is not the one you expect, could create a legitimate cert that you would then attempt to communicate or talk to that server with. But you don't trust that chain even though the certificate chain is trustworthy in terms of its validity and its signature and its et cetera, hasn't been revoked, et cetera, but it's not the one you expect. So you could be talking to a spoof service or to a lot of—
12:04Chris Romeo Is that a malicious? So you're saying that secondary chain that I'm interacting with the web app, I'm trusting it and I shouldn't, that would be potentially a malicious actor who created that secondary certificate.
12:17Matt Coles Potentially malicious, although I think in practice, what I've seen is really, not malicious, but probably not the one you want in terms of service capability, or like an IoT device, for instance, right? Talking to a— IoT devices have limited capabilities as it is in some cases, right? They don't necessarily have the ability to do full cert validation in some cases, right? Because of algorithm choices and software versus hardware implementations. And so your threat isn't necessarily always going to be that the server is malicious, but, but maybe it just isn't the one you intend to be communicating with. Okay.
12:59Chris Romeo Any other threats?
13:00Izar Tarandach At the end of the day, it's threats against the spoofing of the other side and tampering with communication in the middle. And, uh, we, we have had enough, uh, examples of not on the application side, but on the issuer of certificates. Doing some crazy stuff and issuing a perfectly valid certificate to someone who shouldn't have it with good names and all that. And, uh, the fact that you, you can be more selective in terms of which specific certificate you're going to accept rather than just a generic certificate that is valid based on its chain of trust just gives you that one more selection towards, uh, Again, spoofing, but at a business level thing, I think it makes you doubly sure that you are talking to the provider that you are talking to and not to someone who just ended up having a certificate that's sufficiently looking like the one that that person would have.
14:09Matt Coles Right. And again, I guess the other piece here is— I'll just mention, and I don't know if there's a threat. It's an ejection threat, I guess, at the bare minimum. But as I mentioned, some devices may not have cryptographic capabilities to perform proper validation of either of a certificate and its issuer, meaning its signature and that. So there, you're making an explicit choice to trust a particular cert or set of certs.
14:36Izar Tarandach in lieu of the validation that you would do during a traditional X.509 PKI, uh, exchange, which is not ideal, but for that class of device, many times it's the only thing that can be done. So it's better than nothing.
14:53Chris Romeo Okay, better than this. That's good.
14:55Matt Coles Not having TLS.
14:56Chris Romeo Yeah, definitely, definitely. All right, so I feel like you guys have really, uh, You guys really educated me here today around the security table. Like, I learned something and, you know, hey, I think there's a lesson here for people out there and it's taken me 20-plus years of my career. Guess what? You don't know everything. I don't know everything. I'm not afraid to ask people when I don't know something at this point in my career because it helped me. I now have an understanding of certificate pinning that I'll carry with me for at least 6 months before I forget about it and have to ask these guys again.
15:26Matt Coles And by the way, you know, we could be completely full of it. Yeah, so check us. Get comments here.
15:32Izar Tarandach Definitely feel free to to counter us.
15:35Matt Coles And this is, by the way, I think not traditional, not expected browser behavior, right? Yeah, the folks who set the rules for browser behavior, this would be very bad, I believe. So this is definitely a non-standard thing for a lot of people, but it does come up in situations often. In fact, I talk about certificate pinning frequently, and not just in the TLS case, right? There's other uses of certificates, obviously, where cert pinning has validity as a mechanism. Something you touched upon, Chris, I know we sort of had an agenda for today, but I think it's really important what you just highlighted about there is so much stuff for us to know as security professionals. We're not even developers, we're security people, right? And, you know, and it's just, and I think about this actually pretty frequently, you know, I'm Obviously, I'm not a spring chicken anymore, and I've got a lot of years under my belt in my career. And so, a lot of people come to me for advice and guidance, and I work with a lot of teams, you know, in my day job. And the breadth of technology and the business domain and the regulatory domain, and that's just security, not to mention privacy and and other safety and consumer use and a whole host of other things. It's like there's a bucket of information, and then at some point, it starts flowing off the top, and you don't know what you're losing at the top, right? I always joke that— I joke sort of half-heartedly. I do have a sticky note here somewhere that I have a conference presentation waiting to be written called TIL, right? Today I Learned. Probably something, either something I forgot, or something I didn't know that I needed to know that I didn't have the skills for originally. We're in a constant learning pattern. People talk about AI. I'm like, I don't know AI. I don't even have the capacity for it. Kubernetes, sometimes that's a stretch. Although, you know, that's happening more and more frequently for me at least, right? Others are probably already steeped in that. Whereas device security, yeah, that's, I can tackle that. That's in my, still in my bucket of, of knowledge, right? So, it's really important, I think that, and I don't know what your— actually, I do have a question for you because actually, I thought about this. How do other, not only highly successful people, but other security practitioners especially, how do they deal with this? Like, everyone's— a lot of people seem to be very far ahead in a lot of areas and, like, keep a lot of stuff in their head and they know about it and, like, it's like a trivia game. Like, I wouldn't be good at Jeopardy. but some people are really going to be good at security jeopardy, right? And there's esoteric knowledge and there's really some basic stuff. And so how do other people make this work?
18:23Izar Tarandach So I don't know about highly successful people, but I can tell about myself. Let's just say that every time that a new release of Chrome comes out and I go through the release log, the only thing I'm looking for is stuff that's connected to tab management. 'Cause anything that can help me manage 200 tabs that I have open at any given time, that's the valuable stuff for me. So yeah.
18:52Matt Coles Have you discovered the reading tab, the reading pane by any chance?
18:56Izar Tarandach I did. I think that I must have found a buffer overflow in there. 'Cause I just put one more and the whole thing closed. But you know, I think that, One of the most powerful tools that I found in that sense, and Matt, thank you so much for bringing this up. This is something that I don't think that we can explore enough. I think that one of the most powerful tools that I found to deal with this was the ability to get myself to sincerely tell someone, I don't know. I'll get back to you with an answer. But right now at this moment, I don't know and I don't want to talk out of my behind. So I prefer to shut up and come back to it later than to start moving my hands very fast in the air as I usually do and, and say something that I'm going to regret later.
19:51Chris Romeo I mean, that's wisdom. That what you're, what you're describing there is wisdom though, because I would have almost the same way early in my career. I felt a pressure to answer.
20:01Matt Coles Oh, I'm on the spot.
20:02Chris Romeo I gotta, I gotta have an answer. So I gotta dance around and try to, try to dance in such a way that the answer makes sense and people don't. Don't think that I have no idea what I'm talking about. But the older I get, the longer I've been in this industry, the more I'm like, that's— I have no idea what you're talking about. Like, I probably have an opinion if you— if I— let me ask you a couple of questions to you so you can teach me and explain it to me. And, you know, I'm an expert in my own opinion. That's, that's something I'm an expert. I'll claim expertise in. But so I think we've— for folks like us that have been around for a long time, we've, we've seen so many different things. That we can usually, we can form an opinion about the security of something, but often we have to ask questions like, okay, tell me more about this. How does this work? What is that thing you just, you just said that word? I don't know what that means. And once you tell me about it, then I can give you, at least give you an opinion on it. But I think it's important. I think that's something everybody can learn though, is nobody has all the answers in this industry. And if somebody claims to, they're— run, run away from them quickly.
20:59Izar Tarandach Because, and to take this to the place where we double quotes, leave. When you are threat modeling and somebody comes to you with a new design, and sometimes they'll say, hey, in here we put an AWS FoxDigger super duper new box.
21:16Matt Coles And you know what?
21:17Izar Tarandach I have no clue what that service does. I have no clue how it's configured. I have no clue how it can fail. I have to tell people, hey, wait, I'll go check this and get back to you. But I'm not going to start, oh, CFF, SSRF, whatever F. And unfortunately, sometimes that happens. And we have to be honest enough to say, if I do that, I am actually putting a bump on the road, which I don't have the right to do. I'm here to facilitate things, not to make them harder. So yeah, we have to leave our egos in the door when we do that.
21:56Chris Romeo That's good advice. That's good advice. Well, tell you what, we're going to end this episode of the Security Table. Where we kind of started diving into certificate pinning almost on accident. I learned what it was, but then we had some reflections on our careers in the industry and some advice for people that are new to it. But stay tuned, we'll be back shortly with another episode of the Security Table.
Transcript supplied by AssemblyAI. It may contain transcription errors.
View the original transcript on BuzzsproutOriginally published as The Security Table. Part of the AI Security Table archive.
