Shostack + Friends Blog Archive

The Discipline of "think like an attacker"

John Kelsey had some great things to say a comment on “Think Like An Attacker.” I’ve excerpted some key bits to respond to them here.

Perhaps the most important is to get the designer to stop looking for reasons attacks are impossible, and start looking for reasons they’re possible. That’s a pattern I’ve seen over and over again–smart people who really know their system also usually like their system, and want it to be secure. And so they spend a lot of time thinking about why their system is secure. “Nobody could steal our PIN because we encrypt it with triple-DES.”

So this is a great goal. I have two questions: first, is it reasonable? How many people can really step outside their design and regard it with a new perspective? How many people can then analyze the security of a system they’ve designed? (Is there a formal name for this problem? I call it ‘creator-blindness.’) I’m not sure exhorting people to think like an attacker helps. This problem isn’t unique to security, which brings me to my second question: is it effective? I was once taught to read my writing aloud as a way of finding mistakes. I teach people to diagram their system and then use a system we call “STRIDE per element” to help people look at it. By giving people a structure for analysis, we help them step outside of that creator frame.

A second goal of that “think like an attacker” exhortation is to get people to realize that, in order to know whether their system is secure, they need to learn something about what tools and resources an attacker is likely to have.

So, for a moment, let’s assume that this is a reasonable goal, and one we can expect every developer who hears the phrase to go pursue. Where do they go? How much time should they devote to it? Again, I’m not talking about the use of the phrase within the security engineering community, but in software engineering more generally. Secondly (again), there’s the question of “is this the most effective way to push people?”

Third, there’s a mindset of being an attacker. I don’t know how to teach that. It’s not just about intelligence–I’ve worked with stunningly brilliant people who don’t seem to have that mindset, and with people who are much less brilliant in that brute-force impressive brain sense, but who just seem to have the right kind of mind to break stuff.

Well, that I can’t argue with. All I’ll say is that we’ve been exhorting people to think like attackers for years, and it hasn’t helped.

I believe that security analysis is a skill which can be taught. The best have both talent and have worked to develop that talent. I hope and expect that we can figure out how to do so. Figuring that out will involve figuring out what pedagogic approaches have failed, so we can set them aside, and make room for experimentation, chaos, and — we hope — actual improvements. I believe that, when asked of non-security experts, the ‘think like an attacker’ is on that list of things we should set aside.

Finally, a side note on the title. If you’re indisciplined, feel free to skip to about 3:10.

Originally published by adam on 22 Sep 2008
Last modified on 29 May 2017
Categories:   Orientations   Security   threat modeling   Usability