Think Like An Attacker?
One of the problems with being quoted in the press is that even your mom writes to you with questions like “And what’s wrong with “think like an attacker?” I think it’s good advice!”
Thanks for the confidence, mom!
Here’s what’s wrong with think like an attacker: most people have no clue how to do it. They don’t know what matters to an attacker. They don’t know how an attacker spends their day. They don’t know how an attacker approaches a problem. Telling people to think like an attacker isn’t prescriptive or clear. Some smart folks like Yoshi Kohno are trying to teach it. (I haven’t seen a report on how it’s gone.)
Even if Yoshi is succeeding, it’s hard to teach a way of thinking. It takes a quarter or more at a university. I’m not claiming that ‘think like an attacker’ isn’t teachable, but I will claim that most people don’t know how. What’s worse, the way we say it, we sometimes imply that you should be embarrassed if you can’t think like an attacker.
Lately, I’ve been challenging people to think like a professional chef. Most people have no idea how a chef spends their days, or how they approach a problem. They have no idea how to plan a menu, or how to cook a hundred or more dinners in an hour.
We need to give advice that can be followed. We need to teach people how to think about security. Repeating the “think like an attacker” mantra may be useful to a small class of well-oriented experts. For everyone else, it’s like saying “just ride the bike!” rather than teaching them step-by-step. We can and should do better at understanding people’s capabilities, giving them advice to match, and training and education to improve.
Understanding people’s capabilities, giving them advice to match and helping them improve might not be a bad description of all the announcements we made yesterday.
In particular, the new threat modeling process is built on something we expect an engineer will know: their software design. It’s a better starting point than “think like a civil engineer.”
[Update: See also my follow-up post, “The Discipline of ‘think like an attacker’.”]
Saying “think like an attacker” is exhorting people to learn that skill. Telling people to think like a professional chef is appropriate if they’re going into the catering business. And they will have to learn what that means.
If one does not know how to think like an attacker, and is concerned that they might not do things correctly… good. They’re on the right path to correcting that.
Ryan,
Saying “It would be helpful to you if you learned to think like an attacker” is exhorting people to learn that skill. Demanding that they do it, or implying that they’re stupid for not knowing how to do it is actively counter-productive.
Much more important, most software engineers have failed to take even the “learn to” approach to thinking like an attacker. When my advice is ignored over and over again, I try to think of a new approach that will work better.
Dear Adam: thank you for a cogent, thoughtful and very helpful reply. You certainly have a way with words!
Love, Mom
I think this exhortation has several goals.
Perhaps the most important is to get the designer to stop looking for reasons attacks are impossible, and start looking for reasons they’re possible. That’s a pattern I’ve seen over and over again–smart people who really know their system also usually like their system, and want it to be secure. And so they spend a lot of time thinking about why their system is secure. “Nobody could steal our PIN because we encrypt it with triple-DES.”
Back in my consulting days, I don’t know how many times I wound up finding major problems by just sitting there with one of the designers of a system, asking questions from the perspective of someone who wanted to find an attack, instead of someone who wanted to find a reason attacks were impossible. The designer often had all the information needed to find those attacks–that is, often, the attacks weren’t something out of crypto like a replay attack or a side-channel attack, where a non-cryptographer might just not know about them. Instead, it was common for these attacks to be stuff that kind of fell out of the description, things that came out from the second or third probing “why can’t I do X” sort of question. They hadn’t found those attacks because they weren’t looking for them.
A second goal of that “think like an attacker” exhortation is to get people to realize that, in order to know whether their system is secure, they need to learn something about what tools and resources an attacker is likely to have. “Wow, you mean RC4 encryption doesn’t protect the integrity of my data?” But as you said, that does require some studying up on attacks, and there’s never an end to that, there are always more attacks to learn about. (Go read a book on con men and their techniques. Or a paper about lock security from Matt Blaze. Or read about the techniques the commercial botnet/virus criminals are using. Or the tricks being used in espionage. Or….)
Third, there’s a mindset of being an attacker. I don’t know how to teach that. It’s not just about intelligence–I’ve worked with stunningly brilliant people who don’t seem to have that mindset, and with people who are much less brilliant in that brute-force impressive brain sense, but who just seem to have the right kind of mind to break stuff. I suspect (without any data at all to back me up) that this is more like a talent, which can be developed or ignored, but probably not created. A big part of this seems to me to be getting some kind of internal reward from breaking something, so that you’re willing to stand in the shower till the water gets cold thinking about how to break this scheme you just read about, or willing to sit through dinner with a notebook in your hand, muttering to yourself about partitions in the set of pairs of inputs or minimum conspiracy sizes to subvert an election or whatever.
I see, you’re not talking about a person, you’re talking about people. You can teach a programmer to think like an attacker. It does tend to fall apart when you’re talking about programmers.