Certificate pinning is great in stone soup

In his “ground rules” article, Mordaxus gives us the phrase “stone soup security,” where everyone brings a little bit and throws it into the pot. I always try to follow Warren Buffet’s advice, to praise specifically and criticize in general.

So I’m not going to point to a specific talk I saw recently, in which someone talked about pen testing IoT devices, and stated, repeatedly, that the devices, and device manufacturers, should implement certificate pinning. They repeatedly discussed how easy it was to add a self-signed certificate and intercept communication, and suggested that the right way to mitigate this was certificate pinning.

They were wrong.

If I own the device and can write files to it, I can not only replace the certificate, but I can change a binary to replace a ‘Jump if Equal’ to a ‘Jump if Not Equal,’ and bypass your pinning. If you want to prevent certificate replacement by the device owner, you need a trusted platform which only loads signed binaries. (The interplay of mitigations and bypasses that gets you
there is a fine exercise if you’ve never worked through it.)

When I train people to threat model, I use this diagram to talk about the interaction between threats, mitigations, and requirements:

Threats mitigations requirements

Is it a requirement that the device protect itself from the owner? If you’re threat modeling well, you can answer this question. You work through these interplaying factors. You might start from a threat of certificate replacement and work through a set of difficult to mitigate threats, and change your requirements. You might start from a requirements question of “can we afford a trusted bootloader?” and discover that the cost is too high for the expected sales price, leading to a set of threats that you choose not to address. This goes to the core of “what’s your threat model?” Does it include the device owner?

Is it a requirement that the device protect itself from the owner? This question frustrates techies: we believe that we bought it, we should have the right to tinker with it. But we should also look at the difference between the iPhone and a PC. The iPhone is more secure. I can restore it to a reasonable state easily. That is a function of the device protecting itself from its owner. And it frustrates me that there’s a Control Center button to lock orientation, but not one to turn location on or off. But I no longer jailbreak to address that. In contrast, a PC that’s been infected with malware is hard to clean to a demonstrably good state.

Is it a requirement that the device protect itself from the owner? It’s a yes or no question. Saying yes has impact on the physical cost of goods. You need a more expensive sophisticated boot loader. You have to do a bunch of engineering work which is both straightforward and exacting. If you don’t have a requirement to protect the device from its owner, then you don’t need to pin the certificate. You can take the money you’d spend on protecting it from its owner, and spend that money on other features.

Is it a requirement that the device protect itself from the owner? Engineering teams deserve a crisp answer to this question. Without a crisp answer, security risks running them around in circles. (That crisp answer might be, “we’re building towards it in version 3.”)

Is it a requirement that the device protect itself from the owner? Sometimes when I deliver training, I’m asked if we can fudge, or otherwise avoid answering. My answer is that if security folks want to own security decisions, they must own the hard ones. Kicking them back, not making tradeoffs, not balancing with other engineering needs, all of these reduce leadership, influence, and eventually, responsibility.

Is it a requirement that the device protect itself from the owner?

Well, is it?

Threat Modeling and Star Wars

IANS members should have access today to a new faculty report I wrote, entitled “Threat Modeling in An Agile World.” Because it’s May the Fourth, I thought I’d share the opening:

As Star Wars reaches its climax, an aide approaches Grand Moff Tarkin to say, “We’ve analyzed their attack pattern, and there is a danger.” In one of the final decisions he makes, Tarkin brushes aside those concerns. Likewise, in Rogue One, we hear Galen Urso proclaim that the flaw is too subtle to be found. But that’s bunk. There’s clearly no blow-out sections or baffles around the reactor: if there’s a problem, the station is toast. A first year engineering student could catch it.

You don’t have to be building a Death Star to think about what might go wrong before you complete the project. The way we do that is by “threat modeling,” an umbrella term for anticipating and changing the problems that a system might experience. Unfortunately, a lot of the advice you’ll hear about threat modeling makes it seem a little bit like the multi-year process of building a Death Star.

Threat Modeling & IoT

Threat modeling internet-enabled things is similar to threat modeling other computers, with a few special tensions that come up over and over again. You can start threat modeling IoT with the four question framework:

  1. What are you building?
  2. What can go wrong?
  3. What are you going to do about it?
  4. Did we do a good job?

But there are specifics to IoT, and those specifics influence how you think about each of those questions. I’m helping a number of companies who are shipping devices, and I would love to fully agree that “consumers shouldn’t have to care about the device’s security model. It should just be secure. End of story.” I agree with Don Bailey on the sentiment, but frequently the tensions between requirements mean that what’s secure is not obvious, or that “security” conflicts with “security.” (I model requirements as part of ‘what are you building.’)

When I train people to threat model, I use this diagram to talk about the interaction between threats, mitigations, and requirements:

Threats mitigations requirements

The interaction has a flavor in working with internet-enabled things, and that interaction changes from device to device. There are some important commonalities.

When looking at what you’re building, IoT devices typically lack sophisticated input devices like keyboards or even buttons, and sometimes their local output is a single LED. One solution is to put a web server on the device listening, and to pay for a sticker with a unique admin password, which then drives customer support costs. Another solution is to have the device not listen but to reach out to your cloud service, and let customers register their devices to their cloud account. This has security, privacy, and COGS downsides tradeoffs. [Update: I said downsides, but it’s more a different set of attack vectors become relevant in security. COGS is an ongoing commitment to operations; privacy is dependent on what’s sent or stored.]

When asking what can go wrong, your answers might include “a dependency has a vulnerability,” or “an attacker installs their own software.” This is an example of security being in tension with itself is the ability to patch your device yourself. If I want to be able to recompile the code for my device, or put a safe version of zlib on there, I ought to be able to do so. Except if I can update the device, so can attackers building a botnet, and 99.n% of typical consumers for a smart lightbulb are not going to patch themselves. So we get companies requiring signed updates. Then we get to the reality that most consumer devices last longer than most silicon valley companies. So we want to see a plan to release the key if the company is unable to deliver updates. And that plan runs into the realities of bankruptcy law, which is that that signing key is an asset, and its hard to value, and bankruptcy trustees are unlikely to give away your assets. There’s a decent pattern (allegedly from the world of GPU overclocking), which is that you can intentionally make your device patchable by downloading special software and moving a jumper. This requires a case that can be opened and reclosed, and a jumper or other DFU hardware input, and can be tricky on inexpensive or margin-strained devices.

That COGS (cost of goods sold) downside is not restricted to security, but has real security implications, which brings us to the question of what are you going to do about it. Consumers are not used to subscribing to their stoves, nor are farmers used to subscribing to their tractors. Generally, both have better things to do with their time than to understand the new models. But without subscription revenue, it’s very hard to make a case for ongoing security maintenance. And so security comes into conflict with consumer mental models of purchase.

In the IoT world, the question of did we do a good job becomes have we done a good enough job? Companies believe that there is a first mover advantage, and this ties to points that Ross Anderson made long ago about the tension between security and economics. Good threat modeling helps companies get to those answers faster. Sharing the tensions help us understand what the tradeoffs look like, and with those tensions, organizations can better define their requirements and get to a consistent level of security faster.


I would love to hear your experiences about other issues unique to threat modeling IoT, or where issues weigh differently because of the IoT nature of a system!

(Incidentally, this came from a question on Twitter; threading on Twitter is now worse than it was in 2010 or 2013, and since I have largely abandoned the platform, I can’t figure out who’s responding to what. A few good points I see include:

Cyber Grand Shellphish

There’s a very interesting paper on the Cyber Grand Challenge by team Shellphish. Lots of details about the grand challenge itself, how they designed their software, how they approached the scoring algorithm, and what happened in the room.

There’s lots of good details, but perhaps my favorite is:

How would a team that did *nothing* do? That is, if a team connected and then ceased to play, would they fare better or worse than the other players? We ran a similar analysis to the “Never patch” strategy previously (i.e., we counted a CS as exploited for all rounds after its first exploitation against any teams), but this time removed any POV-provided points. In the CFE, this “Team NOP” would have scored 255,678 points, barely *beating* Shellphish and placing 3rd in the CGC.

The reason I like this is that scoring systems are hard. Really, really hard. I know that DARPA spent substantial time and energy on the scoring system, and this outcome happened anyway. We should not judge either DARPA or the contest on that basis, because it was hard to see that that would happen ahead of time: it’s a coincidence of the scores teams actually achieved.

People are The Weakest Link In Security?

Despite the title, end users are rarely the weak link in security. We often make impossible demands of them. For example, we want them to magically know things which we do not tell them.

Today’s example: in many browsers, this site will display as “Apple.com.” Go ahead. Explore that for a minute, and see if you can find evidence that it’s not. What I see when I visit is:

URL bar showing

When I visit the site, I see it’s a secure site. I click on the word secure, I see this:

Dropdown

But it’s really www.xn--80ak6aa92e.com, which is a Puncycode URL. Punycode is way to encode other languages so they display properly. That’s good. What’s not good is that there’s no way to know that those are not the letters you think they are. Xudong Zheng explains the problem, in more depth, and writes about how to address it in the short term:

A simple way to limit the damage from bugs such as this is to always use a password manager. In general, users must be very careful and pay attention to the URL when entering personal information. I hope Firefox will consider implementing a fix to this problem since this can cause serious confusion even for those who are extremely mindful of phishing.

I appreciate Xudong taking the time to suggest a fix. And I don’t think the right fix is that we can expect everyone to use a password manager.

When threat modeling, I talk about this as the interplay between threats and mitigations: threats should be mitigated and there’s a threat that any given mitigation can be bypassed. When dealing with people, there’s a simple test product security engineering can use. If you cannot write down the steps that a person must take to be secure, you have a serious problem. If you cannot write that list on a whiteboard, you have a serious problem. I’m not suggesting that there’s an easy or obvious fix to this. But I am suggesting that as long as browser makers are telling their users that looking at the URL bar is a security measure, they have to make that security measure resist attacks.

Diagrams in Threat Modeling

When I think about how to threat model well, one of the elements that is most important is how much people need to keep in their heads, the cognitive load if you will.

In reading Charlie Stross’s blog post, “Writer, Interrupted” this paragraph really jumped out at me:

One thing that coding and writing fiction have in common is that both tasks require the participant to hold huge amounts of information in their head, in working memory. In the case of the programmer, they may be tracing a variable or function call through the context of a project distributed across many source files, and simultaneously maintaining awareness of whatever complex APIs the object of their attention is interacting with. In the case of the author, they may be holding a substantial chunk of the plot of a novel (or worse, an entire series) in their head, along with a model of the mental state of the character they’re focussing on, and a list of secondary protagonists, while attempting to ensure that the individual sentence they’re currently crafting is consistent with the rest of the body of work.

One of the reasons that I’m fond of diagrams is that they allow the threat modelers to migrate information out of their heads into a diagram, making room for thinking about threats.

Lately, I’ve been thinking a lot about threat modeling tools, including some pretty interesting tools for automated discovery of existing architecture from code. That’s pretty neat, and it dramatically cuts the cost of getting started. Reducing effort, or cost, is inherently good. Sometimes, the reduction in effort is an unalloyed good, that is, any tradeoffs are so dwarfed by benefits as to be unarguable. Sometimes, you lose things that might be worth keeping, either as a hobby like knitting or in the careful chef preparing a fine meal.

I think a lot about where drawing diagrams on a whiteboard falls. It has a cost, and that cost can be high. “Assemble a team of architect, developer, test lead, business analyst, operations and networking” reads one bit of advice. That’s a lot of people for a cross-functional meeting.

That meeting can be a great way to find disconnects in what people conceive of building. And there’s a difference between drawing a diagram and being handed a diagram. I want to draw that out a little bit and ask for your help in understanding the tradeoffs and when they might and might not be appropriate. (Gary McGraw is fond of saying that getting these people in a room and letting them argue is the most important step in “architectural risk analysis.” I think it’s tremendously valuable, and having structures, tools and methods to help them avoid ratholes and path dependency is a big win.)

So what are the advantages and disadvantages of each?

Whiteboard

  • Collaboration. Walking to the whiteboard and picking up a marker is far less intrusive than taking someone’s computer, or starting to edit a document in a shared tool.
  • Ease of use. A whiteboard is still easier than just about any other drawing tool.
  • Discovery of different perspective/belief. This is a little subtle. If I’m handed a diagram, I’m less likely to object. An objection may contain a critique of someone else’s work, it may be a conflict. As something is being drawn on a whiteboard, it seems easier to say “what about the debug interface?” (This ties back to Gary McGraw’s point.)
  • Storytelling. It is easier to tell a story standing next to a whiteboard than any tech I’ve used. A large whiteboard diagram is easy to point at. You’re not blocking the projector. You can easily edit as you’re talking.
  • Messy writing/what does that mean? We’ve all been there? Someone writes something in shorthand as a conversation is happening, and either you can’t read it or you can’t understand what was meant. Structured systems encourage writing a few more words, making things more tedious for everyone around.

Software Tools

  • Automatic analysis. Tools like the Microsoft Threat Modeling tool can give you a baseline set of threats to which you add detail. Structure is a tremendous aid to getting things done, and in threat modeling, it helps in answering “what could go wrong?”
  • Authority/decidedness/fixedness. This is the other side of the discovery coin. Sometimes, there are architectural answers, and those answers are reasonably fixed. For example, hardware accesses are mediated by the kernel, and filesystem and network are abstracted there. (More recent kernels offer filesystems in userland, but that change was discussed in detail.) Similarly, I’ve seen large, complex systems with overall architecture diagrams, and a change to these diagrams had to be discussed and approved in advance. If this is the case, then a fixed diagram, printed poster size and affixed to walls, can also be used in threat modeling meetings as a context diagram. No need to re-draw it as a DFD.
  • Photographs of whiteboards are hard to archive and search without further processing.
  • Photographs of whiteboards may imply that ‘this isn’t very important.” If you have a really strong culture of “just barely good enough” than this might not be the case, but if other documents are more structured or cared for, then photos of a whiteboard may carry a message.
  • Threat modeling only late. If you’re going to get architecture from code, then you may not think about it until the code is written. If you weren’t going to threat model anyway, then this is a win, but if there was a reasonable chance you were going to do the architectural analysis while there was a chance to change the architecture, software tools may take that away.

(Of course, there are apps that help you take images from a whiteboard and improve them, for example, Best iOS OCR Scanning Apps, which I’m ignoring for purposes of teasing things out a bit. Operationally, probably worth digging into.)

I’d love your thoughts: are there other advantages or disadvantages of a whiteboard or software?

What does the MS Secure Boot Issue teach us about key escrow?

Nothing.

No, seriously. Articles like “Microsoft Secure Boot key debacle causes security panic” and “Bungling Microsoft singlehandedly proves that golden backdoor keys are a terrible idea” draw on words in an advisory to say that this is all about golden keys and secure boot. This post is not intended to attack anyone; researchers, journalists or Microsoft, but to address a rather inflammatory claim that’s being repeated.

Based on my read of a advisory copy (which I made because I cannot read words on an animated background (yes, I’m a grumpy old man (who uses too many parentheticals (especially when I’m sick)))), this is a nice discovery of an authorization failure.

What they found is:

The “supplemental” policy contains new elements, for the merging conditions. These conditions are (well, at one time) unchecked by bootmgr when loading a legacy policy. And bootmgr of win10 v1511 and earlier certainly doesn’t know about them. To those bootmgrs, it has just loaded in a perfectly valid, signed policy. The “supplemental” policy does NOT contain a DeviceID. And, because they were meant to be merged into a base policy, they don’t contain any BCD rules either, which means that if they are loaded, you can enable testsigning.

That’s a fine discovery and a nice vuln. There are ways Microsoft might have designed this better, I’m going to leave those for another day.

Where the post goes off the rails, in my view, is this:

About the FBI: are you reading this? If you are, then this is a perfect real world example about why your idea of backdooring cryptosystems with a “secure golden key” is very bad! Smarter people than me have been telling this to you for so long, it seems you have your fingers in your ears. You seriously don’t understand still? Microsoft implemented a “secure golden key” system.[1] And the golden keys got released from MS own stupidity.[2] Now, what happens if you tell everyone to make a “secure golden key” system? [3] (Bracketed numbers added – Adam)

So, [1], no they did not. [2] No it didn’t. [3] Even a stopped clock …

You could design a system in which there’s a master key, and accidentally release that key. Based on the advisory, Microsoft has not done that. (I have not talked to anyone at MS about this issue; I might have talked to people about the overall design, but don’t recall having done so.) What this is is an authorization system with a design flaw. As far as I can tell, no keys have been released.

Look, there are excellent reasons to not design a “golden key” system. I talked about them at a fundamental engineering level in my threat modeling book, and posted the excerpt in “Threat Modeling Crypto Back Doors.”

The typical way the phrase “golden key” is used (albiet fuzzily) is that there is a golden key which unlocks communications. That is a bad idea. This is not that, and we as engineers or advocates should not undercut our position on that bad idea by referring to this research as if it really impacts on that “debate.”

Tacoma Narrows and Security

I always get a little frisson of engineering joy when I drive over the Tacoma Narrows bridge. For the non-engineers in the audience, the first Tacoma Narrows bridge famously twisted itself to destruction in a 42-mph wind.

The Tacoma Narrows bridge collapsing

The bridge was obviously unstable even during initial construction (as documented in “Catastrophe to Triumph: Bridges of the Tacoma Narrows.”) And so when it started to collapse, several movie cameras were there to document the event, which is still studied and analyzed today.

Today, people are tired of hearing about bridges collapsing. These stories undercut confidence, and bridge professionals are on top of things (ahem). When a bridge collapses, there’s a risk of a lawsuit, and if that was happening, no company could deliver bridges at a reasonable price. We cannot account for the way that wind behaves in the complex fiords of the Puget Sound.

Of course, these are not the excuses of bridge builders, but of security professionals.

I always get a little frisson of engineering joy when I drive over the Tacoma Narrows bridge, and marvel at how we’ve learned from previous failures.

The Evolution of Apple's Differential Privacy

Bruce Schneier comments on “Apple’s Differential Privacy:”

So while I applaud Apple for trying to improve privacy within its business models, I would like some more transparency and some more public scrutiny.

Do we know enough about what’s being done? No, and my bet is that Apple doesn’t know precisely what they’ll ship, and aren’t answering deep technical questions so that they don’t mis-speak. I know that when I was at Microsoft, details like that got adjusted as we learned from a bigger pile of real data from real customer use informed things. I saw some really interesting shifts surprisingly late in the dev cycle of various products.

I also want to challenge the way Matthew Green closes: “If Apple is going to collect significant amounts of new data from the devices that we depend on so much, we should really make sure they’re doing it right — rather than cheering them for Using Such Cool Ideas.”

But that is a false dichotomy, and would be silly even if it were not. It’s silly because we can’t be sure if they’re doing it right until after they ship it, and we can see the details. (And perhaps not even then.)

But even more important, the dichotomy is not “are they going to collect substantial data or not?” They are. The value organizations get from being able to observe their users is enormous. As product managers observe what A/B testing in their web properties means to the speed of product improvement, they want to bring that same ability to other platforms. Those that learn fastest will win, for the same reasons that first to market used to win.

Next, are they going to get it right on the first try? No. Almost guaranteed. Software, as we learned a long time ago, has bugs. As I discussed in “The Evolution of Secure Things:”

Its a matter of the pressures brought to bear on the designs of even what (we now see) as the very simplest technologies. It’s about the constant imperfection of products, and how engineering is a response to perceived imperfections. It’s about the chaotic real world from which progress emerges. In a sense, products are never perfected, but express tradeoffs between many pressures, like manufacturing techniques, available materials, and fashion in both superficial and deep ways.

Green (and Schneier) are right to be skeptical, and may even be right to be cynical. We should not lose sight of the fact that Apple is spending rare privacy engineering resources to do better than Microsoft. Near as I can tell, this is an impressive delivery on the commitment to be the company that respects your privacy, and I say that believing that there will be both bugs and design flaws in the implementation. Green has an impressive record of finding and calling Apple (and others) on such, and I’m optimistic he’ll have happy hunting.

In the meantime, we can, and should, cheer Apple for trying.

"Think Like an Attacker" is an opt-in mistake

I’ve repeatedly spoken out against “think like an attacker.”

Now I’m going to argue from authority. In this long article, “The Obama Doctrine,” the President of the United States says “The degree of tribal division in Libya was greater than our analysts had expected.”

So let’s think about that statement and what it means. First, it means that the multi-billion dollar analytic apparatus of the United States made a mistake, a serious one about which the President cares, because it impacted his foreign policy. Second, that mistake was about how people think. Third, that group of people was a society, and one that has interacted with the United States since, oh, I don’t know, someone wrote words like “From the halls of Montezuma to the shores of Tripoli.” (And dig the Marines, kickin’ it old skool with that video.) Fourth, it was not a group that attempts to practice operational security in any way.

So if we consider that the analytical capability of the US can get that wrong, do you really want to try to think like Anonymous, think like 61398, like 8200? Are you going to do this perfectly, or are there chances to make mistakes? Alternately, do you want to require everyone who threat models to know how attackers think? Understanding how other people think and prioritize requires a great deal of work. There are entire fields, like anthropology and sociology dedicated to doing it well. Should we start our defense by reading books on the motivational structures of the PLA or the IDF?

The simple fact is, you don’t need to. You can start from what people are building or deploying. (I wrote a book on how.) The second simple fact is repeating that phrase upsets people. When I first joined Microsoft, I used that phrase. One day, a developer grabbed me after a meeting, and politely told me that he didn’t understand it. Oh, wait, this was Microsoft in 2006. He told me I was a fucking idiot and I should give useful advice. After a bit more conversation, he also told me that he had no idea how the fuck an attacker thought, and if I thought he had time to read a book to learn about it, I could write the goddamned features customers pay for while he read.

Every time someone tells me to think like an attacker, I think about that conversation. I appreciate the honesty that the fellow showed, if not his manner. But (as Dave Weinstein pointed out) “A generalized form of this would be ‘Stop giving developers completely un-actionable “guidance”.’ Now, Dave and I worked together at Microsoft, so maybe there’s a similar experience in his past.

Now, this does not mean that we don’t need to pay attention to what real attackers do. It means that we don’t need to walk a mile in their shoes to defend effectively against it.

Previously, “Think Like An Attacker?,” “The Discipline of “think like an attacker”,” and “Think Like An Attacker? Flip that advice!.” [Edited, also previously, at the New School blog: “Modeling Attackers and Their Motives.”]