Security Rarely Flows Downhill

I was recently in a meeting at a client site where there was a lot of eye rolling going on. It’s about not understanding the ground rules, and one of those groundrules is that security rarely flows downhill.

application, on top of media, on top of core services, on top of core OS, on top of kernel

That is, if you’re in a stack like the one to the right, the application is vulnerable to the components underneath it. The components underneath should isolate themselves from (protect themselves against) things higher in the stack.

I can’t talk about the meeting in detail (client confidentiality), so I’m going to pick an article I saw where some of the same thinking. I’m using that article as a bit of a straw man. This article was convenient, not unique. These points are more strongly made if they are grounded in real quotes, rather than ones I make up.

“The lack of key validation (i.e. the verification that public keys are not invalid) is therefore not a major security risk. But I believe that validating keys would make Signal even more secure and robust against maliciously or accidentally invalid keys,” the researchers explained.

In this farfetched example, researchers explain, communications would be intentionally compromised by the sender. The goal, could be to give the message recipient the appearance of secure communications in hopes they may be comfortable sharing something they might not otherwise.

…
“People could also intentionally install malware on their own device, intentionally backdoor their own random number generator, intentionally publish their own private keys, or intentionally broadcast their own communication over a public loudspeaker. If someone intentionally wants to compromise their own communication, that’s not a vulnerability,” Marlinspike said. (I’m choosing to not link to the article, because, I don’t mean to call out the people making that argument.)

So here’s the rule: Security doesn’t flow downhill without extreme effort. If you are an app, it is hard to protect the device as a whole. It is hard to protect yourself if the user decides to compromise their device or mess up their RNG. And Moxie is right to not try to improve the security of Android or IoS against these attacks: it’s very difficult to do from where he sits. Security rarely flows downhill.

There are exceptions. Companies like Good Technologies built complex crypto to protect corporate data on devices that might be compromised. And best I understand it, it worked by having a server send a set of keys to the device, and the outer layer of Good decrypted the real app and data with those keys, then got more keys. And they had some anti-debugging lawyers in there (oops, is that a typo?) so that the OS couldn’t easily steal the keys. And it was about the best you could do with the technology that phone providers were shipping. It is, netted out, a whole lot more fair than employers demanding the ability to wipe your personal device and your personal data.

So back to that meeting. A security advisor from a central security group was trying to convince a product team of something very much like “the app should protect itself from the OS.” He wasn’t winning, and he was trotting out arguments like “it’s not going to cost [you] anything.” But that was obviously not the case. The cost of anything is the foregone alternative, and these “stone soup improvements” to security (to borrow from Mordaxus) were going to come at the expense of other features. Even if there was agreement on what direction to go, it was going to another few meetings to get these changes designed, and then it was going to cost a non-negligible number of programmer days to implement, and more to test and document.

That’s not the most important cost. Far more important than the cost of implementing the feature was the effort to get to agreement on these new features versus others.

But even that is not the most important cost. The real price was respect for the central security organization. Hearing those arguments made it that much less likely that those engineers were going to see the “security advisor,” or their team, as helpful.

As security engineers, we have to pick battles and priorities just like other engineers. We have to find the improvements that make the most sense for their cost, and we have to work those through the various challenges.

One of the complexities of consulting is that it can be trickier to interrupt in a large meeting, and you have less ability to speak for an organization. I’d love your advice on what a consultant should do when they watch someone at a client site demonstrating skepticism. Should I have stepped in at the time? How? (I did talk with a more senior person on the team who is working the issue.)

Hospital Ransomware

[Update, May 22, added link to “Observing”.]

Good posts by Ross Anderson, George Danezis and Steve Bellovin say much of what I’d wanted to say, and more. So go take a read. [Also worth reading “Observing the WannaCry fallout: confusing advice and playing the blame game.”]

To what Bellovin says, I would add that 15 years ago, Steve Beattie, Crispin Cowan and I did some math for Timing the Application of Security Patches for Optimal Uptime, and estimated that likelihood of attack starts to exceed likelihood of damage from the patch at around 10 days. To my knowledge, no one has updated the dataset or re-run the numbers, but I would expect that improvements in test automation and improvement in attack frameworks make that closer to patch release, not further from it. My experience is that many organizations with dependencies on older technology also have not invested in test automation that enables even fast ‘smoke testing’ of their systems. Such test rigs allow you to quickly start the clock that Steve hypothesizes.

Also, see “Rejection Letter” by Charlie Stross, and “How to Accidentally Stop a Global Cyber Attacks.”

Ross Anderson on Edge

The Edge is an interesting site with in depth interviews with smart folks. There’s a long interview with Ross Anderson published recently.

It’s a big retrospective on the changes over thirty years, and there’s enough interesting bits that I’ll only quote one:

The next thing that’s happened is that over the past ten years or so, we’ve begun to realize that as systems became tougher and more difficult to penetrate technically, the bad guys have been turning to the users. The people who use systems tend to have relatively little say in them because they are a dispersed interest. And in the case of modern systems funded by advertising, they’re not even the customer, they’re the product.

Take the time to listen. Ross’s emphasis is a bit lost in the text.

Warrants for Cleaning Malware in Kelihos

This is an thought-provoking story:

And perhaps most unusual, the FBI recently obtained a single warrant in Alaska to hack the computers of thousands of victims in a bid to free them from the global botnet, Kelihos.

On April 5, Deborah M. Smith, chief magistrate judge of the US District Court in Alaska, greenlighted this first use of a controversial court order. Critics have since likened it to a license for mass hacking. (“FBI allays some critics with first use of new mass-hacking warrant,” Aliya Sternstein, Ars Technica)

One of the issues in handling malware at scale is that the law prohibits unauthorized access to computers. And that’s not a bad principle, but it certainly makes it challenging to go and clean up infections at scale.

So the FBI getting a warrant to authorize that may be an interesting approach, with many cautions about the very real history of politicized spying, of COINTEL, of keeping the use of 0days secret. But I don’t want to focus on those here. What I want to focus on is what did a judge actually authorize in this case. Is this a warrant for mass hacking? It doesn’t appear to be, but it does raise issues of what’s authorized, how those things were presented to the judge, and points to future questions about what authorization might include.


So what’s authorized?

The application for a warrant is fairly long, at 34 pages, much of it explaining why the particular defendant is worthy of a search, and the “time and manner of execution of search” starts on page 30.

What the FBI apparently gets to do is to operate a set of supernodes for the Kelihos botnet, and “The FBI’s communications, however, will not contain any commands, nor will they contain IP addresses of any of the infected computers. Instead, the FBI replies will contain the IP and routing information for the FBI’s ‘sinkhole’ server.”

What questions does that raise?

A first technical point is that for the FBI’s replies to reach those infected computers, there must be packets sent over the internet. For those packets to reach the infected computer’s, they need addressing, in the form of an IP address. Now you can argue that those IP addresses of infected computers are in the headers, not the content of the packets. The nuance of content versus headers is important in some laws. In fact, the warrant para 66 explicitly states that the FBI will gather that data, and then provide it to ISPs, who they hope will notify end users. (I’ve written about that experience in “The Worst User Experience In Computer Security?.”)

Another technical point is that the warrant says “The FBI with the assistance of private parties…” It’s not clear to me what constraints might apply to those parties. Can they record netflow or packet captures? (That might be helpful in demonstrating exactly what they did later, and also create a privacy conundrum which the FBI takes apparent pains to avoid.) What happens if an unrelated party captures that data? For example, let’s say one of the parties chooses to operate their sinkhole in an AWS node. I suspect AWS tracks netflows. A warrant to obtain that data might seem pretty innocent to a judge.


The idea that the FBI will not send any commands is, on the surface, a fine restriction, but it eliminates many possibilities for cleaning up. What could we imagine in the future?

For example, a command that could be useful would be to remove Kelihos from startup items? How about remove C:\Program Files\Kelihos.exe? Removing files from my computer without permission is pretty clearly a form of unauthorized access. It’s a form that’s probably in the interests of the vast majority of the infected. We might want to allow a well-intentioned party to do so.

But what if the commands fail? When Sony built a tool to remove the rootkit their DRM installed, the cleanup tool opened a big security hole. It’s hard to write good code. It’s very hard to write code that’s free of side effects or security issues.

What if there’s disagreement over what fits within the definition of well-intentioned? What if someone wants to remove duplicate files to help me save disk space? What if they want to remove blasphemous or otherwise illegal files?

A Privacy Threat Model for The People of Seattle

Some of us in the Seattle Privacy Coalition have been talking about creating a model of a day in the life of a citizen or resident in Seattle, and the way data is collected and used; that is the potential threats to their privacy. In a typical approach, we focus on a system that we’re building, analyzing or testing. In this model, I think we need to focus on the people, the ‘data subjects.’

I also want to get away from the one by one issues, and help us look at the problems we face more holistically.

Feds Sue Seattle over FBI Surveillance

The general approach I use to threat model is based on 4 questions:

  1. What are you working on? (building, deploying, breaking, etc)
  2. What can go wrong?
  3. What are you going to do about it?
  4. Did you do a good job?

I think that we can address the first by building a model of a day, and driving into specifics in each area. For example, get up, check the internet, go to work (by bus, by car, by bike, walking), have a meal out…

One question that we’ll probably have to work on is how to address what can go wrong in a model this general? Usually I threat model specific systems or technologies where the answers are more crisp. Perhaps a way to break it out would be:

  1. What is a Seattlite’s day?
  2. What data is collected, how, and by whom? What models can we create to help us understand? Is there a good balance between specificity and generality?
  3. What can go wrong? (There are interesting variations in the answer based on who the data is about)
  4. What could we do about it? (The answers here vary based on who’s collecting the data.)
  5. Did we do a good job?

My main goal is to come away from the exercise with a useful model of the privacy threats to Seattleites. If we can, I’d also like to understand how well this “flipped” approach works.

[As I’ve discussed this, there’s a lot of interest in what comes out and what it means, but I don’t expect that to be the main focus of discussion on Saturday. For example,] There are also policy questions like, “as the city takes action to collect data, how does that interact with its official goal to be a welcoming city?” I suspect that the answer is ‘not very well,’ and that there’s an opportunity for collaboration here across the political spectrum. Those who want to run a ‘welcoming city’ and those who distrust government data collection can all ask how Seattle’s new privacy program will help us.

In any event, a bunch of us will be getting together at the Delridge Library this Saturday, May 13, at 1PM to discuss for about 2 hours, and anyone interested is welcome to join us. We’ll just need two forms of ID and your consent to our outrageous terms of service. (Just kidding. We do not check ID, and I simply ask that you show up with a goal of respectful collaboration, and a belief that everyone else is there with the same good intent.)

Threat Modeling and Star Wars

IANS members should have access today to a new faculty report I wrote, entitled “Threat Modeling in An Agile World.” Because it’s May the Fourth, I thought I’d share the opening:

As Star Wars reaches its climax, an aide approaches Grand Moff Tarkin to say, “We’ve analyzed their attack pattern, and there is a danger.” In one of the final decisions he makes, Tarkin brushes aside those concerns. Likewise, in Rogue One, we hear Galen Urso proclaim that the flaw is too subtle to be found. But that’s bunk. There’s clearly no blow-out sections or baffles around the reactor: if there’s a problem, the station is toast. A first year engineering student could catch it.

You don’t have to be building a Death Star to think about what might go wrong before you complete the project. The way we do that is by “threat modeling,” an umbrella term for anticipating and changing the problems that a system might experience. Unfortunately, a lot of the advice you’ll hear about threat modeling makes it seem a little bit like the multi-year process of building a Death Star.

Threat Modeling & IoT

Threat modeling internet-enabled things is similar to threat modeling other computers, with a few special tensions that come up over and over again. You can start threat modeling IoT with the four question framework:

  1. What are you building?
  2. What can go wrong?
  3. What are you going to do about it?
  4. Did we do a good job?

But there are specifics to IoT, and those specifics influence how you think about each of those questions. I’m helping a number of companies who are shipping devices, and I would love to fully agree that “consumers shouldn’t have to care about the device’s security model. It should just be secure. End of story.” I agree with Don Bailey on the sentiment, but frequently the tensions between requirements mean that what’s secure is not obvious, or that “security” conflicts with “security.” (I model requirements as part of ‘what are you building.’)

When I train people to threat model, I use this diagram to talk about the interaction between threats, mitigations, and requirements:

Threats mitigations requirements

The interaction has a flavor in working with internet-enabled things, and that interaction changes from device to device. There are some important commonalities.

When looking at what you’re building, IoT devices typically lack sophisticated input devices like keyboards or even buttons, and sometimes their local output is a single LED. One solution is to put a web server on the device listening, and to pay for a sticker with a unique admin password, which then drives customer support costs. Another solution is to have the device not listen but to reach out to your cloud service, and let customers register their devices to their cloud account. This has security, privacy, and COGS downsides tradeoffs. [Update: I said downsides, but it’s more a different set of attack vectors become relevant in security. COGS is an ongoing commitment to operations; privacy is dependent on what’s sent or stored.]

When asking what can go wrong, your answers might include “a dependency has a vulnerability,” or “an attacker installs their own software.” This is an example of security being in tension with itself is the ability to patch your device yourself. If I want to be able to recompile the code for my device, or put a safe version of zlib on there, I ought to be able to do so. Except if I can update the device, so can attackers building a botnet, and 99.n% of typical consumers for a smart lightbulb are not going to patch themselves. So we get companies requiring signed updates. Then we get to the reality that most consumer devices last longer than most silicon valley companies. So we want to see a plan to release the key if the company is unable to deliver updates. And that plan runs into the realities of bankruptcy law, which is that that signing key is an asset, and its hard to value, and bankruptcy trustees are unlikely to give away your assets. There’s a decent pattern (allegedly from the world of GPU overclocking), which is that you can intentionally make your device patchable by downloading special software and moving a jumper. This requires a case that can be opened and reclosed, and a jumper or other DFU hardware input, and can be tricky on inexpensive or margin-strained devices.

That COGS (cost of goods sold) downside is not restricted to security, but has real security implications, which brings us to the question of what are you going to do about it. Consumers are not used to subscribing to their stoves, nor are farmers used to subscribing to their tractors. Generally, both have better things to do with their time than to understand the new models. But without subscription revenue, it’s very hard to make a case for ongoing security maintenance. And so security comes into conflict with consumer mental models of purchase.

In the IoT world, the question of did we do a good job becomes have we done a good enough job? Companies believe that there is a first mover advantage, and this ties to points that Ross Anderson made long ago about the tension between security and economics. Good threat modeling helps companies get to those answers faster. Sharing the tensions help us understand what the tradeoffs look like, and with those tensions, organizations can better define their requirements and get to a consistent level of security faster.


I would love to hear your experiences about other issues unique to threat modeling IoT, or where issues weigh differently because of the IoT nature of a system!

(Incidentally, this came from a question on Twitter; threading on Twitter is now worse than it was in 2010 or 2013, and since I have largely abandoned the platform, I can’t figure out who’s responding to what. A few good points I see include:

Cassini

This image isn’t Saturn’s Rings, but an image of Saturn from its pole to equator.

Saturn

Sadly, many of the sites reporting on Cassini’s dive through Saturn’s rings — I’m going to say that again — Cassini’s first dive through Saturn’s rings — don’t explain the photos. I’ll admit it, I thought I was looking at the rings. Space.com has the explanations.

“…the Elusive Goal of Security as a Scientific Pursuit”

That’s the subtitle of a new paper by Cormac Herley and Paul van Oorschot, “SoK: Science, Security, and the Elusive Goal of Security as a Scientific Pursuit,” forthcoming in IEEE Security & Privacy.

The past ten years has seen increasing calls to make security research more “scientific”. On the surface, most agree that this is desirable, given universal recognition of “science” as a positive force. However, we find that there is little clarity on what “scientific” means in the context of computer security research, or consensus on what a “Science of Security” should look like. We selectively review work in the history and philosophy of science and more recent work under the label “Science of Security”. We explore what has been done under the theme of relating science and security, put this in context with historical science, and offer observations and insights we hope may motivate further exploration and guidance. Among our findings are that practices on which the rest of science has reached consensus appear little used or recognized in security, and a pattern of methodological errors continues unaddressed.

Cyber Grand Shellphish

There’s a very interesting paper on the Cyber Grand Challenge by team Shellphish. Lots of details about the grand challenge itself, how they designed their software, how they approached the scoring algorithm, and what happened in the room.

There’s lots of good details, but perhaps my favorite is:

How would a team that did *nothing* do? That is, if a team connected and then ceased to play, would they fare better or worse than the other players? We ran a similar analysis to the “Never patch” strategy previously (i.e., we counted a CS as exploited for all rounds after its first exploitation against any teams), but this time removed any POV-provided points. In the CFE, this “Team NOP” would have scored 255,678 points, barely *beating* Shellphish and placing 3rd in the CGC.

The reason I like this is that scoring systems are hard. Really, really hard. I know that DARPA spent substantial time and energy on the scoring system, and this outcome happened anyway. We should not judge either DARPA or the contest on that basis, because it was hard to see that that would happen ahead of time: it’s a coincidence of the scores teams actually achieved.