Consultants Say Their Cyber Warnings Were Ignored

Back in October, 2014, I discussed a pattern of “Employees Say Company Left Data Vulnerable,” and its a pattern that we’ve seen often since. Today, I want to discuss the consultant’s variation on the story. This is less common, because generally smart consultants don’t comment on the security of their consultees. In this case, it doesn’t seem like the consultant’s report was leaked, but people are discussing it after a high-profile issue.

In brief, the DNC was hacked, probably by Russian intelligence, and emails were given to Wikileaks. Wikileaks published them without redacting things like credit card numbers or social security numbers. The head of the DNC has stepped down. (This is an unusual instance of someone losing their job, which is rare post-breach. However, she did not lose her job because of the breach, she lost it because the breach included information about how her organization tilted the playing field, and how she lied about doing so.)

This story captures a set of archetypes. I want to use this story as a foil for those archetypes, not to critique any of the parties. I’ll follow the pattern from “employess vs company” present those three sections: “I told you so”, “potential spending”, and “how to do better.” I also comments on preventability and “shame.”

Was it preventable?

Computer security consultants hired by the DNC made dozens of recommendations after a two-month review, the people said. Following the advice, which would typically include having specialists hunt for intruders on the network, might have alerted party officials that hackers had been lurking in their network for weeks… (“Democrats Ignored Cybersecurity Warnings Before Theft,” Michael Riley, Bloomberg.)

People are talking about this as if the DNC was ever likely to stop Russian intelligence from breaking into their computers. That’s a very, very challenging goal, one at which both US and British intelligence have failed. (And as I write this, an FBI agent has been charged with espionage on behalf of China.) There’s a lot of “might,” “could have,” and other words that say “possible” without assessing “probable.”

I told you so!

The report included “dozens of recommendations,” some of which, such as “taking special precautions to protect any financial information related to donors” might be a larger project than a PCI compliance initiative. (The logic is that financial information collected by a party is more than just card numbers; there seems to be a lot of SSNs in the data as well). If one recommendation is “get PCI compliant,” than “dozens of recommendations” might be a Sysyphean task, or perhaps the Agean Stables are a better analogy. In either case, only in mythology can the actions be completed.

Missing from the discussion I’ve seen so far is any statement of what was done. Did the organization do the top-5 things the consultants said to do? (Did they even break things out into a top-5?)

Potential Spending

The review found problems ranging from an out-of-date firewall to a lack of advanced malware detection technology on individual computers, according to two of the people familiar with the matter.

It sounds like “advanced malware detection technology” would be helpful here, right? Hindsight is 20:20. An out-of-date firewall? Was it missing security updates (which would be worrisome, but less worrisome than one might think, depending on what those updates fix), or was it just not the latest revision? If it’s not the latest revision, it can probably still do its job. In carefully reading the article, I saw no evidence that any single recommendation, or even implementing all of them, would have prevented the breach.

The DNC is a small organization. They were working with a rapidly shifting set of campaign workers working for the Sanders and Clinton campaigns. I presume they’re also working on a great many state races, and the organizations those politicians are setting up.

I do not believe that doing everything in the consultant’s report could reasonably be expected to prevent a breakin by a determined mid-sized intelligence agency.

Shame

“Shame on them. It looks like they just did the review to check a box but didn’t do anything with it,” said Ann Barron-DiCamillo, who was director of US-Cert, the primary agency protecting U.S. government networks, until last February. “If they had acted last fall, instead of those thousands of e-mails exposed it might have been much less.”

Via Meredith Patterson, I saw “The Left’s Self-Destructive Obsession with Shame,” and there’s an infosec analog. Perhaps they would have found the attackers if they’d followed the advice, perhaps not. Does adding shame work to improve the cybers? If it did, it should have done so by now.

How to do better

I stand by what I said last time. The organization has paid the PR price, and we have learned nothing. What a waste. We should talk about exactly what happened at a technical level.

We should stop pointing fingers and giggling. It isn’t helping us. In many ways, the DNC is not so different from thousands of other small or mid-size organizations who hold sensitive information. Where is the list of effective practice for them to follow? How different is the set of recommendations in this instance from other instances? Where’s the list of “security 101” that the organization should have followed before hiring these consultants? (My list is at “Security 101: Show Your List!.”)

We should define the “smoke alarms and sprinklers” of cyber. Really, why does an organization like the DNC need to pay $60,000 for a cybersecurity policy? It’s a little hard to make sense of, but I think that the net operating expenditures of $5.1m is a decent proxy for the size of the organization, and (if that’s right) 1% of their net operating expenses went to a policy review. Was that a good use of money? How else should they have spent that? What concrete things do we, as a profession or as a community, say they should have done? Is there a reference architecture, or a system in a box which they could have run that we expect would resist Russian intelligence?

We cannot expect every small org to re-invent this wheel. We have to help them better.

PCI & the 166816 password

This was a story back around RSA, but I missed it until RSnake brought it up on Twitter: “[A default password] can hack nearly every credit card machine in the country.” The simple version is that Charles Henderson of Trustwave found that “90% of the terminals of this brand we test for the first time still have this code.” (Slide 30 of RSA deck.) Wow.

Now, I’m not a fan of the “ha-ha in hindsight” or “that’s security 101!” responses to issues. In fact, I railed against it in a blog post in January, “Security 101: Show Your List!

But here’s the thing. Credit card processors have a list. It’s the Payment Card Industry Data Security Standard. That standard is imposed, contractually, on everyone who processes payment cards through the big card networks. In version 3, requirement 2 is “Do not use vendor-supplied defaults for system passwords.” This is not an obscure sub-bullet. As far as I can tell, it is not a nuanced interpretation, or even an interpretation at all. In fact, testing procedure 2.1.a of v3 of the standard says:

2.1.a Choose a sample of system components, and attempt to
log on (with system administrator help) to the devices and
applications using default vendor-supplied accounts and
passwords, to verify that ALL default passwords (including
those on … POS terminals…) have been changed. [I’ve elided a few elements of the list for clarity.]

Now, the small merchant may not be aware that their terminal has a passcode. They may have paid someone else to set it up. But shouldn’t that person have set it up properly? The issue is not that the passcodes are not reset, the issue that I’m worried about is that the system appears broken. We appear to have evidence that to get security right, the system requires activity by busy, possibly undertrained people. Why is that still required ten years into PCI?

This isn’t a matter of “checklists replacing security.” I have in the past railed against checklists (including in the book this blog is named after). But after I read “The Checklist Manifesto”, I’ve moderated my views a bit, such as in “Checklists and Information Security.” Here we have an example of exactly what checklists are good for: avoiding common, easy-to-make and easy-to-check mistakes.

When I raised some of these questions on Twitter someone said that the usual interpretation is that the site selects the sample (where they lack confidence). And to an extent, that’s understandable, and I’m working very hard avoid hindsight bias here. But I think it’s not hindsight bias to say that a sample should be a random sample unless there’s a very strong reason to choose otherwise. I think it’s not hindsight bias to note that your financial auditors don’t let you select which transactions are audited.

Someone else pointed out that it’s “first time audits” which is ok, but why, a decade after PCI 1.0, is this still an issue? Shouldn’t the vendor have addressed this by now? Admittedly, may be hard to manage device PINs at scale — if you’re Target with tens of thousands of PIN pads, and your techs need access to the PINs, what do you do to ensure that the tech has access while at the register, but not after they’ve quit? But even given such challenges, shouldn’t the overall payment card security system be forcing a fix to such issues?

All bellyaching and sarcastic commentary aside, if the PCI process isn’t catching this, what does that tell us? Serious question. I have ideas, but I’m really curious as to what readers think. Please keep it professional.

Security 101: Show Your List!

Lately I’ve noted a lot of people quoted in the media after breaches saying “X was Security 101. I can’t believe they didn’t do X!” For example, “I can’t believe that LinkedIn wasn’t salting passwords! That’s security 101!”

Now, I’m unsure if that’s “security 101” or not. I think security 101 for passwords is “don’t store them in plaintext”, or “don’t store them with a crypto algorithm you designed”. Ten years ago, it would have included salting, but with the speed of GPU crackers, maybe it doesn’t anymore. A good library would probably still include it. Maybe LinkedIn was spending more on preventing XSS or SQL injection, and that pushed password storage off their list. Maybe that’s right, maybe it’s wrong. To tell you the truth, I don’t want to argue about it.

What I want to argue about is the backwards looking nature of these statements. I want to argue because I did some searching, and not one of those folks I searched for has committed to a list of security 101, or what are the “simple controls” every business should have.

This is important because otherwise, hindsight is 20/20. It’s easy to say in hindsight that an organization should have done A or B or C. It’s harder to offer up a complete list in advance, and harder yet to justify the budget required to deploy and operate it.

So I’m going to make three requests for 2015:

  • If you’re an expert (or even play one on the internet), and if you want to say “X is Security 101,” please publish your full list of what’s “101.”
  • If you’re a reporter and someone tells you “X is security 101” please ask them for their list.
  • Finally, if you’re someone who wants to see security improve, and you hear claims about “101”, please ask for the list.

Oh, and since it’s sauce for the gander, here’s my list for individuals:

  • Stay up to date–get most of your machines on the latest revisions of software and get patches for security issues installed, especially in your browser and AV software.
  • Use a firewall that blocks most inbound traffic.
  • Ensure you have a working backup of your data.

(There are complex arguments about AV software, and a lack of agreements about how to effectively test it. Do you need it? Will it block the wildlist? There’s nuance, but that nuance doesn’t play into a 101 list. I won’t be telling people not to use AV software.)

*By “lately,” I meant in 2012, when I wrote this, right after the Linkedin breach. But I recently discovered that I hadn’t posted.

[Update: I’m happy to see Ira Winkler and Araceli Treu Gomes took up the idea in “The Irari rules for declaring a cyberattack ‘sophisticated’.” Good for them!]

Threat Modeling At a Startup

I’ve been threat modeling for a long time, and at Microsoft, had the lovely opportunity to put some rigor into not only threat modeling, but into threat modeling in a consistent, predictable, repeatable way. Because I did that work at Microsoft, sometimes people question how it would work for a startup, and I want to address that from having done it. I’ve done threat modeling for startups like Zero Knowledge (some of the output is in the Freedom security whitepaper), and while a consultant or advisor, for other startups who decided against the open kimono approach.

Those threat modeling sessions have taken anywhere from a few hours to a few days, and the same techniques that we developed to make threat modeling more effective can be used to make it more efficient. One key aspect is to realize that what security folks call threat modeling is comprised of several related tasks, and each of them can be done in different ways. Even more important, only one of the steps is purely focused on threats.

I’ve seen a four-question framework work at all sorts of organization. The four questions are: “what are we building/deploying,” “what could go wrong”, “what are we going to do about it” and “did we do an acceptable job at this?”

The “what are we building” question is about creating a shared model of the system. This can be done on a napkin or a whiteboard, or for a large complex system which has grown organically, can involve people working for months. In each case, there’s value beyond security. Your code is cleaner, more consistent, more maintainable, and sometimes even faster when everyone agrees on what’s running where. Discussing trust boundaries can be quick and easy, and understanding them can lead to better marshaling and fewer remote calls. Of course, if your code is spaghetti and no one’s ever bothered to whiteboard it, this step can involve a lot of discussion and debate, because it will lead to refactoring. Maybe you can launch without that refactoring, maybe it’s important to do it now. It can be hard to tell if no one knows how your system is put together.

Understanding “what could go wrong” can happen in as little as an hour for people new to security using the Elevation of Privilege game. You should go breadth-first here, getting an understanding of where the threats might cluster, and if you need to dig deeper.

What are you going to do about it? You’re going to track issues as bugs. (Startups should not spend energy thinking about bugs and flaws as two separate things with their own databases.) If your bugs are post-its, that’s great. If you’re using a database, fine. Just track the issues. Some of them will get turned into tests. If you’re using Test-Driven Development, threat modeling is a great way to help you think about security testing and where it needs to focus. (If you’re not using TDD, it’s still a great way to focus your security testing.)

Total time spent for a startup can be as low as a few hours, especially if you’ve been thinking about software architecture and boundaries along the way. I’ve never spent more than a few days on threat modeling with a small company, and never heard it called a poor use of time. Of course, it’s easy to say that each thing that someone wants a startup to do “only takes a little time,” but all those things can add up to a substantial distraction from the core product. However, a media blow-up or a Federal investigation can also distract you, and distract you for a lot longer. A few hours of threat modeling can help you avoid the sorts of problems that distracted Twitter or Snapchat. The landscape continues to change rapidly post-Snowden, and threat modeling is the fastest way to consider what security investments a startup should be making.

[Update, Feb 5: David Cowan has a complimentary document, “Security for Startups in 10 Steps. He provides context and background on Techcrunch.]

Account Recovery Fail

“Please note that your password will be stored in clear text in our database which will allow us to send it back to you in case you lost it. Try avoid using the same password as accounts you may have in other systems.” — a security conference’s speaker website

This is a silly pattern. At least these folks realize it’s a bad idea, and warn the person, but that’s the wrong approach, since it relies on people reading text, and then relies on them being willing to make up and recall yet another password.

The origin of this problem is a broken model of what people want from their online accounts. As I discuss in chapter 14 of Threat Modeling: Designing for Security, the right goal is almost always account recovery, not password recovery. It’s irksome to see a security conference get this so wrong.

To be fair, the person who selected the conference management system was likely not a security expert, and had other functional goals on which they were focused. It would be easier to be annoyed if someone had a comprehensive and precise list of security problems which they could look for. But while we’re being fair, this isn’t like a XSS or SQLi issue which require some skill to discover. It’s right there in the primary workflow.

How to Ask Good Questions at RSA

So this week is RSA, and I wanted to offer up some advice on how to engage. I’ve already posted my “BlackHat Best Practices/Survival kit.

First, if you want to ask great questions, pay attention. There are things more annoying than a question that was answered while the questioner was tweeting, but you still don’t want to be that person.

Second, if you want to ask a good question, ask a question that you think others will want to hear answered. If your question is narrow, go up to the speaker afterwards.

Now, there are some generic best practice questions that I love to ask, and want to encourage you to ask.

  • You claimed “X”, but didn’t explain why. Could you briefly cover your methodology and data for that claim?
  • You said “X” is a best practice. Can you cover what practices you would cut to ensure there’s resources available to do “X”?
  • You said “if you get breached, you’ll go out of business. Last year, 2600 companies announced data breaches. How many of them are out of business?”
  • You said that “X” dramatically increased your organization’s security. Since we live in an era of ‘assume breach’, can I assume that your organization is now committed to publishing details of any breaches that happen despite X?
      I’m sure there’s other good questions, please share your favorites, and I’ll try for a new post tomorrow.

HHS & Breach Disclosure

There’s good analysis at “HHS breach investigations badly backlogged, leaving us in the dark

To say that I am frequently frustrated by HHS’s “breach tool” would be an understatement. Their reporting form and coding often makes it impossible to know – simply by looking at their entries – what type of breach occurred. Consider this description from one of their entries:

“Theft, Unauthorized Access/Disclosure”,”Laptop, Computer, Network Server, Email”

So what happened there? What was stolen? Everything? And what types of patient information were involved?

Or how about this description:

“Unauthorized Access/Disclosure,Paper”

What happened there? Did a mailing expose SSN in the mailing labels or did an employee obtain and share patients’ information with others for a tax refund fraud scheme? Your guess is as good as mine. And HHS’s breach tool does not include any data type fields that might let us know whether patients’ SSN, Medicare numbers, diagnoses, or other information were involved.

What can I say but, I agree?

Disclosures should talk about the incident and the data. Organizations are paying the PR cost, let’s start learning.

The incident should be specified using either the Broad Street taxonomy (covered in the MS Security Intel Report here) or Veris. It would be helpful to include details like the social engineering mail used (so we can study tactics), and detection rates for the malware, from something like VirusTotal.

For the data, it would be useful to explain (as Dissent says) what was taken. This isn’t simply a matter of general analysis, but can be used for consumer protection. For example, if you use knowledge-based backup authentication, then knowing that every taxpayer in South Carolina has had their addresses exposed tells you something about the efficacy of a question about what address you lived in in 2000. (I don’t know if that data was exposed in the SC tax breach, I’m just picking a recent example.)

Anyway, the post is worth reading, and the question of how we learn from breaches is worth discussing in depth.

The High Price of the Silence of Cyberwar

A little ways back, I was arguing [discussing cyberwar] with thegrugq, who said “[Cyberwar] by it’s very nature is defined by acts of espionage, where all sides are motivated to keep incidents secret.”

I don’t agree that all sides are obviously motivated to keep incidents secret, and I think that it’s worth asking, is there a strategic advantage to a policy of disclosure?

Before we get there, there’s a somewhat obvious objection that we should discuss, which is that the defender instantly revealing all the attacks they’ve detected is a great advantage for the attacker. So when I talk about disclosing attacks, I want to include some subtlety, where the defender is taking two steps to counter that advantage. The first step is to randomly select some portion of attacks (say, 20-50%) to keep secret, and second, randomly delay disclosure until sometime after cleanup.

With those two tactical considerations in place, a defender can gain several advantages by revealing attacks.

The first advantage is deterrence. If an defender regularly reveals attacks which have taken place, and reveals the attack tools, the domains, the IP addresses and the software installed, then the attacker’s cost of attacking that defender (compared to other defenders) is higher, because those tools will be compromised. That has a deterring effect.

The second advantage is credibility. In today’s debate about cyberwar, all information disclosed seems to come with an agenda. Everyone evaluating the information is forced to look not only at the information, but the motivation for revealing that information. Worse, they can question if the information not revealed is shaped differently from what is revealed. A defender who reveals information regularly and in accordance with a policy will gain credibility, and with it, the ability to better influence the debate.

There’s a third advantage, which is that of improving the information available to all defenders, but that only accrues to the revealer to the extent that others don’t free ride. Since I’m looking to the advantages that accrue to the defender, we can’t count it. However, to the extent that a government cares about the public good, this should weigh in their decision making process.

The United States, like many liberal democracies, has a long history of disclosing a good deal of information about our weaponry and strategies. The debates over nuclear weapons were public policy debates in which we knew how many weapons each side had, how big they were, etc. What’s more, the key thinking about deterrence and mutually assured destruction which informed US policy was all public. That approach helped us survive a 50 year cold war, with weapons of unimaginable destructive potential.

Simultaneously, secrecy around what’s happening pushes the public policy discussions towards looking like ‘he said, she said,’ rather than discussions with facts involved.

Advocates of keeping attacks in which they’ve been victimized a secret, keeping doctrines secret, or keeping strategic thinking secret need to move beyond the assumption that everything about cyberwar is secret, and start justifying the secrecy of anything beyond current operations.

[As thegruq doesn’t have a blog, I’ve posted his response “http://newschoolsecurity.com/2013/01/on-disclosure-of-intrusion-events-in-a-cyberwar/“]

Base Rate & Infosec

At SOURCE Seattle, I had the pleasure of seeing Jeff Lowder and Patrick Florer present on “The Base Rate Fallacy.” The talk was excellent, lining up the idea of the base rate fallacy, how and why it matters to infosec. What really struck me about this talk was that about a week before, I had read a presentation of the fallacy with exactly the same example in Kahneman’s “Thinking, Fast and Slow.” The problem is you have a witness who’s 80% accurate, describing a taxi as orange; what are the odds she’s right, given certain facts about the distribution of taxis in the city?

I had just read the discussion. I recognized the problem. I recognized that the numbers were the same. I recalled the answer. I couldn’t remember how to derive it, and got the damn thing wrong.

Well played, sirs! Game to Jeff and Patrick.

Beyond that, there’s an important general lesson in the talk. It’s easy to make mistakes. Even experts, primed for the problems, fall into traps and make mistakes. If we publish only our analysis (or worse, engage in information sharing), then others can’t see what mistakes we might have made along the way.

This problem is exacerbated in a great deal of work by a lack of a methodology section, or a lack of clear definitions.

The more we publish, the more people can catch one anothers errors, and the more the field can advance.

Checklists and Information Security

I’ve never been a fan of checklists. Too often, checklists replace thinking and consideration. In the book, Andrew and I wrote:

CardSystems had the required security certification, but its security was compromised, so where did things goo wrong? Frameworks such as PCI are built around checklists. Checklists compress complex issues into a list of simple questions. Someone using a checklist might therefore think he had done the right thing, when in fact he had not addressed the problems in depth…Conventional wisdom presented in short checklists makes security look easy.

So it took a while and a lot of recommendations for me to get around to reading “The Checklist Manifesto” by Atul Gawande. And I’ll admit, I enjoyed it. It’s a very well-written, fast-paced little book that’s garnered a lot of fans for very good reasons.

What’s more, much as it pains me to say it, I think that security can learn a lot from the Checklist Manifesto. One objection that I’ve had is that security is simply too complex. But so is the human body. From the Manifesto:

[It] is far from obvious that something as simple as a checklist could be of substantial help. We may admit that errors and oversights occur–even devastating ones. But we believe our jobs are too complicated to reduce to a checklist. Sick people, for instance, are phenomenally more various than airplanes. A study of forty-one thousand trauma patients in the state of Pennsylvania–just trauma patients–found that they had 1,224 different injury-related diagnoses in 32,261 unique combinations. That’s like having 32,261 kinds of airplane to land. Mapping out the proper steps for every case is not possible, and physicians have been skeptical that a piece of paper with a bunch of little boxes would improve matters.

The Manifesto also addresses the point we wrote above, that “someone using a checklist might think he’d done the right thing”:

Plus, people are individual in ways that rockets are not–they are complex. No two pneumonia patients are identical. Even with the same bacteria, the same cough and shortness of breath, the same low oxygen levels, the same antibiotic, one patient might get better and the other might not. A doctor must be prepared for unpredictable turns that checklists seem completely unsuited to address. Medicine contains the entire range of problems–the simple, the complicated, and the complex–and there are often times when a clinician has to just do what needs to be done. Forget the paperwork. Take care of the patient.

So it’s important to understand that checklists don’t replace professional judgement, they supplement it and help people remember complex steps under stress.

So while I think security can learn a lot from The Checklist Manifesto, the lessons may not be what you expect. Quoting the book that inspired this blog again:

A checklist implies that there is an authoritative list of the “right” things to do, even if no evidence of that simplicity exists. This in turn contributes to the notion that information security is a more mature discipline than it really is.

For example, turning back to the Manifesto:

Surgery has, essentially, four big killers wherever it is done in the world: infection, bleeding, unsafe anesthesia, and what can only be called the unexpected. For the first three, science and experience have given us some straightforward and valuable preventive measures we think we consistently follow but don’t.

I think what we need, before we get to checklists, is more data to understand what the equivalents of infection, bleeding and unsafe anesthesia are. Note that those categories didn’t spring out of someone’s mind, thinking things through from first principles. They came from data. And those data show that some risks are bigger than others:

But compared with the big global killers in surgery, such as infection, bleeding, and unsafe anesthesia, fire is exceedingly rare. Of the tens of millions of operations per year in the United States, it appears only about a hundred involve a surgical fire and vanishingly few of those a fatality. By comparison, some 300,000 operations result in a surgical site infection, and more than eight thousand deaths are associated with these infections. We have done far better at preventing fires than infections. [So fire risks are generally excluded from surgical checklists.]

Security has no way to exclude insiders the fire risk. We throw everything into lists like PCI. The group who updates PCI is not provided in depth incident reports about the failures that occurred over the last year or over the life of the failure. When security fails, rather than asking, ‘did the checklist work’, the PCI council declares that they’ve violated the 11th commandment, and are thus not compliant. And so we dan’t improve the checklists. (Compare and contrast: don’t miss the long section of the Manifesto on how Boeing tests and re-tests their checklists.)

One last quote before I close. Gawande surveys many fields, including how large buildings are built and delivered. He talks to a project manager putting up a huge new hospital building:

Joe Salvia had earlier told me that the major advance in the science of construction over the last few decades has been the perfection of tracking and communication.

Nothing for us security thought leaders to learn. But before I tell you to move along, I’d like to offer up an alpha-quality DO-CHECK checklist for improving security after an incident:

  1. Have you addressed the breach and gotten the attackers out?
  2. Have you notified your customers, shareholders, regulators and other stakeholders?
  3. Did you prepare an after-incident report?
  4. Did you use Veris, the taxonomy in Microsoft’s SIR v11 or some other way to clarify ambiguous terms?
  5. Have you released the report so others can learn?

I believe that if we all start using such a checklist, we’ll set up a feedback loop, and empower our future selves to make better, and more useful checklists to help us make things more secure.