Category: Reports and Data

Valuing CyberSecurity Research Datasets

There was a really interesting paper at the Workshop on the Economics of Information Security. The paper is “Valuing CyberSecurity Research Datasets.”

The paper focuses on the value of the IMPACT data sharing platform at DHS, and how the availability of data shapes the research that’s done.

On its way to that valuation, a very useful contribution of the paper is the analysis of types of research data which exist, and the purposes for which it can be used:

Note that there has been considerable attention paid to information sharing among operators through organizations such as ISACs. In contrast, we examine data provisioning done primarily for research purposes. Cybersecurity data resides on a use spectrum – some research data is relevant for operations and vice versa. Yet, as difficult as it can be to make the case for data sharing among operators, its even harder for researchers. Data sharing for research is generally not deemed as important as for operations. Outcomes are not immediately quantifiable. Bridging the gap between operators and researchers, rather than between operators alone, is further wrought with coordination and value challenges. Finally, research data is often a public good, which means it will likely be undervalued by the parties involved.

The paper enumerates benefits of research, including advancing scientific understanding, enabling infrastructure, creating parity in access to ground truth(s) for academics, technology developers, and others who don’t directly gather data. It also enumerates a set of barriers to research, including legal and ethical risk, costs, value uncertainty, and incentives.

These issues were highly resonant for me, because our near miss work certainly encounters these issues of value uncertainty and cost as we consider how to move beyond the operational data sharing that ISACs enable.

I’m very glad to see the challenges crystalized in this way, and we haven’t even reached the main goal of the paper, which is to assess how much value we get from sharing data.

While talking about this paper Robert Lemos has a story at Dark Reading, and Ross Anderson liveblogged the WEIS conference.

DNS Security

I’m happy to say that some new research by Jay Jacobs, Wade Baker, and myself is now available, thanks to the Global Cyber Alliance.

They asked us to look at the value of DNS security, such as when your DNS provider uses threat intel to block malicious sites. It’s surprising how effective it is for a tool that’s so easy to deploy. (Just point to a DNS server like 9.9.9.9).


The report is available from GCA’s site: Learn About How DNS Security Can Mitigate One-Third of Cyber Incidents

House Oversight Committee on Equifax

The House Oversight Committee has released a scathing report on Equifax.

Through the investigation, the Committee reviewed over 122,000 pages of documents, conducted transcribed interviews with three former Equifax employees directly involved with IT, and met with numerous current and former Equifax employees, in addition to Mandiant, the forensic firm hired to conduct an investigation of the breach.

I haven’t had time to review the report in detail, but I don’t think it answers questions I had reading the GAO report. Four of their give key findings are about what happened before the breach, but the fifth, “unprepared to support affected consumers,” goes to a point I’ve made consistently over nearly a dozen years: “
It’s Not The Crime, It’s The Coverup or the Chaos
.”

I’m pleased to be able to share work that Shostack & Associates and the Cyentia Institute have been doing for the Global Cyber Alliance. In doing this, we created some new threat models for email, and some new statistical analysis of

It shows the 1,046 domains that have successfully activated strong protection with GCA’s DMARC tools will save an estimated $19 million to $66 million dollars from limiting BEC for the year of 2018 alone. These organizations will continue to reap that reward every year in which they maintain the deployment of DMARC. Additional savings will be realized as long as DMARC is deployed.

Their press release from this morning is at here and the report download is here.

The NTSB has released “Preliminary Report Highway HWY18MH010,” on the Uber self-driving car which struck and killed a woman. I haven’t had a chance to read the report carefully.

Brad Templeton has excellent analysis of the report at “NTSB Report implies serious fault for Uber in fatality” (and Brad’s writings overall on the subject have been phenomenal.)

A few important things to note, cribbed from Brad.

  • The driver was not looking at her phone, but a screen with diagnostic information from the self-driving systems.
  • The car detected a need to brake with approximately enough time to stop had it automatically applied the brakes.
  • That system was turned off for a variety of reasons that look bad (in hindsight, and probably could have been critiqued at the time).

My only comment right now is wouldn’t it be nice to have this level of fact finding in the world of cyber?

Also, it’s very clear that the vehicle was carefully preserved. Can anyone say how the NTSB and/or Uber preserved the data center, cloud or other remote parts of the computer systems involved, including the algorithms that were deployed that day (versus reconstructing them later)?

Calls for an NTSB?

In September, Steve Bellovin and I asked “Why Don’t We Have an Incident Repository?.”

I’m continuing to do research on the topic, and I’m interested in putting together a list of such things. I’d like to ask you for two favors.

First, if you remember such things, can you tell me about it? I recall “Computers at Risk,” the National Cyber Leap Year report, and the Bellovin & Neumann editorial in IEEE S&P. Oh, and “The New School of Information Security.” But I’m sure there have been others.

In particular, what I’m looking for are calls like this one in Computers at Risk (National Academies Press, 1991):

3a. Build a repository of incident data. The committee recommends that a repository of incident information be established for use in research, to increase public awareness of successful penetrations and existing vulnerabilities, and to assist security practitioners, who often have difficulty persuading managers to invest in security. This database should categorize, report, and track pertinent instances of system security-related threats, risks, and failures. […] One possible model for data collection is the incident reporting system administered by the National Transportation Safety Board… (chapter 3)

Second, I am trying to do searches such as “cites “Computers at Risk” and contains ‘NTSB’.” I have tried without luck to do this on Google Scholar, Microsoft Academic and Semantic Scholar. Only Google seems to be reliably identifying that report. Is there a good way to perform such a search?

Dear Mr. President

U.S. President Barack Obama says he’s ”concerned” about the country’s cyber security and adds, ”we have to learn from our mistakes.”

Dear Mr. President, what actions are we taking to learn from our mistakes? Do we have a repository of mistakes that have been made? Do we have a “capability” for analysis of these mistakes? Do we have a program where security experts can gain access to the repository, to learn from it?

I’ve written extensively on this problem, here on this blog, and in the book from which it takes its name. We do not have a repository of mistakes. We do not have a way to learn from those mistakes.

I’ve got to wonder why that is, and what the President thinks we’re doing to learn from our mistakes. I know he has other things on his mind, and I hope that our officials who can advise him directly take this opportunity to say “Mr. President, we do not learn from our mistakes.”

(Thanks to Chris Wysopal for the pointer to the comment.)

Usable Security: History, Themes, and Challenges (Book Review)

Simson Garfinkel and Heather Lipford’s Usable Security: History, Themes, and Challenges should be on the shelf of anyone who is developing software that asks people to make decisions about computer security.

We have to ask people to make decisions because they have information that the computer doesn’t. My favorite example is the Windows “new network” dialog, which asks what sort of network you’re connecting to..work, home or coffee shop. The information is used to configure the firewall. My least favorite example is phishing, where people are asked to make decisions about technical minutiae before authenticating. Regardless, we are not going to entirely remove the need for people to make decisions about computer security. So we can either learn to gain their participation in more effective ways, or we can accept a very high failure rate. The former option is better, and this book is a substantial contribution.

It’s common for designers to throw up their hands at these challenges, saying things like “given a choice between security and dancing babies, people will choose dancing babies every time,” or “you can’t patch human stupidity.” However, in a recently published study by Google and UCSD, they found that the best sites only fooled 45% of the people who clicked through, while overall only 13% did. (There’s a good summary of that study available.) Claiming that “people will choose dancing babies 13% of the time” just doesn’t seem like a compelling argument against trying.

This slim book is a review of the academic work that’s been published, almost entirely in the last 20 years, on how people interact with information security systems. It summarizes and contextualizes the many things we’ve learned, mistakes that have been made, and does so in a readable and concise way. The book has six chapters:

  • Intro
  • A brief history
  • Major Themes in UPS Academic Research
  • Lessons Learned
  • Research Challenges
  • Conclusion/The Next Ten Years

The “Major themes” chapter is 61 or so pages, which is over half of the 108 pages of content. (The book also has 40 pages of bibliography). Major themes include authentication, email security and PKI, anti-phishing, storage, device pairing, web privacy, policy specification, mobile, social media and security administration.

The “Lessons Learned” chapter is quite solid, covering “reduce decisions,” “safe and secure defaults,” “provide users with better information, not more information,” “users require clear context to make good decisions,” “information presentation is critical” and “education works but has limits.” I have a quibble, which is Sasse’s concept of mental ‘compliance budgets’ is also important, and I wish it were given greater prominence. (My other quibble is more of a pet peeve: the term “user” where “people” would serve. Isn’t it nicer to say “people require clear context to make good decisions”?) Neither quibble should take away from my key message, which is that this is an important new book.

The slim nature of the book is, I believe, an excellent usability property. The authors present what’s been done, lessons that they feel can be taken away, and move to the next topic. This lets you the reader design, build or deploy systems which help the person behind the keyboard make the decisions they want to make. To re-iterate, anyone building software that asks people to make decisions should read the lessons contained within.

Disclaimer: I was paid to review a draft of this book, and my name is mentioned kindly in the acknowledgements. I am not being paid to write or post reviews.

[Updated to correct the sentence about the last 20 years.]

Modeling Attackers and Their Motives

There are a number of reports out recently, breathlessly presenting their analysis of one threatening group of baddies or another. You should look at the reports for facts you can use to assess your systems, such as filenames, hashes and IP addresses. Most readers should, at most, skim their analysis of the perpetrators. Read on for why.

There are a number of surface reasons that you might reject or ignore these reports. For example, these reports are funded by marketing. Even if they are, that’s not a reason to reject them. The baker does not bake bread for fun, and the business goal of marketing can give us useful information. You might reject them for their abuse of adjectives like “persistent”, “stealthy”, or “sophisticated.” (I’m tempted to just compile a wordcloud and drop it in place of writing.) No, the reason to only skim these is what the analysis does to the chance of your success. There are two self-inflicted wounds that often happen when people focus on attackers:

  • You miss attackers
  • You misunderstand what the attackers will do

You may get a vicarious thrill from knowing who might be attacking you, but that very vicarious thrill is likely to make those details available to your conscious mind, or anchor your attention on them, causing you to miss other attackers. Similarly, you might get attached to the details of how they attacked last year, and not notice how those details change.

Now, you might think that your analysis won’t fall into those traps, but let me be clear: the largest, best-funded analysis shops in the world routinely make serious and consequential mistakes about their key areas of responsibility. The CIA didn’t predict the collapse of the Soviet Union, and it didn’t predict the rise of ISIS.

If your organization believes that it’s better at intelligence analysis than the thousands of people who work in US intelligence, then please pay attention to my raised eyebrow. Maybe you should be applying that analytic awesomesauce to your core business, maybe it is your core business, or maybe you should be carefully combing through the reports and analysis to update your assessments of where these rapscallions shall strike next. Or maybe you’re over-estimating your analytic capabilities.

Let me lay it out for you: the “sophisticated” attackers are using phishing to get a foothold, then dropping malware which talks to C&C servers in various ways. The phishing has three important variants you need to protect against: links to exploit web pages, documents containing exploits, and executables disguised as documents. If you can’t reliably prevent those things, detect them when you’ve missed, and respond when you discover you’ve missed, then digging into the motivations of your attackers may not be the best use of your time.

The indicators that can help you find the successful attacks are an important value from these reports, and that’s what you should use them for. Don’t get distracted by the motivations.

Navigation