Making Sense of the SANS "Top Cyber Security Risks" Report
The SANS Top Cyber Security Risks report has received a lot of positive publicity (19 online stories, at last count). (TippingPoint and Qualys were partners in the report.) But none of the reporters or bloggers analyzed the report, the methods, or the data. They just repeat the main points from the report.
I applaud the effort and goals of the study and it may have some useful conclusions. We should have more of this type of study, especially at a large scale.
Unfortunately, the report has some major problems, listed roughly in order of severity: (for details, read on…)
- The most basic charts are missing. What was the total number of attacks in the study period? The total number of vulnerabilities? What percentage of identified vulnerabilities were actually attacked? Were the most prevalent vulnerabilities attacked most frequently? What how prevalent are zero-day vulnerabilities as a percentage of total vulnerabilities?
- Some of the most important statements in the report have no backing data or detailed analysis. Here are examples from the Executive Summary:
“Waves of targeted email attacks, often called spear phishing, are exploiting client-side vulnerabilities in commonly used programs such as Adobe PDF Reader, QuickTime, Adobe Flash and Microsoft Office.”
There is no analysis or data in the report regarding targeted email attacks (a.k.a. spear phishing). The rise of spear phishing has been documented elsewhere, but if there was data coming from this study to support that assertion, it didn’t appear in the detailed section of the SANS report.
“Attacks against web applications constitute more than 60% of the total attack attempts observed on the Internet.”
Looking at the detailed sections of the report, I could not find any table or chart that listed the total number of attacks in the study and the percentage of attacks on each on category (i.e. web applications vs. app platform vs. OS). (Figure 1 doesn’t do it. Here is my critique of this bad graphic.)
“Rising numbers of zero-day vulnerabilities”
I couldn’t find any support in the report for the assertion that the number of zero-day vulnerabilities were rising. There was no data analysis and no charts in the “Zero-day Vulnerability Trends” section.
“On average, major organizations take at least twice as long to patch client-side vulnerabilities as they take to patch operating system vulnerabilities. In other words the highest priority risk is getting less attention than the lower priority risk.”
There isn’t any chart that shows the average time to patch client-side vulnerabilities vs. OS vulnerabilities. They don’t mention or analyze server-side application patching, which seems weird. How about a chart with two bars, one for web applications and other for OS, showing the average number of days to reduce vulnerabilities by half? That would be simple and clear. Regarding the charts they do show…
- The charts in the “Vulnerabilities” section are poorly executed and poorly explained. The wiggly “percentage vs. days” graphs leave me scratching my head. The data is probably in there, but it’s very hard to see in this format. Here’s my interpretation of these charts (but I might be wrong). These are supposed to “aging charts” that graph the % of vulnerabilities that remain unresolved or unpatched after X days. If none are resolved, then the line should be flat at 100%. If all are resolved on day one, then it would be vertically down from 100% to 0%. If they are gradually resolved, then the line will slope downward. If some of the vulnerabilities are never resolved, then the line should plateau at some residual level. But why don’t the charts all start with a value of 100% on Day zero? I don’t know and it’s not explained. I’m guessing that it takes some number of days before all the vulnerabilities appear in the scans. More disturbing is the periodic oscillations in data series. These jump out visually. But what does this pattern mean? You might be inclined to think that vulnerabilities were resolved every 5 to 7 days, only to reappear the next week, and so on. That would be a huge process problem. But it turns out that these oscillations are a spurious result of the scanning process and the data analysis method. Consider this phrase from the report:
“Periodic drops in detection rates occur during the weekends when scanning focuses on servers rather than desktop machines and the detection rates of vulnerabilities related to desktop software fall accordingly.”
If this is really true (and it could probably be verified), then it would make sense to drop weekend data points and just include weekday data points for client-side vulnerabilities. The data analysis could be further improved by using a smoothing or curve fitting method, because what counts in this analysis is not the day-by-day squiggles but the overall slope of the curve and whether it plateaus. Finally there should be some marker for “half-life” or something, to allow easy visual identification of how long it takes to resolve half of the vulnerabilities in each class. - One finding was not highlighted but deserves more attention: the intentional use of security-violating methods by legitimate web applications. This blurs the distinction between “good guys” and “bad guys” and makes security management much more difficult. Examples from the report:
“…an advertiser’s banner might be embedded in a web page which is set up to reflect some JavaScript off of the advertiser’s HTTP server for tracking purposes. However, in this case, there is little risk because the site in question (usually) has full control over his/her page, so this request to the advertiser is not generally malicious. It is the “reflection” attacks, along with attacks that leverage flaws in form data handling, that make up the vast majority of XSS attacks that we have seen in the last six months.” [emphasis added]
”A very large spike in SQL Injection attacks in July was caused mostly by an online advertiser who distributed code to many affiliates using SQL injection as functionality. The application was quickly pulled, resulting in a large drop in events for the month of August.”
Related to this trend is the intentional use of security- and privacy-violating technologies to support marketing purposes, e.g. FTC’s action against Sears .
- The analysis of Cross-site Scripting is confusing at best. In the attack section, they say: “Cross Site Scripting (XSS) is one of the most prevalent bugs in today’s web applications.” But looking at the graph immediately below (Fig. 13), it doesn’t look like XSS attacks are frequent – numbering in the thousands while other attacks number in the millions. Looking down to the vulnerabilities section, there is no data or analysis of XSS vulnerabilities as a percentage of the total.
- There is no mention of uncertainty or confidence levels in the analysis or conclusions. For example, was the TippingPoint attack data adjusted for false positives and false negatives? It’s well known that all Intrusion Detection/Prevention Systems (IDPS) are prone to false positives and false negatives, no matter how well designed or tuned. This doesn’t discredit the data or analysis, but it does add uncertainty and reduce confidence levels.
- There was no mention of the limitations of the attack or vulnerability data. Of course, all data sources will have limitations. It’s essential that the authors factor this in to the analysis and make clear that they are not analyzing the full spectrum of cyber attacks or vulnerabilities. For example, there is no coverage of insider threats, social engineering, combined physical and cyber attacks, or data leakage. There is also no coverage in the data sets for complex targeted attacks.
Attack data limitations. As a network appliance, TippingPoint’s Intrusion Detection/Prevention Systems (IDPS) do not detect all types of attacks. Specifically, they cannot detect attacks within encrypted network traffic, attacks that come through wireless protocols, or attacks that utilize application payload, e.g. code injection. Also, the IDPS are susceptible to various types of attacks, most involving large volumes of traffic, which can “blind” the IDPS for a period of time. IDPS appliances also don’t detect authorization or authentication attacks. (I’m no expert in IDPS or intrusion detection in general, so I’m drawing from other sources, including NIST 800-94: Guide to Intrusion Detection and Prevention Systems.)
Vulnerability data limitations. Qualys vulnerability scanning service identifies many vulnerabilities in hardware configuration, operating systems, networking, libraries, platforms (e.g. Adobe Acrobat and Flash), etc., including: Misconfigured or unpatched servers, laptops and desktops; Out-of-date or misaligned security policy; Unauthorized hardware, software or applications; Easily-guessed passwords; and Inadequate controls on traffic from trusted third-party networks.But there are limitations.
Most obvious but unstated, Qualys identifies known vulnerabilities only, not vulnerabilities that have not yet been discovered. While CERT has been quoted as saying “99% of attacks exploit known vulnerabilities”, this does not mean that there is no risk associated with undiscovered vulnerabilities.
More important is that automated vulnerability assessment scanning doesn’t provide a macro view on the relative riskiness of vulnerabilities. According to a Gartner report, vulnerability assessment scanning systems “provide data on devices within the network, but it remains difficult to understand how the overall network is vulnerable, or how vulnerabilities within a device affect their neighbors. What may appear as a benign or low-priority vulnerability on a host may be used as a launching point for an attacker to penetrate other devices on the network.” Thus, the prioritization of vulnerabilities in the report seems to be based on their prevalence in the total population, not based on the relative riskiness of each vulnerability in it’s IT and network context. (That would require other analysis, including attack graphs, penetration testing, etc.)
Finally, Qualys only added Web Application Scanning (WAS) to their QualysGuard Security and Compliance Suite on May 26, 2009. This was half-way through the study period (March – August 2009), and it’s not clear from the report whether WAS produced any data used in this study or how widely it is being used in the customer base. (Qualys suite is software as a service (SaaS), so new functionality should be immediately available to all subscribers. Who knows what percentage of customers started using that functionality.)
- There was very little explanation of the methodology. I’m guessing that they drew conclusions based on total number of attacks of various types, compared to the total number of vulnerabilities of various types. I didn’t see any analysis of the severity of each attack type, or any data regarding how many of these attacks succeeded in exploiting vulnerabilities.
- There is insufficient background on the data sets. For example, what are the demographics of the TippingPoint data (attacks) and the Qualys data (vulnerabilities)? How many TippingPoint customers are also Qualys customers? Some of the context information that was listed seemed irrelevant, e.g. “TippingPoint intrusion prevention systems protect 6,000 organizations” and “vulnerability data from 9,000,000 systems compiled by Qualys”. Yes, that’s a lot of organizations and systems, but compared to what?
- What was the point of the charts and analysis on the geographic origins and destinations of attacks? I presume that this could be very important for understanding the attacking parties (“bad guys”) and the resources they use. Maybe it could also to help guide IT managers on the likelihood of certain attacks in their geography. But the analysis falls short of either purpose. As it is, these charts simply make me say “hmmmmm….”.
As a general comment, the report is poorly organized. I found it hard to read, and it looked like a “cut and paste” job with no overall editing. It was especially hard to trace the line of reasoning from the executive summary to the detailed sections. Also, “…Four Key Attacks” section is confusing. They actually list five key attacks, not four, but then say there are really only two categories: Server-side HTTP attacks and Client-side HTTP attacks. Mis-numbering like this is a basic editing mistake.
Lastly, I wonder if there is any way that this study could be augmented and extended with data from other sources. I bet that other web application scanning services (White Hat, et al), penetration testing companies, managed security services companies, and forensic analysis services could add a lot to get a more comprehensive picture. As a professional community, we need to find ways to do this sort of study in a way that can be extended and combined with other data sets and studies.
Absolutely, brilliant!
I think over half of all resources expended on infosec are for useless “trending” and “high level reporting” efforts like this.
They are useless for exactly the reason you have mentioned: because there is no sound methodology for discovering, characterizing, and reporting “trends” over any long (>3 days, say) time period.
What these reports will tell you that in the past 12 months the community as a whole saw a shift to…the stuff you have been dealing with over the past 12 months. What I need to know from a trending perspective is what response trends appeared at different organizations. Did they spend more or less? Did they hire more people with more security certifications? Did incidents cost more?
None of this stuff is actionable.
What amazes me about this whole process is how the “new media” is often nothing more than an echo chamber, repeating what some “authority” publishes without any evaluation or validation. One blog post ended with something like this: “These guys are experts, so we should listen to them.”
I did see a couple of online articles that did add some information, but only in the form of interviews of people involved in the study or the bosses at SANS.
This pattern of uncritical acceptance of these studies makes me skeptical when ever I hear the phrase “best practices” — as if the practices have been carefully evaluated against alternatives using some objective performance standards. Instead, I wonder how many “best practices” are just this: “I heard it from a guy… who heard it from a guy… who heard it from another guy…”
I can see parallels in lead-up to the Iraq War, esp. the “Iraq Dossier” that was “sexed up”, and the rest. When you have self-interested parties producing reports of “fact” that promote their interests and agenda, we should all be very skeptical.
This kind of impenetrable analysis is easily predictable from the fact that it comes from profit making organizations. The fact that its sponsors, Qualys and TippingPoint, also provide the data, irretrievably contaminates its trustworthiness, the sincere but amateur efforts of the SANS ISC notwithstanding.
This phenomenon is precisely why we have government-sponsored agencies such as Centers for Disease Control and Prevention, and the World Health Organization, doing surveillance for disease prevalence. Imagine if our best data about the H1N1 pandemic came from Roche, based on a geographic breakdown of quarterly sales of Tamiflu. It would be intolerable for public health, and it should be intolerable for public cyberhealth.
The U.S. government already has collection points at all the right locations to provide authoritative measurements, but they’re secret anti-terrorism projects, and inter-agency rivalries (or is it intra-agency now that they’re all within DHS?) prevent them from giving anyone the data needed to “connect the dots” into a credible picture.
@Dean Good points, though I wouldn’t go so far as to say that the SANS study was “irretrievably contanimated” or that for-profit companies can’t usefully participate or lead studies like this.
What we need are checks and balances and transparency. If readers know where the data comes from, has some basis for confidence that the data was collected and recorded reliably, and that the data analysis methods are well-suited to the goals, then we can have some confidence in the outcomes, especially if the study is reproducable or if it faces competing studies.
If for-profit organizations know that they will be facing this sort of scrutiny, they will do better studies. Maybe not perfect or ideal, but at least fairly good.
If, instead, we take a “leave it to the government” approach, we will be depriving ourselves of the resources, creativity, and initiative of the private sector. I’d rather see positive economic incentives for private sector organization to do more and more of this type of study, especially to collaborate with others. IMHO that is the best way to get the richest set of information and insights into a very complex, fast-changing world.
“Attacks against web applications constitute more than 60% of the total attack attempts observed on the Internet.”
They’ve quoted Gartner there. What’s concerning is that Gartner are known for 80/20 results, so this result is suspicious…
These are all good points and it is nice to see some critical analysis of this report. While we may be able to criticize some of the minutiae of the report, I didn’t read anything in it that rang untrue to me. If you work in the industry you know that:
– Many of the vulnerabilities these days are in applications such as Adobe Acrobat Reader and Flash Player.
– There have been lots of zero day vulnerabilities reported this year. I can think of half a dozen in the last few months alone.
– Web application attacks are on the rise and are one of the primary threats on the Internet today.
– Patch management is a time consuming and challenging task that many companies do poorly. This in turn can lead to compromised systems due to the number of malicious web sites and malicious spam email.
We can argue about whether 60% of attacks are against web applications or if 80% of vulnerabilities are SQL Injection or XSS flaws. But it seems to me that the broader picture being painted here is in line with what I see every day in my work, research, and reading.
@Mark — Yes, the general points of this report may, in fact, be true. I certainly don’t have contrary evidence. (See the comments to this NYT blog for some debate: http://bits.blogs.nytimes.com/2009/09/15/security-pros-are-focused-on-the-wrong-threats/)
What my critique is trying to do is hold all of us to a higher standard. This report was presented as a large-scale study based on both attack and vulnerability data. Nearly all of the blogs and online articles that echoed this report mentioned it’s quantitative basis.
The New School manifesto is that it’s not good enough anymore to rely on informed opinion and generally-held beliefs about top security risks, etc. Instead, what we need are robust data analysis, field studies, experiments, and simulations. We need to test hypotheses in a way they can be confirmed or discredited . We need to know more about the limits of our knowledge, unknown-unknowns, and what we need to learn next to improve our understanding. Most of all, we need critical tests that challenge our conventional wisdom.
At the very least, any study should make clear which conclusions are based on solid data analysis and which are based on informed opinion, collective wisdom, or hand-waving.
Yes, some of my critique points were relatively small or cosmetic. But even the cosmetic flaws can get in the way of people making good use of this study, and increase the chances that they will get confused or misinterpret the results.
Lastly, the authors of this report should have been writing for another audience: other organizations who might want to replicate the study internally. Just think how much more compelling the results would be if 5, 10, or 20 large organizations did the same study internally and then published the results in a comparable manner?
Corrected NYT link: http://bits.blogs.nytimes.com/2009/09/15/security-pros-are-focused-on-the-wrong-threats/