Counting In Computer Security
Last week in “Notes from the Security Road,” Mike Nash wrote:
My favorite moment on the trip — which actually resulted in my circumnavigating the entire globe in just a week — was when we illustrated the difference in the number of vulnerabilities in Windows Server 2003 compared to its competitive product, Red Hat Enterprise Linux 3. Steve held Red Hots candies for each vulnerability that he would have had to manage as a Red Hat customer in the last six months. Steve ended dropping quite a few candies on the floor with 217 Red Hots (for 217 vulnerabilities in the last six months) to hold. In contrast, Windows Server 2003 only had 32 vulnerabilities for the same period.
I find this to be a fascinating statement on a whole bunch of levels. Firstly, because it’s such a great visual. Red Hots slipping out of your hands, and bouncing around the floor. 
But then I asked myself, what are those Red Hots? Are they just candy? As Red Hots they are discrete, countable bits of cinnamon goodness. But what is candy, but sugar (and in this case cinnamon)? The bag of sugar that goes into the Red Hots is just that, a bag of sugar which the Ferrara Pan Candy Company separates and crystalizes into Red Hots. But there are other ways to mix that sugar into candy. For example, when you take that same weight of sugar, melt it and add hot air, you get a big blob of cotton candy. (Unfortunately, I don’t have a cotton-candy machine, or you’d have a picture of how big they get.) Or if you melt 217 Red Hots together into a lump, you get something more densely packed, and more manageable. Perhaps its sad, but I’m spending a lot of time lately dealing with questions of taxonomies and atomic units in security configuration, and so I can barely help asking what they measured, and how they chose to divvy up the sweet mess that are vulnerabilities. It’s also interesting because (as I’ll explain) they happen to be slightly factually incorrect in the claim.
More after the break.
So, the statement: “Windows Server 2003 only had 32 vulnerabilities for the last 6 months.” Sorry, that’s almost certainly wrong. When Microsoft gets a private bug report, they go and look for related vulnerabilities in the code, and try to fix them all. This makes lots of sense, although it does increase the amount of change the patch introduces, which has reliability impacts. So because Microsoft does look for other issues and fixes them, then 32 patches implies more than 32 bugs, or vulnerabilities.
Had he said “patches,” this would I think, be an accurate statement.
So I’m going to pretend that he did say that. I’m perfectly willing to believe that Microsoft counted patches, had 32 patches, and since patches fix vulns, the claim became that they fixed 32 vulns.
What’s more, patches are a very good thing to measure. Patches, after all, have become a staple of the system administrator’s job. Sometimes, patches even line up one-to-one with vulns. But patches are not the only thing you could measure. You might measure CVE entries. CVE entries sometimes line up one to one with patches (Microsoft’s MS05-043, Red Hat’s RHSA-2005-307.html), and sometimes not (Microsoft’s MS05-042, Red Hat’s RHSA-2005-365).
It may be interesting to look at vulnerabilities, rather than patches. It’s not so interesting to the system administrator, but it’s far more interesting to the security analyst. Vuln counts are approximated by CVEs, but again if a vendor fixes four stack smashes in a function at the same time, and issues one patch, its likely to get one CVE. That’s true of both open and closed source vendors. Now, with the source, its possible, but time consuming, to examine each change, and see if it fixes a vuln. But that may not be complete or easily reproducable by a second analyst. Changing an int from signed to unsigned may fix several integer-related problems, exploitable in different places. It could be a one line fix to several exploits.
It might be interesting to count how many things are open to an unauthenticated attacker, versus someone local. It might be interesting to split things by code from the vendor, versus external code they’ve included. (Then again, it might not. I don’t think Red Hat gives me that choice on install, and even if they do, they’re still selecting packages and integrating them into a system. Shouldn’t they be held responsible for the integrated system, which, after all, is what they sell?
It could be interesting to count how many are rated critical by the vendor. (Thankfully, both Microsoft and RedHat have moved to a Critical/important/medium/low scale.) It would be better to use an independent measure, like CVSS, or CERT metrics, but CERT’s metric unfortunately includes a concept of scale, making it great for both people worried about the state of the infrastructure, and worthless to anyone else. I guess they know who’s paying their salaries. (As an aside, I talked about the CVSS system, and there’s some good links there.)
I thought, but have only anecdotes, that the one CVE:one patch is more common in open source projects. But in looking at a small data set of Red Hat patches, that theory is contradicted.
The only thing worse than doing analysis on your own prejudices and impressions is doing analysis on really small data sets. With the former, there’s less risk of people misunderstanding what you’ve done. So perhaps at this point, some data would be helpful. So bear with me as I try to collect some data. The first thing I notice is that RHEL3 is the previous release. (Red Hat’s Security Updates page is here.) The second thing is that there are 6 versions of the OS: Enterprise Linux AS, ES, and WS, along with Desktop, Cluster Suite, and Developer Suite. I started by looking at security advisories for RHEL 3 AS. There were 230. Which, incidentally, is not equal to 217. Ahah! Last 6 months, they said. But, now I’m looking at 97, with a generous definition of “last 6 months” (namely, everything back to the end of February, 2005). I’m not actually hand counting, I pulled the HTML table of advisories from the web page, and have been editing it up. That was a good 15 minutes of work, so maybe I’ll email someone at Microsoft and see if I can learn precisely what those Red Hots represented.
Having said all of that, readers paying close attention may have noticed that I haven’t justified the act of counting. Counting (or measuring) is an important part of gathering data to answer questions. And I’ve left the questions implied. The first is “which system is easier to manage securely;” the second, “which system is more secure?” I don’t actually gather enough data to answer either, because I really wanted to focus on the many different things you might count.
The other question which would be a good deal of work but quite interesting would be comparing equivalent features – e.g. should we exclude OpenOffice problems from Red Hat or include all of the MS Office vulnerabilities? Should we exclude overlapping packages since, for example, relatively few people use more than one database?
Chris makes a good point. I’d also add that Red Hat comes bundled with an awful lot of software, some of which you’d get from third parties if you wanted to do similar tasks with Windows Server. That means Microsoft has the luxury of blaming vulnerabilities in those packages on third parties, and not counting them in its totals.
I used to argue that Red Hat’s patch management was much easier, making up for some of the difference, but Microsoft’s has gotten pretty good, too.
No need to go to RedHat’s errata page for counts of patches; RedHat has Data from the security release team up in parseable format — and code to do the parsing up along with it.
I definitely agree with Chris and David — you can’t simply compare Red Hat (any release) directly to MS Windows in terms of “equivelant systems”. There’s large differences in what’s included. You either have to add things like Office to the MS side or subtract things like OpenOffice from the RedHat side. (the adds and subtracts may go the other way in some cases)
You also can’t easily compare “patches” between them. From what I can see, Microsoft uses a patch-based mechanism (like Sun uses) and Red Hat uses an update-based mechanism. That is, MS (and Sun) release something that goes in and fixes everything, mostly by replacing a lot of files with new versions — RedHat has a definition of a “package” and releases new versions of that. This is significant in comparing “patches” because for, say, an SSL vulnerability, MS will release one “patch” and RedHat will release several “updates” because the SSL packages are broken into pieces.
Or to put it better: MS’s patches probably compare more closely to RedHat’s “Errata”. Especially if you exclude Errata for packages that MS doesn’t include an equivelant in the base OS.
Also, when you’re looking at a 6-month timeframe, RedHat releases major (non-security) updates quarterly for the most recent release. So every 3 months or so, there’ll be a large number of minor bug fixes all at once. MS does a similar thing, but it’s one big “service pack” instead of 100 updates.
As a systems administrator, I find the question of “how many patches” pretty irrelevant. The question of “how often do I have to deal with patching this system and how much work is it” is most interesting. In other words, a comparison of the Windows update service and of RHN/up2date. How often is clearly a function of how many vulnerabilities, but only if they’re relevant to my infrastructure. (if I don’t have software installed I don’t have to patch it. And most of the time if it’s installed but we don’t use it, I still don’t have to patch it.)
I haven’t checked out Windows update lately, but RHN’s pretty good at a lot of these things. Applying all required patches to a system is one command. Or, if the system is configured to allow it, clicking a few buttons in a web page (which can just as easily request a thousand machines be patched). Getting a list of which of my systems require a specific update is easy (via a web interface), as is getting lists of which systems require patching at all, etc. And that’s all with something totally usable for patching servers that doesn’t require setting up a special server to handle it.
And of course, the key point is: you can’t just meaningly compare by numbers, with or without candy to help.
So nice of you not to jump on that “circumnavigated the entire world”. What else would he have circumnavigated?
Funny that “Steve” and Mike are resorting to parlor tricks to argue the benefits of Win2003 Server over a competing product.
Time to move the cheese? Let’s talk about up-time.
Eric,
Thanks for your extensive comments.
I’m curious, do you run up2date out of cron? I don’t, I like to look at the advisories and make decisions on how important they are to me. Therefore, the number of pages I have to wade through (and the number of issues fixed) becomes an important contributor to the effort I need to make.
I’m also curious about your saying “ And most of the time if it’s installed but we don’t use it, I still don’t have to patch it. And most of the time if it’s installed but we don’t use it, I still don’t have to patch it.” Could you explain why not? Can’t the attacker use it?
Chris, David,
I see the point you’re making, but isn’t it also fair to count everything in a default install? Shouldn’t distribution vendors be including questions of security and breadth of use in those decisions?
In a way, yes but that’s something of a philosophical difference – I *like* having thousands of packages available for Debian/Ubuntu (it’s a compelling selling point over Windows or Solaris) even though we tend not to use that many of them. One way of recognizing that a default Windows install isn’t very useful for most people would be to compare some roughly equivalent profiles – e.g. database-backed website might be Windows + SQL Server + IIS or Apache + PHP + (MySQL|Postgres), “office worker” would include MS Office or OpenOffice, Adobe Acrobat, etc.
It could be particularly interesting to do this comparison with the applications which are available on multiple platforms – e.g. how many things would affect a dedicated Oracle Financials server.
One other interesting metric would be support burdern – count the number of incidents requiring manual work, service interruptions or reboots. For example, I consider Internet Explorer updates for more annoying than Linux Mozilla/Firefox updates because the practical means of deploying them on Windows (WSUS) requires all of my computers to reboot at least once (depending on whether they’ve provided multiple updates and whether some are flagged as exclusive) whereas it’s an automated, background process on Linux which doesn’t affect anything else on the machine. And Internet Explorer is the best one because the automatic updater at least works reliably – unlike, say, Office – and exists in the first place (almost all third-party products).
Adam,
I would never run up2date out of cron. Okay, maybe up2date in download only mode. It’s too likely that something would break in the middle of the night (or at least require reconfiguration of something afterwards). All patching of production customer-visible systems involves announcements of outage windows, a sysadmin on-site to deal with any problems that might arise, and usually testing the patches at least minimally on a non-production system first. And, yes, also reading the announcements first. Usually I just read the email reports. RHN emails out information about updates that apply to my systems, and there’s also the enterprise-watch-list.
I do use up2date totally on the command-line, though. “up2date -u”, “up2date -fu”, “up2date somepackage”, “up2date –solvedeps=’perl(Some::Module)'”, et cetera.
Well, essentially we do a risk analysis. Usually that’s a quick look at the issues with the vulnerability based on my knowledge of our systems and RedHat’s description of the problem and its seriousness. If there’s any question, I might involve our Security Analyst in the decision about whether an “emergency outage” is justified.
If the vulnerability were, say, in tcpdump (which generally needs to be run as root, is not setuid (doesn’t have elevated priveleges) and doesn’t run without manual intervention), we can put off patching tcpdump until we’re actually going to use it on a system. If the problem is in vi or emacs and it’s on a system where you never edit files from an outside source, there’s no motivation to immediately patch. Those sorts of problems would generally wait for our next quarterly “patch night”.
On the other hand, if the problem were in apache or bind, those are likely to be “public facing” and we’d want to patch them as soon as possible; probably only an hour or two test run and then apply the updates to production systems that evening (with an emergency outage announcement). Same with anything that allows priveleges to be escalated, such as a bug in sudo or some other setuid program. Even if there’s no way for an attacker to get to the shell, the *next* problem might be one that allows it.
Or sometimes issues are kinda in the middle. Maybe because it’s on a system that’s only accessible to a limited population of employees, not to the general public. Those would get a non-emergency announcement of about 2 weeks and a correspondingly longer test period.
And if there were an important tcpdump or vi update (but not apache or bind), I might just “sneak it in” and apply the patch without notification to our user population of a potential problem, since tcpdump breaking simply won’t affect anybody who’s not a sysadmin on the system.
Also, remember: just because RH includes it in the distribution doesn’t mean it’s installed on a system. In many cases they include competing products. It’s very unlikely that you’d be using sendmail, postfix and exim on the same system. I think “install all packages” is pretty unusual in any kind of enterprise linux configuration.
@eric:
Since tcpdump sees every single packet on the wire (as used 99% of the time), with the right kind of bug it can get your system 0wned w/out a single packet being sent to its IP address. That makes the risk analysis a little different, IMO. Do your admins really have the restraint to wait two months before “patch day” arrives and they get to run tcpdump again? That’s one patch I’d apply almost immediately (probably right after vi :^))