Where is Information Security's Nate Silver?
So by now everyone knows that Nate Silver predicted 50 out of 50 states in the 2012 election. Michael Cosentino has a great picture:
Actually, he was one of many quants who predicted what was going to happen via meta-analysis of the data that was available. So here’s my question. Who’s making testable predictions of information security events, and doing better than a coin toss?
Statistical analogies are inappropriate for cybersec, whether it be automobile accident rates, weather events, stock market directions, or election results. These things study large numbers of mostly accidental events, whereas hacking almost always comes down to the decision of a single person.
A good example of this was the MS03-039 bug (the “Blaster” bug). It was the first ‘remote root in the default install’ of Windows. When Microsoft released the bug, we knew within an hour that it was robustly exploitable and wormable, and hence, that a worm was coming that would infect millions of systems. Yet, we couldn’t predict when. We couldn’t predice how long it would take for hackers to release exploits for a bug. We couldn’t predict when a hacker would take an exploit and make it self-propagating. We couldn’t predict things like a bug in Blaster that used 7 UDP packets to copy itself, causing the worm to limit its own spread, because once congestion reached the point where 1 out of 7 packets was dropped, the worm would essentially stop propagating.
Really? If we know a worm is now possible, couldn’t we study past worm releases to get an idea? Can’t we study the set of cases where hackers release exploit code to see what the range is for similar cases?
At the end of the day, automobile accident rates, stock market directions, or election results all hinge on the choices of millions of people, and we somehow manage to study those, along with non-electronic crime rates.
I’ve believed for many years that it is impossible to develop reliable probabilities of information security events. The human element in this area is just too unpredictable. OTOH, the environment these days is so toxic, it’s probably best to just base your security decisions on p=1. This, of course, makes allocation of scarce funds no easier…
Hmmm, here I thought that voting also involved unpredictable humans, and that’s why we did it.
And these comments are why we fail.