Measurement Priorities

Seth Godin asks an excellent question:

Is something important because you measure it, or is it measured because it’s important?

I find that we tend to measure what we can, rather than working toward being able to measure what we should, in large part because some variation of this question is not asked.

I’m going to pick on malware fighting as a case-in-point.  Is there lots of nasty malware out there than can really destroy your infrastructure?  Absolutely, and as a result, IT and IT Security teams put tremendous effort into detecting and cleaning malware infections.

But how much of that malware actually impacts business, either by affecting the availability of the IT environment or producing a material incident?  If your business is like most, then the answer is hardly ever*.

So why does the security industry spend so much money and time (another form of money ) on malware?  Because we can.  Never mind that the stuff you should be truly worried about if you’re talking about protecting The Business (as opposed to The Infrastructure) is the APT/Custom Malware/Targeted Threat stuff, which is invisible on the anti-virus console?

Because they can.


* While that could be changing thanks to innovations like StuxNet, who honestly thinks that messing their business up is worth burning three 0days?  Really?  Get over yourself.

In the meantime, you can test this argument in your own environment.  Compare the number of pieces of malware you’ve detected and cleaned (versus prevented) versus the number that significantly impacted more than the infected person’s machine.

We’ve had one in the past year that might meet the disruption test, versus multiple malware cleanup tickets per week (not daily, but it tends to be spiky, so the average is greater than one per day.  Still…).  It took out a single user who had managed to break his Anti-Virus because either he didn’t like when the full scan ran or it kept stopping him from installing the trojaned, pirated software he’d downloaded–I never quite got a clear answer on this.  The infection jammed the mail queue with outbound spam, causing a degradation (but not disruption) of outbound email for a few hours.

Fixes to Wysopal’s Application Security Debt Metric

In two recent blog posts (here and here), Chris Wysopal (CTO of Veracode) proposed a metric called “Application Security Debt”.  I like the general idea, but I have found some problems in his method.  In this post, I suggest corrections that will be both more credible and more accurate, at least for half of the formula.  The second half is harder to do right and needs more thinking.

Continue reading

Just Because YOU Think Your Clients Are Too Busy and/or Stupid Doesn't Mean Everyone Else Is

Mike Rothman’s “Firestarter” on “Risk Metrics are Crap“.

It’s very difficult to argue with a poorly constructed argument.  Especially when I have no idea what a “risk metric” is.  But best as I can tell, Mike’s position is that unless you are smart and/or have strong resources allocated to your InfoSec team, things like metrics, GRC, application security, SEIM, and a “host of other security processes or technologies.” are “science projects…”

Meaning, I suppose, that they provide no “pragmatic” use to security departments (whatever pragmatic means).

Problem is, for many folks, metrics, risk management, appsec and other “security processes or technologies” can and do have significant value.  In fact, in terms of managing a large, disparate enterprise, the data gathering process for risk analysis alone can be more valuable than the result (experience contrary  to what Rothman writes in comments: “the value of the (assessment) benefit is outweighed by the cost of gathering the data.”).

That said, it’s a shame that his argument is poorly constructed because, by in large, I have to agree that there’s plenty of poopy risk statements to pick on.  As I said in my CSO Magazine interview (shameless self promotion) and in my RSA Risk Management Smackdown panel – there have been times when I’ve counseled an organization to put off making risk statements until their visibility into their environment is much better.  In the public record you should be able to find past statements where I say it’s better for a small business to focus resources away from risk assessment when the required assessment was a bureaucratic quest rather than a quest for knowledge or wisdom.

Bottom Line, for risk and metrics, Rothman shouldn’t generalize for the sake of sensationalism and marketing.  And he probably shouldn’t be doing that for other security processes and technologies beyond risk analysis and security management, too.   But as a consulting firm, what Securosis could/should be doing is giving people help – helping them recognize their organizational maturity, and helping them understand what resource allocation is or isn’t appropriate at various levels of maturity.   Just a thought.