Shostack + Friends Blog Archive

A Letter from Sid CRISC – ious

In the comments to “Why I Don’t Like CRISC” where I challenge ISACA to show us in valid scale and in publicly available models, the risk reduction of COBIT adoption, reader Sid starts to get it, but then kinda devolves into a defense of COBIT or something.  But it’s a great comment, and I wanted to address his comments and clarify my position a bit.  Sid writes:

Alex,

Just imagine (or try at your own risk) this –

Step 1. Carry out risk assessment
Step 2. In your organisation, boycott all COBiT recommendations / requirements for 3-6 months
Step 3. Carry out risk assessment again

Do you see increase in risk? If Yes, then you will agree that adoption of Cobit has reduced the risk for you so far.

You might argue that its ‘a Control’ that ultimately reduces risk & not Cobit.. however I sincerely feel that ‘effectiveness’ of the control can be greatly improved by adopting cobit governance framework & Improvement of controls can be translated into reduced risk.

I can go on writting about how cobit also governs your risk universe, but I am sure you are experienced enough to understand these overlapping concepts without getting much confused.

Nice try, Sid!  However, remember my beef is that Information Risk Management isn’t mature enough.  Thus I’ve asked for “valid scales” (i.e. not multiplication or division using ordinal values) and publicly available models (because the state of our public models best mirrors the maturity of the overall population of risk analysts).

And that’s my point, even if I *give* you the fact that we can make proper point predictions for a complex adaptive system (which I would argue we can’t, thus nullifying every IT risk approach I’ve ever seen), there isn’t a publicly available model that can do Steps One and Three in a defensible manner.  Yet ISACA seems hell-bent on pushing forth some sort of certification (money talks?).  This despite the inability of our industry to even use the correct numerical scales in risk assessment, more or less actually performing risk assessment in a means that can be even used to govern on a strategic level, or even showing an ability to identify key determinants in a population.

Seriously, if you can’t put two analysts with the same information in two separate rooms and have them arrive at the same conclusions given the same data – how can you possibly “certify” anything other than “this person is smart enough to know there isn’t an answer”?

LET’S TALK COBIT

I want to make one thing clear.  My beef isn’t with ISACA, it’s not with COBIT, it’s not with audit.  I think all three of these things are awesome to some degree for some reasons.  And especially, Sid, my beef isn’t COBIT – I’m a big process weenie these days because the data we do have (See Visible Ops for Security) suggests that maturity is  a risk reducing determinant.  However, this is like a doctor telling a fat person that they should exercise based on vague association with a published study of some bias.  How much, what kind, and absolute effectiveness compared to existing lifestyle is (and esp. how to change lifestyle if that is a cause) is still very much a guess.  It’s an expert (if I can call myself an expert) opinion, not a scientific fact.

In the same way your assertion about COBIT fails reasoned scrutiny.  First, there is the element of “luck”.  In what data we do have, we know that while there is a pretty even spread in frequency of representation in data breaches between determined and random attackers.  That latter aspect means that it’s entirely likely that we could dump COBIT and NOT see an increase in outcomes (whether this is an increase in risk is another philosophical argument for another day).

Second, maybe it’s my “lack of experience” but I will admit that I am very confused these days as to a proper definition of IT Security Governance.  Here’s why; there are many definitions (formal, informal) I’ve read about what ITSec G is.  If you argue that it is simply the assignment of responsibility, that’s fine.  If you want to call it a means to mature an organization to reduce risk (as you do above), then we have to apply proper scrutiny towards maturity models, and how the outcomes of those models influence risk assessment outcomes (the wonderful part of your comment there is the absolute recognition of this).  If you want to call it a means to maturity or if ITSec G is an enumeration of the actual processes that “must” be done, then we get to ask “why”.  And once that happens, well, I’m from Missouri – you have to show me.  And then we’re back into risk modeling, which, of course, we’re simply very immature at.

Any way I look at it, Sid, I can’t see how we’re ready for a certification around Information Risk Management.

Side Note: My problem with IT Security Governance is this: If at any point there needs to be some measuring and modeling done to create justification of codified IT Security Governance, then the Governance documentation is really just a model that says “here’s how the world should work” and as a model requires measuring, falsification, and comparative analytics. In other words, it’s just management.  In this case, the management of IT risk, which sounds like a nice way of saying “risk management”.

Originally published by alex on 25 Oct 2010
Last modified on 25 Oct 2010
Categories:   careers   fail   government   measurement   metrics   Science of Risk Management   Uncategorized