Quantitative Analysis of Web Application Usefulness (Or Why Your ROSI is wRONG)
The amazing (in both quality and quantity of blog post production) Lori MacVittie of f5 has a blog post up on their corporate blog called, “A Formula for Quantifying Productivity of Web Applications.”
Basically, Lori proposes that we study web server processes and the time to complete them over a period of time rather than just simple agent utilization. Which is awesome. Except….
THE MISSING PRODUCTIVITY PIECE, “ARE WE BEST SERVING THE USERS”?
It’s not a comprehensive quantitative analysis of web application usefulness, just something to give you a better idea if there’s enough resources available. This doesn’t really give us an idea about how useful our users find the web application. And I think that’s important.
It occurred to me while I was dreaming of ROSI one day recently, that there is a missing piece to the investment picture. User Frustration. You see, we can build applications and measure all sorts of engineering metrics in order to discuss what value IT has to the organization and completely miss the most important question, “do we suck?” Let me give you an example of what I mean:
Recently, I moved to work with Verizon. And as you can guess, my iPhone wasn’t going to be used for business use any longer. I was issued one of our standard corporate smartphones that, well, being nice to the vendor involved, isn’t an iPhone (if you know what I mean). Now I know that the iPhone isn’t perfect, but you see, I only wanted to throw it out the window of a 12 story building maybe once or twice a week, and that was primarily due to the network and not the platform itself. With my “clown-phone” (as I affectionately call it) the network I use absolutely rocks, but I find myself rebooting every other day and wanting to hurl it onto the concrete from great heights about three times a day. I get very frustrated with it and the spinning hourglass.
Alex’s law of phone usefulness: Users shouldn’t ever have to remove the battery just to make a phone call.
As security professionals, we know very well how to frustrate a user. In fact, while I was dreaming of ROSI, I realized that if we were ever going to understand how useful any new InfoSec investment is, we would have to account for how much it was going to piss off our employees. Is security seamless, or does it roadblock getting business done?
So not to put too fine a point on it, I’m going to suggest that CIO’s build a simple web application out of their existing ticket system (irony noted) that measures “Frustration Hours”. Clownphone got you down? Go to the web app, enter the time you just wasted watching the hour glass spin. To Lori’s point above – the web application might be fast and responsive, but if it’s anything like some of the travel expense web apps I’ve seen, it takes 2 hours just to figure it out!
Seriously? It’s a web app to enter in my travel expesnses. If you need to produce a training video teaching me how to use it, you’re doing it wrong! Not only do I now resent the process, but you just cost the corporation roughly $200. Want a useful IT metric? Multiply that by 50,000 employees.
USING THE USER FRUSTRATION METRIC
So it’s real simple. Web app has three fields
- A field to enter in the technology involved
- A field where the user approximates how much time they wasted
- What they believe the main cause of frustration was
You take the technology involved (clown phone, terrible web app, whatever), and you simply measure user frustration hours per month that the ticket site gives you. Then do a simple analysis on the main causes of frustration and fix! Viola!
In terms of ROSI – before we buy the next great silver bullet and try to justify the cost, we could ask ourselves how much frustration will we be adding to the users day? Will it take an extra 15 minutes to boot every morning because we keep increasing the security load but IT won’t upgrade the RAM on these XP Pro machines beyond 1 GB? Are we adding yet another password? Is blocking this physical entryway worth the cost of making everyone walk an extra 5 minutes every day vs. the probability that it will be used in an SE attack? Heck, I’ve heard rumors that in one organization where Op sec and IRM are separated, IRM produced a risk analysis suggesting that compared to the data risk a complex password scheme on a smartphone would reduce, it was more likely that a user would be killed in a car crash trying to enter in the complex password while driving.
And Dilbert IT jokes aside, understand that there’s a security implication to User Frustration Hours. If they are caused by policy, then policy might not be reflecting the organizations risk tolerance. And though policy adherence never guarantees behavior 100% of the time, policy avoidance is even less predictable. And less predictable, in technical risk management terms, is more bad.