Quantitative Analysis of Web Application Usefulness (Or Why Your ROSI is wRONG)
The amazing (in both quality and quantity of blog post production) Lori MacVittie of f5 has a blog post up on their corporate blog called, “A Formula for Quantifying Productivity of Web Applications.”
Basically, Lori proposes that we study web server processes and the time to complete them over a period of time rather than just simple agent utilization. Which is awesome. Except….
THE MISSING PRODUCTIVITY PIECE, “ARE WE BEST SERVING THE USERS”?
It’s not a comprehensive quantitative analysis of web application usefulness, just something to give you a better idea if there’s enough resources available. This doesn’t really give us an idea about how useful our users find the web application. And I think that’s important.
It occurred to me while I was dreaming of ROSI one day recently, that there is a missing piece to the investment picture. User Frustration. You see, we can build applications and measure all sorts of engineering metrics in order to discuss what value IT has to the organization and completely miss the most important question, “do we suck?” Let me give you an example of what I mean:
Recently, I moved to work with Verizon. And as you can guess, my iPhone wasn’t going to be used for business use any longer. I was issued one of our standard corporate smartphones that, well, being nice to the vendor involved, isn’t an iPhone (if you know what I mean). Now I know that the iPhone isn’t perfect, but you see, I only wanted to throw it out the window of a 12 story building maybe once or twice a week, and that was primarily due to the network and not the platform itself. With my “clown-phone” (as I affectionately call it) the network I use absolutely rocks, but I find myself rebooting every other day and wanting to hurl it onto the concrete from great heights about three times a day. I get very frustrated with it and the spinning hourglass.
Alex’s law of phone usefulness: Users shouldn’t ever have to remove the battery just to make a phone call.
As security professionals, we know very well how to frustrate a user. In fact, while I was dreaming of ROSI, I realized that if we were ever going to understand how useful any new InfoSec investment is, we would have to account for how much it was going to piss off our employees. Is security seamless, or does it roadblock getting business done?
So not to put too fine a point on it, I’m going to suggest that CIO’s build a simple web application out of their existing ticket system (irony noted) that measures “Frustration Hours”. Clownphone got you down? Go to the web app, enter the time you just wasted watching the hour glass spin. To Lori’s point above – the web application might be fast and responsive, but if it’s anything like some of the travel expense web apps I’ve seen, it takes 2 hours just to figure it out!
Seriously? It’s a web app to enter in my travel expesnses. If you need to produce a training video teaching me how to use it, you’re doing it wrong! Not only do I now resent the process, but you just cost the corporation roughly $200. Want a useful IT metric? Multiply that by 50,000 employees.
USING THE USER FRUSTRATION METRIC
So it’s real simple. Web app has three fields
- A field to enter in the technology involved
- A field where the user approximates how much time they wasted
- What they believe the main cause of frustration was
You take the technology involved (clown phone, terrible web app, whatever), and you simply measure user frustration hours per month that the ticket site gives you. Then do a simple analysis on the main causes of frustration and fix! Viola!
In terms of ROSI – before we buy the next great silver bullet and try to justify the cost, we could ask ourselves how much frustration will we be adding to the users day? Will it take an extra 15 minutes to boot every morning because we keep increasing the security load but IT won’t upgrade the RAM on these XP Pro machines beyond 1 GB? Are we adding yet another password? Is blocking this physical entryway worth the cost of making everyone walk an extra 5 minutes every day vs. the probability that it will be used in an SE attack? Heck, I’ve heard rumors that in one organization where Op sec and IRM are separated, IRM produced a risk analysis suggesting that compared to the data risk a complex password scheme on a smartphone would reduce, it was more likely that a user would be killed in a car crash trying to enter in the complex password while driving.
And Dilbert IT jokes aside, understand that there’s a security implication to User Frustration Hours. If they are caused by policy, then policy might not be reflecting the organizations risk tolerance. And though policy adherence never guarantees behavior 100% of the time, policy avoidance is even less predictable. And less predictable, in technical risk management terms, is more bad.
Doesn’t Verizon have an IT service desk? Don’t they track calls? Looks to me like this frustration metric is a big part of (what should be…) relatively standard IT service management.
Maybe I drank too much ITIL Kool-Aid…
Chris,
I’m trying to be minimalist here. Simplifying the development of IT Help Desks to only what is too important to jettison. And that is “how much do we suck”?
In other words, while Verizon IT does a good job relative to the industry (*you* try serving 250k users), there’s no way I’m going to call the helpdesk and say “Hey, my phone sucks”. Because they’ll ask “why” and I’ll tell them, and then we’ll spend 3 hours figuring out that it’s just a crappy phone that is operating to specifications. Or my travel app? It’s probably operating to functional and technical requirements. That doesn’t mean it doesn’t unnecessarily cost the company money.
If you follow what I suggest, the user gets to hit an internal website and tell them that my phone sucked again today, or that it took me 30 minutes to enter in 4 receipts. Not only is that “easy” but it’s measuring something that I wouldn’t normally call in about, and something that tangibly costs the company money.
Another example, if you have a Mac and have gone through the learning curve to learn Pages, you might feel, like me, that it’s a much nicer application than other Mac Office Apps. But I would never call into a help desk just to say “Your standard build of Open Office crashed on me again today and wasted 20 minutes of my time”. The help desk would be like “uh, thanks”, or they would take 50 minutes of my life running through a script of “fixes” to my problem.
If you just measure UFH – you might see that while there are few HD calls, this application costs the average user 40 minutes a week in “frustration”. If you want, you can multiply that by a standard “average user cost per hour” and # of users metric.
I’m definitely an example of the behavior that Alex refers to–I consider opening a ticket to the Helpless Desk to be the support of last resort, and usually only do so for hardware issues.
I spend a non-trivial amount of time most weeks dealing with broken apps–as a user, not as a risk & security person–and from what I can tell, I’m better at working around broken tools than most.
This seems like a good approach to prioritising issues that need a fix- however I would be wary of trusting too much the figures users give for the amount of time they have lost. These could be swayed dramatically by a particuarly demanding user.
@Chandler, I agree – in many cases it is quicker to find a fix yourself if you are familar with the software. When you compare the time this can take with to that needed to document the issue and explain exacty how the program should function it seems the “quicker” option. However this could result in a large number of users still experiencing problems which they don’t know how to fix.