A critique of Ponemon Institute methodology for "churn"
Both Dissent and George Hulme took issue with my post Thursday, and pointed to the Ponemon U.S. Cost of a Data Breach Study, which says:
Average abnormal churn rates across all incidents in the study were slightly higher than last year (from 3.6 percent in 2008 to 3.7 percent in 2009), which was measured by the loss of customers who were directly affected by the data breach event (i.e., typically those receiving notification). The industries with the highest churn rate were pharmaceuticals, communications and healthcare (all at 6 percent), followed by financial services and services (both at 5 percent.)
Some comments:
- 126 of the hundreds of organizations that suffered a breach were selected (no word on how) to receive a survey. 45 responded, which might be a decent response rate, but we need to know how the 126 were selected from the set of breached entities.
- We don’t understand the baseline for customer churn. What is normal turnover? Is it the median for the last 3 years for that company? The mean for the sector last year? If we knew how normal turnover was defined, and its variance, then we could ask questions about what abnormal means. Is it the difference between management estimates and prior years? Is it the difference between a standard deviation above the mean for the sector for the past 3 years and the observed?
- Most importantly, it’s not an actual measure of customer churn. The report states that it measured not actual customer loss, but the results of a survey that asked for:
The estimated number of customers who will most likely terminate their relationship as a result of the breach incident. The incremental loss is abnormal turnover attributable to the breach incident. This number is an annual percentage, which is based on estimates provided by management during the benchmark interview process. [Emphasis added.]
The report has other issues, and I encourage readers to examine its claims and evidence closely. I encourage this in general, it’s not a comment unique to the Ponemon report. Some examples from a number of additional surveys, that George Hulme raised in argment in this blog post:
Briefly, the CMO council found concern about security, not any knowledge of breaches. Forrester showed that some folks are scared to shop online, which means brand doesn’t matter, or they’d shop online from trusted brands. Javelin reports 40% of consumers reporting that their relationship “changed,” and 30% reporting a choice to not purchase from the organization again. Which is at odds with even the most ‘consumer-concerned’ estimates from Ponemon, and is aligned with the idea that surveys are hard to do well.
Great post. Needed to be said. To support your comments I recently gave a presentation at a conference where I asked how many people had received “your information may have been compromised” notices. All but a couple of hands went up. When I asked how many people had actually terminated or materially changed their relationship with the organization involved, no hands went up.
There’s a huge difference between “considering changing” and actually changing a business relationship. It’s a pain in the butt to change banks, insurance, and healthcare providers. Less so for retail, but then memories are short when you throw in a sale or two.
Interesting discussion. The real reason that data breaches don’t impact a company’s brand is that the companies have been successful in downplaying the impact of a data breach to the point where people have become clueless.
How many times after a the loss of an unencrypted laptop full of PII/PHI have you heard from the company… “there is no evidence that the data has been misused in any way…”
Really? How could the company possibly know that! The sad part is that people are being duped into believing it. A laptop hard drive can easily be removed, cloned, and put back leaving no trace/evidence (and the criminal has all the data). Social Security Numbers don’t expire, so a thief could put those SSNs and associated data into a database and pull it out 5 years from now and in most cases it will be just as useful as it is today.
If people who receive that “your information may have been compromised” notice really understood the type of data which was lost and the ramifications associated with the lifecycle of that data (e.g. if a SSN is ever compromised, the only real option is to expire the lost SSN and get a new one issued, and all the hassle associated with that), I think you would see more people getting worked up about data breaches.
The problem even extends to our court system. People who have been the victim of a data breach who attempt to be made whole go nowhere because they need to somehow prove “harm” (which is really direct financial harm – loss of money). So unless the data breach resulted in an immediate cleaning out of your bank account, the data breach is pretty much considered a non-event. There is no accounting for the financial cost of having to expire all the data which was breached (i.e. SSN), since the company has used their magic crystal ball to determine that “there is no evidence that the data has been misused”.
When the clueless are told over and over that “nothing bad will happen as the result of a data breach” they will, over time, start to believe it!!
There you go again, unmasking institutions. Keep up the good work.
As an aside, Ponemon seems to subscribe to all of the ethics codes and best practices methodologies, etc. for those in their business. See their Website.
Either those ethics and best practices aren’t being enforced, or aren’t worth much.
Their dollars-per-record may be accurate for a small breach but it’s wildy inaccurate for large ones and business leaders know that. As with anything, it’s “cheaper by the dozen.” My children applied at Ohio State University over the past few years, a veritable bastion of data breaches. They just got the letter saying they were among the 760,000 records breached. They didn’t even go beyond the initial application. And OSU spent four million to investigate it. That’s less than $6 per record. Even if the final cost is ten times that amount it’s nowhere close to Ponemon’s $200+ number. If people stop applying to OSU en masse, then it affects them but that will never happen because the value of the education they deliver far exceeds the personal cost of a data breach and there’s a waiting line to get in the door anyway.
When I talk to the business units I use $50 per record as a possible exposure figure to try and keep some credibility. Customers leave all the time and new ones arrive all the time. Unless the cost of a breach puts you out of business, it’s just another cost.
JJ,
The large release effect does have its moments of reduced cost. For example, alerted people take steps to protect their credit, so the damage per stolen identity is actually worse in small releases where notification may be light, missing or skipped.
Still, how much cheaper by the dozen? Some of the hard costs near $60 a record are simply unavoidable. But some of the soft costs such as disaffected customers does not include adverse viral business losses. An deeply disaffected customer adversely influences the buying decision of 8 other potential customers on the average.
If even one customer made a You Tube Video like, “United Hates Guitars”, that went virual and cost United Airlines millions. True, the number of truly offended talented musicians ready to make YouTube protests is low but truly not zero.
Is it really cheaper by the dozen to offend a larger customer pool? I think not. Even if good sense arguments do not cut it, I am fairly sure that a Monte Carlo Business Case study would not bear out your view, numerically speaking.
@JJ
Per their SEC Filings (Forms 10-k and 10-q), Heartland Payment Systems has spent over $150M on the breach (actual out of pocket and reserves set aside to cover future settlements.)
They aren’t out of business, but $150M is approximately 10% of annual revenue.
Do you think they consider that “just another cost”?
BTW – if you take $150M for the cost and 130M for the number of records, as the DOJ alledged in the Gonzales indictment, that works out to about $1.20 per record.
Is the moral here to expose lots of records?
Of course not 🙂
Point is – cost of breach is a very complicated subject – cost per record may not be the best measure, as several have said.
Patrick
The Ponemon survey report has an entire page of carefully worded caveats compared to one sentence in the Verizon 2010 Breaches tabulation Report that has one sentence saying it is biased: “Although we believe many of the findings presented in this report to be appropriate for generalization (and our confidence in this grows over time), bias undoubtedly exists.”
As far as I know, there are no satistically valid surveys or tabulations in information security sufficiently representative for use in risk analysis for justification and priority-setting of security solutions. However, these two reports and several others such as the CSI/FBI surveys are very useful to identify new and overlooked vulnerabilites, threats, potential perpetrators, assets subject to violation, and security solutions when conducting a(non-risk-based)threat and vulnerability analysis.