Common Criteria
Statistics gleaned from the labs’ Common Criteria work indicates that the testing is improving security, said Jean Schaffer, director of NIAP. Schaffer spoke during a session at a Federal Information Assurance Conference held this week at the University of Maryland.
So far, 100 percent of the products evaluated have been approved, she said. The testing directly improved 30 percent of the products tested by eliminating security flaws that could have been exploited by attackers. About 40 percent of the products evaluated were improved by the addition or extension of security features, Schaffer said.
Critics say Common Criteria testing costs too much and takes too long, but Schaffer argued that these claims are made by those who do not have firsthand knowledge about the testing. Feedback from the labs shows that testing for Evaluation Assurance Level (EAL) 2 — the minimum level of security, which includes products such as firewalls, intrusion-detection systems, routers and switches — costs $100,000 to $170,000 and takes four to six months. The highest level of security — EAL 4, which includes operating systems that support peer-to-peer communications — costs $300,000 to $750,000 and takes one year to two years.
So let me get this straight:
- Spend $100,000 and have a 30% chance of finding a flaw, and a 40% chance of adding features?
Have I got a deal for you! Give me $50,000, I’ll run gcc -W on your code, and give you a trademarked “Adam seal of approval.” No issues? No charge. - Spend half a million dollars to get to EAL 4?
As I said before, too easy, and too expensive. - No product has failed evaluation.
No product fails evaluation because you get to keep coming back for more, secretly. That you had to do so is very useful information. But only the final certification report published; what you did to get there is not. That’s hard to change. I’d sure want a dry run before I got a real report.
But let’s look at this from the sellers perspective. If you’re developing product, what do you get for your $100,000? You could get 3-6 months of high quality security review by an expert. Or you could send a signal that you’re interested in the government market. A signal, you’ll recall, is something that’s hard to fake, and communicates useful information in the situation where the seller knows more than the buyer reasonably can. A signal should be easier for an honest player to send than one who is misrepresenting themselves. The other thing that the CC signals is that you have large piles of documentation. Unfortunately, it doesn’t really imply that your product is any more secure.
(See the original story at Federal Computer Week.)
Not only that. As the one submitting for CC certification, you define define your security target and then you show how you meet it. So it really does devolve to how much money you are willing to spend. Furthermore, there are no limits on how you advertise your CC cert once you have it. Didn’t one smart card manufacturer (rainbow?) got a very small piece of their product certified as EAL7 and then was advertising the entire product as such? At least http://www.commoncriteriaportal.org is finally publishing the security targets so you can at least do a real evaluation.