Science isn't about Checklists
Over at Zero in a Bit, Chris Eng has a post, “Art vs. Science“:
A client chastised me once for making a statement that penetration testing is a mixture of art and science. He wanted to believe that it was completely scientific and could be distilled down to a checklist type approach. I explained that while much of it can be done methodically, there is a certain amount of skill and intuition that only comes from practical experience. You learn to recognize that “gut feel” when something is amiss. He became rather incensed and, in effect, told me I was full of it. This customer went on to institute a rigid, mechanical internal process for web app pen testing that was highly inefficient and, ultimately, still relied mostly on a couple bright people on the team who were in tune with both the art and the science.
Certifications only test the science.
I want to disagree strongly. Science isn’t about checklists. It’s about forming and testing hypothesis. In the case of pen tests, you have an overarching hypothesis, “this thing is secure.” You conduct experiments which demonstrate that hypothesis to be false. (Lather, rinse, repeat, you can’t test security in.)
The design of good experiments is an art. Some people are better at it than others. Great science is driven by a small number of great scientists who have both a comprehension that something is wrong with today’s theories, and a flair for great experiments which illuminate those issues.
The problem isn’t science versus art, the problem is checklist and bureaucracy versus skilled professional.