Models are Distracting

claire-cropped.jpgSo Dave Mortman wrote:

I don’t disagree with Adam that we need raw data. He’s absolutely right that without it, you can’t test models. What I was trying to get at was that, even though I would absolutely love to have access to more raw data to test my own theories, it just isn’t realistic to expect that sort of access in the legal and business environment we have today. So until things change, we have to figure out another way to get at the data.

First off, I don’t disagree with why Dave is going where he’s going, but I think it’s built on a mis-read of where we are, and a strategic error regardless.

Where we are: we do have raw data. It’s coming to us from unexpected sources, and we’re getting more of it day by day. We’d like more details, we’d like more consistency and we’d like more depth, and each of those will come.

But far more important is the strategic error of asking for something that isn’t the fullness of what we want, and the risk that the cover-up club will use it to avoid the real goal by talking about how much progress we’ve made sharing models.

You almost never get anything you don’t ask for. If we have a list of requests, the top of the list is data, data and data.

Further, I declare that this is a realistic request, and attach precisely the level of proof that the good Mr. Mortman did when asserting that “it just isn’t realistic.”

Not that I’m opposed to model sharing. We just need to recognize it for the poor substitute that it is, and keep our eyes on the real goal.

Speaking of where your eyes are, that’s Claire, she’s represented by Specs Model Management. And as the title says, quite distracting.

Security is About Outcomes, FISMA edition

Over at the US Government IT Dashboard blog, Vivek Kundra (Federal CIO), Robert Carey (Navy CIO) and Vance Hitch (DOJ CIO) write:

the evolving challenges we now face, Federal Information Security Management Act (FISMA) metrics need to be rationalized to focus on outcomes over compliance. Doing so will enable new and actionable insight into agencies’ information and network security postures, possible vulnerabilities and the ability to better protect our federal systems.
(“Moving Beyond Compliance: The Status Quo Is No Longer Acceptable”)

I’m tremendously excited to see this because back in April I wrote “Security is about outcomes, not about process.” I don’t know that I can claim credit for this, but it’s nice to see how far the meme has gone.

Meta-Data?

So awhile back, I posted the following to twitter:

Thought of the Day: We don’t need to share raw data if we can share meta-data generated using uniform analytical methodologies.

Adam, disagreed:

@mortman You can’t test & refine models without raw data, & you can’t ask people with the same orientation to bring diverse perspectives.

We went back and forth a bit until it became clear that this needed an actual blog post, so here it is:

I don’t disagree with Adam that we need raw data. He’s absolutely right that without it, you can’t test models. What I was trying to get at was that, even though I would absolutely love to have access to more raw data to test my own theories, it just isn’t realistic to expect that sort of access in the legal and business environment we have today. So until things change, we have to figure out another way to get at the data.

One thing that has become increasingly popular is for vendors to publish aggregate data about what they’ve seen with their customers or on their networks. Verizon and WhiteHat have used this model to great effect. Not only has it generated a lot of press for them, but we as an industry have learned a lot from these reports.

What would be even better is if people would share the models they are using when generating their data. This way, other organizations could use the models and as reports were published, the rest of us could actually compare apples to apples. This would also allow us to more quickly identify issues/errors in the models, allow for public discussion of necessary tweaks and then test said changes while limiting liability for the data owners.

This is really where I was going with my initial thought above; that we need common models so we can have an intelligent discussion. This is also how things generally work in the sciences (yes, Alex, I know, we’re not a science yet :). Researchers almost never publish their raw data, but just their models, methods and results. I feel strongly that until we can convince people to share raw data more openly, this is our best shot to figuring real information about what’s going on in the security world. It’s also what drove me to start developing the soon to be renamed Mortman/Hutton Model that Alex and I presented at Blackhat and BSides Las Vegas.

More data, even if it’s aggregate, is better then no data.

Gates Was Hardly An Exception

There was a lot of news when Henry Lewis Gates was arrested back in July, essentially for mouthing off to a cop. What happened was a shame, but what is more of a shame is that this sort of thing isn’t that rate. Time magazine had a recent article about this, Do You Have the Right to Flip Off a Cop? which you should read. One of my best friends from High School, Jeff Miller, linked to this article from his own blog and summed up the issue as only he can:

You can be rude to Taylor Swift, you can be rude to a tennis line judge, you can even be rude to the President … none of these things will get you arrested. But if you’re rude to a cop, get ready for some handcuffs.
This is a problem, no?

You said it Jeff!

Happy Banned Books Week!

banned-books.jpgQuoting Michael Zimmer:

[Yesterday was] the start of Banned Books Week 2009, the 28th annual celebration of the freedom to choose what we read, as well as the freedom to select from a full array of possibilities.

Hundreds of books are challenged in schools and libraries in the United States each year. Here’s a great map of challenges from 2007-2009, although I’m sure it under-represents the nature of the problem, as most challenges are never reported. (Note the West Bend library controversy is marked on the map.)

According to the American Library Association, there were 513 challenges reported to the Office of Intellectual Freedom in 2008.

I’m somewhat surprised by how many bluenoses dots there are in the northeast. Does anyone know of a good tutorial that would help me to re-map the data against population?

Visualization Friday – Beautiful, Functional, and Effective

We can all learn from this great role model, aimed at personal nutrition awareness and education: Nutritiondata.com. If only security awareness web sites were this good.

We can all learn from this great role model, aimed at personal nutrition awareness and education: Nutritiondata.com .

I encourage you to click on the images below to visit the site and explore interactive features. 

nutritiondata-dot-com1
nutritiondata-dot-com2

 

 

 

 

 

 

 

 

 

 

 

 

 

If only security awareness web sites aimed at end-users and consumers were this good.

A Little Temporary Safety

allstate-new-jersey-ad.jpg
So I saw this ad on the back of the Economist. (Click for a larger PDF). In reading it, I noticed this exhortation to “support the STANDUP act of 2009:”

The STANDUP Act* (H.R. 1895) creates a National
Graduated Driver Licensing (GDL) law that [limits nighttime driving, reduces in-car distractions, puts a cap on the number of friends in the car and increases the required hours of training and supervision. ] congressional representatives When states have implemented comprehensive GDL programs, the number of fatal crashes among 16 year old drivers has fallen by almost 40%.”

Now I was curious as to how many lives that was, and so I went looking. I found a lot of interesting stuff. For example, “Beginning with Florida in 1996, graduated licensing systems also have been adopted in most U.S. states.” That’s from the “Insurance Institute for Highway Safety/Highway Loss Data Institute.” But they also tell us: “A national evaluation reported that states with 3-stage graduated systems had 11 percent fewer fatal crashes per population of 16 year-olds during 1994-2004 than states without such systems.” Last I checked, 11 is not almost 40.

It also turns out that the number of teens killed in New Jersey last year was 60. Now, I don’t want to minimize the pain for the families who lost their children, or those injured by teens driving like, well, teens. But based on Allstate’s high number, these laws about graduated driving privileges may save as many as 25 lives a year. Based on the IIHS assessment, it may be 6 or 7.

Now there’s an old saw “Where are you from? New Jersey. Oh, what exit?” The truth is that life in New Jersey is car-centric, and saving those lives involves restricting the behavior of about 110,000 teens. (Or so I estimate, based on New Jersey Quickfacts from the US Census, who say that there are 8.6MM people, and roughly 24% are under 18, and so I figure that roughly 1.3% of the population is 16.) Those teens are in the process of exploring who they are, and asserting their independence from their parents and geography. They’re in the process of growing up. Part of that growing up is taking risks, and I suspect that some of the risk taking is simply delayed, not removed.

The other thing I don’t get about Allstate’s ad is that the insurance industry says “most states” already have such laws. Setting a national law is hard, and Congress is busy investigating baseball players. So clearly, they have important tasks to be working on. What’s more, phrases like “A national evaluation reported that states with 3-stage graduated systems had 11 percent fewer fatal crashes … than states without such systems.” A stronger argument for continued experimentation by laboratories of democracy is hard to imagine.

But stepping back, the real issue I have here is the desire to drive one particular danger to zero without consideration of the costs or alternatives. These folks are dedicated to stopping deaths in cars (which is appropriate for the IIHS, less so for Allstate). But what fraction of teen deaths are in cars that a teen is driving? What are the costs of a little temporary safety for teens?

[updates: corrected quote, added link to text]
[update2: Don’t miss Kenneth Finnegan’s comment about having 5 teens all drive separately from point A to point B, with attendant environmental and parking impact.]

VP's residence is still blurred on Google Earth (political influence on data and its long shadow)

Politics and power can manipulate the “ground truth data” we depend upon. Case in point: the VP residence image on Google Earth is still blurred, even though VP Dick Cheney has been out of office for almost a year. Could similar things happen in InfoSec data if it were more visible and public? You bet.

Amusement: Some of you may have heard that former VP Dick Cheney pulled some strings to get Google (or rather their third party supplier) to blur the image of his residence in DC  (One Observatory Circle), presumably for security reasons.   Cheney is out and Biden is in, so you’d think that the image would now be unblurred.  Not so.  Here’s the current image.   Compare it to the neighboring buildings across the street  and you can clearly see that the VP’s residence is still blurred.    What about the more important targets?  Both the White House (1600 Pennsylvania Ave.) and 10 Downing St. in London  are not blurred.  Maybe they didn’t have the same clout as Cheney. 

Lesson:  Politics and power can manipulate the data, and also leave a shadow.   Could this happen to information security data if it were more visible and public?  You bet.  I’m not being cynical, just realistic.  Reminds me of a team motto from a project long ago:  “Trust no one.  Believe nothing.”  In other words, don’t take any data on face value.  Always inquire about the interests of the parties who produce or publish the data.