Shostack + Friends Blog Archive

 

National Cyber Leap Year: Without a Good Running Start, There Might Be No Leap

— Hi, Alex here, today I want to welcome guest blogger Russell Thomas.  Those on the metrics mailing list are already pretty familiar with Russell, and we’re delighted to have him post with us.  For those who don’t know Russell, an independent consultant specializing in modeling the business value and risks of information technology.  Even though he’s got an EE degree, he’s more of a business guy than a technologist, and certainly not an InfoSec technologist.  For the last four years, he’s been focused on research to advance the state of the art in the economics of InfoSec. Russell lives in the Bay Area.–

There might be more US government research funding for security metrics in coming years.   This is hugely important because there are major unsolved research problems in security metrics and incentives.  This has been known for years.  It’s also been well known that  funding for research in this area (both public and private) has been chronically low.

But this depends on whether the US Feds and Congress are be persuaded by the report from the NCLY Summit held recently. The Summit report will be published in few weeks.  I hope they succeed this time, but I have doubts.  Either way, the NCLY is not a good model for public-private collaboration going forward.

In case you haven’t heard, 2008-9 was “National Cyber Leap Year” (NCLY) in the US.  It has been sponsored by the White House Office of Science and Technology Policy (OSTP) and the Network and Information Technology R&D council (NITRD).  The main and final event was the invitation-only National Cyber Leap Year Summit, held Aug. 17-20.  The Summit reports are now being written and will be made public in few weeks.  (My focus is on “Cyber Economics” track, one of five tracks, because it was most relevant to security metrics and associated incentives.)

I wasn’t at the Summit, but I was involved both before and after, and my doubts arise because of the preparation and collaboration process leading up to it (or lack thereof).  People who were at the Summit have blogged about what happened there:

In a nutshell, here’s why I’m critical about the NCLY process:

  1. Most of the year was wasted on redundant RFIs,
  2. The Summit had inadequate preparation and pre-work
  3. The Summit itself was a glorified brainstorming process, which isn’t adequate to deal with problems and ideas of this depth and complexity
  4. The organizers missed the opportunity to promote true collaboration through out the year.
  5. There is no follow-on process other than to submit the recommendations to the sponsors.

(I also have concerns about who participated and who didn’t, based on the blog posts above.  But since the participant list isn’t public and I wasn’t there, I won’t comment further.)

Maybe NCLY will lead to good things in spite of the shortcomings in the process.  I hope so.  But I’m blogging about this with hopes that the next public-private collaboration will be done much better.  Keep reading if you want to know the details…

The Process

This whole NCLY idea arose because Congress was bashing the federal agencies for not investing enough in R&D for radical or “game-changing” cyber security solutions.  It fell into the lap of OSTP and NITRD to do something about it.  Someone came up with the idea of soliciting “game-changing” ideas from the public. The basic thesis seemed to be that there were Silver Bullet ideas already be out there in industry and academia, and all the Feds needed to do was find them and then fast-track them.  Based on the timeline made public, the organizers seemed to think they could do all this in a year and show meaningful progress.

The first step was to sign up a contractor to manage the whole process – QinetiQ North America.   Their business is government contracting and outsourcing, including cyber security services.  As far as I can tell, the contractor ran the whole NCLY process, with perhaps communication and guidance from the NITRD steering committee.  I’ve seen no evidence that QinteliQ NA, or their subcontractor, Martin Ross, has ever run this sort of breakthrough collaboration process before.

RFI…Rinse and Repeat …Rinse and Repeat Again

Next step was a public Request for Input (RFI), in Oct. 2008, with Dec. 15, 2008 deadline.  [Cool!]  The RFI was framed in a cutesy way by extending the “game changing” metaphor.  [Hmmm.  Maybe too clever by half, but we’ll see how it works.]  They said they would make all the submissions public on their web site to enable collaboration, synergy, etc.  They are were posted here.    [Excellent]  However, the format was limited to two pages.  [This seems to limit submissions to either be narrow/focused with some detail, or broad but superficial.  Is that adequate to the task?]

But then something weird happened.  Instead of moving forward on their original plan, they issued a second RFI, identical to the first, except they made provision for handing confidential/proprietary submissions.  [Why didn’t they do that in the first place?  Seems like they wasted 3 months.]  The new deadline was Feb 20, 2009.

Then, more weirdness.  A third RFI was issued, identical to the first two, except that they identified specific categories or themes that they were seeking, based on submissions to the first two RFIs.  [Why didn’t they organize a workshop at this point, or at least an on-line discussion among submitters to hash this out?  So far, all the communication has been one-way: from the public to the organizers.]  New deadline was April 15, 2009.

The good news to me was that one of the five categories was “Cyber Economics”, defined as: “Security decision-making frameworks that  incorporate economic insights; understanding and altering economic value-chains to make cyber security exploits increasingly expensive for attackers”.  [Disclosure: I made two RFI submissions in this category].  I was worried that NCLY might be focused only on technology solutions, so I was heartened by the inclusion of economics.

Then a month passed with no feedback to submitters or to the public or on their web site.  I called in late May to see if NCLY was still on, and the organizers called back to say, “Yes”, and that they were planning a summer workshop to be announced shortly.

The month of June passed with no announcement and no feedback to submitters.  What was going on?  Who knows.

The Summit is Born as Brainstorming Workshop

Finally, in early July, they announced on the NITRD web site that there would be an invitation-only workshop (“Summit”) at the end of August, with tracks for each of the five categories.  Rather than prusue any specific ideas submitted to the RFIs, the organizers decided to have a brainstorming workshop on the five themes.

With this announcment, many red flags appeared:

  1. They had not sent any “Save the date” communication to potential invitees.  Normally, this is done four to six months in advance for major conferences and workshops.  Plus it was the middle of summer vacation season.  Would the right/best people even be available to attend?  Don’t bet on it.
  2. They had not yet recruited program co-chairs for each track.  Normally, this is done months in advance.
  3. There was no clear pre-workshop preparation process.  Where people just going to show up with no pre-work and no pre-introductions?  That’s not a formula for success when bringing together diverse people to chew on hard problems and out-of-the-box solutions.
  4. The agenda seemed to focus on brainstorming (“Six Hats” method), led by facilitators who had no domain knowledge.  This would probably lead to superficial discussions, I thought.
  5. The “Cyber Economics” track had been truncated.  In the description, it only focused on disrupting attacker economics.  Why this truncation?  Who decided?
  6. The program co-chairs were only selected in mid- to late July.  They had no say in the Summit agenda, process, scope, and they were expected only to take notes and write up the final report, not to participate in the Summit discussions.  [This is very different from all the conferences and workshops I’ve been to, where the co-chairs largely shape their session, and moderate/facilitate the process.]

Participant invitations went out in late July, less than one month before the Summit.  Or should I say “Participant Applications” were sent out.  Yes, the organizers sent emails to people where they asked them to write an essay as to why they should be invited to this event — with a 24-hour deadline (later extended).   After another week and a half, the real invitations were finally sent.  The program co-chairs were not involved in the invitation process and had no say in who was invited or not.  [As should be obvious, this is not the way you get the best and brightest to participate.]  On the plus side, the government paid for travel, lodging, and per diem for participants.

Summit Happened

The good news is that the Cyber Economics track was not limited to disrupting attacker economic models.  The bad news is that the agenda and process did exclude other important discussions (see Gene Spafford’s blog post).

The draft discussion notes on Cyber Economics are here .  The co-chair’s report will be much more compact and synthesized.

Nearly everything they came up with has been discussed and presented before in other visioning workshops or blue-ribbon commissions, for example:

The only exception might be “Disrupting attacker’s profitability”, but then no game-changing ideas emerged from that discussion.

The Report and Next Steps

Who is their target audience for the report? It’s the OSTP and the NITRD, so they can incorporate the recommendations in the FY2010 federal budget.  Thus, it may not be very compelling or useful to the broader audience, and certainly no advance on the reports listed above.

This target audience explains why there are no follow-on activities or workshops involving the participants or others.  That’s why I call the Summit a “glorified brainstorming workshop”.  For these discussions to have real value, all the bright people, their organizations, and the government sponsors are going to have to spend more time figuring out how to make them happen.  That’s hard work.  Without any specific plan or follow-on events scheduled, I’m doubtful.

But Why?

Why did NCLY turn out this way?  I can only guess.  Following Bruce Schneier’s colorful phrase, was NCLY really just “Innovation Theater”?  Or was this just poor execution on the part of the organizers and/or poor supervision by their government bosses?  Or maybe some combination?

Either way, I assert that the NCLY process is not a good model for public-private collaboration to drive breakthrough cyber security innovation.  Next time, do it right.

I’m also doubtful whether the report will be compelling enough to motivate additional funding for security metrics research or any big changes in how research is organized.  When the report is published, I will blog with further reactions.

Originally published by Russell on 11 Sep 2009
Last modified on 11 Sep 2009
Categories:   Uncategorized  

One comment on "National Cyber Leap Year: Without a Good Running Start, There Might Be No Leap"

Comments are closed.