Emergent Design Issues

It seems like these days, we want to talk about everything in security as if it’s a vulnerability. For example:

German researchers have discovered security flaws that could let hackers, spies and criminals listen to private phone calls and intercept text messages on a potentially massive scale – even when cellular networks are using the most advanced encryption now available.

Experts say it’s increasingly clear that SS7, first designed in the 1980s, is riddled with serious vulnerabilities that undermine the privacy of the world’s billions of cellular customers. The flaws discovered by the German researchers are actually functions built into SS7 for other purposes – such as keeping calls connected as users speed down highways, switching from cell tower to cell tower – that hackers can repurpose for surveillance because of the lax security on the network. (“German researchers discover a flaw that could let anyone listen to your cell calls.” Washington Post, 2014).

But these are not vulnerabilities, because we can have endless debate about it they should be fixed. (Chrome exposing passwords is another example.) If they’re not vulnerabilities, what are they? Perhaps they’re flaws? One definition of flaws reads:

“Flaws are often much more subtle than simply an off-by-one error in an array reference or use of an incorrect system call,” the report notes. “A flaw might be instantiated in software code, but it is the result of a mistake or oversight at the design level.”

An example of such a flaw noted in the report is the failure to separate data and control instructions and the co-mingling of them in a string – a situation that can lead to injection vulnerabilities. (IEEE Report Reveals Top 10 Software Security Design Flaws)

In this sense, the SS7 issues are probably not “flaws” in the sense that the system behavior is unanticipated. But we don’t know. We don’t know what properties we should expect SS7 to have. For most software, the design requirements, the threat model, is not clear or explicit. Even when it’s explicit, it’s often not public. (Larry Loeb makes the same point here.)

For example, someone decided to write code to run a program on mouse over in Powerpoint, that code was tested, dialog text was written and internationalized, and so on. Someone documented it, and it’s worth pointing out that the documentation doesn’t apply to Powerpoint 2016. Was there a debate over the security of that feature when it shipped? I don’t know. When it was removed? Probably.

There’s a set of these, and I’m going to focus on how they manifest in Windows for reasons that I’ll get to. Examples include:

The reason I’m looking at these is because design questions like these emerge when a system is successful. Whatever else you want to say about it, Windows was successful and very widely deployed. As a system becomes more successful, the easily exploitable bugs are fixed, and the hard to fix design tradeoffs become relatively more important. As I wrote in “The Evolution of Secure Things:”

It’s about the constant imperfection of products, and how engineering is a response to perceived imperfections. It’s about the chaotic real world from which progress emerges. In a sense, products are never perfected, but express tradeoffs between many pressures, like manufacturing techniques, available materials, and fashion in both superficial and deep ways.

That chaotic real world exposes a set of issues that may or may not have been visible during product design. In threat modeling, identification of issues is the most crucial step. If you fail to identify issues, you will not manage those issues well. Another way to say that is: identifying issues is a necessary but not sufficient step.

The design choices listed above almost all predate threat modeling as a structured practice at Microsoft. But there have been other choices, like Windows Wifi sense or new telemetry in Windows 10. We can disagree with those design choices, but it’s clear that there were internal discussion of the right business tradeoffs. So we go back to the definition of a flaw, “a mistake or oversight at the design level.” These were not oversights. Were they design mistakes? That’s harder. The designers knew exactly what they were designing, and the software worked as planned. It was not received as planned, and it is certainly being used in unexpected ways.

There are interesting issues of composition, especially in backup authentication. That problem is being exploited in crypto currency thefts:

Mr. Perklin and other people who have investigated recent hacks said the assailants generally succeeded by delivering sob stories about an emergency that required the phone number to be moved to a new device — and by trying multiple times until a gullible agent was found.

“These guys will sit and call 600 times before they get through and get an agent on the line that’s an idiot,” Mr. Weeks said.

Coinbase, one of the most widely used Bitcoin wallets, has encouraged customers to disconnect their mobile phones from their Coinbase accounts.

One can imagine a lot of defenses, but “encouraging” customers to not use a feature may not be enough. As online wallet companies grow, they need to have threat modeled better, and perhaps that entails turning off the feature. (I don’t know their businesses well enough to simply assert an answer.)

In summary, we’re doing a great job at finding and squishing bugs, and that’s opening up new and exciting opportunities to think more deeply about design issues.

PowerPoint Screen capture via Casey Smith.

[Update Dec 13: After a conversation with Gary McGraw, I think I may have read the CSD definition of flaw too narrowly.]

20 Year Software: Engineering and Updates

Twenty years ago, Windows 95 was the most common operating system. Yahoo and Altavista were our gateways to the internet. Steve Jobs just returned to Apple. Google didn’t exist yet. America Online had just launched their Instant Messenger. IPv6 was coming soon. That’s part of the state of software in 1997, twenty years ago. We need to figure out what engineering software looks like for a twenty year lifespan, and part of that will be really doing such engineering, because theory will only take us to the limits of our imaginations.

Today, companies are selling devices that will last twenty years in the home, such as refrigerators and speakers, and making them with network connectivity. That network connectivity is both full internet connectivity, that is, Internet Protocol stacks, and also local network connectivity, such as Bluetooth and Zigbee.

We have very little idea how to make software that can survive as long as AOL IM did. (It’s going away in December, if you missed the story.)

Recently, there was a news cycle around Sonos updating their privacy policy. I don’t want to pick on Sonos, because I think what they’re experiencing is just a bit of the future, unevenly distributed. Responses such as this were common: “Hardware maker: Give up your privacy and let us record what you say in your home, or we’ll destroy your property:”

“The customer can choose to acknowledge the policy, or can accept that over time their product may cease to function,” the Sonos spokesperson said, specifically.

Or, as the Consumerist, part of Consumer Reports, puts it in “Sonos Holds Software Updates Hostage If You Don’t Sign New Privacy Agreement:

Sonos hasn’t specified what functionality might no longer work in the future if customers don’t accept the new privacy policy.

There are some real challenges here, both technical and economic. Twenty years ago, we didn’t understand double-free or format string vulnerabilities. Twenty years of software updates aren’t going to be cheap. (I wrote about the economics in “Maintaining & Updating Software.”)

My read is that Sonos tried to write a privacy policy that balanced their anticipated changes, including Alexa support, with a bit of future-proofing. I think that the reason they haven’t specified what might not work is because they really don’t know, because nobody knows.

The image at the top is the sole notification that I’ve gotten that Office 2011 is no longer getting security updates. (Sadly, it’s only shown once per computer, or perhaps once per user of the computer.) Microsoft, like all tech companies, will cut functionality that it can’t support, like <"a href="https://www.macworld.com/article/1154785/business/welcomebackvisualbasic.html">Visual Basic for Mac and also “end of lifes” its products. They do so on a published timeline, but it seems wrong to apply that to a refrigerator, end of lifeing your food supply.

There’s probably a clash coming between what’s allowable and what’s economically feasible. If your contract says you can update your device at any time, it still may be beyond “the corners of the contract” to shut it off entirely. Beyond economically challenging, it may not even be technically feasible to update the system. Perhaps the chip is too small, or its power budget too meager, to always connect over TLS4.2, needed addresses the SILLYLOGO attack.

What we need might include:

  • A Dead Software Foundation, dedicated to maintaining the software which underlies IoT devices for twenty years. This is not only the Linux kernel, but things like tinybox and openssl. Such a foundation could be funded by organizations shipping IoT devices, or even by governments, concerned about the externalities, what Sean Smith called “the Cyber Love Canal” in The Internet of Risky Things. The Love Canal analogy is apt; in theory, the government cleans up after the firms that polluted are gone. (The practice is far more complex.)
  • Model architectures that show how to engineer devices, such as an internet speaker, so that it can effectively be taken offline when the time comes. (There’s work in the mobile app space on making apps work offline, which is related, but carries the expectation that the app will eventually reconnect.)
  • Conceptualization of the legal limits of what you can sign away in the fine print. (This may be everything; between severability and arbitration clauses, the courts have let contract law tilt very far towards the contract authors, but Congress did step in to write the Consumer Review Fairness Act.) The FTC has commented on issues of device longevity, but not (afaik) on limits of contracts.

What else do we need to build software that survives for twenty years?

Threat Modeling “App Democracy”

Direct Republican Democracy?” is a fascinating post at Prawfsblog, a collective of law professors. In it, Michael T. Morley describes a candidate for Boulder City Council with a plan to vote “the way voters tell him,” and discusses how that might not be really representative of what people want, and how it differs from (small-r) republican government. Worth a few moments of your time.

Building an Application Security Team

The Application Security Engineer role is in demand nowadays. Many job offers are available, but actual candidates are scarce. Why is that? It’s not an easy task as the level of skills needed is both to be broad and specialized at the same time. Most of the offers are about one person, one unicorn that does all those wonderful things to ensure that the organization is making secure software.


Looking at where we are coming from

For the sake of simplicity, let’s say that the application security industry is traditionally split between two main types of offering: those who are selling products and those who are selling pen test services. In many occasions both are from the same vendor, but from different teams internally. Let’s break down how they are judged on their success to have an idea how it managed to evolve.

Continue reading “Building an Application Security Team”

Worthwhile books, Q3

Some of what I’ve read over the past quarter, and want to recommend each of the books below as worthy of your time.


  • The Internet of Risky Things, Sean Smith. This was a surprisingly good short read. What I gained was an organized way of thinking and a nice reference for thinking through the issues of IOT. Also, the lovely phrase “cyber Love Canal.”
  • American Spies, Jennifer Stisa Granick. Again, surprisingly good, laying out with the logical force that really good lawyers bring, explaining both sides of an issue and then explaining the frame in which you should understand it.
  • Saving Bletchley Park, Sue Black. (Title links to publisher, who sells ebook & print, or you can go to Amazon, who only sells the hardback.) The really interesting story of the activism campaign to save Bletchley Park, which was falling apart 20 years ago. Dr. Black is explicit that she wrote the book to carry the feel of an internet campaign, with some stylistic bits that I found surprising. I was expecting a drier style. Don’t make my mistake, and do read the book. Also, visit Bletchley Park: it’s a great museum.

Nonfiction, not security


  • N. K. Jemisin’s Broken Earth Series. Outstanding writing, interesting worldbuilding, and the first two books have both won Hugos. First book is “The Fifth Season.” Bump it up in your queue.
  • The Rise and Fall of D.O.D.O, Neal Stephenson and Nicole Galland. I’m not (yet) familiar with Galland’s work, much of which seems to be historical fiction. This fairly breezy and fun time travel read, much less dense than most of Stephenson’s recent books.

Previously: Q2.

Emergent Musical Chaos

Global jukebox framed
The New York Times reports on how many of Alan Lomax’s recordings are now online, “The Unfinished Work of Alan Lomax’s Global Jukebox.” This is a very interesting and important archive of musical and cultural heritage. The Global Jukebox. I was going to say that Lomax and Harry Smith were parallel, and that the Anthology of American Folk Music is a similar project, but I was wrong, Smith drew heavily on Lomax’s work.

Simultaneously, the Internet Archive has been working to put 78 RPM records online, and has a feed.