Category: breach analysis

House Oversight Committee on Equifax

The House Oversight Committee has released a scathing report on Equifax.

Through the investigation, the Committee reviewed over 122,000 pages of documents, conducted transcribed interviews with three former Equifax employees directly involved with IT, and met with numerous current and former Equifax employees, in addition to Mandiant, the forensic firm hired to conduct an investigation of the breach.

I haven’t had time to review the report in detail, but I don’t think it answers questions I had reading the GAO report. Four of their give key findings are about what happened before the breach, but the fifth, “unprepared to support affected consumers,” goes to a point I’ve made consistently over nearly a dozen years: “
It’s Not The Crime, It’s The Coverup or the Chaos.”

GAO Report on Equifax

I have regularly asked why we don’t know more about the Equifax breach, including in comments in “That Was Close! Reward Reporting of Cybersecurity ‘Near Misses’.” These questions are not intended to attack Equifax. Rather, we can use their breach as a mirror to reflect, and ask questions about how defenses work, and learn things we can bring to our own systems.

Ian Melvin was kind enough to point out a GAO report, “Actions Taken by Equifax and Federal Agencies in Response to the 2017 Breach.” As you’d expect of a GAO report, it is level headed and provides a set of facts.


However, I still have lots of questions. Some very interesting details start on page 11:

Equifax officials added that, after gaining the ability to issue system-level commands on the online dispute portal that was originally compromised, the attackers issued queries to other databases to search for sensitive data. This search led to a data repository containing PII, as well as unencrypted usernames and passwords that could provide the attackers access to several other Equifax databases. According to Equifax’s interim Chief Security Officer, the attackers were able to leverage these credentials to expand their access beyond the 3 databases associated with the online dispute portal, to include an additional 48 unrelated
databases.

The use of encryption allowed the attackers to blend in their malicious actions with regular activity on the Equifax network and, thus, secretly maintain a presence on that network as they launched further attacks without being detected by Equifax’s scanning software. (Editor’s note: I’ve inverted the order of the paragraphs from the source.)

So my questions include:

  • How did the attackers get root?
  • Why wasn’t the root shell noticed? Would our organization notice an extra root sell in production?
  • How did they get access to the other 48 databases?
  • Why didn’t the pattern of connections raise a flag? “As before, Equifax
    officials stated that the attackers were able to disguise their presence by
    blending in with regular activity on the network.” I find this statement to be surprising, and it raises questions: Does the dispute resolution database normally connect to these other databases and run the queries which were run? How was that normal activity characterized and analyzed? Encryption provides content confidentiality, not meta-data confidentiality. Would we detect these extra connections?

Specifically, while Equifax had installed a device to inspect network traffic for evidence of malicious activity, a misconfiguration allowed encrypted traffic to pass through the network without being inspected. According to Equifax officials, the misconfiguration was due to an expired digital certificate. The certificate had expired about 10 months before the breach occurred, meaning that encrypted traffic was not being inspected throughout that period.

Would your organization notice if one of hundreds or dozens of IDSs shut up for a week, or if one ruleset stopped firing?

More published incident reports will help us get smarter, and provide better answers to the questions that CEOs and boards are asking: could this happen to us? With this report we an answer that better, but still not well.

It’s Not The Crime, It’s The Coverup or the Chaos

Well, Richard Smith has “resigned” from Equifax.

The CEO being fired is a rare outcome of a breach, and so I want to discuss what’s going on and put it into context, which includes the failures at DHS, and Deloitte breach. Also, I aim to follow the advice to praise specifically and criticize in general, and break that pattern here because we can learn so much from the specifics of the cases, and in so learning, do better.

Smith was not fired because of the breach. Breaches happen. Executives know this. Boards know this. The breach is outside of their control. Smith was fired because of the post-breach chaos. Systems that didn’t work. Tweeting links to a scam site for two weeks. PINS that were recoverable. Weeks of systems saying “you may have been a victim.” Headlines like “Why the Equifax Breach Stings So Bad” in the NYTimes. Smith was fired in part because of the post-breach chaos, which was something he was supposed to control.

But it wasn’t just the chaos. It was that Equifax displayed so much self-centeredness after the breach. They had the chutzpah to offer up their own product as a remedy. And that self-dealing comes from seeing itself as a victim. From failing to understand how the breach will be seen in the rest of the world. And that’s a very similar motive to the one that leads to coverups.

In The New School Andrew and I discussed how fear of firing was one reason that companies don’t disclose breaches. We also discussed how, once you agree that “security issues” are things which should remain secret or shared with a small group, you can spend all your energy on rules for information sharing, and have no energy left for actual information sharing.

And I think that’s the root cause of “U.S. Tells 21 States That Hackers Targeted Their Voting Systems” a full year after finding out:

The notification came roughly a year after officials with the United States Department of Homeland Security first said states were targeted by hacking efforts possibly connected to Russia.

A year.

A year.

A year after states were first targeted. A year in which “Obama personally warned Mark Zuckerberg to take the threats of fake news ‘seriously.’” (Of course, the two issues may not have been provably linkable at the time.) But. A year.

I do not know what the people responsible for getting that message to the states were doing during that time, but we have every reason to believe that it probably had to do with (and here, I am using not my sarcastic font, but my scornful one) “rules of engagement,” “traffic light protocols,” “sources and methods” and other things which are at odds with addressing the issue. (End scornful font.) I understand the need for these things. I understand protecting sources is a key role of an intelligence service which wants to recruit more sources. And I also believe that there’s a time to risk those things. Or we might end up with a President who has more harsh words for Australia than the Philippines. More time for Russia than Germany.

In part, we have such a President because we value secrecy over disclosure. We accept these delays and view them as reasonable. Of course, the election didn’t turn entirely on these issues, but on our electoral college system, which I discussed at some length, including ways to fix it.

All of which brings me to the Deloitte breach, “Deloitte hit by cyber-attack revealing clients’ secret emails.” Deloitte, along with the others who make up the big four audit firms, gets access to its clients deepest secrets, and so you might expect that the response to the breach would be similar levels of outrage. And I suspect a lot of partners are making a lot of hat-in-hand visits to boardrooms, and contritely trying to answer questions like “what the flock were you people doing?” and “why the flock weren’t we told?” I expect that there’s going to be some very small bonuses this year. But, unlike our relationship with Equifax, boards do not feel powerless in relation to their auditors. They can pick and swap. Boards do not feel that the system is opaque and unfair. (They sometimes feel that the rules are unfair, but that’s a different failing.) The extended reporting time will likely be attributed to the deep analysis that Deloitte did so it could bring facts to its customers, and that might even be reasonable. After all, a breach is tolerable; chaos afterwards may not be.

The two biggest predictors of public outrage are chaos and coverups. No, that’s not quite right. The biggest causes are chaos and coverups. (Those intersect poorly with data brokerages, but are not limited to them.) And both are avoidable.

So what should you do to avoid them? There’s important work in preparing for a breach, and in preventing one.

  • First, run tabletop response exercises to understand what you’d do in various breach scenarios. Then re-run those scenarios with the principals (CEO, General Counsel) so they can practice, too.
  • To reduce the odds of a breach, realize that you need continuous and integrated security as part of your operational cycles. Move from focusing on pen tests, red teams and bug bounties to a focus on threat modeling, so you can find problems systematically and early.

I’d love to hear what other steps you think organizations often miss out on.

Breach Vouchers & Equifax 2017 Breach Links

[Thursday, September 21th is the latest of 5 updates.]

When I wrote “The Breach Response Market Is Broken,” I didn’t expect one of the players to validate everything I had to say. What I said was that the very act of firms contracting with breach response services inhibit the creation of a market for breach response, and the FTC should require them to give vouchers to consumers.

Vice Motherboard is reporting that “Firm Hired to Monitor Data Breaches Is Hacked, 143 Million Social Security Numbers Stolen.”

It’s not clear what database was accessed. On their website, Equifax says “No Evidence of Unauthorized Access to Core Consumer or Commercial Credit Reporting Databases” and “Company to Offer Free Identity Theft Protection and Credit File Monitoring to All U.S. Consumers.”

But here’s the thing; I don’t trust Equifax to protect data that … they just failed to protect. I want protection from an independent firm.

Equifax’s self-dealing in providing breach response services is unfair. No rational, well-informed consumer would select Equifax’s service in this situation. Equifax’s offering of credit file monitoring to all US consumers is also an unfair trade practice, which undercuts innovation, and limits the ability of new entrants to deliver effective services.

The FTC should require Equifax to send a voucher to each impacted individual which can be used to purchase any identity theft protection service on the market as of August, 2017.

Usually I don’t try to blog fast moving stories, but I may make an exception.

Update 1, later that day:

Update 2, Sept 9:

  • The International Business Times reports “Equifax Lobbied To Kill Rule Protecting Victims Of Data Breaches.” They report Equifax wrote “a rule blocking companies from forcing their customers to waive class action rights would expose credit agencies ‘to unmanageable class action liability that could result in full disgorgement of revenues’ if companies are found to have illegally harmed their customers.” It’s a nice life, having the government block your victims from suing you, especially if you’re worried that the harm is great enough to result in ‘full disgorgement of revenues.’ Now, you might argue that’s hyperbole, but maybe it’s a real fear.
  • The Onion reports “Equifax Impressed By Hackers’ Ability To Ruin People’s Finances More Efficiently Than Company Can.”
  • Equifax once brought me to a Nine Inch Nails concert, and under the payola rules, I ought to have disclosed that when writing about them. It was over a decade ago, and had slipped my mind.

Update 3, Sept 12:

Update 4, September 16:

Update 5, September 21:

Security Lessons from Healthcare.gov

There’s a great “long read” at CIO, “6 Software Development Lessons From Healthcare.gov’s Failed Launch.” It opens:

This article tries to go further than the typical coverage of Healthcare.gov. The amazing thing about this story isn’t the failure. That was fairly obvious. No, the strange thing is the manner in which often conflicting information is coming out. Writing this piece requires some archeology: Going over facts and looking for inconsistencies to assemble the best information about what’s happened and pinpoint six lessons we might learn from it.

There’s a lot there, and I liked it even before lesson 6 (“Threat Modeling Matters”). Open analysis is generally better.

There’s a question of why this has to be done by someone like Matthew Heusser. No disrespect is intended, but why isn’t Healthcare.gov performing these analyses and sharing them? Part of the problem is that we live in an “outrage world” where it’s easier to point fingers and giggle in 140 characters and hurt people’s lives or careers than it is to make a positive contribution.

It would be great to see project analyses and attempts to learn from more projects that go sideways. But it would also be great to see these for security failures. As I asked in “What Happened At OPM,” we have these major hacks, and we learn nothing at all from them. (Or worse, we learn bad lessons, such as “don’t go looking for breaches.”)

The definition of insanity is doing the same thing over and over and hoping for different results. (Which may includes asking the same question or writing the same blog post over and over, which is why I’m starting a company to improve security effectiveness.)

What Happened At OPM?

I want to discuss some elements of the OPM breach and what we know and what we don’t. Before I do, I want to acknowledge the tremendous and justified distress that those who’ve filled out the SF-86 form are experiencing. I also want to acknowledge the tremendous concern that those who employ those with clearances must be feeling. The form is designed as an (inverted) roadmap to suborning people, and now all that data is in the hands of a foreign intelligence service.

The National Journal published A Timeline of Government Data Breaches:
OPM Data Breach

I asked after the root cause, and Rich Bejtlich responded “The root cause is a focus on locking doors and windows while intruders are still in the house” with a pointer to his “Continuous Diagnostic Monitoring Does Not Detect Hackers.”

And while I agree with Richard’s point in that post, I don’t think that’s the root cause. When I think about root cause, I think about approaches like Five Whys or Ishikawa. If we apply this sort of approach then we can ask, “Why were foreigners able to download the OPM database?” There are numerous paths that we might take, for example:

  1. Because of a lack of two-factor authentication (2FA)
  2. Why? Because some critical systems at OPM don’t support 2FA.
  3. Why? Because of a lack of budget for upgrades & testing (etc)

Alternately, we might go down a variety of paths based on the Inspector General Report. We might consider Richard’s point:

  1. A focus on locking doors and windows while intruders are still in the house.
  2. Why? Because someone there knows how to lock doors and windows.
  3. Why? Because lots of organizations hire out of government agencies.
  4. Why? Because they pay better
  5. [Alternate] Employees don’t like the clearance process

But we can go down alternate paths:

  1. A focus on locking doors and windows while intruders are still in the house.
  2. Why? Because finding intruders in the house is hard, and people often miss those stealthy attackers.
  3. Why? Because networks are chaotic and change frequently
  4. [Alternate] Because not enough people publish lists of IoCs, so defenders don’t know what to look for.

What I’d really like to see are specific technical facts laid out. (Or heck, if the facts are unknowable because logs rotated or attackers deleted them, or we don’t even know why we can’t know, let’s talk about that, and learn from it.)

OPM and Katherine Archuleta have already been penalized. Let’s learn things beyond dates. Let’s put the facts out there, or, as I quoted in my last post we “should declare the causes which impel them to the separation”, or “let Facts be submitted to a candid world.” Once we have facts about the causes, we can perform deeper root cause analysis.

I don’t think that the OIG report contains those causes. Each of those audit failings might play one of several roles. The failing might have been causal, and fixing it would have stopped the attack. The failing might have been casual and the attacker would have worked around it. The failing might be irrelevant (for example, I’ve rarely seen an authorization to operate prevent an attack, unless you fold it up very small and glue it into a USB port). The failings might even have been distracting, taking attention away from other work that might have prevented the attack.

A collection of public facts would enable us to have a discussion about those possibilities. We could have a meta-conversation about those categorizations of failings, and if there’s other ones which make more sense.

Alternately, we can keep going the way we’re going.

So. What happened at OPM?

California gets a strengthened Breach Notification Law

Governor Brown of California has signed a strengthened breach notification bill, which amends Sections 1798.29 and 1798.82 of the California Civil Code in important ways. Previous versions had been repeatedly vetoed by Arnold Schwarzenegger.

As described[.DOC] by its sponsor’s office, this law:

  • Establishes standard, core content — such as the type of information breached, time of breach, and toll-free telephone numbers and addresses of the major credit reporting agencies — for security breach notices in California;
  • Requires public agencies, businesses, and persons subject to California’s security breach notification law, if more than 500 California residents are affected by a single breach, to send an electronic copy of the breach notification to the Attorney General; and,
  • Requires public agencies, businesses and persons subject to California’s security breach notification law, if they are utilizing the substitute notice provisions in current law, to also provide that notification to the Office of Information Security or the Office of Privacy Protection, as applicable.

    • senatorsimitian.com

    This makes California the fifteenth (!) state with a central notification provision on the books, the others being: Hawaii, Iowa, Maryland, Massachusetts, Minnesota, New Hampshire, New York, North Carolina, Oregon, Vermont, Virginia, West Virginia, Wisconsin, and Wyoming. Puerto Rico also has such a requirement. Ibid.

    I’m looking forward to the resulting information, and I hope California’s Attorney General has the good sense to post all received notification letters. This will undoubtedly be easier for the state than dealing with the inevitable FOIA requests, and serves the public interest by increasing transparency.

    J.C. Penny knew best

    JC Penney, Wet Seal: Gonzalez Mystery Merchants

    JCPenney and Wet Seal were both officially added to the list of retail victims of Albert Gonzalez on Friday (March 26) when U.S. District Court Judge Douglas P. Woodlock refused to continue their cloak of secrecy and removed the seal from their names. StorefrontBacktalk had reported last August that $17 billion JCPenney chain was one of Gonzalez’s victims, even though JCPenney’s media representatives were denying it.

    and

    [The judge said] both retailers should have announced their involvement from the start, that consumers had the right to know. He said he would not provide the companies “insulation from transparency.” The judge stressed that the companies were seeking privacy as though they were individual victims, which he said was like “hermaphroditing a business corporation.”

    Wired picked up the story and wrote:

    It’s a bit jarring to see a lucid pro-transparency, pro-security argument from a federal prosecutor. For years, law enforcement has had an informal policy of protecting companies from the public relations consequences of their poor security — a kind of omerta among intruders, the companies they hack and the feds, where only the public is left in the dark. To be sure, it’s never been set in stone, and not all feds have played ball. But it’s a common practice, and it corrodes accountability.

    Biggest Breach Ever

    Precision blogging gets the scoop:

    You’re probably talking about this terrible security disaster already: the largest database leak ever. Arweena, a spokes-elf for Santa Claus, admitted a few hours ago that the database posted at WikiLeaks yesterday is indeed the comprehensive 2009 list of which kids have been naughty, and which were nice. The source of the leak is unclear. It may have come from a renegade reindeer, or it could be the work of a clever programmer in the Ukraine. Either way, it’s a terrible black eye for Santa. Arweena promised that in the future, access to this database would be restricted on a “need to know” basis. And you know who that means!

    Let’s see if customers really change their behavior. I know which way I’m betting.

    Connecticut Attorney General On The March

    It’s been a bad couple of weeks for residents of Connecticut and their personal health information. First Blue Cross Blue Shield had a laptop stolen with enough PHI that over 800K doctors were notified that their patients were at risk, including almost 19K in Connecticut.

    Connecticut’s attorney general said Monday that he’s investigating insurer Blue Cross Blue Shield’s loss of confidential information about health care providers, which was on an employee’s stolen laptop computer.
    Richard Blumenthal said Monday that the company and its affiliates may have broken state law by losing the information and taking too long to notify doctors.

    And if that wasn’t enough, Health Net lost Information for 450,000 Connecticut residents.

    Blumenthal said he’s “outraged” that the company never told customers or police and only told the AG on Wednesday.
    Blumenthal is investigating and demanding that Health Net provide consumers with at least two years of identity theft protection, identity theft insurance, reimbursement for credit freezes and credit monitoring for at least two years for all 446,000 consumers.

    I wonder how many other State AGs are investigating Health Net at this point. There were a total of 1.5 million records lost at least count.
    At bare minimum Arizona’s AG is also investigating.

    Health Net officials said they were not able to determine which information was on the disk, so they investigated and learned the information was saved in an image format that cannot be read without special software.

    So anyone have any clue what this supposed image format is? And what makes them think that someone who was smart enough to grab that drive wasn’t smart enough to grab a copy of the software? Assuming of course that wasn’t just all in pdf…