That is all.
Category: SysAdmin
Who Watches the FUD Watcher?
In this week’s CSO Online, Bill Brenner writes about the recent breaks at Kaspersky Labs and F-Secure. You can tell his opinion from the title alone, “Security Vendor Breach Fallout Justified” in his ironically named “FUD watch” column.
Brenner watched the FUD as he spreads it. He moans histrionically,
When security is your company’s business, even the smallest breach is worthy of scorn. If you can’t keep the bad guys out of your own database, how can customers reasonably expect that you’ll keep theirs safe?
Oh, please. Spare us the gotcha. Let me toss something back at Brenner. In the quote above, he says, “theirs” but probably meant to say “them.” The antecedant of “theirs” is database, and Kaspersky isn’t strictly a database security company, but an anti-virus company. “Them” is a much better turn of phrase, and I hope what he meant to say. How can we possibly trust CSO Online as a supplier of security knowledge when they can’t even compose a simple paragraph? And how can we even trust your own tagline:
Senior Editor Bill Brenner scours the Internet in search of FUD – overhyped security threats that ultimately have little impact on a CSO’s daily routine. The goal: help security decision makers separate the hot air from genuine action items.
Why is FUD Watch creating the very sort FUD they claim to watch? Who watches the FUD watchers? I do, I suppose.
Is my criticism unfair and picayune? Yup.
People make mistakes, even Kaspersky and F-Seecure. And heck, even CSO Online. I forgive you.
Brenner came very close to writing the article that should have been written. If even the likes of Kaspersky and F-Secure fall victim to stupid things like SQL injection, what does that say about the state of web programming tools? How can mere mortals be safe if they can’t?
The drama about these breaks is FUD. It shows that no one is immune. It shows that merely being good at what you do isn’t good enough. It means that people need to test, verify, buy Adam’s book, read it, and act on it.
The correct lesson is not schadenfreude, but humility. There but for the grace of God, go all of us.
The Hazards of Not Using RFC 1918
RFC 1918 is a best-current-practicies RFC that describes network address ranges that we all agree we won’t use globally. They get used for private networks, NAT ranges and so on. There are three ranges:
10.0.0.0 to 10.255.255.255
172.16.0.0 to 172.31.255.255
192.168.0.0 to 192.168.255.255
They are thus the Internet equivalent of the American phone system not using the exchange 555, only more useful. If you need to give an example IP address, you can use one of those without causing anyone consternation or irritation.
An example of why you want to use one of these addresses can be found (at least for the next few minutes) at Microsoft’s site for the IE 8 beta. One of the IE 8 features is the “SmartScreen Filter” which can tell you IP addresses you’re best not going to. An example is the picture accompanying my post.
If you check out that address, 207.68.196.170, at ARIN Whois, you find out that it’s owned by Microsoft themselves.
I suppose that using one of your own addresses as a hazardous address is better than using someone else’s, but immature people like Your Friendly Author will titter over it and point it out to other people as well.
There’s a reason RFC 1918 exists, and this is one of them. Oh, by the way, be sure to look at RFC 2606, which reserves the domains example.com, example.net, and example.org. It also reserves the top-level domains .test, .example, .invalid, and .localhost. Remember them.
Brightening up the day from an unexpected place
I would estimate that 2/3 of the calls I get are from people trying to sell me things I neither need nor want. Of those, over half are outsourcing services. Of the remainder, recruiters are over half.
There are also people who call me for their services once a week. There’s one particular outsourcing firm whose name is burned into my brain because of the number of times I’ve been subjected to it. I don’t know how to spell their name, but I can sure pronounce it. There’s also a recruiting firm that I know well, too. Each of these people I have asked to take me off their list, asked to talk to supervisors, talked to supervisors, yelled at them, ranted at them, and finally sworn at them, and yet I still get my weekly call.
As I was doing office stuff a few moments ago, I played a voicemail, and it was from my friends at Hadron Infotech, letting me know about their services just in case I have (a) developed a need I didn’t have last week and (b) forgot their name. (One of my rants included telling them that when I do need such services, they will be the last people I call and sadly for them, I have no trouble remembering their name.)
Since I was doing office stuff, I let the message drone on, and got the litany of things they can do for me including, Java, Jay-mumble-E, Dot-Net, Pee-Haitch-Pee, AJAX, Perl, Ruby on Trains, updating your web site, ….
Wait a minute. Did he say what I thought he said? Ruby on what? I ran over to my computer, backed up the player, and … Yes! Ruby on Trains! How delightful!
I’m still laughing. I hope you are, too. Maybe I’ll get another laugh next week.
Photo “Ruby on Train” by theresa_l_reed.
There's got to be an IT secret handshake
I’ve been in the hotel I am in for over a week now. It is a European hotel that has wireless, and you have to get an access card and type a six-character string into an access web page. That authenticates you, and you can go.
The problem I have today is that I can browse the net completely. But I can’t do anything else. No email, no vpn, no ping, no traceroute, no nothing. If I telnet to a useful port on my own servers, I get a syn/ack/syn and no flow.
My hypothesis is that whatever does a redirect on port 80 to get you to the authentication web page is broken.
I’ve talked to first-line tech support at the provider who let it slip that he thinks its in the firewall at the hotel. This is consistent with my evidence. However, he won’t let me talk to anyone who actually knows what “ping” is. I have talked to someone at my front desk, who has talked to the local IT person, and we’ve had mediated back-and-forths.
If I could actually talk to someone who knows what a web redirect is or even what a “port” is, I could let them know. If I knew the URL of the authentication page, I could tell them the problem. The local IT guy is presently talking to the ISP, but I told the gal at the desk that I’m an IT person, too, and if their IT guy will call me, then I will help explain the problem.
As a matter of fact, while writing this, I just connected to an https url, which redirected me to the authentication page, and now everything is working. This is how you’re reading this today. So I know what their problem is and can tell them how to fix it. They just have to know that I know, and that I’m not a mere luser.
We need an IT secret handshake. Perhaps Randall Munroe can help. Remember those old stories about the Freemasons in some pickle or another who suddenly showed the handshake? We need one.
Update: The gal at the front desk has called back. The ISP and the local IT people have decided this is actually my problem. However, she also says that another guest has this problem. I explained this as much as I could to her, and told her to tell the other guest to go to an SSL web page to fix it.
Photo courtesy of photos.tjweb and selected because it matched a search for “authentication web page”
Splunk'd?
I have been playing with Splunk, for about 45 minutes.
So far, I like it.
I’ve previously been exposed to Arcsight, but what I have more of an affinity for psychologically is not so much a correlation engine, but a great visualization tool that automagically can grok log formats without making me write a hairy regex. I have no idea whether I will use Splunk for anything real, but it made a good first impression. Since my budget is zero, the price of the non-enterprise version looks good, too. I am sure that for those of a less penurious station, there are many more fine contenders.
Security Through Stupidity
In my last post on security, I promised a tale, and I ought to deliver on that before it becomes nothing more than a good intention.
Some time ago, so long ago that it no longer matters, I bought a piece of network stereo equipment. It was one of these little boxes that lets you play MP3s, etc. through your stereo. I got it because it was a cute little system running Linux, had a MIPS processor, a web site for developers, extension and enhancement tools in Java, and so on.
I used it for a couple of months, and played with the Java-based remote control application for it and then decided to do some more serious work on it. I rolled my eyes that it only had telnet to get to it, but telnetted to it and was met with:
#
which I just stared at for a moment. It didn’t even register for a good twenty or thirty seconds before I had the wit to type
ls
and was met with something akin to:
bin dev home mnt proc tmp boot etc lib usr sbin
and that didn’t even register with me until I finally then typed
pwd
and was met with
/
and I made a loud two-word exclamation, of which the former was “oh” and the latter is left as an exercise for you, Gentle Reader, but there are two obvious candidates.
Yup, for the last couple of months, sitting bear-ass nekkid on the Internet was a Linux box with open telnet and a root shell. No username, no password, just a root shell. I said the other obvious candidate word. I also considered (again) getting a firewall. My network doesn’t have a firewall. Part of it is that I like the road feel of the packets whizzing by. Part of it is that by the time I open up enough ports to do useful things, I’m just closing down the ones that don’t have services on them anyway. Part of it is also that of the three times I’ve had serious security problems on my network, one of them was because my IDS box got rooted, and one was because the firewall got rooted. For me, adding a firewall adds complexity, and that lowers security. (That last time was when I was traveling with my SO who wanted to send me an email from an utterly ancient netnews program that knows nothing of SMTP-AUTH. Never reconfigure your email infrastructure from five thousand miles away while jetlagged. A couple of days later, you will ask yourself, “I wonder why the SMTP server logs have gotten so big.” Fortunately for me, I caught it before the blacklists did.)
I yanked the music box off the network and connected to it directly (one cable, just it and me). Looking through the thing, I didn’t see what anyone who was now using it for anything. I checked the IDS logs and there was nothing that leapt out at me to as suspicious traffic. That seemed odd, because how could it not have been owned? I thought about it for a bit, and thought about it more as I reflashed the critter. Then I laughed, because I realized that the tools that probe for vulnerable boxes are not going to be looking for #. It was then too late to tell, but I allowed myself to think that maybe the box hadn’t been compromised, as the evidence suggested.
With the machine rebuilt, I connected to it directly with telnet and started probing around for putting a password (like /etc/passwd). There was none. There was no SSH, either. I fulminated on the developer fora about this security stupidity. I found the instructions on how to build the right cross-compiled Linux setup to build binaries for it, and it was filled full of warnings about how to make sure you did this, set that compiler switch, and if you didn’t, things wouldn’t work, and you get to reflash the box.
This wasn’t how I was wanting to spend my Saturday, so I turned the box off, and went to do something else. As I did, I thought about the situation. I became increasingly amused that (apparently) the box hadn’t been compromised. I convinced myself that this is because the bad guys wouldn’t recognize the box as vulnerable.
As I grumbled and thought more about how to lock down the box and then something occurred to me — anyone who wants to own the box has to go to the same trouble to make it be a productive member of their botnet community as I do to do the opposite, but they’re at a disadvantage because they also have to protect it from me. Since it’s easier to find some unpatched Windows box than it is to set up a MIPS cross-compile sandbox, even if they can tell that has an open root shell, it’s not economically viable. Think of it as Mutual Assured Annoyance, Economic-Based Intrusion Prevention, Security Through Stupidity, or proving old adage, “In the land of the blind lion, the one-eyed zebra doesn’t have to run very fast.”
A couple of weeks later, I solved the whole problem when a new product was introduced that did exactly what I wanted (to be able to play music on my laptop on my stereo) at half the price and no icky telnet. The poor little music box now sits face-down, forlorn, and dust-covered on a shelf.
Holding a Lighted Brand up to Damage
Adam comments on some breach commentary, and quotes Nick Owen saying that breaches are a sign of incompetence.
I can’t let this stand un-commented-upon. I believe that that is a dangerous comment, and one that needs to be squashed early. It’s like saying that a bug tracking system with lots of bugs in it is a sign of engineering incompetence. It actually means the opposite. A truly incompetent management team wouldn’t know they’d been breached. A slightly less incompetent team would bury it under the rug. This is true for software developers as well as operations people.
This is a very dangerous comment because it rewards the truly incompetent who don’t know how screwed up they are. It is a dangerous comment because it rewards the mendacious, who hide that they’ve been breached — or who design their operations so they won’t know when they’re breached. Stop. You’re going to set us backwards if you keep that up.
It doesn’t matter how good you are, some day you will be breached. Accept that. As a consumer, that’s a mildly unpleasant thing to think of, but it’s true. However, you want people who lose your data to have the wit to know they’ve lost it, and the morality to own up to it.
I also want to comment on Allan Friedman’s comment about Iron Mountain, as I’ve noticed the same thing, that many breaches involved Iron Mountain losing tapes. But I’m not an economist, I’m a guy who’s spent times in operational groups, and I have an alternative hypothesis.
Let us assume an organization that makes daily backups and sends them to a data warehouse. Let us suppose that the tape monkeys have a Very Bad Day. Sam’s on vacation. Ginger broke up with her boyfriend and came in late. Two tapes verified bad and had to be re-done, Networking misconfigured something and you couldn’t get to C Building at all. The Iron Mountain guys come in to get the tapes from you, and you tell them the horror story. They say hey, no problem, just give them what you have. They’ll take it off to the warehouse, and as long as there’s no disaster tomorrow, it’ll all be taken care of in the next incremental. The CIO never has to know. Whew! Thanks, Iron Mountain! You’re a life saver.
Iron Mountain is being smart. The real customer is the supervisor of the tape monkeys, and if you help him shine, he helps you shine. Alas, they’re being smart until lost data is not simply a gap in the backup history, it’s a breach. Then this habit of mutual back-scratching all falls apart. If someone does an audit and finds out that a backup of the Order Database is missing, Iron Mountain takes the fall. All the paperwork says that the database was backed up, put onto tape 1723-A5, and sent to the warehouse. And therefore, so it was. Iron Mountain can’t say, “Um, actually, for years now, we’ve been covering for our customers and letting them claim data was in the warehouse when we all know it wasn’t.” They just have to take it on the chin.
You know what? The real customers, the tape monkeys who have been let off the hook yet again know that Iron Mountain kept them out of even bigger trouble. They know that the Iron Mountain guys can’t let them hand over an empty box any more. But they aren’t going to switch to another company, either.
My hypothesis could be wrong. I don’t know if it is. I can’t admit to ever having been in a situation like my hypothesis. I am, however, a cynic, and I know that if Iron Mountain were in the habit of losing tapes, it may or may not show up in their stock price. But if they were in the habit of making the tape monkeys look more competent than they actually are, it is consistent with observed phenomena. It doesn’t mean my hypothesis is right; heck, the magic blue smoke theory of semiconductor physics is consistent with observed phenomena. But when I noticed Iron Mountain showing up in a number of breaches, the smoke I smelled seemed to have a hint of electrolytic capacitor in it, and whiff of insulation.
Department of pre-blogging, II
A bit of background.
Sun recently got hit with a 0-day that was 13 years in the making, by seemingly repeating a coding worst practice that bit AIX back in 1994 — trusting environment variables under the control of an attacker. A slightly more complex variant bit Solaris’ telnetd in 1995.
From the advisory (NSFW) at http://www.com-winner.com/0day_was_the_case_that_they_gave_me.pdf, as sent to me in an email:
/usr/src/cmd/cmd-inet/usr.sbin/in.telnetd.c
3198
3199 } else /* default, no auth. info available, login does it all */ {
3200 (void) execl(LOGIN_PROGRAM, "login",
3201 "-p", "-h", host, "-d", slavename,
3202 getenv("USER"), 0);
3203 }
Anyway, to save you valuable time, the pre-blogging department at EC has prepared a short summary of some posts you will see elsewhere on this topic.
“This proves open source is less secure than closed source”
This vulnerability is so Old Skool, it could have lain dormant like some sort of unexpressed genetic flaw. By making source available, Sun provided a road map to their own weakest link. You can hear the chortling in Redmond already.
“This proves the value of open source”
Low-hanging fruit like this is ripe for the picking. By harnessing the people power of a million eyeballs, cruft like this will be much more quickly eliminated. For all we know, Vista’s telnet service is even worse.
“Don’t they teach these kids to write half-decent code???”
[This one will be found in a blog containing a “Created with vi” emblem in the corner, and embedded RCS keywords showing its last mod time.]
(Note: The pre-blogging department accepts no responsibility for metaphors mixed in the provision of this valuable service. YMMV. Do not taunt happy fun ball.)