Archive for November, 2007

More on (moron?) Vulnerability Research Business Models

Dave G. | November 9th, 2007 | Filed Under: Industry Punditry

In a comment on my last post, cmlh writes:

@Dave G,

Based on the failing (due to agenda) of (particular) Researchers, Coordinators (i.e. FIRST Members) and Vendors - Which “trusted person or organization” is left “that can represent vulnerability researchers whose reputation is at stake when dealing with vendors.”?

In a word, there are plenty individuals that could fulfill this role. What I was really trying to say is that rather than the auction model, maybe the way to make this all work is to go for more of the agent model, like for screenwriters or novelists. While no one likes agents, the fact is they play an important role. They are responsible to both parties. When they fail, they lose customers fast. And there aren’t enough people around buying vulnerabilities that you can afford to lose customers. Also, it all happens in private, which reduces risk. Finally, everyone involved can be contractually bound. Who knows, maybe one day this will take off and there will be a vulnerability researchers strike!

BTW, If this was a lay-up question where I was supposed to say Matasano, thanks, but no thanks. Spitballing about vulnerability markets is fun, but that’s where it ends.

Comment Bubble 11 Comments

WabiSabiLabi Co-Founder Arrested

Dave G. | November 6th, 2007 | Filed Under: Industry Punditry

WabiSabiLabi, formerly most famous for bringing to market the first public vulnerability market, has once again made the headlines. This time, one of their co-founders, Roberto Preatoni, has been folded into an ongoing Italian wiretapping scandal. This investagation has been going on for 10 plus months.

Prior to WabiSabiLabi, Roberto worked at Telecom Italia as part of their penetration testing team. Four members of this team have already been arrested back in January for using a Trojan Horse to compromise and monitor Vittorio Colao, the former CEO of the Rizzoli Corriere della Sera publishing group.

From Robert McMillan:

According to the reports, Preatoni helped staff a 10-member “Tiger Team,” ostensibly set up to test Telecom Italia’s information security system. Members of this team are now charged with hacking and spying on Carla Cico, CEO of Brasil Telecom; the Kroll investigative agency; and journalists Fausto Carioti and David Giacalone of the newspaper Libero.

This might actually be one of the biggest challenges for vendors and vulnerability researchers. How far can you really trust that everyone is doing the right thing? If I were a vendor, I would not make the assumption that the vulnerability researcher is trustworthy. This isn’t to say that you should be hostile towards vulnerability researchers. It is simply that you have absolutely no idea how many people a researcher has told about a vulnerability. Given that, I think it makes sense to treat vulnerability reports as if you just found out about your vulnerability through BUGTRAQ.

While obvious, this also speaks to why it is hard to implement a vulnerability market. It is all about trust. And if the buyers and sellers utilizing (or considering utilizing) WSL can’t get past this, I’d say it’s game-over.

As I think about it, probably the best way for vulnerability researchers and vendors to be bridged is through a vulnerability broker. This could be a trusted person or organization that can represent vulnerability researchers whose reputation is at stake when dealing with vendors.

Of course, I am personally not sold on the idea that the sale of vulnerabilities is a good idea.

Finally, from the ‘There’s No Such Thing as Bad Press Dept’:


WSLBlogTN

Comment Bubble 12 Comments

Old Fashioned Data Theft

Dave G. | November 5th, 2007 | Filed Under: Industry Punditry

On friday, The Register reported that CI Host, a Chicago-based co-lo was subjected to a burglary. This was not their first time.

In spite of “multiple layers of 24x7 security cameras, proximity card readers, biometric access controls and key pads, double-locking mantraps at data center entrance and 360-degree perimeter and roof surveillance”, CI Host was burglarized at least four times in the past 2 years.

In the most recent incident, “at least two masked intruders entered the suite after cutting into the reinforced walls with a power saw,” according to a letter C I Host officials sent customers. “During the robbery, C I Host’s night manager was repeatedly tazered and struck with a blunt instrument. After violently attacking the manager, the intruders stole equipment belonging to C I Host and its customers.” At least 20 data servers were stolen, said Patrick Camden, deputy director of news affairs for the Chicago Police Department

This might be the way of the future for mass credit card theft happens. I wonder how many individual credit cards are located in that facility? Or more importantly, on the servers that were plundered from that facility.

It’s 4:50PM do you know if your servers are still in your colo?

Comment Bubble 8 Comments

The Insidious Insider Threat

Dave G. | November 5th, 2007 | Filed Under: Industry Punditry

The Mogull writes about his pet peeves around the abuse of the insider threat, and lists some principles for talking about the insider threat. There is no doubt that it is a concept used to sell security technologies. No doubt it is an overloaded term (like the word “hacker” or “democracy”).

To me, the term “Insider Threat” conjures up images of employees and/or contractors that are given legitimate access to the internal resources of an organization. It is assumed by said organization that they either know or can find out the identities of this subset of the world’s population. This allows them to utilize a number of different mitigation techniques to reduce risk. This provides them with a comfort level that allows them to trust employess and contractors more than the rest of the people on the Internet.

Which, as far as an approach goes, is about as good as it is going to get for now. It is easier to make it really difficult for most external attackers to succeed. The attack surface is better defined, and your employees expect some levels of inconvenience due to the fear of hackers, viruses and worms. It’s also cheaper.

As a penetration tester, I can tell you with a fair amount of certainty that if you put us on your internal network, we are going to be able to compromise many things that you care deeply about. Internal networks are large, diverse, rapidly-changing ecosystems. And even though we find a lot of crazy new vulnerabilities on these engagements, we often end up simply taking advantage of the privilege we already have.

This is precisely why it is used to sell product. The Insider has so many avenues to attack internal resources. And for each avenue, there is someone willing to build a product to save you. On internal penetration tests, it is pretty common for a customer to learn about systems they didn’t even know were on their networks.

The difference between the external and internal threat can probably be ascertained by asking these questions:

  1. Motivation. Do they want to do bad things?

  2. Skillset. Do they know how to do bad things?

  3. Access. What can the do bad things to?

Most of the people on Earth are not going to be motivated to attack an organization. And of the ones that are, a pretty small subset are actually capable. For example, my mom is really not capable of launching an internet based attack against a F500 enterprise(1). However, when she was an office manager, I am reasonably sure she had the ability to do lots of bad things (2).

Not unlike relationships, the ones that can hurt you the most are the people that know you the best. Almost every external intrusion ends up with someone stealing a whole bunch of PII. And yes, that stinks, and can be totally devastating. But internal attacks tend to be about how some guy that knows how the business works uses that knowledge to directly damage the business, typically via theft. This actually holds true for your local convenient store as it does to your online brokerage.

I think one reason why we focus (maybe overfocus) in the external threat is that it can’t be controlled. When an employee does something bad, it doesn’t have to become a major media event. You can handle things civilly. And by that I mean civil law.

(1) She is however capable of launching an internet based attack against my conscience for not calling enough.

(2) She wouldn’t.

Comment Bubble 3 Comments

Quick Hit On Open Source Routers, Vyatta, and NetworkWorld

Thomas Ptacek | November 3rd, 2007 | Filed Under: Industry Punditry, Uncategorized

I agree with Matt Franz that router security isn’t very interesting (though there’s real work to be done in mainstreaming the IOS reversing work that’s been done). And I agree that the Network World article on Vyatta, the open source router startup, is silly. I can answer Jim Duffy’s question (“is open source routing closing the router gap?”) pretty quickly: no. Last I checked, the ISR had revitalized routers for Cisco, and they were doing better than they ever had.

Comment Bubble 10 Comments

Thoughts on Ten Years of qmail Security

Thomas Ptacek | November 2nd, 2007 | Filed Under: Defenses, Uncategorized

Ahh, qmail. What is it about Daniel J. Bernstein’s full-featured Sendmail replacement that we can’t shut up about? Oh yeah, maybe it’s the nine years that have gone by since its last major release in 1998 which saw not one exploitable vulnerability. qmail is one of the top 5 Internet mail servers. Not one other project of similar standing shares this track record. It’s actually kind of spooky.

So, we’ve written about qmail before. And, if Georgi Guninski reads our blog, our qmail writing could conceivably cost me $500. What excuse can we make to keep talking about it?

How about a just-published retrospective on qmail and software security by DJB himself?

DJB isn’t really talking about qmail; he’s using it to illustrate lessons learned about software security. Here’s an example: three answers for eliminating security vulnerabilities from software:

  1. Eliminate bugs from code. As Theo said, “we assume all bugs are vulnerabilities”. Kill bugs, close vulnerabilities. How do we kill bugs? Engineering has wrestled with that for decades. Some answers: formalized software engineering. Automated QA and coverage testing. Unit testing. Code review. Up-front security design.

    It works. qmail won by replacing 80’s era C library interfaces with new libraries that made it hard to write incorrect code. Microsoft won by beating the living crap out of their code and release schedule with security-focused QA.

  2. Eliminate code. More code? More potential bugs. Minimizing code footprint reduces exposure to bugs. “Complexity is the enemy of security”. How do we eliminate code? Bernstein’s best answer is reuse: expend effort so your programs exploits your platform, instead of reinventing the wheel.

    It works. qmail won by taking advantage of somewhat obscure Unix features like programmable resource limits. The major enterprise web stacks like J2EE and .NET allow web apps to eliminate code by building on well-tested session, database, and templating libraries.

  3. Eliminate trusted code. If you have to spend lines of code to deliver a feature, at least keep the code out of the line of fire. How do we minimize trusted code? Bernstein grapples with this question, arriving at two answers: use the operating system to compartmentalize code, so that vulnerabilities can’t do real damage, and run sensitive code in interpreters, trading speed for security.

    It works. Minimizing trusted code (or what consultants would call “the attack surface”) in qmail is the signature architectural trait of qmail. But you get the impression that Bernstein is least satisfied with qmail’s execution here. At the same time, it’s impossible to miss the wins other projects see with this approach: the vast majority of deployed enterprise software is innoculated against memory corruption vulnerabilities because they’re run in high-level languages like C# and Java.

Bernstein has a few grenades to toss. I disagree with his take on “least privilege”; revoking program rights to operating system resources like files and sockets isn’t a distraction. Yes, there’s a (subtle) difference between sandboxing a broken chunk of code and designing it properly to begin with; and yes, it’s true that access control schemes are plagued with “game-over-equivalence” problems (“you can read everyone’s mail, but at least you didn’t get root”). But I found his argument mushy, seeming to contradict his second “minimize the TCB” strategy, and Saltzer’s principals still seem vital to me.

There’s a conspiracy theory among those who have audited qmail’s idiosyncratic codebase that Bernstein didn’t actually write it in C, but rather in a language he compiled down to C. Here he in on the subject:

When I wrote qmail I rejected many languages as being much more painful than C for the end user to compile and use. I was inexplicably blind to the possibility of writing code in a better language and then using an automated translator to convert the code into C as a distribution language.

Of course, now that he’s spending his time writing compiler backends, maybe the new plan is to ship the entire language stack along with his applications.

Bernstein had the benefit of hindsight with buffer overflows, but had to weather integer vulnerabilities along with everyone else. qmail fared remarkably well (the only known integer overflow in qmail isn’t exploitable on any real deployment). The jury’s still out on what the next major bug class will be (our money has been on uninitialized stack variables). We may eke out a few more qmail bugs over the next 10 years. But relying on it seems like a safe bet.

Comment Bubble 32 Comments

Excellent Explanation of Leopard’s Firewall Behavior

Thomas Ptacek | November 1st, 2007 | Filed Under: Apple, Uncategorized

Some of Rich Mogull’s new writeup on what the 10.5 firewall does seems to refute Heise’s article: without “stealth mode” enabled, services show up in port scans but can’t actually be used. It’s still a mess.

Rich seems to have the mysterious test case that demonstrates code signing; let’s see if he’ll give us a step-by-step on it!

Comment Bubble 24 Comments

A Quick Data Point On Sandboxes

Thomas Ptacek | November 1st, 2007 | Filed Under: Apple, Uncategorized

Sandboxes are implemented via the “seatbelt” kext. You can run “lipo” on “seatbelt” to extract the i386 kernel module, and and pull it into a disassembler. Ralf and I are doing that now. Here’s what we now know:

Sandboxes are built, at least in part, on the new security/ subsystem of XNU (the source is available for that), which is derived from TrustedBSD (and, presumably, SEDarwin).

The Sandbox/seatbelt policy layer itself is Apple-proprietary, and I don’t think the source is available; more as I figure out more from the binary.

Comment Bubble 3 Comments

The Silly New Mac OS X Trojan or HoHum.A

Dave G. | November 1st, 2007 | Filed Under: Apple

As I am sure most of our readers are aware, there is a new Mac OS X trojan floating around. The authors seem to target users of pornographic websites, and requires full user interaction to install (e.g. allowing the program to run and typing in your administrator password). While it is definitely a sign that the botnet side of Internet crime is beginning to target Mac users, I don’t agree with Gadi Evron who states:

For whoever didn’t hear, there is a Macintosh trojan in-the-wild being dropped, infecting mac users. Yes, it is being done by a regular online gang—itw—it is not yet another proof of concept. The same gang infects Windows machines as well, just that now they also target macs.

http://sunbeltblog.blogspot.com/2007/10/screenshot-of-new-mac-trojan.html http://sunbeltblog.blogspot.com/2007/10/mackanapes-can-now-can-feel-pain-of.html

This means one thing: Apple’s day has finally come and Apple users are going to get hit hard. All those unpatched vulnerabilities from years past are going to bite them in the behind.

I can sum it up in one sentence: OS X is the new Windows 98. Investing in security ONLY as a last resort losses money, but everyone has to learn it for themselves.

It definitely does not mean that one thing. Here are some quick points:

  1. What unpatched vulnerabilities is he referring to?
  2. This didn’t exploit any vulnerabilities. This same exact trojan exists for Windows. It could also be written for just about any OS.
  3. To this day, I am not entirely convinced that it makes sense to invest in security before it costs you.
  4. OSX is NOT the new Windows 98. It is a pretty unfair comparison. We may be critical of some of Apple’s security efforts, but at least we try to be objective.

One thing I would say is that Mac OS X users may not be as battle hardened as Windows users are when it comes to malware. If there is an increase in this style of Dupe-The-User attack, I wonder what the success rates would be.


  1. If you read this blog, you are not the average Mac user.

Comment Bubble 16 Comments

What We’ve Since Learned About Leopard Security Features

Thomas Ptacek | November 1st, 2007 | Filed Under: Apple, Uncategorized

There are over 100 comments accumulated on my last Leopard post. As usual, they’re better than the post itself. Since you’re probably in a hurry, I’ll spare you the effort of poring over them, and instead present our findings to date.

OS X Runtime Stack Security

A commenter asked if Leopard’s compiler included ProPolice. ProPolice (and/or SSP) is a C compiler extension that guards the call stack of a program, injecting tripwires onto the stack that will be set off by buffer overflows.

Leopard gcc ships with stack protection. There’s probably a simple answer about what OS X programs are compiled with it, but the best I can tell you is that some OS X programs appear to use it; you can see for yourself by loading a program in “gdb”, and disassembling some functions. SSP’d functions have an idiosyncratic prologue and call a “check stack” function in their epilogue.

Do we care? Meh. Stack protectors defend against the oldest, easiest-to-find memory corruption errors. You still find stack overflows in obscure enterprise code, or on embedded platforms that are hard to test. Also on AIX. But you’d be a little shocked to find one in privileged OS X code.

OS X Memory Randomization

A commenter asked if the OS X heap and stack were randomized. Stack memory stores the call stack, which in turn stores the sequence of functions and subroutines being used at any given time. Stack memory also stores most of the variables a program knows about when the program is compiled. Heap memory stores dynamic variables, which depend on the programs inputs rather than on the code itself.

I could now waste your time with a discussion of how valuable stack and heap randomization is, but it’s a moot point: the OS X stack and heap don’t appear to be randomized.

Do we care? A little. Heap overflows are relatively common, because dynamic memory usage is always a bit more complicated than stack memory usage.

Library Randomization

An interesting point was made that the Mach-O ABI was inherently hard to randomize. We had noted that even Leopard’s Library Randomization was imperfect, as it kept the dynamic linker (and, as Ralf pointed out, the program text) at an exposed fixed address. Until those problems are fixed, you might as well not randomize. The commenter basically predicts that it will be awhile before this is resolved.

Do we care? Yes, in that Library Randomization is a major advertised security feature of Leopard. If you don’t randomize program text, it is straightforward to exploit memory corruption vulnerabilities.

W^X and Heap Security

Someone posited that the OS X memory model was now W^X. “Write XOR Execute” is an OpenBSD design idiom; it says that if something in memory is writeable, and therefore exposed to memory corruption, it should not at the same time be executable.

The OS X stack has been non-executable for quite a while. The OS X heap remains executable, a fact you can verify with a trivial piece of C code. Someone involved with PaX, the Linux runtime memory security extension, gave test results verifying that, and also showing that both the stack and the heap could be made executable by returning through the BSD “mprotect” system call.

Do we care? Yeah. This is an area where Leopard is noticeably lagging behind Vista. Read Marinescu’s talk at Black Hat; the Vista heap has an intricate protection scheme; Leopard seems to lack anything comparable.

Sandboxes

OS X Sandboxes —- my favorite Leopard feature, one I’ll have more to say about later —- allow users to write policies that firewall the operating system off from different programs. It is possible to use a Sandbox to prevent iChat from running any other programs, or touching any sensitive files.

Sandboxes are apparently enforced by a kernel extension called “seatbelt”. Seatbelt is a cooler name than Sandbox. Seatbelt calls a program called “sandbox-compilerd” from the kernel when a sandboxed program runs. You’d want OS X to be careful with “sandbox-compilerd”, since it consumes complex input (whole Scheme programs) and runs out of the kernel. In GA Leopard, “sandbox-compilerd” is itself sandboxed (wooo a paradox) and runs under your own credentials.

Do we care? We do, but you shouldn’t; this is just trivia.

Sandboxes and Watson’s Vulnerabilities

Sandboxes were inspired by RBAC features in other operating systems, most notably Niels Provos’ OpenBSD Systrace. Systrace has a well-known vulnerability, first documented by Robert Watson and published formally this year at Usenix WOOT. The problem is a classic TOCTTOU (time-of-check-to-time-of-use) race condition. As an example, Systrace goes to look up a file in the filesystem to see if you can touch it. Between the time Systrace OK’s the operation and the kernel actually performs it, you can swap the safe file with a sensitive one. The kernel’s second lookup will return a different result, which Systrace cannot verify.

Nobody (that we know of) has audited OS X Sandboxes for race conditions. It’s hard to know whether they are present. It wouldn’t surprise us either way.

Do we care? No, not really. First, it’s just speculation. Second, I don’t have any evidence that TOCTTOU races in kernel wrappers are ever actually exploited. Right now, someone actively beating OS X Sandboxing is not writing commodity virus programs; you did something to piss them off.

A Brief Interlude

It is taking me longer to write this up than I expected. Sorry!

The Leopard Firewall

The consensus opinion is that it’s a step backwards. Most notably, it doesn’t filter outbound connections. Multiple commenters note that you can get outbound filtering from programs like Little Snitch.

Do we care? We don’t, but our Moms do. Outbound filtering is more valuable than inbound filtering; it catches “phone-home” malware. It’s not that hard to implement, and I’m surprised Leopard doesn’t do it.

Code Signing

Apple says, “Leopard can use digital signatures to verify that an application hasn’t been changed since it was created.” You can create these signatures with the “codesign” tool, verify them on the command line with “codesign -v”, and display them with “codesign -dvv”. To create a key to sign them under, go to Keychain Access, select “Certificate Assistant” from the app menu, and generate a new “Code Signing” certificate.

Code signatures appear to be enforced from the TMSafetyNet kernel extension. I was just wrong about this, thanks Ralf.

Awesome. Two problems, though.

First: I haven’t yet found a place that checks these signatures. I tried Parental Controls and I tried Saved Passwords in Safari, both times testing by corrupting the binary in the same fashion as a virus. Evidently, the only thing “protected” by signatures is the Keychain, and the “protection” means that instead of accessing the Keychain transparently, you get a confirmation dialog that looks substantially similar to a Keychain dialog you probably click through several times a week.

Second: Even if they were validated, you can still inject unsigned libraries into applications when they launch; this is a core feature of the dynamic linker, which you enable with the “DYLDINSERTLIBRARIES” environment variable.

Do we care? It is very, very, very hard to build systems that gain security from code signing. There are like 10 posts, each longer than this one, that could go into explaining why that is. So, our take is, “no”. There was no way this was going to be a straightforward security win for Leopard. You care to the extend that you are irritated with Apple for marketing hyperbole.

Parental Controls

Here’s an interesting one. You can lock down what executables an account can use. “Parental Controls” undersells this feature. Enterprises pay tens of dollars per desktop for aftermarket software to lock down desktops to trusted applications.

I assumed with a name like “Parental Controls” that the threat model was my 8 year old son. It’s not. Parental Controls are enforced in the kernel, which you can demonstrated by allowing an account Terminal.app and nothing else. Parental Controls will keep you from executing arbitrary programs; it’s enforced at execve()!

Here’s where it gets weird.

Terminal.app is not very useful without the several hundred Unix command line tools you invoke Terminal to get access to. And you can run these programs. They aren’t individually allowed or denied; that would be a nightmare to configure.

You can even execute the compiler, and build new programs. But you can’t execute them!

I originally thought, “Eureka! A place to actually witness Code Signing in action!” No such luck. Copy /bin/ls to /tmp (its signature remains intact), and you can’t run it. Copy “hello world”, with no signature, into “/bin” (as root, of course) and you can. This appears to be “trusted path execution” —- programs in certain directories run, others must be individually allowed.

Unfortunately, the feature is broken in the same way Code Signing is. Want to run an arbitrary program under Parental Controls lockdown? Change its “main()” to a GCC “contructor” function, and compile it as a dylib. Then “DYLDINSERTLIBRARIES” it into any allowed program. Your code runs, and has full access to the system.

Do we care? Kind of a lot, yeah. Not that we’re disappointed. Actually, even though we seem to be able to walk past this feature, it works way better than we expected it to; it is one silly flaw away from parity with expensive aftermarket Windows tools. This feature should be fixed, exposed somewhere besides “Parental Controls”, and relabeled “Secure Desktop”.

Comment Bubble 25 Comments

Who We Are

Matasano is a team of internationally respected security experts who have led security efforts at @stake, Microsoft, ISS, Secure Computing, Arbor Networks, Secure Networks, Bloomberg, Sandia Labs, and others. Read more about our team and how we can help you today.