Vulnerability Sportsmen vs. Vulnerability Hunters
Dino Dai Zovi | November 29th, 2006 | Filed Under: Disclosure, Industry Punditry
First comes the vulnerability. Then comes the fame. And then the women. Finding vulnerabilities is sexy and makes great news these days. Unfortunately, this is attracting the type of vulnerability finders that may only be in it for the fame. In a time long ago and a land far away, the only people who discovered vulnerabilities either were doing so to improve products or take advantage of them for any number of reasons. That necessitated finding the actual cause of the vulnerability and/or method of exploiting it. Now, however, when fame is the only goal, neither of these is actually necessary. One may just find a crash due to a null pointer dereference, say “Risk: High, code execution”, and go straight to the recognition. Over the years, we have seen plenty of bugs and bug classes that were not thought to be exploitable later proven to be so by some very ingenious ways. Now we know better. Unfortunately, we have gone so far in our exploitability paranoia that we assume that all NULL pointer dereferences can lead to code execution. That perception is misguided for a number of reasons.
Assuming that vulnerabilities may be exploited in the future by yet-to-be-discovered means is not unjustified, but it must be tempered with some analysis of the specific vulnerability that has been discovered. Some very smart and talented people have advanced the state of the art of exploitation. Three that come to mind are Solar Designer’s JPEG exploit, Gobbles’ Apache exploit, and Skywing and Skape’s Exploiting the Otherwise Non-Exploitable on Windows paper. SD’s BUGTRAQ post clearly and eloquently brought heap structure corruption exploits into the mainstream. Also note how even though no exploit was provided, he did not just throw up a stack trace and register dump and say, “Code execution!”, but instead explained just how it was possible leaving no doubts as to whether the vulnerability was exploitable. Similarly, Gobbles laid down the law with apache_scalp.c proving that the Apache chunked encoding bug was, in fact, quite exploitable and exploited in the real world.
Skywing and Skape’s paper deserves a little more discussion. Most people understand their discovery to be a technique to exploit NULL pointer dereferences in Internet Explorer. They discovered that by loading and unloading certain COM objects out of order, they can leave the Unhandled Exception Filter function pointer dangling. Then, whenever the browser crashes (NULL pointer deref or any other crash), the dangling UEF pointer will be used. If you use javascript to populate the heap with controlled values, you can often control the function pointer that will be used. There are three ingredients to this type of exploit: dangling UEF, heap population, and a crash. The dangling UEF pointer is actually the vulnerability, and Microsoft patched it. Controlled heap population is also not possible in every application. So under no circumstances should the takeaway from this paper be, “NULL pointer dereferences are exploitable now.”
What is the difference between a software penetration tester and a QA tester and why do you pay more money to the former? Both make software crash. The penetration tester, however, provides value after the crash. They identify the risk of the vulnerability by identifying where and how the vulnerability is triggered and whether it is exploitable (and if so, how reliably). They should understand what types of vulnerabilities are exploitable by currently known methods and what types are highly unlikely to be exploitable. Developers and vendors need to prioritize security issues among other development needs and security is going to be ignored when risk is blown out of proportion. A good penetration tester provides this analysis, giving clients verified, reproducible vulnerabilities that may be appropriately prioritized.
Therein lies the difference between vulnerability sportsmen and vulnerability hunters. Sportsmen are out to get the biggest trophy for their wall, so they hunt endangered species with bazookas. After the kill, they are done. A hunter respects the hunted, learns about it, and eats it. In this metaphor, that means identifying the cause of the vulnerability and even exploiting it. Sportsmen go quail hunting blitzed and blast their friend in the face with a shotgun. Hunters track prey silently through the forest and kill cleanly with a bow and arrow. Vulnerability Sportsmen fuzz blindly and don’t even sort out the bodies, but instead print out stacktraces, register dumps, disassembly (in open source products, no less) and other random data that doesn’t even identify the vulnerability. The true hunter finds the code that is the cause, identifies how much control the attacker has, whether it may be exploited, and proves it if there may be any doubt.
In the current security climate where posting of working exploit code is frowned upon, there needs to be an alternate mechanism that provides accountability and transparency in security vulnerability reporting. Otherwise we are left with the “he said, she said” debate between vendors and researchers where neither party shows their cards and the validity of all independent security research is harmed in the process. A trusted third party vulnerability verifier or broker with actual vulnerability exploitation expertise would help rescue the process of independent vulnerability disclosure. That needed service is hopefully not too far away.
Mangoboy
November 29th, 2006 1:55 pmThanks Dino. It’s good to see the blog flame wars and meeting plugs punctuated with a bit of actual wisdom every now and again.
sp
November 29th, 2006 4:33 pmI’m neither a vulnerability hunter nor a vulnerability sportsmen so let me ask you a question. You dismiss the work of people you describe as vulnerability sportsmen but I can see at least two positive aspects of their work.
First, a software project (open source or otherwise) can work with every published bug. The programmers don’t need to know intrinsic details about the nature of the bug and how the bug can be abused. They just have to know that the bug exists and how to reproduce it reliably. If they have that information they can find the bug in their code and fix it.
Second, the vulnerability shot by the sportsmen is dead but still up for autopsy. Even if sportsmen find the bugs, the hunters can still analyze them and see whether they’re exploitable. The total number of vulnerabilities the sportsmen + the hunters find is bigger than the total number of vulnerabilites only the hunters find. Therefore the work of the sportsmen gives the hunters a bigger base of vulnerabilities to work with.
Dino Dai Zovi
November 29th, 2006 5:53 pmHello sp,
I do not mean to dismiss any bug finding work as not being a positive contribution. I am simply looking for more accuracy and transparency. If the exploitability of a vulnerability has not been verified, state as such. If the advisory says the issue is exploitable, I want to be able to believe it. Vendors and users need to be able to differentiate a wormable, reliable, remote vulnerability from a remote DoS that cannot lead to code execution through any publicly known technique.
Joost Pol’s kernel bug advisories (e.g. http://www.pine.nl/press/pine-cert-20040201.php) are great examples of how to clearly demonstrate that a complex security vulnerability has been researched and verified to be exploitable, thus accurately reporting its risk.
dre
November 29th, 2006 8:47 pmi blame capitalism.
plus, we have a lot more to worry about than vulnerability sportsmen. how about organized criminals?
i would like to make one point, as you metaphor rocks and you are dead on for all those issues otherwise.
my point is that vulnerability hunters should not stop from seeing the big picture. responsible disclosure and private disclosure are not always the right things to do. cow-towing to corporations will never allow progress in the vulnerability disclosure process.
sometimes exploits are necessary to move things forward, especially if a vulnerability is publically discussed but no interim fix or patch is made available. this allows plenty of time for blackhats to do the exploit work while joe administrator/user sits and waits. this is not fair or just.
reversely, when a patch exists and no exploit or vulnerability information exists, how are administrators and risk assessers supposed to test? now we have a situation where blackhats can binary diff the patch and come up with an exploit and hide it from the public, but now the public has no way to verify. nor can they protect themselves without strict, clean, secure patch management. not to mention most patch management systems (configuresoft aside) don’t work with remote users well or at all.
this also goes back to the whole white/black/gray arguments. white hats are sellouts because they get paid for the “expertise” (measured or not - hey let’s talk about CISSP) by corporations/government to secure insfrastructure/applications. blackhats get paid in fame or profit by trading/selling exploits or working for online organization crime.
you won’t like my solution, but i’ll give you my opinion on this. don’t be a whitehat if you find something with a client. break your contract. post a working exploit or all the information you know right away to full-disclosure from a remailer. never brag or talk to anyone about it. then let the rest work its course.
i call it ‘irresponsible disclosure’.
Thomas Ptacek
November 29th, 2006 8:55 pmIt’s hard enough as it is talking to a less-than-clueful vendor and trying to explain memory corruption and why it’s a serious threat.
It gets much, much harder when people salt the earth with findings that cast doubt on how serious the work is. I don’t want to have to bill 2 days of exploit development work just to prove that I’m not full of it.
Chris_B
November 30th, 2006 2:52 amDino,
Nice post. You also nailed the root of the problem with the two fellows from Secureworks rather nicely.
dre,
I for one dont like your solution and I cant imagine anyone with repeat clients who would. As far as blaming “capitalism” I’ll just give you the benefit of the doubt and assume that was supposed to be a joke.
I for one look forward to the day when the whole hat thing is passe, when “vulnerability hunters” are standard practice professionals verifying software much in the same way that any business uses an accountant to verify their books. It aint glamorous, but the fact is it shouldnt be.
djm
November 30th, 2006 7:04 amAs a software developer, I don’t care whether questionable bugs are exploitable or not. I’ll just kick myself for missing them, fix them quickly and be grateful that someone found them and was good enough to give them away. If this someone is just doing it for ego or fame, I don’t really care.
And yes, I have had NULL pointer dereferences (among other stupidities) sent to bugtraq@ for software that I help maintain. It sucks, but having people looking critically at your software, effectively for free, is a good thing regardless of their motivation.
I can see how it could annoy people if you make a business out of building real exploits, but a false report of exploitability can be pretty easily debunked.
Thomas Ptacek
November 30th, 2006 11:06 amIt’s not just annoying.
I feel like I’m pretty careful about not sniping at the way people disclose legitimate findings. So if you want to take your new Windows remote and plaster an exploit across Bugtraq and send a copy to Brian Krebs, I’m not going to say much. If you want to embargo it for 10 years, or sell it to the Russian Mafia, or whatever — we don’t do those things, but we don’t get to make those calls for other people.
But when you make up bullshit findings — and I’m not pointing any fingers specifically — you do more than annoy people. You make our lives harder. It means that when we find problems and get registers under control or find a crypto weakness or whatever, we now have to write exploits to convince people that attacks are real.
I don’t like billing people for writing exploits when all anybody needs is a simple patch to fix a problem. To get around that silliness, researchers need credibility. People who make shit up are spending credibility they didn’t earn, and costing all of us.
alastair
November 30th, 2006 1:41 pmhttp://alastairs-place.net/2006/11/dmg-vulnerability
If you’re a bona-fide security researcher, that should annoy you because lmh has damaged the reputation of the entire industry. As a developer, and as someone who does know what they’re about, it annoys me that lmh should get credit (not to mention all the publicity) for finding a security exploit when he did such a poor job of researching it.
On top of which, there’s always what he wrote about me on his blog; see:
http://kernelfun.blogspot.com/2006/11/more-mokb-20-11-2006-related-news.html
if you haven’t already.
Josh Daymont
November 30th, 2006 3:55 pmNothing regarding the motivation of vulnerability researchers has changed from what I can tell in the last 10+ years. There has always been people in it for the fame, usually fame for building the brandname of some new security company or research group, others mostly in it for the short term financial payoff. Technology changes. People as a group do not.
Thomas Ptacek
November 30th, 2006 4:07 pmI feel like it’s gotten easier for people to come up with stuff they can publish.
Amrit
December 1st, 2006 12:26 pmNice post! I agree with all the observations you made Dino.
As Thomas notes, It has certainly easier for people to come up with stuff they can publish and the speed at which the noise flys around has increased dramatically as well. Lots of noise out there.
I wanted to focus on the last paragraph of the post…
How does this vision of an independent 3rd party actually play out?
What motivation would an indepent 3rd party have to provide the level of transparency and accountability that is needed (I can think of tons of reasons), what motivation would vendors like Oracle have to cooperate, and more importantly what motivation would researchers, especially the deviant ones, have to work with them?
ivan
December 5th, 2006 11:10 am@Amrit: It does not matter. If some researchers and some vendors exercise a high degree of transparency and accountability in the process of discovering and disclosing vulnerabilities they will set the bar for the others, in the end it is the users who will determine who’s playing fairly and who’s just hiding behind bombastic 0day announcements, ridiculous press releases and smoke screens
I am not particularly fond of ‘independent’ 3rd parties (because there are none in sight and apparent no motivation to become one). I believe that if vendors and researchers (at least some of them) embrace the idea of transparency and accountability there would be no need for a 3rd party anyway.
Leave a reply