Vulnerability Research In Numbers

Thomas Ptacek | April 5th, 2006 | Filed Under: Disclosure, Industry Punditry

Lindstrom’s still at it. If you haven’t been paying attention, Peter Lindstrom believes vulnerability research should be outlawed, because of this syllogism:

  1. There is an indefinite, practically inexistinguishable supply of security-relevant bugs.
  2. The disclosure of any one of those bugs imposes cost on the industry.
  3. Therefore, vulnerability research is pointless (#1) and damaging (#2) and should cease.

Obviously, the premises are broken. You can’t extinguish errors, but you can suppress them.

The cost to a research team of finding a remote code exec flaw in a popular target is increasing. And less is spent (in time, effort, and money) on research than you think. I cleaned up and crunched the SecurityFocus bug database and came up with the following:

  • 2685: CVE numbers assigned (but not confirmed) for 2004
  • 1229: The subset of these with SecurityFocus Bug IDs that weren’t obvious XSS/PHP detritus
  • 1004: Non-vendor, non-anonymous credited findings therein
  • 581: Distinct credited researchers in 2004 according to SecurityFocus
  • 1.7: Mean findings per researcher in 2004
  • 1-5: Findings for a typical researcher in 2004
  • 60: Findings from Luigi Auriemma, the most prolific researcher in 2004
  • 8: Findings from Nick Gudov, the 10th place researcher in 2004
  • $130,000: SWAG fully-loaded headcount cost of a mid-level FTE researcher
  • $75.5MM: Crazy-talk ceiling on dollars spent for vulnerability research in 2004 (if this number was true, a single published “real” finding in 2004 cost $61,025 —- so it’s obviously much lower)
  • 140: Security companies in the public portfolios of 120 well-known venture capital firms
  • 10%: Annual vulnerability research dollars as a percentage of a SWAG (average 5MM/per) total investment in security companies
  • 22%: As a percentage of SYMC’s 2005 R&D expense
  • 12%: Or Google’s
  • 1%: Or Microsoft’s
  • $500: Cost of a default-install Microsoft remote code exec if it takes one day to find, as it did in 1997
  • $10000: Cost of that vulnerability if it takes a month to find; note: nobody publishes 12 MSFT remotes a year.

You could SWAG dollars spent on QA across the top 20 technology vendors (estimate % of R&D expense attributeable to headcount, then take an industry standard ratio of developers to QA, and extrapolate). That number will obviously beat public vulnerability research by more than an order of magnitude.

(Why 2004? Because I assume the numbers have settled down by now.)

Viewing 8 Comments

    • ^
    • v
    I believe the question, at some point, becomes, "Are vulnerabilities being introduced into major code-bases at a rate faster than they are being discovered?". If they are, we're perpetually chasing our own tail. I think we've sufficiently proved, thus far, that attackers have knowledge and exploitation techniques of vulnerabilities (in most cases), months prior to a bug's discovery by the community at large.

    Maybe this has changed in the last 5 years, but I doubt it has. Unemployed college kids will always have more time for vulnerability research than any other demographic. Couple that with the diverse range of available hardware at your average .edu, and the fact that amongst the kiddies there will be a few 'diamonds in the rough', who are actually capable hackers, and you quickly realize who is the cat and who is the mouse here.

    Full disclosure/vulnerability reporting obviously drives the IS economy, without it InfoSec would be a dying breed (wouldn't you love to see IDS vendors having to create 'blind' signatures solely by watching network traffic, software vendors doing their own in house code auditing (Hi Microsoft!, Hi Oracle!, Hi Sun!)?)

    In all reality we, as security professionals, are being paid to keep our clients as close to the curve as possible. FD makes the community, as a whole, better and more stable, but we seriously are all just playing with ourselves.

    And one can probably make a fairly solid case for the fact that exploit methodology and proof-of-concept disclosure inherently causes more harm than good, it's undebatable.

    I just find it hard to believe that the hundreds of millions of dollars, that the disclosure of exploit methodology for MS03-026, MS04-011, MS05-039, CVE-2002-0392, CVE-2002-0656, etc., has caused in the last 5 years, can constitute anything but proof that disclosure of exploit methodolgy fundementally costs more, than partial disclosure does.
    Dr. Strangelove | 04.06.06 - 12:46 pm | #

    I'm not sure you've succeeded in drawing a line between hundreds of millions spent and disclosure of exploits. Want to help me understand your point better?
    Thomas Ptacek | Homepage | 04.06.06 - 2:37 pm | #

    I believe all of the worms designed to propogate around those particular vulnerabilities were based on publicly available exploit code. Are you trying to tell me that security spending would not have been drastically decreased by a lack of public knowledge of specific exploit techniques and methodologies here? No code == No worm.

    I suppose the counterpoint to that claim would logically be "Well, even if there was a decrease in overall IR spending as a result of a diminished threat base, you will still see comparable Information Security costs as a result of a new found need to be more proactive in your deployment of security appliances."

    I'm still willing to bet, in spite of that statement, that the cost of establishing and maintaining the integrity of a data environment becomes drastically larger when faced with a Full Disclosure vulnerability research policy; even if it adheres to some sort of esoteric "disclosure timeline" ala RFPolicy.

    It all comes down to a qualatative/quantatative analysis of the threat-base. Public exploits and techniques mean a broader, yet less skilled, threat base which results in a smaller portion of the average security budget being spent on proactive countermeasures (Firewall/IPS/VPN) and more being spent on reactive threat response countermeasures (AntiVirus, a patch management solution, outbreak containment, forensic investigations etc.). The flip side of the coin is, without public exploits and techniques the threat base shrivels, but retains a much greater potency. The insuing result is a greater emphasis being placed on proactive countermeasures, as opposed to reactive ones, which I think we'll both agree are more expensive. Weighting to the side of proactive defense, packet filters, content filters, end point encrpytion, and so on has got to be inherently cheaper than having to run after your own tail, chasing Nachi/Blaster-esque worms in circles around your Win2k environment.

    Sure, back in '95, people were stealing much more interesting stuff from higher profile machines, that were insanely more vulnerable than your average machines are today. But I don't think jsz or Mitnick were jumping at the chance to post Sun/Motorola/Nokia source code to usenet. Maybe they were/did and I'm too young to remember.

    I conclude that while the threat base will become smaller, and more potent, the cost of defending your average network will go down; as there will be fewer threats to respond to.
    Dr. Strangelove | 04.07.06 - 11:01 am | #
    • ^
    • v
    Is IPS "proactive"?

    Is antivirus really part of the "security budget", or is it just part of the cost of deploying desktops? Either way, what does it have to do with vulnerability research?

    If you factor AV out, firewall spending dwarfs "reactive" security spending.

    At any rate, the thing that I'm pretty sure DOESN'T work is "half-assed" full disclosure, like we had with CORE/INFOHAX. What that does is guarantee that the bad people get the info before the good people. Disclosure should either be full-assed or no-assed.

    You know what side of this I come down on. There's no no-assed disclosure solution that doesn't leave us dramatically less secure, by effectively ending all vulnerability research of any stripe.
    • ^
    • v
    Hmm, depends how granular, and efficient, your protcol-based anomaly detection is I suppose (and whether or not you're configuring it to do active resets) ;) If it's hopelessly broken, then no, it's just as reactive as an IDS. If it works for most cases, even when dealing with in-the-wild 0day, then you can call it proactive because it stops an attack prior to having packets forwarded to the destination device. It's all relative, Vulnerability Research requires a tangible and justifiable proof of ROI, like everything else does. Basically, all I'm saying is that while Lidstrom's overall goal is fairly ludicrous, he's not too far off in the points he makes to get there.

    You've got me on this point, I'm kind of in a corner. Do viral threats exist exterior to the general 'security' space, absolutely. Are remote/local exploits a mandatory requirement of self-propagated code, absolutely not. The existence of email and P2P filesharing networks will insure propagation methods external to remote exploits for years to come. I'm going to agree with you here, because if you remove Anti-Virus from consideration as a 'security appliance', you're absolutely right that Firewall spending dwarfs "reactive" security spending.

    I think we have a divergence on the disclosure issue though. How can one truthfully argue that providing exploit details to the community at large is doing more of a service, than a disservice, to end-users, especially when based on current disclosure processes 98% of the major bugs (Read: Non-CGI, Non-XSS, Non-SQL Injection) aren't released without a patch hand in hand.
    More to the point, how can you argue that Full Disclosure isn't costing businesses more than "'Half-Assed' Disclosure" would?

    If someone can take the patch and reverse engineer it, good for them! Other than strongarming pennywise executives, what overall good is served by releasing the exploit methodology and/or code?

    It would be interesting to trend the number of break-ins as a ratio to distribution of machines on the internet in the 10-12 years that full disclosure has been popularized, and the 10 years prior. I'm willing to bet that the percentage of machines getting owned even when _everyone_ was ownable, prior to 1994 is only a fraction of the amount of the percentage of machines which have been getting owned in the last 10 years as the result of publicly disclosed exploits.

    It might be safer to have full disclosure in the mix, but it will definitely cost a shitload more to do it.
    • ^
    • v
    I want to write a more detailed response to your post, but I can't get past "98% of major bugs aren't released without a patch hand-in-hand".

    Are you saying most bugs are released without patches? Or with them? And where do you get the number from?

    Either claim (the good one, where we have a consensus minimum standard of disclosure ethics, or the bad one) is pretty dramatic and would be valuable to refer back to. Pick one and defend it.
    • ^
    • v
    Hmm, depends how granular, and efficient, your protcol-based anomaly detection is I suppose (and whether or not you're configuring it to do active resets) ;) If it's hopelessly broken, then no, it's just as reactive as an IDS. If it works for most cases, even when dealing with in-the-wild 0day, then you can call it proactive because it stops an attack prior to having packets forwarded to the destination device. It's all relative, Vulnerability Research requires a tangible and justifiable proof of ROI, like everything else does. Basically, all I'm saying is that while Lidstrom's overall goal is fairly ludicrous, he's not too far off in the points he makes to get there.

    You've got me on this point, I'm kind of in a corner. Do viral threats exist exterior to the general 'security' space, absolutely. Are remote/local exploits a mandatory requirement of self-propagated code, absolutely not. The existence of email and P2P filesharing networks will insure propagation methods external to remote exploits for years to come. I'm going to agree with you here, because if you remove Anti-Virus from consideration as a 'security appliance', you're absolutely right that Firewall spending dwarfs "reactive" security spending.

    I think we have a divergence on the disclosure issue though. How can one truthfully argue that providing exploit details to the community at large is doing more of a service, than a disservice, to end-users, especially when based on current disclosure processes 98% of the major bugs (Read: Non-CGI, Non-XSS, Non-SQL Injection) aren't released without a patch hand in hand.
    More to the point, how can you argue that Full Disclosure isn't costing businesses more than "'Half-Assed' Disclosure" would?

    If someone can take the patch and reverse engineer it, good for them! Other than strongarming pennywise executives, what overall good is served by releasing the exploit methodology and/or code?

    It would be interesting to trend the number of break-ins as a ratio to distribution of machines on the internet in the 10-12 years that full disclosure has been popularized, and the 10 years prior. I'm willing to bet that the percentage of machines getting owned even when _everyone_ was ownable, prior to 1994 is only a fraction of the amount of the percentage of machines which have been getting owned in the last 10 years as the result of publicly disclosed exploits.

    It might be safer to have full disclosure in the mix, but it will definitely cost a shitload more to do it. You also increase your creton factor by about 10+ fold.
    • ^
    • v
    Qouting dr. strangelove;
    "Are you trying to tell me that security spending would not have been drastically decreased by a lack of public knowledge of specific exploit techniques and methodologies here? No code == No worm."

    Yeah and worlwide medicine spending would have been drastically decreased if there was no knowledge about AIDS or cancer. Ignorance is bliss for many

    The diatribe about disclosure policies vs. infosec spending is pointless and suffers from severe logical flaws. It is closer to whining and wishful thinking that to a logical argument.

    Maybe it is time to get it into our heads: vulnerability research is social phenomena and something that does not require or necesarilly fit any given business model and hardly ever follows the methodology of a quasi-scientific discipline.
    • ^
    • v
    You should run the same numbers against a different data set for comparison.
    • ^
    • v
    Based on my own estimates from CVE, I think about 25% of issues are disclosed with a patch (or at least solid attempts at vendor coordination), and ultimately about 50% are proven fixable. The real numbers are probably a little higher, because CVE has very strict requirements about when to say that an issue has been acknowledged by a vendor.

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus