Mozilla vs. Klocwork: 611 “defects”, 72 “vulnerabilities”, 3 verified bugs, 99.5% useless?

Thomas Ptacek | September 14th, 2006 | Filed Under: Defenses, Uncategorized

My readers can do a better job on the technical details of static analysis tools than I can (I can rock the mic with yacc, but did I miss the paper that solved the halting problem?). I’ll just say that 2-3 bugs —- here defined implicitly as “coding errors that have an operational impact” —- 2-3 from 611 defect reports is a damning statement.

One gets the impression that among the “71 potential vulnerabilities found” (I assume a subset of the 611 defects), many were “overflows” that actually turned out to be out-of-bound reads, apparently often in string comparisons. How many of the Klocwork findings were “potentially” exploitable, even on paper?

The publicity Klocwork got by Slashdotting their Mozilla “findings” seems in danger of backfiring.

O’Callahan’s sense that testing tools are beating static analysis tools seems sensible. Code-informed testing (“model-checking”) is the future. It’s a good post; read it.

PS: Fortify. Ounce. Coverity. I curse you for talking SNI out of static analysis in 1996, Tim. You and your children and your children’s children! For 2 weeks.