Penetration Testing: Dead But Not Really Dead.

Dave G. | December 24th, 2008 | Filed Under: Industry Punditry

Yah, late commentary.  Sorry, been a little busy.  Brian Chess kicked the hornet’s nest beautifully by declaring:

Penetration Testing: Dead in 2009

with:  

“Death doesn’t mean it goes away, it means it transforms. Pen testing will be reborn in the area of production monitoring and measurement,” Chess said. “The goal won’t be that failure is found and must be fixed. The goal is that failures will become a much rarer event.”

That is a great goal.  However, the goal of penetration testing (which in my world is synonymous with security assessments, so if you are going to get all semantic on me, go nuts, most of my customers use these words interchangably, and they are some of the most sophisticated puchasers of security), is not to prove you have a problem, it is a last minute check to find what you missed elsewhere during the construction of a given application or environment.  In the future, where security in the SDLC is considered mature, penetration testing will still be there for assurance purposes, but also perform continuous improvement, where the findings get looped back into the process so that developers learn about the latest security flaws that they aren’t defending their applications against.

Spending more in other parts of the lifecycle is essential to effectively manage security.  But, no matter how much you spend this year in other parts of your development lifecycle, you will have NO idea what still lurks inside of your application until a qualified party investigates it.  The success rate of any top rate security consultancy in finding critical vulnerabilities that customers think were worth the cost of the engagement in 2008 is going to be over 90%.  

What should really die in 2009 is the low hanging fruit findings that still plague applications going into production today.  SQL injection and Cross Site Scripting were all to prevalent in 2008.  

What I actually think is going to happen is that SDLC efforts are going to have an incredibly hard time in 2009.  Strategic security initiatives are expensive and time consuming, especially to developers.  If you are going to be effective at security in 2009, you are going to have to be low-drag.  Dev teams are not going to have the time to delay product releases, by increasing the overhead involved in cumbersome security at each phase.

The status quo is likely to remain just that over the next 12 months.  On our list of blog posts coming up will be one on lightweight SDLC.  I promise.

ps: if it isn’t at all obvious, it is in Brian Chess’ best interest to say that penetration testing is dead, and it is in mine to say that it is alive and well.

pps: I just read this and I am apparently really rusty at writing blog posts.  Please be patient as I evolve from 16 year old emo livejournal quality to the slightly higher Matasano quality.

ppps: Read Ivan’s excellent response and Brian’s response to Ivan’s response.

Viewing 8 Comments

    • ^
    • v
    It's raging confirmation bias. Fortify sells embedded Security QA. Hence, Fortify's customers have reached the point where they've matured enough to actually buy tools in-house and do source code review, and even actually fix crap prior to go-live.

    I mean it! Security testing. During the development process. No, for reals! Srsly! Why are you laughing? These are the kind of companies that actually have security guys in their dev teams. The ETradestercardachovias and the AmaGoogaBays. People with the budgets to buy tools that do things like create pie charts of all the SELECT statements your apps have.

    But is this everybody in the F500? Hells no. Now, to Bri, it sure feels like it, because they're the ones buying his lattes, and goshdarnit, it feels good to be right. People have been saying it was cheaper to test on the front end of the SDLC for years, and here he is, with a software company of his vewy vewy own, doing just that.

    Prob is, Fortify's customers are probably a fraction of a percent of the companies in the world doing Serious Business on the Interwebs. Even inside Fortify's own customers there are KLocks upon KLocks of code in whose dark bowels no one will ever dare to inflict the eColonoscopy of a source code scanner. To say nothing of the Legacy Fortran App That Dare Not Speak Its Name.

    So, enter pen-testing. It's ugly, it's unscientific. It's imperfect. It Does Not Offer Complete And Verifiable Code Coverage (tm). It may or may not succeed. But it confirms, within reason, that a very smart person spent some time trying to break your (app|network|building|crypto|drm|kneecaps), and failed. Or succeeded.

    And until we reach the point where every app has been carefully verified by Fortify's Ironclad Patent Pending Business Software Assurance Process (FIPPBSAP for short) before ever ever ever going prod, guess what?

    You need pen testers. Suck it up.
    • ^
    • v
    Do not fail to notice how the CSO author gladly proselytizes Fortify's new self-serving industry achronym "BSA" for business software assurance.....if Fortify's claim is that security assurances are now embedded in an industry-wide QA-gone-mature process, then they ought stop trying to differentiate themselves as a discoverer of a new niche process. If independent attack and pen efforts are dead, then any source control tool can commoditize their value-add for a far lower per seat cost.
    • ^
    • v
    I agree with the author that,m be frank, evensecurity process engaged in SDLC is becoming a trend and need, it is not a replacement of Penetration Test. It is because system designer has not got a priority to build a system with thought like a hacker to attack to system(except security software/system).

    However, I would like to bring out the issue and the low quality of penetration test nowadays are those "Click-Once" scanning becomes the major trend without studying and harvesting system infrastructural information, studying the possible weaknesses and conducted exploitation.

    From my perspective, vulnerability could be classified into two major types:
    1. Systematic - It is related to OS/devices/software itself vulnerability. We may need to go for "Catch and Patch" approach for remedy.

    2. Design/Implementation Flaws - the device itself may not exhibit any vulnerability, but if connected to another one or implemented/designed, it gives chances for malicious attack.

    Hopefully, companies recruiting pentest service should engage this kind of approach, not just take a scan-once report for their homework to their headquarter and authority.
    • ^
    • v
    Brian Chess needs a reality check. Penetration Testing is not going away anytime soon. Source Code Analysis has it's place within the SDLC, but it is not the cure all for application weaknesses. Security is undergoing drastic changes as week speak. As many companies try to turn security into a commodity, the value is brought down.

    What do I mean by this? Companies are hiring less skilled testers, it's becoming a point and click world and so many vulnerabilities and weaknesses are being missed. On numerous occasions Jr testers are being promoted to Sr testers within 6 months to a year. Do they have the proper skillset? NO! They run WebInspect, AppScan, and import Nessus, Retina and Qualys into CORE Impact and Boom. You are Secure!

    Whats the problem? Application Scanners find 20% of the vulnerabilities and Network Scanners find 50%. So that means there is another 80% of application weaknesses and 50% of network weakesses not discovered or tested.

    Now Brian Chess claims Fortify will fix all the application vulnerabilities.........NOT REALLY!!! It has the same issues, it won't find Logical errors that lead to vulnerabilities.

    THE TRUTH......If you combine Threat Modelling, Source Code Analysis, Application Pen Testing and Network Pen Testing. Roughly a 2 week process.....you will achieve the best results. and by Best I mean 80-90%. Why not 100%??? First of all no one can be 100% secure. TOO MANY unpublished vulnerabilities and malware out there.

    Testers skillsets vary drastically, so desired results vary depending on the tester, time of year, mental state of the tester, how quick the project is being rush, how limited of scope is the project, etc.

    Will Penetration Testing Die????? NOT ANYTIME SOON!!!!
    • ^
    • v
    Christ on a tricycle, Tommy!1!! If you can do a Threat Model, SCA, and App+Net Pen in 2 weeks I'm hiring your ass. Maybe that assumes a Big4-level army of interns and a handful of token hackers plus two partners to fellate the customer, for somewhere around the price of a chop-topped Bentley with white gold spinners?

    Still, dude has a point, even if it is in caps. Takes a lot of tools, time and testing to keep up with the level of Kung Fu of your average garden-variety Bulgarian botmaster or Eastern Bloc carding gang.
    • ^
    • v
    What you think about ISO 27001? A lot of company that need ISO compliance or PCI-DCC, needs penetration testing
    • ^
    • v
    It is not necessary so to worry about it.
    Technological progress not to avoid.
    If you are connected with computer technologies can always earn. In it also there is a main plus.
    • ^
    • v

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus