Cigital Ponders: Is Penetration Testing Security Testing?

Dave G. | April 9th, 2008 | Filed Under: Development, Industry Punditry

My initial response was: duhhhhhhhh

My second response was: uhm yes.

My third response was: Maybe I should read past the subject before I respond.

The short answer is still in the yes category.

The longer answer is that it is a form of security testing.

The blogging (and now featuring blockquotes) answer is:

Some people start “Security Testing” by buying and using a pen-test tool on project. Such tools uncover security vulnerabilities (though they seldom help with root cause analysis or even obtaining double-digit code coverage).

I would argue that uncovering security vulnerabilities is a major motivation for security testing. or Security Testing. Or even “Security Testing”. It is great to talk about Root Cause Analysis and code coverage, but i’ll take single digit code coverage over zero digit code coverage. Pen test tools are what I like to call Least Common Denominator testing. It will find the flaws you can expect an unsophisticated attacker to uncover. It is an excellent place for people who are “just getting started”. Because they don’t have anything in place, and want to hit the ground quickly. Rather than quote the next piece, I will try to sum it up. It is a classic security debate about blackbox/white box and requirements. If you read this blog you have heard it 1000 times and/or you already don’t read my posts.

The point I will nit pick on is:

Because black box tools to a large extent run canned tests they will not satisfy my security testing goal (see previous entry) of having run tests that one traces back to requirements. ‘Requirements that one created as a result of doing risk analysis that determines exactly what behaviors (and their impacts) should be avoided were the software attacked. Arguably, security folk have “cached” this risk analysis and these implicit requirements in the pen testing tool. Fine, this is that small benefit that I mentioned pen tests do provide. And, they DO find bugs. Again, this is at best a degenerate case of security testing in the same way running a fuzz testing tool is a degenerate way of conducting functional testing.

I’m really not sure this is “cached” or arguable. These tools discover flaws that are generally applicable to 99% of the requirements applications that need to get tested. Of course there are exceptions. But those are generally, you know, exceptions. Pen-test tools are useful. They identify a lot of vulnerability quickly and in a cost effective manner. Even if some of the flaws it finds aren’t applicable to a given app, most of them will be. And you can always say “Not going to fix it” for the flaws you don’t care about.Finally, your tests shouldn’t be limited to the tests that map back to requirements. Maybe that is controversial, but honestly, most of the times security requirements are:

1. Poorly documented
2. Vague or Poorly articulated
3. Non existent (Especially if you are in the “just getting started” phase)

When you are specific about security requirements, you are in the battle of staying current against attacks. When you are vague about security requirements, you become test cases become all inclusive.SDLC is great. If you are just getting started, you need to be pragmatic.

My final response: Pen test tools are not pen tests. both are forms of security testing. If the question is, “Is it enough?” or “Do these things work?”, then we have a different conversation.