Cigital Ponders: Is Penetration Testing Security Testing?

Dave G. | April 9th, 2008 | Filed Under: Development, Industry Punditry

My initial response was: duhhhhhhhh

My second response was: uhm yes.

My third response was: Maybe I should read past the subject before I respond.

The short answer is still in the yes category.

The longer answer is that it is a form of security testing.

The blogging (and now featuring blockquotes) answer is:

Some people start “Security Testing” by buying and using a pen-test tool on project. Such tools uncover security vulnerabilities (though they seldom help with root cause analysis or even obtaining double-digit code coverage).

I would argue that uncovering security vulnerabilities is a major motivation for security testing. or Security Testing. Or even “Security Testing”. It is great to talk about Root Cause Analysis and code coverage, but i’ll take single digit code coverage over zero digit code coverage. Pen test tools are what I like to call Least Common Denominator testing. It will find the flaws you can expect an unsophisticated attacker to uncover. It is an excellent place for people who are “just getting started”. Because they don’t have anything in place, and want to hit the ground quickly. Rather than quote the next piece, I will try to sum it up. It is a classic security debate about blackbox/white box and requirements. If you read this blog you have heard it 1000 times and/or you already don’t read my posts.

The point I will nit pick on is:

Because black box tools to a large extent run canned tests they will not satisfy my security testing goal (see previous entry) of having run tests that one traces back to requirements. ‘Requirements that one created as a result of doing risk analysis that determines exactly what behaviors (and their impacts) should be avoided were the software attacked. Arguably, security folk have “cached” this risk analysis and these implicit requirements in the pen testing tool. Fine, this is that small benefit that I mentioned pen tests do provide. And, they DO find bugs. Again, this is at best a degenerate case of security testing in the same way running a fuzz testing tool is a degenerate way of conducting functional testing.

I’m really not sure this is “cached” or arguable. These tools discover flaws that are generally applicable to 99% of the requirements applications that need to get tested. Of course there are exceptions. But those are generally, you know, exceptions. Pen-test tools are useful. They identify a lot of vulnerability quickly and in a cost effective manner. Even if some of the flaws it finds aren’t applicable to a given app, most of them will be. And you can always say “Not going to fix it” for the flaws you don’t care about.Finally, your tests shouldn’t be limited to the tests that map back to requirements. Maybe that is controversial, but honestly, most of the times security requirements are:

  1. Poorly documented
  2. Vague or Poorly articulated
  3. Non existent (Especially if you are in the “just getting started” phase)

When you are specific about security requirements, you are in the battle of staying current against attacks. When you are vague about security requirements, you become test cases become all inclusive.SDLC is great. If you are just getting started, you need to be pragmatic.

My final response: Pen test tools are not pen tests. both are forms of security testing. If the question is, “Is it enough?” or “Do these things work?”, then we have a different conversation.

Viewing 2 Comments

    • ^
    • v
    It appears that everyone uses the term "penetration testing" in ways that convey multiple & different meanings and that creates a bit of confusion. In the context of Cigital's blogpost -and generally among the CS and software engineering crowds biased towards SDLC as the "right thing" to "really" improve security- penetration testing is implicitly assumed to mean "finding bugs in software artifacts" and "penetration testing tools" are software artifacts that find bugs in other software artifacts.

    That interpretation of the term "penetration testing" is substantially different from the practice aimed at finding weaknesses in an *organization's security posture and exploiting them* as an attacker would (or as close as possible to that) and from the tools that assist that practice.

    In the later context, penetration testing is much more than finding bugs in software and it may/should not even be considered a part of the software development lifecycle. There are tools that fit both definitions of "penetration testing" and I posit that they are both quite useful.

    ...or putting it in another way: l0phcrack and toneloc are definitely considered penetration testing tools within a large set of security practitioners (hm ok, I'm not sure about Toneloc) yet they provide 0% code coverage... are they useful?
    • ^
    • v
    Dave,

    You have a crucial point. Penetration testing is a very common and valuable technique for security testing. The question I intended to ponder was more accurately, "Am I done security testing after I've done a penetration testing?" I believe the answer is no.

    I'd like to respond to your notion of security requirements. I agree that documenting security controls/features/widgets is untenable. Almost every team is ill-equipped to do so. What teams do have--uniquely in fact--is an understanding of what failures would impact their system's business objectives. Security requirements (and misuse/abuse cases) should focus here, not on the technology-specifics of a particular control (use SHA-256 for...). Technology specifics can be pulled from security standards, reused across the organization to relieve individual teams of this burden.

    This doesn't let teams off the hook for specificity. Once a stakeholder has figured out which impacts pose the greatest risk from intentional attack, they can engage in a rather detailed process of specifying how the application should resist attack. This proceeds much like performance and scalability requirements elicitation--entwined with design and architecture discussions. While a multitude of tactical vulnerabilities could be fleshed out by each attack, they shouldn't be at this level. Leave that to test design and test case creation. At the conceptual level, attacks just don't change that often. They shouldn't produce undue burden on the requirements management process.

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus