On The Different Types Of Penetration Tests

Thomas Ptacek | March 7th, 2007 | Filed Under: Industry Punditry, Matasano, Uncategorized

There are two types of “penetration testing” services, each different. I’ll spell them out, and introduce a “third”.

Network Penetration Testing involves a team attempting to break into your network or servers. This is what jumps into the minds of most firewall people, security admins, operations teams, and IT security groups when you say “penetration test”.

Network pentests involve tools, a (usually somewhat proprietary) grab-bag of tricks and exploits, network scanning, social engineering, port scanning, OS versions, Metasploit and CORE, or what-have-you.

The value of net pentests is identifying weak-leak dependencies between servers that you wouldn’t have thought of. For instance, a trust relationship between two otherwise locked-down machines, perhaps based on Active Directory credentials and local security policy or mapped network drives. These problems are hard to detect with automated tools, originate from the kinds of operational compromises that happen all the time in large enterprises, and are best smoked out with a set of “fresh eyes”.

Application Penetration Testing is the other type, and it’s what we tend to be talking about here. An app pentest isn’t usually looking at servers or the network. Instead, an app pentest team is given a piece of software to look at, with the assumption that “the servers we’re running will be fine as long as there is no horrible unknown vulnerability in this program”.

There are two common kinds of app pen-tests, which are really the same thing but are staffed so differently that you’ve got to call them out seperately:

  • Web Penetration Testing (or “web pesting”, as we like to say —- OK, as only I like to say) involves a web application: J2EE, ASP.NET, PHP, and the like. In a web pentest, your team is usually given a set of accounts on an app, at varying levels of privilege, and is tasked with finding OWASP-type vulnerabilities in the service. A web pentest will tell you if you have SQL injection problems, XSS, XSRF-able forms, bypassable authentication, and exposed source code.

    Web pentests are different from network pentests because they on a specific app. Candidly, web pentests are harder to perform than net pentests. On the network, you work from a list of known problems, checking them off a list. In a web app you’re doing QA, working to prove a negative.

    The tools of the web pest trade are web proxies like Paros, fuzzers, curl, and Python. Solid web pentests often involve custom code.

    Web pentests are becoming common; they’re de rigeur for any application which handles money, or has an Internet-facing audience, or is tainted by HIPAA, GLBA, or PCI.

  • Shrink-Wrap Penetration Testing (“product” penetration testing as @stake used to say) is the same thing, but for closed-source applications that don’t use HTTP as their transport. The big difference between shrink-wrap and web pesting is that web pesters start each engagement with the ability to talk to the application and lots of ways to figure out how it works; shrink-wrap pesters do not. Almost all shrink-wrap engagements involve custom code.

    It’s on shrink-wrap engagements that you start getting into things like reversing, differential debugging, protocol debugging and protocol fuzzing. Where web pentests turn up SQL injection, shrink-wrap engagements turn up overflows.

    Shrink wrap engagements are much less common, though they shouldn’t be —- whatever dollar amount you spend every year on signature IPS to keep up with Bugtraq, you should spend at least that much to stay ahead of Bugtraq. Smart enterprises do shrink-wrap tests for products they expose to third-parties and customers. Smart vendors do shrink-wrap tests for anything they ship.

When you talk to security testing providers, some things to keep in mind:

  • Net pentests, web pentests, and shrink-wrap pentests are all staffed differently. There are fewer talented web pesters than there are net pentesters. There are fewer shrink-wrap testers still. No large services firm is uniformly staffed with people who can do all of these services. A company with more than 20 testers that says all of their testers can all deliver a shrink-wrap project is lying to you.

  • The firms you talk to should spell out these differences. Be careful about companies that have a one-size-fits-all “Penetration Test” service on their rate card; these are radically different services with different cost structures and different value props, and if a firm’s marketing management doesn’t know the difference, neither do their engagement managers.

  • Different firms have different focuses. There are plenty of firms that specialize in net pentests, but who can keep a couple web pentests staffed year-round with competant people. On the other hand, there are also several firms who specialize in web pentesting and who can reliably staff app pentests. This matters to you:

    • The web pester at the network firm occasionally has to do network pentests. The web pesters at the web firm don’t. Guess which one has an easier time recruiting and retaining talent?

    • The web pester at the network firm knows the OWASP vulnerabilities and what’s being talked about at Black Hat. The web firm testers know what to look for first in a JBoss application versus a WebSphere app.

    I’m not being at all fair here, and I’ll take heat for it. Oh, well.

  • It works the other way, too. Jeremy and Dave are excellent network pen-testers and excellent app-pentesters, but I think they’re an anomaly. Counterexample: I am not a very good network pentester. Different skills, different disciplines, different degrees of patience required for different things.

And now I’ll say something transgressive, that is almost certainly going to get me in trouble, and all the Bay Area security people who think I talk too much and don’t publish enough research are going to find lots more fuel to say that I’m a blowhard:

Make-it-or-break it attributes for each type of tester:

  • Pen-Testers In General: cleverness, adversarial nature, competitiveness, ego: you should work exclusively with testers who love this work.

  • Network Pentester: Methodology.

  • Web Pentester: Experience.

  • Shrink-Wrap Pentester: Published research or shipping code.

Viewing 9 Comments

    • ^
    • v
    TP,

    The transgressive bit didnt come at the end. The thing which is going to upset alot of pester divas of all types is the fact that this stuff really is QA processes.
    • ^
    • v
    I agree, a lot, but from experience a shrink-wrap pentest usually requires more code than an average QA person can turn out in a tight time frame.
    • ^
    • v
    Chris_B, though pen testing can be classified as QA and the lines are becoming increasingly blurred between the two, QA tends to be more scripted than pen testing and usually doesn't require building customized tools each time. Product (or shrink-wrap) pen tests require a very different skillset, and they are few and far between.
    • ^
    • v
    Great writeup, Thomas. If I thought I could get away with it, I'd use web pester and web pesting in an article I'm writing now on webapp scanners.

    One minor nitpick -- I'd suggest that methodology is crucial for web pentesting as well. In fact, I think it's even more crucial than network pentesting.

    Mainly because a big web app actually has a much bigger attack surface area that has to be mapped out than your average network (assuming we're discounting web-apps from the network segment), and the details of that surface area are theoretically totally new to you each time. There are so many nooks and crannies that it's really important to have a good methodology to be repeatable and thorough. Of course, you need experience as well, I definitely agree with that.
    • ^
    • v
    Purely subjectively, the word "methodology" scares me. In the best circumstances, a methodology is a safety net, keeping you thorough and accurate even if your head isn't in the game. But if your head isn't in the game, don't test. People rely on your results to assure my mom's credit card numbers.
    • ^
    • v
    TP,

    OTOH a methodology can be good for people whose heads are too much in the game, forcing them to mind the shot clock and get the rest of the work done.

    Chris E,

    Even now a certain percentage of pesting can be automated and can spot the low hanging fruit. I expect that this percentage will rise over time. Of course human eyes will always spot more, but eliminating some problems before shipping is better than waiting for someone else to identify those problems for you.
    • ^
    • v
    Just the description of web pentest strikes me as silly... "A team is given accounts"... How about looking at the code, Sherlok? As Ranum beautifully puts it - you have an application you don't understand and you shell out money for someone who doesn't understand it, and doesn't even look at the code, in the hopes they'll make it secure. When you think about it it's just ludicrous. Don't give me the "simulate an attacker" talk, that's just idiotic.

    There are people hired to do these things couldn't program themselves out of a hello world and they're supposed to say what's broken with an application? What's broken is that some clueless person is shelling out money for "penetration testing"... How about shelling out money for code reviews and for having top notch programmers that understand security?

    How about spending your time coming up with solutions for secure coding and design of these things instead of searching for the 1001th variation of a flaw here and there? Yes, I know it's not so cool and you won't be a "mercenary" badass, but maybe you'll actually change the world for the better? I'm tired of this arrogant talk. It's always me, me, me - I can break, I can hack, all books suck (unless of course they teach how to pentest), all certifications suck, all software sucks, and no advice on how to actually improve anything. Blargh.
    • ^
    • v
    Anonymous:

    We agree! However, most companies just don't want to share source code to their applications with third parties. And most companies have a business need to have an independent third party review their application. For example, their customer might demand it. So, on one side, a company needs to have someone outside of its organization perform a security review, on the other side, they want to protect their intellectual property.

    Code reviews and penetration testing will find different bugs. Most companies that do one will also do the other.

    Some people are really good at software development and penetration testing. You are pigeonholing an industry based on some of the people inside of it.
    • ^
    • v
    Anonymous -

    First, the goals of a web app "pen test" can be different than trying to secure it. Sure, securing it is the ultimate goal, but 100% of the multi-billion dollar companies that I consult with do pen tests to confirm that they don't have any vulnerabilities. The reality is, I've never seen a web app without some flaw - however serious. These organizations usually DO have a security development lifecycle, have very intelligent and security-saavy developers, and incredibly layered security review processes. This doesn't mean that there will never be vulnerabilities, and no amount of developer training will ensure apps are always secure.

    Another thing I'd like to state is that I usually don't need the code. When I test an app, I can usually tell you what the code looks like behind the vulnerability. In my experience, rarely does a code review turn up new vulnerabilities - though it might locate more instances through root-cause analysis and point out other broken security practices.

    I guess my point for this entire post is to say the pen testing definitely does have value. Our customers go much further knowing where the issues are really at. It can then be used as a metric to improve future development practices and create better and more secure apps.

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus