Seven Deadly Pen Test Sins

Dave G. | March 11th, 2008 | Filed Under: Industry Punditry

This is a quick list of sins that I think most people that do this are or have been guilty of in the past:

  1. Managing Time.  One of the most common yarns about the difference between pen testing and hackers is that pen testers have a limited amount of time to look for vulnerabilities in a small window of time in the applications’ life.  As a result, time is precious.  Knowing when you are on to something that you can confirm in a reasonable period of time is probably the biggest place where good pen-testers go bad.  It is really easy to turn a pen-test into a research project.  And while you toil away on a tiny sub-section of the application, you may never get to that remote code execution flaw that is lurking elsewhere.  This is also where coverage vs. depth plays a huge role.
  2. Smugness.  Devaluing findings that customers care about.  Yes, XSS and CSRF are lame findings to people used to exploiting memory corruption or even compared to sql injection and auth bypass.   This also extends to “I found one, let the dev team find the rest of them”. Smugness can also be extended into overconfidence.  And overconfidence equals underestimation.  This all results into missing vulnerabilities that you needed to find.
  3. Never understanding the app.  It is easy to just treat an application as a series of inputs, and not bother to understand what the application is actually for or what it is actually doing under the hood.  Good penetration testers are often trying to get into the developers’ heads.
  4. Over-automation.  While I am a big fan of utilizing tools to make people more effective, there are two problems with relying on these tools:
    1. They generally create as much work as they eliminate.  False positives in the popular web app scanning tools are still common enough that you waste a lot of time, especially on a small website.  
    2. After you run them, you don’t have any better understanding about the application.  It encourages #3.
  5. Sloth.  This raises its ugly head in a couple of ways.  It is usually either in avoiding the difficult parts of the test or conversely the easy parts of the test.  Total human nature.  One is hard and the other is boring.  As a result, usually the team that comes in after this sinner finds serious flaws that are either hard-core or are embarrassing.
  6. Stagnation.  Given the difficulties of the job, it is easy to not evolve one’s skillset.  This is compounded by the fact that even in 2008 there still aren’t enough resources out there to keep evolving.   This is also an organizational problem inside of every company.  It is why the evil M word raises its ugly head.  
  7. Communication/Soft Skills.  Where it’s a project manager or the customer, you need to understand what the customer cares about, manage their expectations, and oh yeah… if you can’t actually care, at least pretend to.  Lots of people think that doing a good job is simply breaking the app.   That is enough until someone who is as technically savvy as you rolls in and also has that polish (you know, the one that you can’t stand).  
The results of these sins all generally lead to missed findings and an overall craptastic job. 

Viewing 9 Comments

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus