Seven Deadly Pen Test Sins
Dave G. | March 11th, 2008 | Filed Under: Industry Punditry
This is a quick list of sins that I think most people that do this are or have been guilty of in the past:
- Managing Time. One of the most common yarns about the difference between pen testing and hackers is that pen testers have a limited amount of time to look for vulnerabilities in a small window of time in the applications’ life. As a result, time is precious. Knowing when you are on to something that you can confirm in a reasonable period of time is probably the biggest place where good pen-testers go bad. It is really easy to turn a pen-test into a research project. And while you toil away on a tiny sub-section of the application, you may never get to that remote code execution flaw that is lurking elsewhere. This is also where coverage vs. depth plays a huge role.
- Smugness. Devaluing findings that customers care about. Yes, XSS and CSRF are lame findings to people used to exploiting memory corruption or even compared to sql injection and auth bypass. This also extends to “I found one, let the dev team find the rest of them”. Smugness can also be extended into overconfidence. And overconfidence equals underestimation. This all results into missing vulnerabilities that you needed to find.
- Never understanding the app. It is easy to just treat an application as a series of inputs, and not bother to understand what the application is actually for or what it is actually doing under the hood. Good penetration testers are often trying to get into the developers’ heads.
- Over-automation. While I am a big fan of utilizing tools to make people more effective, there are two problems with relying on these tools:
- They generally create as much work as they eliminate. False positives in the popular web app scanning tools are still common enough that you waste a lot of time, especially on a small website.
- After you run them, you don’t have any better understanding about the application. It encourages #3.
- Sloth. This raises its ugly head in a couple of ways. It is usually either in avoiding the difficult parts of the test or conversely the easy parts of the test. Total human nature. One is hard and the other is boring. As a result, usually the team that comes in after this sinner finds serious flaws that are either hard-core or are embarrassing.
- Stagnation. Given the difficulties of the job, it is easy to not evolve one’s skillset. This is compounded by the fact that even in 2008 there still aren’t enough resources out there to keep evolving. This is also an organizational problem inside of every company. It is why the evil M word raises its ugly head.
- Communication/Soft Skills. Where it’s a project manager or the customer, you need to understand what the customer cares about, manage their expectations, and oh yeah… if you can’t actually care, at least pretend to. Lots of people think that doing a good job is simply breaking the app. That is enough until someone who is as technically savvy as you rolls in and also has that polish (you know, the one that you can’t stand).
Stefano Zanero
March 12th, 2008 7:03 amGreat post Dave
bobdole
March 12th, 2008 7:59 pmoh come now, you just posted that so that I would continue to read this site even though I see no content day after day…
Dr. Neal Krawetz
March 13th, 2008 9:15 am“Seven Deadly Pen Test Sins”? You have a good list, but what about:
- Get it in writing. If you don’t have a written agreement to do the testing, signed by someone with the authority to grant permission, then don’t start. (Don’t do the crime if you can’t do the time.)
- Scope and Target. You brushed this topic in 1, 2, and 7. Basically, you need to know what to look at and what to look for. It’s great to identify that they are using an F5 load balancer and showing that there is an inherent vulnerability in them. But unless the customer is F5, there is no point in doing a deep pen-test of the load balancer.
- No plan. A good pen-test should end with a map of the system, dependencies, etc. If you’re testing a network, then there should be a detailed map of the network discovered by the pen-tester. If you’re testing an application then you should have a design architecture as seen by the pen-tester. Without this, the customer will know that the tester was just bouncing around from random exploit to random exploit.
Vitaly McLain
March 13th, 2008 1:11 pmbobdole: Well, that’s what RSS feeds are for
Great post, though I think because I try not to be guilty of #5 I become guilty of #1.
sigsegv
March 14th, 2008 8:30 amDr. Neal Krawetz: I’m going to have to disagree with you here. I’ve had more success with semi-randomly going through a network letting my intuition guide me rather than following some overly “structured” approach.
For me, hacking is a creative process rather than a 9 to 5 day job.
Vitaly McLain
March 14th, 2008 9:10 amsigsegv: I think you’re right, but there are also multiple approaches to pen-testing. Some people spend months at one client, meticulously mapping a network. Others, due to various factors, spend a week and have to take a less formal approach. I think as long you do your due-diligence in scanning and documenting the network in the beginning (at least as far as IPs/ports), you can responsibly sample and creatively assess it without being too structured.
Tyler Shields
March 14th, 2008 10:44 amThis is a fantastic post. I’ve written before about some of the qualities that make a penetration tester a good one, but over time even an expert can fall prey to the problems you listed out. Even the best can get lazy.
Sven Weizenegger
March 15th, 2008 4:20 pm@sigsegv
in term of QA is the pure hacking (more chaos related ;)) the wrong approach. a good pentester should work with prepared checklist (testcase description., passed/failed criteria) and a well defined method and more structured (reusability and repeatability ). no one says that a pentester with a QA background cant hack with creativity…
to point number 7: a pentest should understand and explain the BUSINESS IMPACT of a vuln.
Qualities of good pen-testers | tssci security
March 18th, 2008 12:15 am[…] posted last week on the Seven Deadly Pen-Test Sins, with follow-ups from two other blogs I’m aware of: Mike Andrews and BlogFranz. This should […]
Leave a reply