More on Pen Testing
Dave G. | March 13th, 2007 | Filed Under: Industry Punditry
Bruce Schneier and mjr faced-off on penetration testing in the latest issue of Information Security Magazine.
Structurally, here are my issues with this piece:
It’s pretty clear that there was no real point counterpoint. These were two seperate articles,
Which makes them boring because they aren’t very far off in opinion,
Which would be fine, but it also sounds like neither of them have been anywhere near a penetration test since 1997
Bruce’s points really come down to his last graf:
There are two reasons why you might want to conduct a penetration test. One, you want to know whether a certain vulnerability is present because you’re going to fix it if it is. And two, you need a big, scary report to persuade your boss to spend more money. If neither is true, I’m going to save you a lot of money by giving you this free penetration test: You’re vulnerable.
Only the least sophisticated of customers buy a penetration test just to know that they are vulnerable. People genuinely want to fix these problems.
Let’s see what mjr has to say:
The only useful outcome of a pen test is the worst one: The pen testers walk in and demonstrate, conclusively, that system security is horrible. Then you’ve got a 50-50 chance you’ll end up with a mandate to fix it. Here’s the sad fact: Organizations with poor security already know it, and it is not going to be improved a great deal by having an outsider show up and point that out.
What about organizations that have good security and want it to get better? We have customers like that. Also, organizations with bad security might find this to be a good way to start improving it.
So what’s the realistic alternative to pen testing? It’s obvious: Have a good security design, and then verify that it is in place and working correctly. If your management wants to hire outsiders because they don’t trust you, or they think you’re stupid, hire outsiders to review your security design and help you improve it; then you’ll actually have something to test. Isn’t that a bit more scientific and logical? Your security design is your plan; then you validate your implementation against the plan, note deviations, and reassess.
I mostly agree with marcus. I think the problem here is twofold. There are probably a lot of people out there doing a really bad job on penetration testing/assessments (hate to break it to the world, but they are interchangable terms at this point), and/or mjr hasn’t been near a penetration test in forever.
Saying that you will have a good security design and test that it is working sounds really smart. Most people refer to the test that your security design actually works is called a penetration test. Also, plans are built on poor assumptions.
Testing your assumptions can be as valuable as testing your implementation against your design.
If we were going to have a face-off where everyone pretty much agrees, I would have like to have seen Bruce and mjr write about how to get value from security testing.
yoshi
March 13th, 2007 3:50 pmsometimes (ok all of the time) I think marcus lives in some fantasy world that I just don’t get.
in the world i inhabit at least, applications are frequently installed or designed without someone knowledgeable about the risks they are introducing -or- you don’t have full control over an implementation (for whatever reason). security assessments and pen tests are useful tools to understand and to document that risk.
Chris W.
March 13th, 2007 4:06 pmPenetration testing can be especially effective when you DON’T have a good security design. I know it won’t come as a shock to people that not everyone has a good security design or that many people don’t want to pay for a good security design.
From where I sit many applications are composite. They have legacy code, outsourced code, shared internal code, binary only components, and open source code. This is done for a practical reason. It is faster to reuse than to create from scratch. Where is the design spec for the binary only component or the open source component or the legacy code, etc…?
Security testing has the advantage of being able to find problems you didn’t know was in the code because you either didn’t have access to the source or the design documentation or the designers.
Without security testing how are you going to find problems like a leftover debug interface in the code or a insecure side effect of a binary component such as a predictable temp file name? What about a DWR interface that is missing its authorization or session check? A secure design can mitigate these vulnerabilities but not eliminate them.
It’s not use a secure design OR do security testing. It’s do both! If your development process makes assessing the design difficult or impossible then security testing is all the more important.
No wonder quality assurance professionals are the poor step children of the software development community compared to developers and architects. I see the same biases being played out here in the security space as security professionals who review software design pooh pooh security testers.
-Chris
Chris Rohlf
March 13th, 2007 4:17 pmBruce makes the best point. A lot of the time IT staff knows their networks are vulnerable. But they need an outside, 3rd party/independent piece of paper to prove it to management in order to get the budget.
Internal audits don’t always work and sometimes its out of your control (i.e. you legally MUST have a third party come in and look depending on the nature of your organization).
And as a side note, I think the more and more ’security testing’ and ‘QandA’ converge the more security pro’s are going to distance themselves from the term or try to reinvent their purpose.
mcwresearch.com » Pen testing
March 13th, 2007 5:40 pm[…] I’m seeing a lot of posts about Pen Testing, specifically Tom’s over at Matasano. On that post, Tom is commenting on an article for Information Security Magazine. […]
Nate
March 13th, 2007 5:49 pmChrisW, agree with your conclusion.
Also, I think we should stop calling vulnerability reporters “security researchers” and instead call it what it is, “security QA”. Developing a new class of attack (i.e. format strings) is security research. Finding the Nth instance of it in a product? Security QA.
Security QA is very important, and it should get its due. But when you look at it as just another type of QA, you can see why it is plagued with the problems it has. Vendors file reports in their normal QA database, fixes later show up in software updates, issues get dropped if you’re not some major customer, just like every other type of QA.
Chris_B
March 13th, 2007 8:42 pmWasnt it Deming who talked about the “PDCA model” of management? PDCA or Six Sigma type ideas should mandate good design and verification of implementation.
There sure do seem to be lots of Chris’s round lately!
LonerVamp
March 14th, 2007 1:02 amOf course, any discussion on pen-testing should start with defining what you mean by pen-testing, especially in relation to vuln testing. I think Bruce is talking more about vuln testing, which means his conclusion is fairly accurate.
I think Marcus has a great point, though he maybe didn’t make it obvious enough. Penetration testing should not just come in and prove vulnerability. Like Bruce said, he can “prove” that in just a few words. Pen-testers should give something back. No one likes someone who just points out the bad things then sticks their nose up in the air and walks away. Rather, point them out and give plenty of constructive criticism and feedback.
Maybe because that can be an added service that costs extra beyond what some companies want to pay (consulting?) they don’t get that level, but *I* certainly would want that level. Even if it just means having the IT/security team sit down informally over dinner with the testers and pick their brains and share stories.
In addition, there are way too many “bad” assessments being done, and this is not always the assessor’s fault. My last company had 4 assessments of varying degress and while us techs in the trenches were praying for plenty of holes to dog mgmt to give us resources or just plain managerial backing, the assessments were just done because clients required them. And they were met with dancing, smoke, mirrors, and utilizing ignorant middle managers as the liaisons to the assessors. Lip service and a cheap company… Just like Marcus said, companies with poor security tend to know it, some just deny that they know it while others don’t deny it but wiggle and dance so they don’t have to do anything about it.
PaulM
March 14th, 2007 9:33 amSo far everybody has failed to mention the best real-world business driver for a third-party pen-test: making an effort to demonstrate to business partners, customers, and auditors that your security doesn’t suck.
Seriously, here’s some spending on security where you can put the deliverable RIGHT IN THE SALES CYCLE.
Of course, this means that 1) your security rocks and you come up clean on the first run, 2) your third-party auditor sucks (but it’s almost impossible for your customers to tell #1 from #2, so tell them it’s #1), or 3) you have to go through several test-fix cycles to finally come up clean, which costs more money.
I know I sound jaded on the topic, and I probably am a little bit, but it’s true - you can use a pen-test deliverable just like a SAS or ISO audit report to help inspire confidence in your organization. …and maybe improve security along the way.
Doug
March 14th, 2007 10:34 amSeems to me that the purpose of any comprehensive security test is to validate that the organization achieved what it set out to achieve. Ideally, the organization set out to limit its risks to some residual level. That limitation comes about through the three components of governance and technology (1) design, (2) implementation, and (3) operation.
Isn’t it best to design appropriately then gain assurance that the design is appropriate, and implemented and operated according to the design?
If so, how well do modern pen tests identify the root causes of flaws, and isolate and categorize issues between enterprise design, implementation, and operation? And how well do modern pen tests provide continual assurance of appropriate operation?
Around the Blogosphere and Back | RiskAnalys.is
March 15th, 2007 10:18 am[…] Dave G. over at Matasano cracked me up talking about the Ranum vs. Schneier cage match articles on penetration testing in the recent issue of Information Security Magazine. […]
Leave a reply