What Common Criteria Certification Means
Thomas Ptacek | June 19th, 2006 | Filed Under: Industry Punditry
One of the good things about consulting work is that it forces you out of your own little world. So, it wouldn’t have occurred to me to write a post about Common Criteria certification —- “of course CC is a joke, get back to writing about shell-script fuzzers” is what I (wish) they’d say. But I did a quick job for a client last week, and it came up in conversation. “This product is EAL2 certified. Wasn’t it tested and verified secure by the government?”
People seem to have three basic misconceptions about Common Criteria certification:
That CC testing addresses modern threats, such as would be addressed in a Bugtraq post. No. CC testing confirms the existence of features such as a password prompt, unless you don’t have one, in which case you can just document that anyone can log in and still get certified.
My guess is that the threat environment contemplated by CC testing dates back to the early ’90s, when “unpassworded command and control software” was a real risk.
That CC testing is overseen in good faith by NIST, the NSA, or the DOD. No. CC testing is conducted by “CCTLs”, which are often tiny divisions of the IT practices of companies like Lockheed and SAIC, like every other commercial enterprise that seeks to do business with the government.
That vendors get CC testing so they can get an independent assessment of their security. No. Vendors get CC tested so they can close product sales to the government, particularly the DoD, where lack of CC status can delay or kill a deal.
Of course, if you’re in marketing, you’d have to be made of stupid to receive a security certification that appears to come from the NSA and not tout it as a major achievement. So we get things like:
UnityOne is the Only IPS to Pass Internationally-Recognized, Government-Approved, Security Certification
and
To achieve this added level of certification, the NSA performed vulnerability and penetration testing on the product in conjunction with source code analysis, verifying it meets a specific set of stringent security requirements.
and
Nothing Common in “Common Criteria”: How Microsoft Customers Can Utilize the Unprecedented Security Recognition Awarded to Windows 2000
The confusion is abetted by the Common Criteria “Evaluation Assurance Levels” (EALs), which appear to establish “grades” of security for products:
- Functionally Tested
- Structurally Tested
- Methodically Tested and Checked
- Methodically Designed, Tested, and Reviewed
- Semiformally Designed and Tested
- Semiformally Verified Design and Tested
- Formally Verified Design and Tested
You can find an unintentionally revealing glimpse about the real meaning of EAL grades in this Xerox FAQ on their certification process:
At EAL2, the CC requires that the vendor use a configuration management (CM) system, and keep track of the configuration items that make up the system. EAL3 adds access control requirements to the CM system
Presumably lack of integer overflows resides somewhere up towards EAL9.
I can’t really speak to levels 5-7, which apparently can’t be achieved through commercial labs. And EAL1 apparently doesn’t win you the DoD purchasing merit badge, so nobody goes for it. But here’s what I think about EAL’s 2-4:
OS X 10.3.6 (is EAL3+)
BEA WebLogic after BEA-05-107.00 (but before 30 other advisories) (is EAL2+)
BMC Patrol (is EAL 2)
Groove Workspace/Server (is EAL2+)
HP OpenView (EAL2)
DB2 (is EAL4 [it must be unbreakable!])
WebSphere 5.0.2.8 (is EAL2+)
IRIX + 4354, 4451, 4452 (is EAL3. No, not kidding. Irix.)
Windows XP, Server ‘03 (is EAL4+)
Red Hat Linux (is EAL4+)
Windows 2000 Advanced Server SP3 (is EAL4+)
Does an EAL grade correlate with better resilience against modern attacks? It doesn’t seem that way. What it does correlate with is the amount of money you’ll spend with Lockheed or SAIC to purchase a better grade for your product than your competitors have.
My experience with CC certification is, you talk to some consultants. You call up 4 or 5 “CCTLs”. You pick one based on availability and rates. We wound up with SAIC. You get on the phone for 15 minutes with them to discuss what your product does. Then your tech writer goes into a room and transforms your product documentation into the literary equivalent of punch cards. You hand off documentation. You sign a purchase order. Two weeks later, you appear on an “In Process” website that enables your GSA purchases with the DOD to go through.
There may be steps after cutting a check and appearing on that website, but I don’t think anyone cares about them.
Security researchers get obnoxiously cynical about CC certification, and this stuff is why. CC is pretty close to a defining example of the “pay for play” practices that dominate government security work.
IBMer
June 19th, 2006 5:27 pmYou forgot to mention that AIX has been evaluated at EAL4+ against LSPP and CAPP.
Thomas Ptacek
June 19th, 2006 5:44 pmAre you trying to say something nice about AIX or mean about AIX? =)
ScummyVendor
June 19th, 2006 7:37 pmThe CC process is nothing more than yet another “full employment act” for government contractors. At the levels that DoD requires to sell into govt organizations (EAL2) it pretty much just certifies that your product documentation describes what the product actually does. The ability of the CCTLs to do any credible level of testing on network security technologies like a hardware-based IPS engine is laughable when the vendors themselves have a hard time coming up with a complete test matrix.
CC is nothing more than a government mandated ~$250k barrier to entry for startups and other small vendors. To say it is worthless is overstating its value.
Tyler Durden
June 19th, 2006 9:44 pmThe first rule of the CC certification process is: you do not talk about the CC certification process.
The second rule of the CC certification process is: you DO NOT TALK ABOUT THE CC CERTIFICATION PROCESS.
Since this whole deal is so similar to a mob business, you should watch out for the EAL hitmen.
Seriously, thanks for debunking this extortion racket in such brutally honest terms.
Ray Potter
June 19th, 2006 11:52 pmI think there is another misconception: that a product evaluated against the Common Criteria is a secure product. Are products that are evaluated hacked less often? Does evaluation yield fewer (or even no) vulnerabilities? These metrics are hard to quantify, and this is a rather dicey issue in the Common Criteria community.
That said, the purpose of Common Criteria is to provide assurance that a product’s specified security features work as the vendor claims. And there can be some benefits to the development process.
Regarding your misconception #2, there actually is Government review of testing. You’re correct in that the CCTLs conduct the testing, but there is a review conducted by a government Validator (or Certifier in other global evaluation schemes). Each Validator checks that the CCTL evaluation of evidence documentation is thorough according to the Common Evaluation Methodology, and at the end of the evaluation, the Validator will write a Validation Report.
Otherwise, I think your other points are spot-on.
The Other Tyler Durden
June 20th, 2006 10:29 amI wanted to destroy something beautiful.
Thomas Ptacek
June 20th, 2006 8:08 pmRay, I hear what you’re saying about how hard these things are to quantify. The problem is, they aren’t hard to quantify. We have years of data now about the histories of popular software.
CC certification hasn’t correlated with stronger products.
This doesn’t surprise people who have been through the process before, because most of them share the experience of having the overwhelming majority of the work done by their tech writers, not their engineers.
Ray Potter
June 21st, 2006 10:25 pmI’d like to restate my previous comment of “Otherwise, I think your other points are spot-on.” By this I was referring to your other stated misconceptions.
>What it does correlate with is the
>amount of money you’ll spend with
>Lockheed or SAIC to purchase a better
>grade for your product than your
>competitors have.
While it’s true that evaluation costs typically increase as the assurance level increases, but I don’t think that’s really the point you’re trying to make here. Anyone who has been through Common Criteria evaluation knows that a product vendor doesn’t just cut a check for a certificate. CC evaluation requires involvement from senior engineering resources (and yes, their tech writers) to support the effort - trust me… I ran the program for one of the most active vendors in the CC community. I certainly wished it was as simple as you make it sound to be.
Don’t get me wrong… I see a lot of issues and a lot of room for improvement and expansion to add value. But folks need to know and remember the focus of CC, why it’s there, and what it does and doesn’t do. And while it may be a checkbox exercise for procurement, it’s certainly not a checkbox exercise for product vendors, (most) consultants, or evaluation labs.
AnAAGReader
June 22nd, 2006 6:13 amRe: ‘That CC testing addresses modern threats, such as would be addressed in a Bugtraq post. No. CC testing confirms the existence of features such as a password prompt, unless you don’t have one, in which case you can just document that anyone can log in and still get certified.’
I don’t think there is intent for this to hold true with CC 3.0. I believe there is intent to ensure that external evaluators will be urged to check for vulnerabilities, i.e., CERTs on the product being evaluated and verify whether or not they impact the TOE.
Also re: ‘That CC testing is overseen in good faith by NIST, the NSA, or the DOD. No. CC testing is conducted by “CCTLs”, which are often tiny divisions of the IT practices of companies like Lockheed and SAIC, like every other commercial enterprise that seeks to do business with the government.’
I have to at least partially agree with Ray’s dispute on this. I’m confident that either people directly representing NSA/NIAP are involved or that people who have received “validator training” from NSA/NIAP are involved in the evaluation process, particularly at EAL4 and above. At EAL5 and above TOEs are pen-tested by teams affiliated with NSA. I think it would be accurate to say that direct government involvement is limited at low assurance levels; but that involvement increases with respect to the level. The team that did the EAL7 “data link diode” would be in a great position to answer that.
Re: ‘That vendors get CC testing so they can get an independent assessment of their security. No. Vendors get CC tested so they can close product sales to the government, particularly the DoD, where lack of CC status can delay or kill a deal.’
I think Johnathan Shapiro of EROS/CoyoteOS put this best when he said (re the Windows evaluation): ‘Security experts have been saying for years that the security of the Windows family of products is hopelessly inadequate. Now there is a rigorous government certification confirming this.’
(http://eros.cs.jhu.edu/~shap/NT-EAL4.html) CC is (in general) reflection that the TOE does what you says it does; and a measure of the steps you’ve taken to provde it.
Thomas Ptacek
June 22nd, 2006 10:27 amRay, thank you for writing. You are making interesting points. I’d love to be able to offer you some form of equal-time defense of the program beyond blog comments.
That said: I just disagree with you. I don’t know you personally but I have every reason to believe that you’re a dedicated and competant professional. I think that’s probably true of the majority of the people who work at CCTLs. But that doesn’t mean CC evaluation is a serious process for vendors.
I’m speaking from knowledge of several vendors, including the successfuly-certified vendor who coached me through the CC process when I was managing the product Arbor got evaluated. In each case these vendors saw the CC process as largely irrelevant to the functionality of their product, meaning engineering resources (beyond tech writing) were not committed to it.
There are other processes — a Neohapsis evaluation, an @stake assessment, a customer pen-test, even Cisco AVVID certification — that I have been a party or witness to. And I see a night-and-day difference between those and CC evaluations.
Here’s another way of making the same point:
At Matasano we’ve done many, many customer application and product pen tests. We do ours under insanely tight schedules — one week is probably a good median allowance. I can think of only one instance in which we did not come back with shattering vulnerabilities (that was a storage HBA, which had firmware built on a synthesized CPU, with minimal “everything”, for which we had just a few days testing time).
So the obvious question is: how many vulnerabilities can you point to that were discovered by the CC process?
I’m saying this to offer you a chance to quantifiably stick up for the program, not just to make a rhetorical point.
Ray Potter
June 22nd, 2006 10:17 pmThomas-
>So the obvious question is: how many
>vulnerabilities can you point to that
>were discovered by the CC process?
From my experience at Cisco: none.
From my experience as a consultant: none.
Remember, EAL2-EAL4 evaluation isn’t meant to be a “find the vulnerability” exercise. It’s a level of assurance on the implementation of a product’s security functionality. That’s it.
So the next obvious question is: is that valuable?
My take (in brief) is maybe, maybe not. It’s certainly not a panacea. As far as the other evaluations (and the work you do at Matasano), that’s all very good and useful and needed stuff. I certainly think it yields more tangible, measurable, and quantifiable benefits/results. Personally, I’d like to see some increased requirements for these methods to complement Common Criteria or FIPS 140. Those types of activities are usually conducted as part of a systems-level certification & accreditation process. Products will undergo pen testing as prescribed by various requirements of policies and systems owners, and this testing is conducted on the product(s) in their deployed configuration.
One thing is certainly true: Whether it’s disparate customer requirements, difference in evaluation strategies, or variance of involvement of engineering, everyone has their own experience, insights and emotions regarding Common Criteria and FIPS 140. Mileage certainly varies.
I look forward to continued off-line discussions.
Jon
June 23rd, 2006 2:09 pmWhile the original post makes valid points, it also shows a lack of understanding of Common Criteria and the complexities of assurance methodologies. I think Mr. Potter addressed most of these so I will not rehash that.
A few points I would like to add.
There is a difference between the common criteria methodology and how that methodology is applied. I personally feel that much of the problems with CC today are due to problems with government oversight. They are the ones that have to power to raise the bar with penetration testing etc. That being said without over site the process will quickly degrade in quality as vendors will generally not choose a lab that will break their product and fail them. Much of the documentation CC asks for is just good software engineering; my experience is that companies that do not have design documentation generally do not have very good products.
As a consumer of a product it is also difficult to have faith in a “certification” performed by a “high end” independent lab. How would one compare these labs? Is matasano better than @stake? Also labs generally have core competencies but will happy to do evaluations for all product types (example I would not trust a hardware assessment by @stake but I may if Ross Andersons group does it). Additionally within each company individuals skills differ. Has a vulnerability in the product been found but due to NDA not been made public or in fact fixed (yes this DOES happen).
I have seen a number of vulnerabilities found in CC evaluations and have seen vendors change to a lower EAL level because they could not easily address them and the lab/scheme lost some faith in the product. That being said these generally tended to be hardware based higher assurance systems with the evaluations performed by specialized labs.
So yes CC needs to be revamped. BUT you can be sure that if labs/schemes start failing evaluations there will be a lot of political ramifications.
Jeff Jones
June 23rd, 2006 7:59 pmRay - Hadn’t found your blog until (former Colleague) Window reach out to me this past week to see how things were going, but I like the topics so far.
I recently also wrote a bit about the Common Criteria process at (http://blogs.technet.com/security/articles/430098.aspx)The Importance of the “Evaluated Configuration” in Common Criteria Evaluations.
I’m pretty critical of the process too, but there are some benefits it has brought - largely to get governments away from large custom (arbitrarily spec’d) systems and to provide impetus for driving research forward in the industry well before security became “hot”.
The other key problem is that there is a lot of variation in the scope of existing process by the different evaluators AND oversite bodies worldwide, which basically enables a “race to the bottom”, or lowest common denominator. That is part of why the mutual recognition agreement ends at EAL4 - USGov wouldn’t trust anyone who says they evaluated something at EAL7 unless NSA folks reviewed it themselves.
sargon
September 25th, 2006 8:15 pmJust to keep the old blog current, this just in (from NIAP website):
Due to fiscal constraints, beginning on October 1, 2006, for FY07, the NIAP CCEVS will only accept Medium and High Robustness PP compliant products in support of National Security customers. Product submissions meeting the above criteria will be queued and validation resources will be allocated as they become available. As a condition of acceptance, detailed letters of intent that identify the intended DoD or IC customer (containing POC name, organizations, email, phone number) will be required.
CCEVS will continue to provide updates on the status of the program via this website. Please direct questions or concerns to NIAP at (410) 854-4458.
So not only are CC evaluations crap, you can’t even do them anymore
Matasano Chargen » Defense in Depth, Reconsidered: Is Information Security Anything Like War?
April 21st, 2008 6:16 pm[…] Principals for the Protection of Information . It does, however, feature prominently in the Common Criteria —- which should tell you […]
Leave a reply