The AV Doth Protest Too much (Consumer Reports)
Igor Muttik, over at AVERT, McAfee’s research team, has post up disputing the consumer reports process of testing AV products.
1. It is claimed that created viruses were “the kind you’d most likely encounter in real life” which is, of course, something the testers cannot know.
While no one can predict what new techniques viruses are going to use in the future. These were variants of existing viruses. If you can’t detect these well, then we already know you can’t detect whatever a next generation virus will be.
2. Creating new viruses for the purpose of testing and education is generally not considered a good idea - viruses can leak and cause real trouble (you can read an open letter on the AVIEN site about that).
A valid point. But something that can be controlled. They can have processes and procedures in place to make sure nothing bad happens. Over all, low risk.
3. There is a more scientific way of measuring real proactive detection of AV products on future malware - it is called “proactive testing” or “retrospective testing”. The idea is to measure, say, 3-month old AV product against real field viruses that appeared within these last 3 months. The discussion of the methodology of such tests can be found here and some real test results with common AV products are on the AV-comparatives.org site.While I don’t think one is more scientific than the other, I do concede these are different tests. And I also agree that Consumer Reports should have made sure it tested AV products to make sure they detect the viruses they claim to detect (retrospective testing). However, what Consumer Reports essentially did was perform a proactive test on all of the AV products. It seems reasonable to me that a large percentage of future viruses will be based on the ones that exist today. If anything, they are likely to be more sophisticated than what Consumer Reports did.
The biggest difference here is the number of viruses created would be way smaller with av-comparitives.org study (they appear to do quarterly reviews). Also, AV comparitives rating system has 3 ratings:
- Standard
- Advanced
- Advanced+
Lets compare this with other rating systems that have three categories.
| AV-Comparitives | Reality | Starbucks | Stock Analyst |
|---|---|---|---|
| Standard | Bad | Tall | Weak Buy |
| Advanced | Average | Grande | Buy |
| Advanced+ | Good | Venti | Strong Buy |
4. Objection #1, that ConsumerReports.org cannot know what viruses we are going to face in future could be moot as their testing team apparently invented a time machine and shifted themselves forward to September.My objection #1 is that AV products can’t detect viruses generated from known toolkits and can’t detect simple variants of viruses they already can detect.
I have a different objection to this testing methodology. What will most likely happen now is that consumer reports will probably end up giving these viruses to the AV companies. The AV companies will spend [read: waste] a lot of time to make sure their software detects/removes all of these lab viruses. Which in and of itself is unfortunate. But now there will be stickers on AV boxes saying how the new version of their product protects you from ‘5500+ new threats’. The total malware counts will raise by approx. 5,500.
While I am sure R&D wants no one else to ever do this type of test again, Sales and Marketing is probably pushing for quarterly reviews like this.
In all honesty, I do believe that the AV researchers genuinely care about this, and are truly concerned about the release of new viruses. There is just an interesting dichotomy in the practice of AV versus the business of AV.
8 Comments so far
Leave a reply
Just to be overly picky and pain in the ass, a better stock analyst rating would be
Sell
Hold
Buy
And what the hell is up with the
Underweight
Hold
Overweight
???
Anyway, we all know that we don’t have to worry about viruses now that we are on OS X.
Lucas.-
Those of us who worked for companies McAfee devoured under the guise of NAI learned not to trust the business people there but the AVERT folks tended to be good at heart. Its too bad that they probably had to go through some of the corporate vipers in their “official” communications.
I agree with Chris_B that AVERT is a strong team, and I’m sure someone in the PR group at least got to edit this response before it came out. It’s not like security companies are ever in the middle of public relations problems right?
However, I don’t think that it is relevant if AVERT or the business people came up with this response. MFE along with the other AV companies have built a good business and they want to protect it. As if it wasn’t enough that their margins and market share are being squeezed by Microsoft, but now their product is being called into question by a group a people that review blenders and washing machines. The AV business is full of some very smart and very proud people, and they we’re just going to take this one laying down.
Kudos to Consumer Reports for turning the standard security product review on its head and testing the core value of the product. These types of reviews will lead to better products and more intelligent consumers.
How does the industry respond to articles like this?
Eighty percent of new malware defeats antivirus http://www.zdnet.com.au/news/security/soa/Eighty_percent_of_new_malware_defeats_antivirus/0,2000061744,39263949,00.htm
This 80% number was confirmed in a posting on the offensivecomputing.net blog which has a database of 33,000+ pieces of malware.
Here is an anonymized quote from a friend who works at a very well known security product company,
“At XXXXX we have a few honeypot boxes that we use to capture malware that is actually in the wild (none of this we found it in our lab). We then run it through an engine that uses 27 different AV products to try and identify the malware. The results obviously vary but out of the 27 it is common to only have 2 or 3 products actually identify the code.”
It seems clear that catching old malware is easy and catching new malware is hard, even new malware that is a slight variation on old.
So the efficacy of current AV must be proportional to the churn rate of malware. The faster virus writers are able to make modifications, the more likely they are to be successful.
The number of “hockey stick” graphs in this trend report tells the tale:
http://www.viruslist.com/en/analysis?pubid=182974451
Is there a point where the current AV technology just cannot keep up with the churn rate? Have we reached it?
-Chris
Actually the point was passed a few years ago.
It’s far worse than expected. I’ve written more about this on my blog — http://sunbeltblog.blogspot.com/2006/08/consumer-reports-testing-scandal-its_25.html
[…] The classic example of chain designs is antivirus. A release ships with N “signatures”, each tuned to a specific type of virus (at varying levels of genericity). If you can take a virus and change it just enough to beat the signature that’s tuned for it (like Consumer Reports did), the system fails catastrophically. […]
[…] the reasoning that led to a testing lab writing viruses, while other security researchers argued that it’s reasonable to measure the performance of antivirus software against previously […]