The AV Doth Protest Too much (Consumer Reports)

Dave G. | August 21st, 2006 | Filed Under: Industry Punditry

Igor Muttik, over at AVERT, McAfee’s research team, has post up disputing the consumer reports process of testing AV products.

1. It is claimed that created viruses were “the kind you’d most likely encounter in real life” which is, of course, something the testers cannot know.

While no one can predict what new techniques viruses are going to use in the future. These were variants of existing viruses. If you can’t detect these well, then we already know you can’t detect whatever a next generation virus will be.

2. Creating new viruses for the purpose of testing and education is generally not considered a good idea - viruses can leak and cause real trouble (you can read an open letter on the AVIEN site about that).

A valid point. But something that can be controlled. They can have processes and procedures in place to make sure nothing bad happens. Over all, low risk.

3. There is a more scientific way of measuring real proactive detection of AV products on future malware - it is called “proactive testing” or “retrospective testing”. The idea is to measure, say, 3-month old AV product against real field viruses that appeared within these last 3 months. The discussion of the methodology of such tests can be found here and some real test results with common AV products are on the AV-comparatives.org site.
While I don’t think one is more scientific than the other, I do concede these are different tests. And I also agree that Consumer Reports should have made sure it tested AV products to make sure they detect the viruses they claim to detect (retrospective testing). However, what Consumer Reports essentially did was perform a proactive test on all of the AV products. It seems reasonable to me that a large percentage of future viruses will be based on the ones that exist today. If anything, they are likely to be more sophisticated than what Consumer Reports did.

The biggest difference here is the number of viruses created would be way smaller with av-comparitives.org study (they appear to do quarterly reviews). Also, AV comparitives rating system has 3 ratings:

  1. Standard
  2. Advanced
  3. Advanced+

Lets compare this with other rating systems that have three categories.

AV-ComparitivesRealityStarbucksStock Analyst
StandardBadTallWeak Buy
AdvancedAverageGrandeBuy
Advanced+GoodVentiStrong Buy
As you can see, in ‘Reality’, when we break up and rate widgets into three groups, we have Good, Average and Bad. So, when we apply that mapping, we are left with the notion that in the AV world, standard equals bad.

4. Objection #1, that ConsumerReports.org cannot know what viruses we are going to face in future could be moot as their testing team apparently invented a time machine and shifted themselves forward to September.
My objection #1 is that AV products can’t detect viruses generated from known toolkits and can’t detect simple variants of viruses they already can detect.

I have a different objection to this testing methodology. What will most likely happen now is that consumer reports will probably end up giving these viruses to the AV companies. The AV companies will spend [read: waste] a lot of time to make sure their software detects/removes all of these lab viruses. Which in and of itself is unfortunate. But now there will be stickers on AV boxes saying how the new version of their product protects you from ‘5500+ new threats’. The total malware counts will raise by approx. 5,500.

While I am sure R&D wants no one else to ever do this type of test again, Sales and Marketing is probably pushing for quarterly reviews like this.

In all honesty, I do believe that the AV researchers genuinely care about this, and are truly concerned about the release of new viruses. There is just an interesting dichotomy in the practice of AV versus the business of AV.

Viewing 6 Comments

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus