Ignore Igor Muttik’s “Retrospective” Antivirus Testing Method

Thomas Ptacek | August 22nd, 2006 | Filed Under: Defenses, Malware, Uncategorized

Dave wrote yesterday about the McAfee AVERT lab objections to ISE’s Consumer Reports test of antivirus engines. I have more to add.

The ISE team manufactured a stream of “new” viruses, in part by generating semantically-equivalent variants of existing viruses. This data set showed suprising problems with well-known AV engines.

Igor Muttik, a longtime member of McAfee’s antivirus research group, complained, writing that the more “scientific” way to approach this problem is to conduct a “retrospective” test. From his 2001 paper, which he cited:

Retrospective testing is used to test proactive abilities of scanners. The idea idea is to take, say, a three-month-old scanner and compare the detection rates over the viruses that appeared within the last 3 months. Naturally, as scanners were released before said viruses appeared, we would measure pure capability to detect new viruses.

This is more scientific than the ISE test? Nonsense, Igor.

Using currently-known viruses to measure the performance of older AV engines builds a huge assumption into the test: that viruses known today, but unknown three months ago, are representative of new viruses in general. But of course they aren’t. They’re representive of the new viruses that AV labs and AV engines have learned how to detect in those last three months.

Even if we’re charitable —- assuming retrospective tests aren’t simply diffs of the results of different product revisions —- these tests only sample viruses that AV vendors are bound to detect anyways. Because if they can’t detect them, they can’t sample them!

Like Igor’s retrospectives, the ISE tests are also built on an assumption. ISE claims you can assess performance in part by making small variations to pre-existing viruses. These variant viruses are also a small subset of new viruses in general. It just so happens that this subset knocks McAfee into the middle of the pack.

You can make up your own mind about whether a virus born out of modifications to an existing virus is a more serious threat than any of the thousands of historical curiousities and QA test lab anomalies that get replayed during a “retrospective” test. Personally, I look at the genealogy of other forms of malware —- shellcode, bots, worms, and exploit tools —- and I notice that the most malicious attackers tend not to write things from scratch, and I think the ISE guys can make a good case for having designed the most relevant test in the industry.

What part of this test is most embarassing for McAfee?

  1. That they didn’t come in first place?

  2. That a test harness ISE created as a one-off consulting project clearly outdid over a decade of R&D research by mainstream AV vendors?

  3. That tests like ISE’s will, over the long term, expose antivirus products for the freely substitutable commodity products that they are?

I assume that in 2006 AV purchasing decisions in large enterprises are made from the “big 3”, and that the decision hinges on price, price, price, and management offerings. Which over the short term bodes well for McAfee and Symantec. But it’s cold comfort for the long term, when the AV product lines that account for the overwhelming majority of both companies revenue finally come under sustained assault by Microsoft.

The blogs and trade press is full of sound and fury about ethics and “objective testing”. I’ll do my best to hold in how little the “virus ethics” argument signifies. And I’ll stop belaboring the obvious point that a commoditized AV market will belong to Microsoft. Because what’s really happening here might be much more annoying.

Is it possible that the AV researchers —- people with doctorates who are published principally, if not exclusively, in venues like “Proceedings of the International Virus Prevention Conference” —- don’t want the new kids playing with their toys, or setting foot in their sandbox? Is that why universally respected consultancies like ISE* have to play by the perverse rules of “virus science”? It wouldn’t be the first time: just look how the old-timer security people handled eEye.

.* I get to be credible complimenting ISE because we compete with them.

Viewing 36 Comments

    • ^
    • v
    how do i disagree with thee, let me count the ways...

    1) "This is more scientific than the ISE test? Nonsense, Igor."

    yes it is more scientific... it tests a product against viruses that it *actually* faced in the months after it's release rather viruses similar to what we think it will face (based on an overly simplistic model) in the months to come... good science tries to measure and model the real world - the closer something represents the real word the better it is from a scientific standpoint...

    2) "Using currently-known viruses to measure the performance of older AV engines builds a huge assumption into the test: that viruses known today, but unknown three months ago, are representative of new viruses in general."

    there is no such thing as a statistically representative sample in the context of virus detection testing - you cannot select a subset of the population at random and expect detection results for that subset will scale up to the entire population... that's why good tests use as much of the entire population as possible or limit their scope to some significant subset (like viruses known to be in the wild)...

    3) "these tests only sample viruses that AV vendors are bound to detect anyways. Because if they can’t detect them, they can’t sample them!"

    this is false, the viruses are not drawn from a pool of what products can detect but rather from a pool of what researchers have analyzed and determined to actually be viruses... instead of making assumptions about how viruses are 'sampled' for testing purposes i suggest you do some actual research into the practice of av testing - i'd suggest starting with vesselin bontchev's "analysis and maintenance of a clean virus library"...

    4) "It just so happens that this subset knocks McAfee into the middle of the pack."

    it just so happens that it's not just mcafee complaining... a half dozen or more companies have issued public statements condemning the test, not just mcafee... in fact, i have yet to see a single person from within the av industry or research community say anything positive or even just neutral about what CR/ISE did....

    5) "You can make up your own mind about whether a virus born out of modifications to an existing virus is a more serious threat than any of the thousands of historical curiousities and QA test lab anomalies that get replayed during a “retrospective” test."

    the ISE's variants *ARE* lab anomalies, and retrospective tests already contain many variants of previous viruses that were actually made by the bad guys...

    6 "What part of this test is most embarassing for McAfee?
    1. That they didn’t come in first place?"

    they often don't in other tests as well... no one gets to come in first all the time...

    "2. That a test harness ISE created as a one-off consulting project clearly outdid over a decade of R&D research by mainstream AV vendors?"

    outdid how? just because the viruses didn't get detected? unless you've got some clever solution to the halting problem it will always be possible to write viruses that anti-virus products can't yet detect regardless of the technology being used...

    "3. That tests like ISE’s will, over the long term, expose antivirus products for the freely substitutable commodity products that they are?"

    that particular revelation has already been made years ago... av products don't distinguish themselves from the competition based on detection rates anymore and haven't for a long time now... it's all features, UI, and quality of support now...

    7) "I’ll do my best to hold in how little the “virus ethics” argument signifies."

    it may mean little to you but it's extremely important in the av domain...

    8) "Is it possible that the AV researchers —- people with doctorates who are published principally, if not exclusively, in venues like “Proceedings of the International Virus Prevention Conference” —- don’t want the new kids playing with their toys, or setting foot in their sandbox?"

    it's possible but so are death rays from mars... i think david harley's observation that the security community is largely still ignorant of av technology, practices and issues has more to do with it (and with the mistrust you're displaying here)...

    9) "Is that why universally respected consultancies like ISE*"

    they aren't *universally* respected... widely respected perhaps, but it should be clear there is one field in which they currently have no credibility...

    10) "have to play by the perverse rules of “virus science”?"

    how is 'thou shalt not write viruses' a perverse rule? it seems a perfectly reasonable expectation to place on those who are supposed to be the good guys if you ask me...
    • ^
    • v
    "But of course they aren’t. They’re representive of the new viruses that AV labs and AV engines have learned how to detect in those last three months."

    how can they learn if the products/updates were frozen 3 months ago? think about what you wrote ;) [or try to read again how a retrospective test works]
    • ^
    • v
    Thanks for a great comment. We couldn't disagree more.

    1.
    2.
    3.

    The fallacy you are promoting is that "all known viruses" is the equivalent of "all viruses likely to exist today". In other words: collecting every virus every researcher knows about is not an accurate representation of all viruses.

    I've read enough "virus research" to understand the ongoing debate about "sampling known viruses" versus "using all known viruses". Neither are "scientific" tests.

    This fallacy is particularly painful when you attempt to build on it. For instance, the claim that the set difference between "all viruses known on 8/23/06" and "all viruses known on 8/22/06" is the equivalent of "all viruses created between 8/22 and 8/23". The problem, obviously, is the word "known".

    There's a bogus mystique that virus marketers attempt to create around "virus research" and the finding of new viruses. In reality, a new, currently-undetectable virus for Windows XP is infinitely easier to create than, say, a new remote code execution vulnerability in Windows XP. Because it is so incredibly easy to create new viruses, it seems like claims to knowing about all viruses are dubious, similar in essence to knowing about the existence of all "hello world" programs.

    The evidence virus marketing people offer to support their claim is that the viruses they know about are more likely to represent a real threat than those manufactured without their knowledge. But there's no evidence, anywhere, to back this claim up.

    4.

    Many of the other companies complaining are not winners in the ISE tests. Is ZoneLabs complaining? Why are the other complainants more credible than McAfee, or for that matter more credible than ISE? I note ISE is the only company in the fray without product revenue riding on their position.

    In both cases, my argument and your response, we are relying on appeals to the authority of third parties. We are't going to bridge this gap: I'm going to continue to assert that people with unimpeachable reputations in security research are more qualified than those whose authority resides primarily in working for McAfee for a decade. When have I ever had to cite Igor Muttik? How often do people cite Avi Rubin?

    5.

    As with 1-3, "Actual examples of things made by bad guys" in your words needs to be amended to "things known to have been made by bad guys". ISE demonstrates that bad guys can easily make a variety of things AV vendors won't know about.

    6.

    The reason ISE's test harness should be embarassing is that, in the course of executing a one-off consulting project, they devised a way to generate extremely realistic data sets, based on empirically sound baselines (well-known existing viruses), that evade detection by many well-known AV engines. Presumably, they could keep generating more, if they wanted the numbers to look even worse for MCAF.

    This is embarassing because given the $1+Bn MCAF has riding on AV, and the apparent ease with which ISE executed this project, they should have been able to invest in a similar testing system.

    7.

    I'll address "virus ethics" elsewhere. I think it has more to do with self-importance than altruism.
    Suffice it to say, concern over the abuse of Turing Completeness seems silly from the perspective of someone who deals with remote code execution vulnerabilities being routinely sold to the Russian Mafia.

    8.

    We should be precise about what's at issue here: it's not that the security community is oblivious to the systems and methods of antivirus researchers, it's that the security community disdains those methods.

    9.

    I'll argue that trying to impeach ISE's credibility on this issue will, over the long run, do more damage to the antivirus community's credibility than Avi's.

    10.

    "Thou shalt not write viruses" is a perverse rule because it retards innovation in antivirus, as ISE's tests made perfectly clear: any group willing to break this arbitrary commandment can readily de-pants mainstream AV engines.
    • ^
    • v
    "Thou shalt not write viruses." That sounds like a pretty good rule. I would probably be pretty upset if I heard that McAfee was writing a bunch of viruses and one got loose, and I had to buy an AV product to detect it. But, ISE is definitely no AV vendor. They're trying to test AV products. How do you test an AV product without throwing a virus at it? Ok, so you can use existing viruses (virii?), but that clearly doesn't thoroughly test the virus engine. Throwing new viruses at the AV products allowed ISE to perform more thorough testing. Hmm, but AV vendors arent performing this test themselves? Surely more testing is a good thing!
    Hmm.. Maybe its not such a good law afterall...
    • ^
    • v
    Andi: I think you misread me. I'm saying, instead of throwing stale viruses at stale virus engines, dynamically generate an inexhaustable supply of artificial viruses in the testing environment. You can even base those artificial samples off well-known examples. Then you're testing virus engines without the bias of only using data that "virus researchers" have on hand.

    That's what ISE did.

    The concern about "viruses escaping from the lab" is silly. On hundreds of computers around the world there are samples of exploit code that, if given to organized crime, could compromise hundreds of thousands of computers and untold millions of credit card accounts. Any single one of those hundreds of machines is multiple orders of magnitude more valuable to criminals than a single virus, which can simply be commissioned for hundreds of dollars.
    • ^
    • v
    By the way, in case anyone is misunderstanding me: I am implying that the real reason "virus researchers" haven't constructed similar tests to ISE is not that they have an ethical quandary, but rather that they lack the technical ability.
    • ^
    • v
    @thomas
    1-3) "The fallacy you are promoting is that “all known viruses” is the equivalent of “all viruses likely to exist today”."

    it is as close as we can get to an accurate representation of all viruses... you cannot build a model that includes unknown viruses because in order to do so you would need to have the viruses in your hot little hands and then they wouldn't be unknown anymore...

    "Because it is so incredibly easy to create new viruses, it seems like claims to knowing about all viruses are dubious"

    it's not dubious, it's blatant snake-oil and you don't need to create new viruses to prove the fact... if you find an anti-virus vendor actually making that claim then call them on it publicly... name names, don't just use it as some tired justification for unethical behaviour...

    so please, enlighten us, who exactly is making that claim so that they can be publicly castigated... don't forget to include the supporting evidence of the violation...

    heuristics are going to miss viruses, and not just a few but a lot... heuristic technology is a stop-gap measure for new/unknown malware, it can not and will not ever detect all new malware for reasons quite well understood in the av field...

    4) "Is ZoneLabs complaining?"

    zonelabs is a consumer of av technology, not part of the av industry or research community...

    "I note ISE is the only company in the fray without product revenue riding on their position."

    instead they have service revenue riding on their position... they didn't do the test for free after all...

    "I’m going to continue to assert that people with unimpeachable reputations in security research are more qualified than those whose authority resides primarily in working for McAfee for a decade. When have I ever had to cite Igor Muttik? How often do people cite Avi Rubin?"

    if you're going to make a point about something in the AV field, you're better off drawing your citations from the pool of expertise in the AV field rather than the one in the general security field... expertise in one field does not impart expertise in the other, or are you unfamiliar with the concept of false authority syndrome?

    5) "As with 1-3, “Actual examples of things made by bad guys” in your words needs to be amended to “things known to have been made by bad guys”. ISE demonstrates that bad guys can easily make a variety of things AV vendors won’t know about."

    pure sophistry... the samples created by the bad guys were unknown to the vendors at the moment their products were snapshotted too... both test beds are using viruses unknown to the vendors at the time their products were sampled... the *ONLY* difference is that one contains viruses created in the wild and the other is made up entirely of ones manufactured in a lab...

    6) "The reason ISE’s test harness should be embarassing is that, in the course of executing a one-off consulting project, they devised a way to generate extremely realistic data sets, based on empirically sound baselines (well-known existing viruses), that evade detection by many well-known AV engines."

    they are not realistic data sets... the class of viruses (variants of known viruses) in their test bed is a proper subset of the classes of viruses used in retrospective testing... in reality new virsuses *ARE* made from scratch quite often and ISE's test bed does not reflect that reality...

    as for why it should or shouldn't be embarrasing in your mind, you aren't an av vendor, you aren't part of av industry or research community, and you clearly aren't familiar with the retrospective tests your criticizing or you'd realize they show precisely the same things and are just as 'embarassing' to vendors as ISE's test was... mcafee had an overall detection rate of only 30% in the most recent one at av-comparatives.org ...

    "This is embarassing because given the $1+Bn MCAF has riding on AV, and the apparent ease with which ISE executed this project, they should have been able to invest in a similar testing system."

    and why should mcafee (or any other vendor) manufacture new threats when the stream of new threats coming in from the wild is so plentiful (60-70 per day)?

    7) "I’ll address “virus ethics” elsewhere. I think it has more to do with self-importance than altruism."

    what it has to do with is commercial survival - competitors will roast each other over ethical breaches...

    8) "We should be precise about what’s at issue here: it’s not that the security community is oblivious to the systems and methods of antivirus researchers, it’s that the security community disdains those methods."

    i didn't say oblivious, i said ignorant, as in you know some things but not enough... certainly not enough to know when you don't know something...

    9) "I’ll argue that trying to impeach ISE’s credibility on this issue will, over the long run, do more damage to the antivirus community’s credibility than Avi’s."

    credibility is something that must be built... they are too new to the anti-virus field to have built any credibility there yet...

    10) "“Thou shalt not write viruses” is a perverse rule because it retards innovation in antivirus, as ISE’s tests made perfectly clear: any group willing to break this arbitrary commandment can readily de-pants mainstream AV engines."

    and once again, it's possible to do the same thing without creating new viruses, so you wind up manufacturing new threats needlessly...

    11) "I’m saying, instead of throwing stale viruses at stale virus engines, dynamically generate an inexhaustable supply of artificial viruses in the testing environment."

    a) while the av engines are stale, the viruses are not...
    b) the supply of viruses from the wild is also inexhaustible and has the benefit of not being artificial...

    "The concern about “viruses escaping from the lab” is silly."

    [sarcasm]
    yes, concern about things that have already happened before and are therefore known to be a real possibility is silly...
    [/sarcasm]

    "On hundreds of computers around the world there are samples of exploit code that, if given to organized crime, could compromise hundreds of thousands of computers and untold millions of credit card accounts."

    do you really not see how the risks associated with exploit code are not comparable with the risks associated with self-replicating code?

    12) "By the way, in case anyone is misunderstanding me: I am implying that the real reason “virus researchers” haven’t constructed similar tests to ISE is not that they have an ethical quandary, but rather that they lack the technical ability."

    it's become abundantly clear that you really don't know what you're talking about - there is nothing particularly complicated about program that takes an arbitrary binary and replaces opt-codes in it with semantically equivalent alternatives... if you really think the av industry and research community are unable to do that then you are completely out of touch with the av field and are thus not in any position to be criticizing it...

    algorithmic variant generators are generally not how variants are made in the wild - they're generally made by adding/modifying text, adding functions, tweaking variables, and fixing bugs... as such, variants created by algorithmic variant generators are not representative of real world viruses...
    • ^
    • v
    The "av industry" smacks of elitism. So what if one is not part of the industry? Is the "av industry" not part of the security industry?

    If I "change some op-codes" in a virus, I would EXPECT my AV software to catch it. I don't think I need to part of an industry in order to criticize it. And creating virus variants in order to test an AV product is un-ethical? OMG. I had no idea that the "av industry" was that far out there. Security professionals can and do work with malware all the time.

    "new virsuses *ARE* made from scratch quite often and ISE’s test bed does not reflect that reality"

    is there any support to this claim? I always thought variants outnumbered new new virii by a large margin, hence the relevance of ISE's tests.
    • ^
    • v
    @obijuan
    "The “av industry” smacks of elitism. So what if one is not part of the industry? Is the “av industry” not part of the security industry?"

    sure it is, in the same way that musicians are part of the entertainment industry...

    the av industry is a highly specialized part of the security industry and most on the outside prefer to ignore that fact and treat it as little more than an afterthought...

    "If I “change some op-codes” in a virus, I would EXPECT my AV software to catch it."

    which illustrates an ignorance of av technology (and to some extent computational complexity)...

    "I don’t think I need to part of an industry in order to criticize it."

    you don't need to be part of the industry, but you do need to have some familiarity with the subject your criticizing... few outside the field (industry and community) have such familiarity with the more technical issues (which unfortunately do come into play with criticism of 'i expect it to detect X')... without that familiarity your criticism are as hollow as me criticizing the government for not bringing about world peace and ending hunger...

    "And creating virus variants in order to test an AV product is un-ethical? OMG. I had no idea that the “av industry” was that far out there. Security professionals can and do work with malware all the time."

    indeed they do, and sometimes they wind up being part of the problem when the virus they modified and were experimenting with gets loose (which happens often enough to have made it into sarah gordon's paper "generic virus writers II")...

    to borrow a phrase from schneier (who probably borrowed it from someone else) creating viruses makes you 'part of the problem rather than part of the solution'... especially when it's done needlessly (which is the case here)...

    "“new virsuses *ARE* made from scratch quite often and ISE’s test bed does not reflect that reality”

    is there any support to this claim? I always thought variants outnumbered new new virii by a large margin, hence the relevance of ISE’s tests."

    variants do outnumber new virus families but that doesn't change the fact that new virus families are made quite often... just watch the alerts as they come out of the vendors and count the number that have an 'A' for their variant identifier - and remember that those viruses the vendors issue alerts on are just the tip of the iceberg as far as what's actually getting generated each month...
    • ^
    • v
    "the real reason “virus researchers” haven’t constructed similar tests to ISE is not that they have an ethical quandary, but rather that they lack the technical ability."

    or perhaps because it's a waste of time. There are near limitless ways to modify known malware to make it unknown, and none of them terribly clever.

    Sure you could sit and play the better virus/antivirus arms race by yourself, but in the end, it would be just that, playing with yourself..
    • ^
    • v
    Martin, if none of the techniques required to make malware "unknown" to an antivirus engine are clever, what does that say about the engines themselves?

    I feel like you're making my point for me.
    • ^
    • v
    Kurt, apart from knowing the secret password to VCL and not knowing what the "op" in "opcode" is short for, what qualifies you to determine who is and is not part of the "antivirus industry"?

    I'm sorry to be such a jerk. I just think the "AV research" community is a big silly clique. "Real AV researchers have ways of getting access to what the bad guys write WITHOUT writing new viruses" is just code for "you can't play in our sandbox unless you have a job with one of 10 approved companies". What Halvar Flake does to diff vulnerabilities out of patches seems 10x more sophisticated than the ability to flag a decryptor in an instruction stream based on the specific combination of instructions one guy in Bulgaria decided to use when he wrote the thing.

    Some undergraduate in a dorm room at umich is going to totally revolutionize OS security someday, or figure out a way to reliably detect shellcode in network packets without buffering or reassembling. Nothing like that is ever going to happen in antivirus, and I think part of the reason is that AV people don't want it to.

    Warmest regards.
    • ^
    • v
    @thomas
    "Martin, if none of the techniques required to make malware “unknown” to an antivirus engine are clever, what does that say about the engines themselves?"

    that they aren't written with the goal of detecting every conceivable transformation of an existing virus because it's just as intractable as detecting all possible viruses...

    "Kurt, apart from knowing the secret password to VCL and not knowing what the “op” in “opcode” is short for,"

    oooOOoo, i mistakenly typed opt-code instead of op-code... you should see what my brain used to have me type instead of himem.sys back in the days of dos...

    "what qualifies you to determine who is and is not part of the “antivirus industry”?"

    i guess after 17 years following the field i just have a knack for determining whose opinions come from knowledge of the field (and therefore are likely a part of it) and those whose opinions do not...

    "I’m sorry to be such a jerk. I just think the “AV research” community is a big silly clique."

    while i'll agree that CARO is a clique, i don't really see why it should be called 'silly'... it's a clique formed out of trust relationships rather than the more conventional cliques you see in highschool...

    "“Real AV researchers have ways of getting access to what the bad guys write WITHOUT writing new viruses” is just code for “you can’t play in our sandbox unless you have a job with one of 10 approved companies”."

    complete misinterpretation... real av researchers only share samples with those they know they can trust (to not do something bad, stupid, or irresponsible with the sample)... with enough of those kinds of connections you form a web of trust, which is the network by which many researchers are able to get samples...

    those are precautions that developed out of necessity, by the way, as there have been cases 'tagged' viruses escaping due to being shared with the wrong people...

    "What Halvar Flake does to diff vulnerabilities out of patches seems 10x more sophisticated than the ability to flag a decryptor in an instruction stream based on the specific combination of instructions one guy in Bulgaria decided to use when he wrote the thing."

    and if we could package him up with a product we'd all be much better off, but unfortunately av products are software, not wetware (yet)... see, it's not what halvar's tools do on their own unattended, it's what *HE* does with them... experts can do a lot more than finite state automata right now and there's very little we can do to change that in the short term...

    "Some undergraduate in a dorm room at umich is going to totally revolutionize OS security someday, or figure out a way to reliably detect shellcode in network packets without buffering or reassembling. Nothing like that is ever going to happen in antivirus, and I think part of the reason is that AV people don’t want it to."

    and this, once again, is an expression of mistrust that comes from not knowing enough... revolutionizing virus detection would give whichever company controlled the new technology a huge competitive advantage and after all is said and done money is what motivates the industry - if they can develop it or buy out whoever else might have done so they will... the industry wants that revolution to happen because it means lots of money for them...
    • ^
    • v
    No it wouldn't. Look at what happened to TBAV. The AV industry does not run on innovation; it runs on and is forever locked into a stifling symbolic relationship with a bunch of teenagers and low-rent criminals who's best avenue to create havoc, in 2006 mind you (!), is to write malicious Office macros.

    There are smart people who work in AV, and they have my sympathy. With respect to the core CS problems AV grapples with, their research community has been eclipsed by OS and application security. In my field, people write instruction set emulators as plugin scripts, and release disassemblers and tracers as open-source freeware.
    • ^
    • v
    By the way, with regard to the "trust relationship" CARO is built on: trust is nice, but smart people shouldn't have to ask permission to improve the state of the art.
    • ^
    • v
    @thomas
    "No it wouldn’t. Look at what happened to TBAV."

    yes, let's look at what happened to tbav, shall we?

    tbav's technology was bought by norman data defence and what does norman have now? sandbox technology that goes above and beyond conventional heuristics...

    "The AV industry does not run on innovation; it runs on and is forever locked into a stifling symbolic relationship with a bunch of teenagers and low-rent criminals who’s best avenue to create havoc, in 2006 mind you (!), is to write malicious Office macros."

    a) i believe you meant 'symbiotic'...
    b) macro viruses are considered old school now - that fad wore itself out some time ago...
    c) i believe you are referring to the continued reliance on signature scanning - clearly you don't see that signature scanning remains the most effective technique in so far as it can deal with more of the set of known malware than behaviour based systems can and because unlike behaviour based systems it can operate on the malware *BEFORE* the malware gets cpu resources and is able to defend itself...

    "There are smart people who work in AV, and they have my sympathy. With respect to the core CS problems AV grapples with, their research community has been eclipsed by OS and application security. In my field, people write instruction set emulators as plugin scripts, and release disassemblers and tracers as open-source freeware."

    emulation can be detected, disassemblers and tracers require skilled operators...

    reverse engineering an arbitrary suspect sample is still an intractible problem and neither your field nor the av field, nor any other field is ever going to be able to change that fact... as such, what can be done in a virus lab and what can be done on a user's desktop are 2 entirely different things...

    "By the way, with regard to the “trust relationship” CARO is built on: trust is nice, but smart people shouldn’t have to ask permission to improve the state of the art."

    building trust relationships is not the same thing as asking for permission... it's not even comparable...

    smart people shouldn't have to ask permission to improve the state of the art with respect to biological viruses either, but you don't see the cdc handing out free samples at the local grocery store...

    smart people can do whatever it is they want to do, but being smart does not automatically entitle one to hazardous materials, nor would there be any good way to make that distinction if it did entitle one...
    • ^
    • v
    http://www.av-comparatives.org/weblog/?cat=7
    I totally agree in nearly all points with Kurt Wismer :)
    • ^
    • v
    I think the underlying point of contention here is that Tom (and others like him, perhaps myself included) disdain anti-virus since it is nothing more than pattern matching against known threats and weak hueristics to catch a few unknown threats. Its not a sound analysis to detect all threats, and it can never be. The virus community understands that, and hence views the ISE tests as silly and without merit. But, the general population of people using virus scanners do not know this. The fact that ISE's tests point this out is a thorn in their side. The fact that Consumer Reports, a reputable name in consumer advocacy and education, is pronouncing this loudly and clearly is a major fiasco. This warms the hearts of people like Tom.
    That's the downside of building a product around undecidable problems. As a consolation prize, the AV community gets job security in perpetuity (so long as they don't mind working in redmond).
    • ^
    • v
    @newsham
    "The fact that ISE’s tests point this out is a thorn in their side."

    existing retrospective testing shows the same things that ISE/CR's test showed, and retrospective testing does so *without* making new viruses...

    the av industry and research community don't care about the results of ISE/CR's test because the results aren't anything that can't be found elsewhere, what the av industry and research community care about is the manner in which the test was performed...
    • ^
    • v
    We're well aware that the AV industry doesn't care about the ISE test. ISE didn't play by your rules or ask permission or "earn" their right to test your stuff by putting years into the trenches and making up Sara Gordon jokes.

    Is there really even a "virus research community" to speak of? What's a good example of a journal-quality article done recently in AV?

    By the way, the "you can't create new viruses!" argument? That's code for "you don't work for an antivirus company! Go away!" If real security companies aren't actually simulating worm outbreaks (you know, real malware?), I'm disappointed. The "hazmat" a typical security researcher works with is 1000x more "dangerous" than some random virus that nobody is ever going to get infected with. Somehow we manage to talk about it, study it, and generate it without "ethics" hysterics.
    • ^
    • v
    Except for Peter Lindstrom, who wants to put us in jail.

    Good luck with the IBM lawyers now, though, Peter. :P
    • ^
    • v
    Hi,

    There's thread going on over here - http://www.dslreports.com/forum/remark,16725030... - that many figures in the " Biz " have contributed to, in various ways. I and others have also included our thoughts and ideas etc in there.

    I recently referenced this thread, and some of your views about Variants + Retrospective testing in there, which has been responded to, so you may like to add your comments etc.

    Spanner

    SpannerITWks
    • ^
    • v
    Hi,

    There's thread going on in the Security forum over here - Our unique antivirus testing: How we did it - http://www.dslreports.com/forum/remark,16725030... - that several figures in the " Biz " have contributed to, in various ways. I and others have also included our thoughts and ideas etc in there.

    I recently referenced this thread, and some of your views about Variants + Retrospective testing in there, which has been responded to, so you may like to add your comments etc.

    Spanner

    SpannerITWks
    • ^
    • v
    sorry for the tardy response - b-day celebrations take precedence over these sort of things

    @thomas
    "We’re well aware that the AV industry doesn’t care about the ISE test. ISE didn’t play by your rules or ask permission or “earn” their right to test your stuff by putting years into the trenches and making up Sara Gordon jokes."

    no one needs to ask permission or earn any right - they simply have to do it right...

    "Is there really even a “virus research community” to speak of? What’s a good example of a journal-quality article done recently in AV?"

    you mean like the stuff that's published each month in virus bulletin? or do the av fields own periodicals not count?

    "By the way, the “you can’t create new viruses!” argument? That’s code for “you don’t work for an antivirus company! Go away!” "

    no, it's code for "don't be part of the problem"...

    "If real security companies aren’t actually simulating worm outbreaks (you know, real malware?), I’m disappointed."

    while i can think of a paper or two on computer virus epidemiology that most likely involved controlled outbreaks in a closed test network, i'm not sure what else you think can actually be learned from outbreaks that would have people orchestrating them as part of a commercial product or service...

    "The “hazmat” a typical security researcher works with is 1000x more “dangerous” than some random virus that nobody is ever going to get infected with."

    that remains to be seen... while i'm sure there are things that create a bigger 'bang' than a virus or worm, nothing is as hard to control as self-replicating code... unlike other types of malware or exploits that stop being an immediate threat when the people using them stop doing so, viruses and worms keep going and going without intentional assistance... stoned.empire.monkey was in the wild and on the wildlist for over a decade in spite of the fact that anti-virus software could detect and remove the virus for almost that entire time... few other threats are as completely autonomous as self-replicating code...

    "Somehow we manage to talk about it, study it, and generate it without “ethics” hysterics."

    concern over the needless manufacture of new hazards is not 'hysterics'...
    • ^
    • v
    1.

    "They simply have to do it right" is code for "they need to do it the way we tell them to". You don't decide what "right" is.

    2.

    Virus Bulletin and the "field's own periodicals" don't count. You're a smart guy. Go find me real research. Journal articles. ACM Transactions. Things where the majority of the "peer community" don't work at AV companies. Virus Bulletin is a magazine. I might as well cite my Dark Reading press hits, or this blog.

    By the way, I'm sure quality research on viruses exists. Just not in anything resembling the numbers that quality systems security research does. Best of luck tracking it down, though!

    3.

    The fact that you can't see why a countermeasure vendor would want to replicate an outbreak to test a product tells me you don't spend a lot of time developing security product. A hint: what people first think will solve security problems usually doesn't.

    4.

    A "new virus", exhibiting no novel capabilities other than the fact that antivirus companies haven't heard of it and thus can't detect it, developed by a researcher with her own private resources, presents almost no meaningful risk. The notion that this activity is even comparable to the development of a new zero-day Win32 remote isn't just ignorant. It's dishonest.
    • ^
    • v
    "suprising problems" No. Not surprising at all. AV Engines from any company tend to be pretty much as good as the latest DAT file. AV companies are

    Oh and yes, test case viruses do get out of the lab. Happned more than once while I was at NAI Japan.
    • ^
    • v
    @thomas
    "“They simply have to do it right” is code for “they need to do it the way we tell them to”. You don’t decide what “right” is."

    i grow tired of having my words twisted to mean something entirely different than what i said... "doing it right" is simply a restatement of "not doing it wrong"... the CR/ISE test did things wrong that have already been explored in depth in the past and have been pointed out here and elsewhere...

    "Virus Bulletin and the “field’s own periodicals” don’t count. You’re a smart guy. Go find me real research. Journal articles. ACM Transactions. Things where the majority of the “peer community” don’t work at AV companies. Virus Bulletin is a magazine. I might as well cite my Dark Reading press hits, or this blog."

    fine - http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?...
    but that's the last time i do your homework for you...

    excluding the av field's own periodicals is pointless and arbitrary as it is where the lion's share of the relevant hits will be... your concerns about who the peers will be is meaningless as they will be drawn from the same places regardless of what publication a research paper is published in... peers in the av field don't magically change depending on where you publish...

    "By the way, I’m sure quality research on viruses exists. Just not in anything resembling the numbers that quality systems security research does."

    oh great, so now it's a "mine's bigger" argument is it? well duh - of course a highly specialized field is going to have less published research than a more general one...

    "Best of luck tracking it down, though!"

    oh yeah, so hard - google scholar is your friend...

    "The fact that you can’t see why a countermeasure vendor would want to replicate an outbreak to test a product tells me you don’t spend a lot of time developing security product."

    and the fact that you think it's necessary to replicate an actual outbreak to test anti-viral countermeasures tells me once again that you do not understand anti-virus technology, practice, or issues...

    "A hint: what people first think will solve security problems usually doesn’t."

    a hint: trying to apply principles from your field of expertise to a different field is the cornerstone of false authority syndrome...

    "A “new virus”, exhibiting no novel capabilities other than the fact that antivirus companies haven’t heard of it and thus can’t detect it, developed by a researcher with her own private resources, presents almost no meaningful risk."

    perhaps you meant to include more conditions there that might actually support your conclusion of no meaningful risk because so far you've failed... a virus developed by a researcher with his/her own private resources is able to spread just as well as one written by your average virus writer - in fact virus writers generally develop their viruses with their own private resources... some even do research...

    "The notion that this activity is even comparable to the development of a new zero-day Win32 remote isn’t"

    i assume you meant "remote exploit" and no, i don't think they're comparable - development of the virus is worse by far in the long run and since viruses in general don't rely on software/hardware flaws as exploits do there is no patch or other correction that can be made to mitigate the threat it poses...

    "just ignorant. It’s dishonest."

    great, so now instead of just speaking in a deceptive code i'm actually lying... tell me oh guru what could possibly motivate me to be dishonest about the threat posed by viruses when i am neither part of the anti-virus industry nor do i have any published anti-virus research to defend...

    perhaps if i am being dishonest that dishonesty can be revealed by say identifying a statement of fact which is false... are viruses not able to find new victims autonomously? did stoned.empire.monkey not survive in the wild for over a decade? perhaps viruses really are based on software flaws that can be patched? or maybe it's something that hasn't yet been explicitly said, like maybe the fact that the set of people who qualify as the "wrong hands" for viruses to fall into is orders of magnitude larger (making the chances of encountering one much much higher) than the set of people who qualify as the "wrong hands" for exploits - perhaps that's false somehow...
    • ^
    • v
    One of the aspects of this argument that may be frustrating you is that I give absolutely no credence to the argument that "self-replicating code" is so dangerous that it can't be created for the purposes of research or testing.

    And no, I don't believe "viruses", as we are both likely to define them, "find new victims autonomously". I think that in the manner that viruses use computers, their actions are deterministic and bounded.

    Give me a scenario in which _I_ am likely to spread a virus (say, an OSX 10.4 kernel/dylib infector) that I write myself to test "AV" techniques on my Mac.
    • ^
    • v
    So, the best example of credible virus research you can cite from the literature:

    1. Is from a security generalist

    2. Is a guided tutorial of how to use IDA Pro

    3. Has 1 journal cite, as a place to learn more about the Bagle worm

    Again I believe there's credible virus research out there of a quality comparable to the best work done, even in malware, by, say, Stefan Savage, Vern Paxson, or Nick Weaver. You haven't found it yet, though.
    • ^
    • v
    @thomas
    "One of the aspects of this argument that may be frustrating you"

    frustrating to me? i'm not the one leveling accusations of talking in code and dishonesty...

    "is that I give absolutely no credence to the argument that “self-replicating code” is so dangerous that it can’t be created for the purposes of research or testing."

    not "can't"... *shouldn't*... the risks are real whether you choose to believe in them or not...

    "And no, I don’t believe “viruses”, as we are both likely to define them, “find new victims autonomously”. I think that in the manner that viruses use computers, their actions are deterministic and bounded."

    autonomous and deterministic are not mutually exclusive... even a computer science undergrad would have heard of deterministic finite state automata...

    if you truely doubt the autonomous nature of viruses then we are once again left to conclude that you do not understand what it is you're talking about... although certainly deterministic given a set of known inputs (drawn from the data, resources, and environment), the inputs are not knowable in the wild... you cannot accurately predict what kind of environment a virus will encounter, what data it will find, what kind of network resources it will have at it's disposal, or who the infected files will subsequently be shared with...

    "Give me a scenario in which _I_ am likely to spread a virus (say, an OSX 10.4 kernel/dylib infector) that I write myself to test “AV” techniques on my Mac."

    since i did not say that anyone who creates a virus will innevitably release it the above challenge is irrelevant... you may well not spread it (though i certainly don't trust you enough to share viral materials with you)... the fact of the matter is that the most likely avenue for a virus to escape through an accident and accidents happen - are you perfect? have you never had an accident?

    "So, the best example of credible virus research you can cite from the literature:"

    i'm going to nip this in the bud right now - it is not the best i could find, it's simply one from the first page of results from google scholar... you wanted something recent that was published outside of normal places one would find anti-virus research published so i found one from 2005 that was published by the ieee...

    "Again I believe there’s credible virus research out there of a quality comparable to the best work done, even in malware, by, say, Stefan Savage, Vern Paxson, or Nick Weaver. You haven’t found it yet, though."

    http://members.tripod.com/~k_wismer/papers.htm

    you still think i haven't found it yet? all 3 of the people you mention are listed there... you wanted something recent and since i don't keep track of the publication dates i had to go to google, otherwise i would have just pointed you here to start with...

    of course, my collection is not comprehensive, nor strictly constrained to just anti-virus research, however it should be adequate to demonstrate that there is indeed an anti-virus research community...
    • ^
    • v
    Give me a break. This isn't a page of virus research; it's a page of all security research, in networking, systems, algorithms, and architecture that apply in any direct way on malware in general.

    Vern Paxson is not a virus researcher, and neither is Nick Weaver.

    How about this: make a list of people who have published more than once in an exclusively virus-oriented "journal", like the proceeding of Virus Bulletin conferences. Then cross reference how many times those people publish in ACM journals, or Usenix.
    • ^
    • v
    @thomas
    "Give me a break. This isn’t a page of virus research; it’s a page of all security research, in networking, systems, algorithms, and architecture that apply in any direct way on malware in general."

    the main focus is still self-replicating code...

    "Vern Paxson is not a virus researcher, and neither is Nick Weaver."

    if they research viruses then they are virus researchers - at least part of the time...

    "How about this: make a list of people who have published more than once in an exclusively virus-oriented “journal”, like the proceeding of Virus Bulletin conferences. Then cross reference how many times those people publish in ACM journals, or Usenix."

    how about this... see up there where i said i wasn't going to do your homework for you anymore - try reading that a few more times until it sinks in...

    debating the very existence of an anti-virus research community is insipid... despite the current employment status of it's members the computer anti-virus research organization is not an industry organization, and the european institute for computer anti-virus research is not run by the vendors... the association of antivirus asian researchers is also independant from the industry, plus there are a variety of universities that are active in virus research such as (off the top of my head) the university of hamburg and the university of tampere...
    • ^
    • v
    So list the three "most prestigious" venues for research purely on "self-replicating code", and I'll do the homework for you.
    • ^
    • v
    @thomas
    if you're looking to evaluate the existence of the anti-virus research community, give it up already... it's real even if you don't believe in it...

    if you're looking for actual research i've already provided a page with links to literally thousands of papers, most of which concern viruses... if you need more, google scholar will save you from having to search through multiple sources...

    conferences such as vb and eicar tend to be more prestigious than various publications (as people such as yourself tend to disrespect the publications)... there is the journal of computer virology, however...
    • ^
    • v
    Dr. O'Donnell jumps into the fray: http://www.virusbtn.com/virusbulletin/archive/2...
    • ^
    • v
    I can't read this; there's zero chance I'm going to pay $175 to read "Virus Bulletin". Can you summarize?

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus