Anti-virus companies pissed with Consumer Reports

[doob]

Consumer Reports recently conducted one of the most thorough tests ever of antivirus programs. But to really put these security programs through the paces, the magazine hired a firm to create 5,500 new viruses, using them to test the antivirus software products for their ability to detect unexpected threats.

Now antivirus companies are crying foul, saying the magazine ignored a long-standing principle not to invent new viruses.

"Creating new viruses for the purpose of testing and education is generally not considered a good idea,” wrote Igor Muttik of McAfee's antivirus lab on a public company blog this week. “Viruses can leak and cause real trouble." The entry helped touch off a firestorm.

From MSNBC article

Now now... mighty interesting,McAfee shaking after all this specially when their software didnt came out on top instead BitDefender and unexpectedly for me Zone Labs!

Also taking the words from a user in another forum where i accidently got to read this the first time here

How dare you actually crash a car to see how it does in a real collision!!!!

If car manufacturers would inform the government to only run side impact on car "A" but go ahead and do a front impact on car "B" as long as it's under 50 MPH. Every car could look GREAT with those restrictions.

You want to test a Antivirus product... you need to really test it with new stuff and see how each product performs.

IMO this is almost the best way to actually test these anti-virus softwares. Hope they do another test like this next year perhaps also for M$ Vista and see how well their new OS behaves in security

In case you don't notice, here's the source

Edit: ahm... maybe i should have writen mad instead of pissed at the topic, mods please change if you feel that language is to inapropriate, my excuses if such.

 

[Bouncing Zabaglione Bros.]

If you create new viruses, you are no longer testing the complete ability of a scanner to protect your PC from viruses during every day use. You're testing the scanner's heuristics for unknown virus signatures, which is normally just one part of a virus scanners job. If it were a real-world infection, you might also be testing the scanners ability to update it's signatures.

As with most virus tests, you have to understand what the testing methodolgy was, as it's often not very clear what is actually being tested as opposed to what the big headline is screaming.

For instance, I've seen "industry tests" where major functionality of an anti-virus product was turned off to test the one specific anti-intrusion part of the package. In the real world, with all parts of the package working (as would normally be the case), the results would be far different, as the whole package is designed to work together, and provides a much stronger defence than just having one part of a suite of defence measures enabled.

I'd rather anti-virus companies developed robust solutions to the problem of virii/trojans/worms, than pandered to get high scores on this this kind of high publicity, flawed methodolgy testing.

 

[zsouthboy]

Antivirus companies suck.




The end.

 

[mjtdevries]

From MSNBC article

Now now... mighty interesting,McAfee shaking after all this specially when their software didnt came out on top instead BitDefender and unexpectedly for me Zone Labs!

I STRONGLY suggest people read the MSNBC article, and don't just read doob's selective quoting.

The article makes it very clear that other sources than McAfee oppose this method far more strongly then McAfee did. It's very unfair to make it sounds as if just McAfee is complaining about it.

From the article:

"Creating new viruses for the purpose of testing and education is generally not considered a good idea,â€￾ wrote Igor Muttik of McAfee's antivirus lab on a public company blog this week. “Viruses can leak and cause real trouble." The entry helped touch off a firestorm.

Other antivirus commentators were far more inflammatory, accusing Consumer Reports of being irresponsible.

"The antivirus community has always been very strongly opposed to the creation of new malware for any purpose," wrote John Hawes, the technical consultant at antivirus Webzine Virus Bulletin. "There's just no need for it. Plenty of new viruses are being written all the time, why would anyone in a responsible position want to add to the glut?"

As mail admin for a large firm, I am very much aware of the virus thread.
But the MOST important part of antivirus software is NOT the heuristic scanning, but the speed at which the antivirus companies update their definition files.

Heuristic scanning has never been and will never be the primary means to detect viruses, because it is simply not reliable enough for that.

 

[doob]

My bad, i should'v wrote "and others" after McAfee. I didn't mean to specificly target at McAfee, its one of the most populars. Next time when it comes to companies, ill leave all of them out of the context and use none as example, so that it doesn't seem im targeting specificly one wichever may be, even if just as a well known/popular company and try to maintain myself more politicaly correct if i may say so.
Plus the topic doesn't say specificly it's/was a company but companies, it's in plural, so in principle whoever starts reading should be aware it's more than just one (McAfee) company, and that that selective quote from the original article is just part of several others mentioned, more wasn't quoted due to forum rules about not posting entire articles.

Anyway, personally doesn't change my point of view and i disagree with the idea that heuristics detection systems shouldn't be of higher importance, because so far to today the primary methodology is letting it (virus) act first and do some damage, let it spraid into a contained system, analyze it and release the patch later, in other words, let it break some pots and pans before it gets cleaned. That case could probably (although currently not very likely) avoided with better heuristics or detection methodologies.

 

[Bouncing Zabaglione Bros.]

Anyway, personally doesn't change my point of view and i disagree with the idea that heuristics detection systems shouldn't be of higher importance, because so far to today the primary methodology is letting it (virus) act first and do some damage, let it spraid into a contained system, analyze it and release the patch later, in other words, let it break some pots and pans before it gets cleaned. That case could probably (although currently not very likely) avoided with better heuristics or detection methodologies.

Heuristic detection of virii is just one of a number of techniques used by the newest generation of anti-virus software. It's a useful addition, but there are other techniques such as virus signatures, behaviour detection, integrity validation, etc, that are just as valuable and important in defending against a unknown virus attack in the real world.

Just focussing on heuristics for unknown virii is like trying to find out how fast a car goes by taking the engine out and measuring it's power output on a dyno. You only get a small part of the picture, not the whole picture.

 

[Colourless]

Heuristic detection if it's too agressive also has the lovely side effect of causing false positives (not that Anti-virus software packages regularly get false positives that cause problems for some people).

When releasing Exult, we've had quite a number of problems with antivirus software detecting us as being infected because of the stub used on the self extracting archive. Always annoying and we've only got a small amount of users... as far as I know.

 

[doob]

At the end of the line you can read "and other methodologies." or other methods, that would include: "virus signatures, behaviour detection, integrity validation, etc"
And stop making conclusions that im all "pro-heuristics".

Fact is that that showed how anti-virus software is pretty much incapable or very un-effective at detecting new viruses or simple slightly modified old ones, they relly largely on the company efforts of maintaining their database updated and broadcasted.
Probably current hardware architectures don't let much efficiency room to conduct more efficient code in real time, maybe some AI-PU (Artificial Inteligence Processing Unit) maybe used in the future for such task and not only games! Who knows!

 

[mjtdevries]

It would be very nice if heuristic scanning would be more effective.
The antivirus companies would like that a lot too. After all, making and testing all those signatures files costs them a LOT of effort.

If there was any chance of making heuristics the primary means of detection, they could save a LOT of money with it.

The problem is that it is very difficult to detect a virus using heuristics. Most of the characteristics of a virus are also present in innocent programs.
Not only virusses send mails. Not only virusses spread files around (think about installers) etc etc.
Especially trojans aren't all that different from normal applications.

 

[Bouncing Zabaglione Bros.]

A
And stop making conclusions that im all "pro-heuristics".

Anyway, personally doesn't change my point of view and i disagree with the idea that heuristics detection systems shouldn't be of higher importance, because so far to today the primary methodology is letting it (virus) act first and do some damage, let it spraid into a contained system, analyze it and release the patch later, in other words, let it break some pots and pans before it gets cleaned. That case could probably (although currently not very likely) avoided with better heuristics or detection methodologies.

Well that's what you said above! Or did you mean to say something else again?

Fact is that that showed how anti-virus software is pretty much incapable or very un-effective at detecting new viruses or simple slightly modified old ones, they relly largely on the company efforts of maintaining their database updated and broadcasted.

Fact is, that test doesn't test how effective an antivirus scanner is - it just tests it's ability to heuristically identify unknown viruses, which is only one small part of what today's modern virus scanners do.

 

[_xxx_]

It would be very nice if heuristic scanning would be more effective.

If it would be any more effective, it would always imediately delete IE, Media Player, hell probably the whole Windows right on the initial check...

 

[doob]

Well that's what you said above! Or did you mean to say something else again?

Not exactly, maybe saying or/and detection methods will clear things as including improved methods curently used and partially mentioned.

True the test doesn't show how effective they are (and i never wrote it was or implyed to), but shows how poor they are facing slightly changed/new virus's. And that imo is more important than detecting (and/or removing) already known ones. That should be more than expected to work flawlessly. If some one/group is planning to proceed with industrial software attacks/spying it won't be done with common know virus's and detected variations.
So with that in mind the article shows how poorly those reviewed software behave under aone of many possible situations.

 

[doob]

In one of my posts i wrote effective but meant to write efficient, it has been corrected.
And i assume that was also what mtjdevries meant. Some users mother language isnt english and slight miss-writes/interpretations happen, wich was probably the case.

 

[Bouncing Zabaglione Bros.]

Not exactly, maybe saying or/and detection methods will clear things as including improved methods curently used and partially mentioned.

True the test doesn't show how effective they are (and i never wrote it was or implyed to), but shows how poor they are facing slightly changed/new virus's. And that imo is more important than detecting (and/or removing) already known ones. That should be more than expected to work flawlessly. If some one/group is planning to proceed with industrial software attacks/spying it won't be done with common know virus's and detected variations.
So with that in mind the article shows how poorly those reviewed software behave under aone of many possible situations.

But if you think about it, how many people get hit by a completely new virus within the first few hours? You are far, far more likely to get hit by a current virus in circulation.

Some of the newer virus scanners even have the ability to increase the rate at which your system updates itself from their virus signatures in the event of increased virus activity, so unless you are one of the very first people to get a new virus in the first few hours of it's existence, heuristics won't help you.

Even if you do get hit by a new virus, other methods, like application intergrity and behaviour scanning will pick it up.

Heuristics just arn't that easy to do well, because it requires your software to be able to make value judgements about what the code in a virus might be capable of doing, and that is something that is easily hidden, and is duplicated by many forms of legitimate software. That's why virus companies additionally use other methods, and mantain hundreds of thousands of virus signatures - because heuristics alone can't do the job.

To just test a product based on just that one part of it's functionality, while ignoring the rest of the functionality which is designed to work together as one whole defence suite produces a nice headline for the website, but also a completely distorted view of whether the product actually protects your PC.

It's rather like saying that if you blindfold the driver, tie his hands behind his back and ask him to drive up a mountain, he'll go over a cliff, so therefore all cars are useless and dangerous to drive. All car manufacturers should be working on cars that drive themselves up mountains when the driver is blindfolded and tied up.