Retrospective Test November 2011

[Guest]

[GOA]

New Retrospective Test (Heuristic) from av-comparatives

http://www.av-comparatives.org/en/comparativesreviews/detection-test

[Tech]

Just to help cross posting:
Wilders: http://www.wilderssecurity.com/showthread.php?t=312396 and http://www.wilderssecurity.com/showthread.php?t=312445
avast justification of the poor qualification: http://forum.avast.com/index.php?topic=88672.msg709388#msg709388

[HeffeD]

avast justification of the poor qualification: http://forum.avast.com/index.php?topic=88672.msg709388#msg709388

That's a pretty weak justification... In other words, once something gets to your machine, you can pretty well write Avast off... 

[panic]

That's a pretty weak justification... In other words, once something gets to your machine, you can pretty well write Avast off... 

But interesting to note that their focus is shifting to prevention as opposed to detection.

[Tech]

That's a pretty weak justification... In other words, once something gets to your machine, you can pretty well write Avast off... 

If a driver is loaded, nothing can be done, by avast, by CAV, by CIS or by anything else...
It's pretty too late.

But interesting to note that their focus is shifting to prevention as opposed to detection.

Interesting uh? I've arise a discussion about that in a reserved part of the forum and it does not go further... or, better, I've got bashed there because I was trying to say that

[HeffeD]

If a driver is loaded, nothing can be done, by avast, by CAV, by CIS or by anything else...
It's pretty too late.

So you're saying that all of the malware that wasn't detected by Avast had installed a driver?

Edit: Just to clarify, I'm not trying to be accusing, I'm just curious. I haven't looked at the results of the test.

[Tech]

So you're saying that all of the malware that wasn't detected by Avast had installed a driver?

No, I'm not an insider from av-comparatives
What I can say, now, is that the av-comparatives team (IBK) has posted in avast forum and specifically mention it (http://forum.avast.com/index.php?topic=88672.msg710147#msg710147).

Edit: Just to clarify, I'm not trying to be accusing, I'm just curious. I haven't looked at the results of the test.

No problems HeffeD. You're welcome.

[Melih]

Hi Vlk, I disagree with you.
1) only about the half is pointing directly to binaries/files. The rest are exploits. In your misses you for sure also encountered some exploits and not only direct links. The "problem" is (and it is even written in the report) that practically all products (including of course Avast) are good are blocking/detecting exploits/drive-by downloads. That's also why the % are so high. If you look at the latest research of Microsoft, the biggest issue for users are not 0-day exploits (according to their paper its even close to 0%) but social-engineered malware, which includes also tricking users in clicking on links pointing to files. If you miss malware from the web, the test will and does reflect that. But I am glad to hear that the next version will improve further in this regard.
2) too less samples: others use 10 samples for such a test and base ratings based on that. We use usually 50x that size. Arguing that sample size is too small doesn't sound fair. If it would be 1 million someone would say "who surfs to 1 million malicious sites...?" missing the whole point.
3) How user-dependent cases are interpreted is up to the user. I do not believe that a product which would ask the user for everything should get the same like a product which is able to distinguish between malware and goodware without letting the decision up to the user. Anyway, only on chart2 you can sort based on the green bar. In chart3 you can combine blocked+userdependent.
4) I expected that also Whole Product Dynamic Tests would be criticized (like any other test) in future if the scores are unfavorable for someone, despite the internal promotion for such sophisticated tests.


*****************************************

the above is, I am assuming from av comparitives.....

few things require clarification:

"social-engineered malware, which includes also tricking users in clicking on links pointing to files." what do you call this when its brand new if not Zero day malware?

""who surfs to 1 million malicious sites...?" missing the whole point." Who has 1M malware in their computer??? but they do their detection testing with 1M malware??? I don't understand the logic they are presenting here as it contradicts what they do with detection testing.

When will they have the capability to test "innovation" like CIS with its "Automatic Sandboxing"? Spreading these old style tests is old now....give users information about what matters which is "Protection" not "detection" by putting dead viruses on your HD and then detect them using Antivirus.....Seriously...lets get serious about Testing. And testing should be FREE!!!! Any financial relationship between testing organisation and AV companies could be seen as a negative. Testing organisations getting money from AV companies should be changed.

Melih

[spainach_12]

I have a few hunch...

"social-engineered malware, which includes also tricking users in clicking on links pointing to files." what do you call this when its brand new if not Zero day malware?

Social-engineering has been around for quite a long time. Maybe not for computers, but I suppose this kind of attacks are common and has been around for quite some time, only unrecognized since their "malware-ness" so to speak was back then, hardly threatening. I remember back in 2009, there was a site advertising a product as a legitimate, even stellar antivirus. Turns out it was a sham. It does a fake scan and nothing more. My brother was fooled. In this sense, they're not zero-day, are they?

""who surfs to 1 million malicious sites...?" missing the whole point." Who has 1M malware in their computer??? but they do their detection testing with 1M malware??? I don't understand the logic they are presenting here as it contradicts what they do with detection testing.

Well, in this sense I'm supposing that malicious sites are entities different from malware residing in your system. What I'm saying is that by using 1M local test samples, you're stressing the capacities of the av being tested. Though unrealistic, it does have some sense to strain a product to find out its limits.

On the other hand, a million malicious sites isn't quite necessary since the malicious codes for sites aren't really that much varied, or am I mistaken? In this view, you can test, for example, 10 different kinds of sites and they can represent as much as 1M others. Malware, however, are varied (and very much so) and much more complex in coding, hence, 1M malware may have individual properties that define them from other malware.

When will they have the capability to test "innovation" like CIS with its "Automatic Sandboxing"? Spreading these old style tests is old now....give users information about what matters which is "Protection" not "detection" by putting dead viruses on your HD and then detect them using Antivirus.....Seriously...lets get serious about Testing. And testing should be FREE!!!! Any financial relationship between testing organisation and AV companies could be seen as a negative. Testing organisations getting money from AV companies should be changed.

Old as they may be, I still find them rather relevant as they do show you the capacity of AV's in case of emergencies. Prevention is indeed a better option, but it is not expected that every malware can be prevented. This is still as serious as it can be because if every other av company focused on prevention, and it so happens that by some misfortune a prodigious cracker manages to slip a virus inside computers, then what of the capacities of the av's to remedy such things? What would become of the users?

Yet, in spite of all these, I must agree that financial relationships do mar the lines between business and honest testing of products. I do not suggest that paying for being tested should be altogether discarded or worse, banned (what they're doing is a form of advertising after all, and advertisements should be paid. Moreover, testing becomes less serious since no benefit on the behalf of the tester is gained from this. Hence, testing would be done however questionable the means and nothing can be said about it. A few might make the charitable deed, but sooner or later, that'll come to an end). What I am suggesting is that av-comparatives change the mode of payment and/or state the transactions that were made (were the payments equal? who tested the av's and are they credible? are they part of av-comparatives?) and the mechanics of this financial relationship, or they can offer a free, but limited version of testing. In this, we could limit one cause of doubt.

well, these are all just a bunch of hunch.  I'm not claiming any expertise in this field nor have I any solid proof of what I have claimed. I have only deduced from what I have come to learn in my experience and from what I have previously read. Hopefully, you won't hold it against me from trying.

Have a nice day.

[spainach_12]

Oh, and on a side-note, each time I refer to variation of codes, I refer to the variation of known codes. 

[Melih]

http://forums.comodo.com/melihs-corner-ceo-talkdiscussionsblog/avcomparativesorg-bullying-and-financial-deals-with-anti-virus-vendors-t78869.0.html;msg564687#msg564687

[spainach_12]

http://forums.comodo.com/melihs-corner-ceo-talkdiscussionsblog/avcomparativesorg-bullying-and-financial-deals-with-anti-virus-vendors-t78869.0.html;msg564687#msg564687

While it's saddening for them to get a wee bit greedy (though I'm not altogether discarding the possibility of a miscommunication seeing so many instances of possible variations of meanings in statements), I still stand-by what I said that it is still recommended for payments to be made. It's just that the mechanics they employ/implore are unsuitable, even perhaps grievously faulty. Disappointing, yes it is.

[Melih]

While it's saddening for them to get a wee bit greedy (though I'm not altogether discarding the possibility of a miscommunication seeing so many instances of possible variations of meanings in statements), I still stand-by what I said that it is still recommended for payments to be made. It's just that the mechanics they employ/implore are unsuitable, even perhaps grievously faulty. Disappointing, yes it is.

AV-Comparatives Force AntiVirus vendors to deny even existence of a financial deal between them. Why?

[WinDefend]

AV-C's reply to Comodo: http://www.av-comparatives.org/forum/index.php?page=Thread&threadID=1054

[Melih]

Andreas of AV-Comparitives said: "When I (Andreas) started doing the public tests in 2003, I did it for free and asked users if they wanted to donate something. Practically no one donated, and based on the high demand for continuing the tests, and the increasing complexity of the tests, I had to start asking all vendors to pay a fee."


But why do they force the Antivirus companies to deny the "existence" of the financial relationship? Why are they trying to hide this?

Here is the clause look at the highlighted section.

Why are they scared of letting public know? How can they be trusted by public if they don't trust public with this information in the first place?

[Tags:]