I have a few hunch...
"social-engineered malware, which includes also tricking users in clicking on links pointing to files." what do you call this when its brand new if not Zero day malware?
Social-engineering has been around for quite a long time. Maybe not for computers, but I suppose this kind of attacks are common and has been around for quite some time, only unrecognized since their "malware-ness" so to speak was back then, hardly threatening. I remember back in 2009, there was a site advertising a product as a legitimate, even stellar antivirus. Turns out it was a sham. It does a fake scan and nothing more. My brother was fooled. In this sense, they're not zero-day, are they?
""who surfs to 1 million malicious sites...?" missing the whole point." Who has 1M malware in their computer??? but they do their detection testing with 1M malware??? I don't understand the logic they are presenting here as it contradicts what they do with detection testing.
Well, in this sense I'm supposing that malicious sites are entities different from malware residing in your system. What I'm saying is that by using 1M local test samples, you're stressing the capacities of the av being tested. Though unrealistic, it does have some sense to strain a product to find out its limits.
On the other hand, a million malicious sites isn't quite necessary since the malicious codes for sites aren't really that much varied, or am I mistaken? In this view, you can test, for example, 10 different kinds of sites and they can represent as much as 1M others. Malware, however, are varied (and very much so) and much more complex in coding, hence, 1M malware may have individual properties that define them from other malware.
When will they have the capability to test "innovation" like CIS with its "Automatic Sandboxing"? Spreading these old style tests is old now....give users information about what matters which is "Protection" not "detection" by putting dead viruses on your HD and then detect them using Antivirus.....Seriously...lets get serious about Testing. And testing should be FREE!!!! Any financial relationship between testing organisation and AV companies could be seen as a negative. Testing organisations getting money from AV companies should be changed.
Old as they may be, I still find them rather relevant as they do show you the capacity of AV's in case of emergencies. Prevention is indeed a better option, but it is not expected that every malware can be prevented. This is still as serious as it can be because if every other av company focused on prevention, and it so happens that by some misfortune a prodigious cracker manages to slip a virus inside computers, then what of the capacities of the av's to remedy such things? What would become of the users?
Yet, in spite of all these, I must agree that financial relationships do mar the lines between business and honest testing of products. I do not suggest that paying for being tested should be altogether discarded or worse, banned (what they're doing is a form of advertising after all, and advertisements should be paid. Moreover, testing becomes less serious since no benefit on the behalf of the tester is gained from this. Hence, testing would be done however questionable the means and nothing can be said about it. A few might make the charitable deed, but sooner or later, that'll come to an end). What I am suggesting is that av-comparatives change the mode of payment and/or state the transactions that were made (were the payments equal? who tested the av's and are they credible? are they part of av-comparatives?) and the mechanics of this financial relationship, or they can offer a free, but limited version of testing. In this, we could limit one cause of doubt.
well, these are all just a bunch of hunch.
I'm not claiming any expertise in this field nor have I any solid proof of what I have claimed. I have only deduced from what I have come to learn in my experience and from what I have previously read. Hopefully, you won't hold it against me from trying.
Have a nice day.