CAV3 detection rate test [2008.10.05] - 10.26% [CLOSED]

CAVS BETA Corner
41

Alot of viruses are created each and every day, Some are a high risk, Some are low risk that 99.999% of us will never discover and only exist in someones collection.

If it only exists in someones collection then it won’t be a problem. But if it is active, or has the potential to mass spread or cause serious damage or loss then it should be added.

If you think about it, handing out malware samples to antivirus vendors would be what testers do if they cared about money first and foremost. Antivirus vendors have vested commercial interests in performing well in tests they participate in; it would make no sense in forking out testing fees and yet have their reputation dented by poor results.

If testers want money, their priority would be to encourage vendor participation (which, of course, translates to testing fees), and this is done by handing out samples all around so everyone achieves a 99% detection rate. AV-Comparatives gained nothing when DrWeb, Panda and F-PROT decided to boycott their testing, and neither did Virus Bulletin when Trend Micro did theirs. Revenue, reputation as a renowned independent tester, and public trust all go down. I’m not a professional tester; I merely collect malware as a hobby, and I can’t say I know all the insider tricks of the trade. But I do think that Melih trying to take the moral high ground and blaming testers for CAV3’s poor results is way off the mark

EXACTLY: If the Testors handed out their collections, Every hard working vendor would have 100% detection! but then… The testing looses it’s point, because everyone would get 100%.

I don’t think that Melih is trying to shift blame on anyone, He wants to set internet standards because there are none today/very little - “Creating Trust online”

EDIT: I’m not saying CAVS3 is god, It has a long way to go until it’s up to the top with the big boys.
i won’t participate in this thread as it’s silly disscussing detection rates for something so young.

42

So how do you think these samples ended up in my collection? Did I write them myself?

It’s common knowledge that you use only harvested samples - and NOT self-created ones - for testing purposes. It isn’t the Number One rule of antivirus testing, but it does come close. And if it can be harvested, that means it’s actively propagating or being propagated around in most cases.

Products in AV-Comparatives have been getting 97-99% detection for years. VB100% awards are a dime a dozen. But they still exist all the same, rendering your argument null and void.

Perhaps it’s to be able to provide credible tests and unbiased data that testers do not fork over samples on a whim. Users trust testers to conduct accurate testing. Protecting users, on the other hand, is the duty of antivirus vendors.

As an aside, just because samples are handed over doesn’t mean a vendor will achieve 100% detection anyway. Comodo seems to be a fine example of this.

And why not? It’s an antivirus product, and detection rate is one of the most heavily weighed factors in gauging its quality. After all, what is an antivirus supposed to do, if not detect viruses? Certainly not to sit there and look pretty, I hope.

43

I hear and can appreciate what you are saying, Solcroft. I have to agree in that testing must remain unbiased to remain credible and fair to all involved in the process.
Melih has given himself 12 months to place CAV on top. A major undertaking I think, but I among others believe he will accomplish this goal. (Time will be his judge).
I myself question an AV with a score of 100%. Why? With all the Internet talk of the millions of viruses, clones, and variants of themselves, I cannot see any AV reaching 100%. 97-99% I consider quite good in the detection business. Even better if the AV’s can effectively clean out the detected viruses.
Melih’s first step, prevention, has been accomplished. (This still leaves the human factor in the equation, hence the AV requirement we see today).
One just needs to take a step back, a deep breath, and see where things sit now. Then take the necessary steps to reach the goal originally set, even if those steps are small and numerous.
I personally thank you for your time.

44

Athough I may not be able to fully understand AV-Comparative Methodological paper the ondemand comparative scores could also be explained by this snippet.

[i]...snip...[/i]

Test Lab Security
…snip…

AV‐Comparatives sends (missed) samples only AFTER the main tests in February and August to trusted representatives of vendors whose products were publicly tested. We do not send any samples to unknown/untrusted vendors/individuals, no matter what they say or offer.

This appears to be a fair practice to me if there is also in place a fair, verifiable and not discriminatory participation charter.

Simply scoring the number of detected samples bias the testing process only to evaluate in an indirect way the sample gathering abilities of AV vendors.

But AFAIK most malware doesn’t persist in the wild for long timeframes so I guess these non-sharing policies could be used to exploit those baseline testing methodology and possibly be abused to create a closed market.

Is the absolute number of detected samples the only way to estimate the quality of an AV?
Sure it is the most stressed market feature most users are educated to consider.

It is undeniable that even knowing that a threat cannot possibly found in the wild anymore anyone would be thinking about the “lucky” chance of such a dreadful encounter.
So if there are two good and reliable AV brands with the only dicriminant being the detection rates any user would choose the one able to detect most samples.

I read many times the statement that a reliable AV brand should be able to gather malware samples in an autonomous way without the need to rely on AV testers but there is no public test about that.

Just looking at AV comparatives Retrospective/ProActive Tests starting form May 2006 it would be possible to have an idea about heuristic detection and get an idea about the differences among many trusted and ethically outstanding, high-quality AV brands.
I’m not going to quote the average minimun score for those Retrospective/ProActive Tests but http://www.av-comparatives.org/ is there for anyone interested about this aspect and does a great job on that.

The average time to create a new signature over 1000 samples (or any other reliable high number) as per your previous post would be a valuable score to estimate the AV brands analyzing backends. It would be interesting if AV comparatives could add such test to fill in the three months gap between Retrospective/ProActive Tests and On-demand comparatives.
This test will obviously be unable to check the quality of the hash signatures so it could useful to also fin a way to check for False positives (no clue about a reliable way) and prevent the use of hash signatures like CRC32 (this should be simple enough for all kind of tests).

I guess there could be plenty of ways to estimate the quality of an AV product and the non-disclosure policy and restriction of sample sharing may be a legit practice today only because computer viruses are considered a second rate threat in a time where possibly pandemic biological viruses are researched with a much different attitude and with a totally different media awareness.

Edit: corrected innacuracies and typos

45

Gibran

You raise a very good issue here.

Can you imagine a Drug company who discovers a new malicious virus in the wild and keep that to themselves and not sharing it with everyone else while this virus is causing damage to human race!

Thanks for this great point!

The testing of AV products should NOT be about who has managed to keep the virus samples to themselves!!! That shows that you have somehow access to viruses (this even creates doubt about the AV vendor and whether AV vendor itself generating the viruses in order to continue competing)

Testing of AV products should be about “Response time” to “Cure”, overall system performance of the “AV product” on your machine etc. These are what should be tested, NOT HOW MANY VIRUSES YOU HAVE GIVEN SAFE HAVEN TO AND NOT SHARED WITH OTHER AV VENDORS IN AN ATTEMPT TO EXPLOIT THIS MALWARE FOR THEIR OWN BENEFITS WHILE RISKING MILLIONS OF USERS COMPUTERS"

Anybody encouraging this kind of practice to motivate AV vendors into the practice of keeping the malware to themselves in an attempt to gain a top spot in these “So called” testing orgs should immediately change their ways of testing into more credible testing methodologies!

Melih

46

Yes I can. In fact we even have a similar, government-enforced system already in place. It’s called medical drug patents.

And it’s not. I’ve never seen any credible antivirus test where one of the tested categories was “most number of viruses kept secret from other companies”. They’re straight-up flat file scanning detection tests. Very simple.

As I’ve already mentioned, researchers often share samples and information among their peers whom they know and trust, even though they may be from different companies competing commercially against each other on the sales market. Again, I can only wonder why Comodo researchers seem to not be privy to the trust of their fellow colleagues in the field.

47

Solcraft

You are confusing the “Solution” which drug companies patent with “problem” that exist in the wild. I would love to see you show us a deadly virus that affected people and the patent to this deadly virus is owned by a drug company!

We eagerly await your references to the above.

thanks
Melih

48

Oh dear Melih,

don’t waste your time arguing with someone like that. you have a noble quest to to be fulfilled in 12 months time! Just show him that Comodo will have the last laugh!

49

“We”? Or just you personally?

I think it’s time you got over word games and focus on the issue at hand, Melih. To be honest I find it amazing that you would come up with all sorts of preposterous (not to mention non-existent) excuses why antivirus testing should be abolished, simply because they show the low quality of your product. Improving CAV3 is the way to solve these problems, not trying to silence the people who reveal them.

50

That we will foxman. Thank you.

Its the eternal optimist in me, thinking that people are truly honest and positive in their everyday lives, hence I go out of my way to help them and sometimes I get caught by people who see the glass half empty and waste their lives and my time. thats life I guess.

Melih

51

We do look forward to seeing the references you have made about drug companies owning patents to viruses in the wild.

thank you

PS: If you can’t backup your claim, do you want to take back what you said and edit your post?

Melih

52

It’s very telling how you choose to harp and nitpick on an irrelevant side issue while deftly sidestepping the main problem at hand.

Your excuses for why antivirus testing should be abolished are incorrect at best, and an attempt at deliberate misinformation at worst.

But simply for the sake of argument, you were (erroneously) claiming how antivirus companies keep their samples “secret” so as to ensure only their product can detect those viruses. All I’m pointing out is that we do have such an analogy in the pharmaceutical industry, a government-sanctioned system that ensures only the products of select companies can cure certain illnesses. Contrary to your claims, this of course does not happen at all in the antivirus industry, where detection is an issue of manpower, logistics and technology rather than shady backroom practises or patents.

The notion that viruses can be kept “secret” by one or more antivirus vendors is ridiculous at best. If it’s so secret, how did it end up in the tester’s sample set to be included in the antivirus tests, and how did the vendor find it in the first place anyway?

53

The only relation that the patent system could bear with the non-disclosure practice would be patent-trolling besides patents apply to the vaccines (kinda like AV signarures) not on the samples themselves.

What really happen for something taken seriously is this:

Every year, the World Health Organization predicts which strains of the virus are most likely to be circulating in the next year, allowing pharmaceutical companies to develop vaccines that will provide the best immunity against these strains

Yes it’s very simple and yet deceiving and encourage a closed market shifting the focus on a single aspect. What about the others?

Wouldn’t be more meaningful to know in a month timeframe how many new samples each brand detect?
Even sharing those samples after a month wouldn’t affect the value of that test.

And what about testing how much time a Vendor need to create signature for a thousand samples?
Is there anyone who test that? Wouldn’t be interesting to know?
Then again sharing samples won’t impact this kind of tests.

What about false positives? does sharing samples affect that?

The most used test to score the reliability of AVs has the intrinsic property of not being accurate if samples are shared.
You know you gonna have enough samples and reach a high enogh detection rate to think about creating a new AV.
Who cares about the fact that some malwares cannot be possibly found anymore.

After all malware is a second rate threat. It is not necessary to cooperate to eradicate them and encourage competition on different aspects other than the absolute number of detected samples.

54

No such thing takes place. Tests provide users with a measure of a product’s detecion rate: one of the most important factors of an antivirus product’s quality, which also happens to be one that many users cannot readily ascertain by themselves. There are of course other factors: compatibility, system impact, ease of use etc. But most of these factors can be easily determined by installing a trial version and letting it run for a few days, and besides they tend to vary from system to system. Not so easily evaluated, however, is the ability (or lack thereof) of the product to detect viruses. And this is where professional testers come into the picture.

Many people, of course, rate an antivirus very heavily by its detection rate. Yet is this necessarily an inappropriate point of view? After all, who are we to tell them otherwise if they decide that they want a product with high detection rates?

If you’re of the opinion that testing needs to be diversified, that’s all well and good. Many reputable testing organizations do in fact offer such testing. All these, however, do not detract from the primary importance of detection rate testing.

55

If detection rate is what sells the product why should a vendor share a sample?

Melih

PS: Solcroft: Still waiting for the reference to backup the statement you made about drug companies owning patents to viruses in the wild. Or you can simply take back the statement you made and edit your post.

56

Perhaps because they aren’t in it entirely for the money. I can only speculate, but perhaps researchers are driven by the desire to do good, or maybe they see themselves fulfilling their bit of collective public responsibility. One of the first things DrWeb did when they discovered the Rustock.C driver sample was to send copies to their competitors. Aside from samples, researchers often gather at events and conferences to share news, technological developments and other bits of info. I can’t claim to know fully why they do it. But they do nonetheless.

And with this little remark of yours, Melih, I think I may have the first clue as to why your company appears to be shunned by other researchers at large.

I’ve already addressed it, for your sake more than out of any necessity with respect to this discussion, since you insist on harping on it endlessly. Perhaps you’re afraid to inspect my reply?

57

How many of the AV firms we collobrate with? do you know?

Melih

58

You’re the one who’s been crying all this while that you aren’t getting any samples. For the sake of your customers, I do hope, of course, that Comodo is a company worthy of the industry’s trust.

59

You are the one making a statement with no facts to back it up.

first you said detection ratio of 10%… again with basis for your statement
now you made a statement about us not working with others… of which both are totally false!

Please, either come here armed with facts, or just ask questions to learn. You seem to be throwing out baseless statements without understanding our motives!

Our motive is to secure end users! As long as people make detection their “selling point” there will be no material collobration that will result in “speedy” responses to malware outbreaks! Our intention is to create a unified front against malware by making sure AV vendors collobrate almost in real time about new threats. We are trying to solve the “problem”! The “symptom” of our AV having very high detection is just a matter of time. However unless the “problem” is resolved end users will suffer!

Melih

60

This would simply ruin the AV business. And they don’t want that. I would like it either, cos there won’t be any competition :slight_smile: