AV-Comparatives.org, bullying, censorship and financial deals-continued...

[Guest]

[clockwork]

Gizmo criticizes the Matousec's tests. It's a technical reading, but seems fair (http://www.techsupportalert.com/content/matousec-personal-firewall-tests-analyzed.htm).

That was interesting.
Now i see.

We can compare comodo to a degree with the other full tested programs. But we can not make assumptions about all the other programs. Apart from the fact, that they were falling over one test in the progress, which doesnt allow them to proceed.

Another interesting point: The claims of the products themself, compared to the "claims" that the matousec tests are showing as general claims. That was what i meant with my question, if some products just dont fit to that test. Like comodo doesnt fit 100% in antivirus-only tests.

[Tech]

Are you sure you posted the correct article Tech? The cited Gizmo article is not technical at all, the author has an issue with Matousec's scoring method, results presentation and what they consider a critical failure (including its consequence).

Thanks Kail. Seems that the article was changed... (Updated 8. October 2011). I was almost sure it criticizes the metodology (using technical language). I cannot understand... But, after all, sorry. I'll edit my earlier post.

But, after all, does anybody know how Matousec's tests are financed?

[Tech]

Just for the record, my earlier post was a "canned" text that I had posted before.
http://forum.avast.com/index.php?topic=78200.msg647015#msg647015 (here seems to be one of the first time: Wednesday, 18-05-11, 22h14).

[pc_pete]

Let's pretend that I agree with this statement...Now let's put it to the test. I have programmed myself to look for an AV with the HIGHEST rate of DETECTION because according to the 'experts' detection is the most important and reliable factor in determining the EFFECTIVENESS of an anti-virus engine. Now where would I find the AV produt with the highest detection rate? Duh! av-comparatives!

Yay! Now that I have found and purchased 'the best' product out there according to the 'experts', I feel safe. However, one day my system got heavily infected and now my computer will no longer startup. I cannot blame the AV because after all their detection rate was about 97%. I guess that malware was part of the 3%.

However right before my system got infected, I sent this weird file that messed up my system to my friend who runs Comodo Antivirus. Like my AV, Comodo did not DETECT it either but somehow my friends system is perfectly fine!? How can this be!? Because it was SANDBOXED!

So in this scenario we have person A who purchased the best AV according to AVC and ended up with a non booting computer, and we have his friend who has Comodo Antivirus (with a relative low detection rate) and yet person b's computer is perfectly fine!

Would it not be logical to say that Comodo Antivirus performed better than AVC's most recommended product, SPECIALLY because Comodo AV's detection rate is lower and yet it saved your computer in the end WITHOUT asking questions???

Please explain to me again why detection is still more important than protection!? In the end of the day all that matters is if your computer is still wokring properly rather than how many popups your AV did shoing off it's detection power?

Apart from being a bit 'now-what-if' dependent (not to mention the assumption that your "friend" has not torn his hair out trying to stop safe programs being made inoperable by being 'partially limited'), I've no argument with your conclusion re the comparative effectiveness of 2 AV products.

However I'm not sure what it has to do with the statement from AVC via Melih that you quoted.

"Malware detection rate is still one of the most important and reliable factors in determining the effectiveness of an anti-virus engine which works without asking for user interaction, decision or opinion."

Even if we assume that this statement is tailored to support the worth of AVC testing, it does specify "an anti-virus engine", not an AV product (or even an "AV produt" ), not a firewall, HIPS, hash/whitelist checker, sandbox, virtual machine, etc. So while I understand what you're saying, I also agree with the quote.

[pc_pete]

Guys, let's don't forget about false positives. According to AVC CIS has 'crazy many' fps. That's a great disadvantage for CIS.

Its easy to get FPs with any AV.

the question is: How wide spread the app is that causes FP. Just saying it has FP is at best misleading...it could well be one application you wrote yourself that causes an FP, this would not affect other users. Also, did anyone validate the statement they made?

Right! I think we've learned something today.

• If CIS flags many FPs in comparative testing, user-written apps could be to blame.
• If CIS flags many FP's on the "average user's" PC, tell them to stop writing apps.
• Note to self. Take authorship credit for the 100s of FPs I've reported to Comodo.

[Solarlynx]

Right! I think we've learned something today.

• If CIS flags many FPs in comparative testing, user-written apps could be to blame.
• If CIS flags many FP's on the "average user's" PC, tell them to stop writing apps.
• Note to self. Take authorship credit for the 100s of FPs I've reported to Comodo.

Lol.
Unfortunately Comodo persistently disregards the FP issue. That's a great minus to CIS, sorry to say that.

[dariovolaric]

Even if we assume that this statement is tailored to support the worth of AVC testing, it does specify "an anti-virus engine", not an AV product (or even an "AV produt" ), not a firewall, HIPS, hash/whitelist checker, sandbox, virtual machine, etc. So while I understand what you're saying, I also agree with the quote.

You are correct, detection rate might be the most reliable way of testing an AntiVIRUS product yes. But these days antiVIRUS as simply not enough. It might have been enough 10 years ago, but these days you simply need MORE than something that only looks for Viruses. You know that, I know that and everyone else here knows that. But the sad fact is that the average user does NOT know that. To them an antivirus product or an internet security suite is the same. It protects your computer from harm and infections. And AV-comparatives and all the vendors that put the AVC logo on their box exploit this fact that people don't know about computer security.

Let's not forget we are talking about the AVERAGE computer user, not about experienced users like us. If you take a look at the comments about the av-comparatives website in WOT (Web Of Trust) you will see comments like:

Good website for information regarding virus testing as well as anti-virus programs co-operation in a network of threading information and making it available for public viewing. A+.

and

Good company/group that compares antivirus suites.

and

One of the most reliable AV test sites. Unbiased and accurate from what I have always seen.   

and

Very informative, a must to visit if shopping for antivirus software.

Just to name a few...and the last comment is what I am talking about. Their tests DO influence what people choose to purchase. Yes I agree that they do a good job in testing how effective an AV product DETECTS a VIRUS, but is that really what people want? Just a product that detects viruses? I thought they were looking for something that keeps their computer SAFE in the end? You honestly believe that people say "I just want something that RECOGNIZES viruses, I am not too concerned about other types of malware and the 3% of malware that is NOT detected by the AV."

It is fascinating to see that the world of preventing and treating diseases is exactly the same in the digital world as in the physical world. Chemo and radiation treatment is THE industry standard to treat cancer and THERE IS NO OTHER way! This has been proven by medical experts, the FDA, the scientists and every 'educated' person you ask. To simply even entertain the thought that there is an 'alternative' cure for cancer it preposterous! If there was, it would have been on TV, in the newspapers, on the internet, everyone would have been notified of this, and certainly the FDA would not keep this from us would they? It is the same in the digital world. People see 'experts' like AVC and others as someone they can trust that they will advise them what to get. Sure people have the freedom to do their homework and look at other sites, but the sad fact is that people do not want to read and learn. They simply want an 'expert' to tell them what to buy. And as you see in these comments about the site, they actually do use that site to decide what AV product they will buy. But is what they buy really going to protect their system as good as other products?

And then there is the question about fairness and honesty about the tested AV products. If it is really unbiased, then why don't they test ALL av products, regardless if they pay thousands of dollars or not? And why don't they give out more details about HOW they test this? And why is there not third part involved in checking on them? Where is the transparency?

I believe the best method of testing AV products is the way Languy99 does it. Simply install an AV product, leave it in STOCK SETTINGS (as most regular users would) and simply go to dangerous sites and run malware (zero day). If the AV product asks you anything, simply answer "YES" without reading (as most average users would) and see what happens. Is your computer infected in the end of the ride or not? Here you can actually SEE what is going on and HOW it was tested. Then you can decide for yourself if you liked what you see or not, rather than look at some statistical numbers on the site. But I guess it takes less time to simply look at the highest scoring number rather than watch all the review video's...

[clockwork]

Very informative, a must to visit if shopping for antivirus software.
Their tests DO influence what people choose to purchase. Yes I agree that they do a good job in testing how effective an AV product DETECTS a VIRUS, but is that really what people want? Just a product that detects viruses? I thought they were looking for something that keeps their computer SAFE in the end? You honestly believe that people say "I just want something that RECOGNIZES viruses, I am not too concerned about other types of malware and the 3% of malware that is NOT detected by the AV."

You can buy a car, but that doesnt mean that bikes shouldnt be tested anymore.

You are correct, detection rate might be the most reliable way of testing an AntiVIRUS product yes. But these days antiVIRUS as simply not enough.

....one of the most

And then there is the question about fairness and honesty about the tested AV products. If it is really unbiased, then why don't they test ALL av products, regardless if they pay thousands of dollars or not? And why don't they give out more details about HOW they test this? And why is there not third part involved in checking on them? Where is the transparency?

The testers of the testers. I am sure, this tester tester company would take money, and the tester companies would pay because they can advertise with that results, and then someone would say, .... the same as you

I believe the best method of testing AV products is the way Languy99 does it. Simply install an AV product, leave it in STOCK SETTINGS (as most regular users would) and simply go to dangerous sites and run malware (zero day). If the AV product asks you anything, simply answer "YES" without reading (as most average users would) and see what happens. Is your computer infected in the end of the ride or not?

You can bet, it will be infected, whatever product you use

Average users can read.
You think, "those" users would watch videos or would read test interna? But you dont think they could read the question, instead of clicking blind yes?

[dariovolaric]

You can buy a car, but that doesnt mean that bikes shouldnt be tested anymore.

People can clearly see the difference between cars and bikes, but not between Antivirus and Internet Security suites.

....one of the most

To the average user it is THE most.

The testers of the testers. I am sure, this tester tester company would take money, and the tester companies would pay because they can advertise with that results, and then someone would say, .... the same as you

Correct, even testers of testers can be corrupted.

You can bet, it will be infected, whatever product you use

I have seen live tests of Comodo Internet Security by Languy99 and the system was NOT infected with zero day threats.

Average users can read.

Yes they do have the ability to read, but they CHOOSE not to. So many times I see people NOT READ the 'internet explorer has blocked a popup, lalalala" message and yet they do not know what to do, or they complain "I keep getting this message over and over! Even though it gives you do option to add the site to the safe list. So either people DO NOT READ or they are too dumb. And this is just one simple example of not reading. Want another one? How many average users do you know with at least one obsolete google / yahoo / facemoods / etc... toolbar? If they could READ they would select CUSTOM INSTALL and select that they do NOT want the toolbar, rather than "yes yes yes next next next whatever" and then later complain "why do I have this facemoods toolbar???".

You think, "those" users would watch videos or would read test interna? But you dont think they could read the question, instead of clicking blind yes?

Absolutley! Because they are not LOOKING for the question or an answer to that question. They WILL read the test results or watch the videos because they are LOOKING for an answer, because the WANT to know. Plus blindly clicking yes is also an automated response to many. Because clicking on NO usually does not get you what you want, and clicking on YES does. So without reading, they just want to get what they want and simply click "yes", or they trust that the software vendors know what's best for you. So if it 'recommends' you to install the facemoods toolbar, then you will run it.

Even worse, AV products that do not recognize the virus will not even ask you if you still want to run it, but just blindly run it for you, while Comodo places it in the sandbox without asking you. You are still able to run it, install it, etc...But it won't do any permantent harm to your computer. That is still better than not 'bothering' the user and silently running the program. It's like an unknown person knocks on your door and you ask the security man "do you think this person is a threat" and the security man looks in a list and says "hmm...no I did not find this person in the list of bad people, so I have no reason to stop him". While Comodo says "I don't have this person in the list of good nor bad people, so I will restrict his actions in your home and keep an eye out on him for you". What doorman would you prefer? The one that does allow you to have strangers in your home do whatever they want or the one that restricts strangers from doing harm to your home by placing them in a 'copy' of your home?

[clockwork]

When i read your texts, i think, antivirus is very important, while you try to explain the opposite

--------------------
Quote from: clockwork
You can bet, it will be infected, whatever product you use

And you answered:
I have seen live tests of Comodo Internet Security by Languy99 and the system was NOT infected with zero day threats.

I say: You will most probably infect your computer or files if you blindly press yes, like you suggested for testing of security programs.... (Languy doesnt blindly press yes).
If "yes" would lead to the safe action, the program could excute "yes" without user consent ....



Btw, i dont know why you think that anybody would discuss about if its not better to have more than an antivirus. Thats totally not the point of this topic. And who would say, no?

[The CEO]

Its easy to get FPs with any AV.

the question is: How wide spread the app is that causes FP. Just saying it has FP is at best misleading...it could well be one application you wrote yourself that causes an FP, this would not affect other users. Also, did anyone validate the statement they made?

Doesn't that sound just a teensy bit like the defense used by some AV vendors when their product scores a False Negative? Justifying a failiure to detect that brand new so-called "zero-day" malware.

[Solarlynx]

+1 to the previous post
FPs must not be disregarded. Unfortunately Comodo ignores negative effects of FPs using far-fetched explanations.

[Chiron]

I do agree that false positives are still a problem.

Its easy to get FPs with any AV.

the question is: How wide spread the app is that causes FP. Just saying it has FP is at best misleading...it could well be one application you wrote yourself that causes an FP, this would not affect other users. Also, did anyone validate the statement they made?

While I don't entirely agree with this statement I do have to ask this, does anyone know where they get their safe files to be checked? I know that I've been downloading a lot of different programs lately, and although a few weren't whitelisted (which I fixed) I didn't run into a single FP. Where do they get these programs to check?

From what I've seen, they cannot be popular downloads, as I've run many of these without a single popup. They must be more obscure downloads.

Now, putting CIS aside for the moment, isn't this a bit unfair to the other AV's? Why should they, who are mainly meant to work on the system of an ordinary user, not be judged by testing against the types of programs that these types of users would use? Isn't this actually who the tests are aimed at?

Sorry, I'm actually a bit confused about the methodology and who the tests are aimed at.
This isn't meant to be pro-Comodo , or anti-anybody, I just want to understand the reasoning behind testing for FP's they way that they do. Which files are more important? Why aren't there different categories (ie: FP's for popular software or FP's for unpopular software) etc...

[Tech]

I know that I've been downloading a lot of different programs lately, and although a few weren't whitelisted (which I fixed) I didn't run into a single FP. Where do they get these programs to check?

Chiron, I were running CCE and got quite some FP in the setup files of mine. Some are PUPs but others are clearly FP. I have already uploaded them in the past, but they're there all the time...

Now, putting CIS aside for the moment, isn't this a bit unfair to the other AV's? Why should they, who are mainly meant to work on the system of an ordinary user, not be judged by testing against the types of programs that these types of users would use? Isn't this actually who the tests are aimed at?

The more aggressive you go with heuristics, behavior analysis, proactive detection... the more change to get FP. What I can say is that, in some tests and awards, a single FP is enough to lose the "game". Avira always push forward in proactive. avast is much more conservative (and some users are complaining about this).

[Solarlynx]

What I can say is that, in some tests and awards, a single FP is enough to lose the "game".

Mostly FPs prevent Comodo to get Virus Bulletin award .

[Tags:]