exactly, once you agree to be tested, good or bad your results will be published. If you don’t agree to this don’t even bother to be tested. I like how most testing organizations show only the good results, I believe this artificially inflates how good the AV’s are doing today. From my testing most AV’s are actually between 70-80% effective ( not counting proactive parts), but if you look at the results from the so called experts they show most of them over 90%. That is hard for me to believe from first hand experience.
+1
Undoubtedly misleading for the users.
I am pleased that these type of people are being exposed :-TU
Well done.
from AVC
http://www.av-comparatives.org/forum/index.php?page=Thread&threadID=1054
Our reply to posts done recently by Comodo:In our opinion, software evaluations should be provided free of charge to the public.
The costs that arise from the tests and demands from vendors should be covered by the vendors, as it is like an external quality assurance assessment (and they also use it for marketing). That vendors have to pay a fee (which is the same for all vendors, and of course has no influence on results) is publicly known, and also written inside the old methodology document which has been available on our website since at least 2008.
We are legally a Not-For-Profit Organization, which allows us to keep the costs relatively low compared to certain other testing labs, because of the support of the Austrian Government and partnerships with Universities.
When I (Andreas) started doing the public tests in 2003, I did it for free and asked users if they wanted to donate something. Practically no one donated, and based on the high demand for continuing the tests, and the increasing complexity of the tests, I had to start asking all vendors to pay a fee. Nevertheless, my goal of providing independent tests has not changed at all - it allows us to further improve the tests and their relevance for the public.
While what they say is true, why do they not publish their prices? Is it because they don’t want the readers their results to see how much they charge?
Comodo, on the other hand, promotes itself as providing AV software for free (although on their website they promote the paid versions very prominently). Comodo gets money for services promoted in the paid version they charge users (which is OK, as Comodo cannot live on fresh air).
Right comodo offers the most free security compared to anyone else in the business, they give it all to you for free, you only pay for the extra services. What other company does that?? None that I can think of off the top of my head.
Melih continuously stated in his forum that we get paid by vendors, and insinuated that the payments have an influence on the results. This is clearly not the case, as the confidential results we provided to Comodo (and which he published without permission) show.
So how come in your tests other av companies do so well when in my tests most fail miserably? Remember my latest Norton review? Also he would not have published the results until AVC mailed him threatening to expose the results if he did not remove the posts. You are just mad that he beat you to the punch and you have nothing to reveal anymore.
Comodo paid us several times for conducting static detection tests of their software, and although they promised users in their forums that they will publish the results, they never did (they said to us it would be bad for their marketing and therefore they preferred to keep it internal).
True it would be bad for marketing because most users don’t understand how to read the numbers. In then end comodo could not publish the results until they paid you more money anyway.
Of course my co-worker got angry about reading such blatant insinuations/misinformation and asked Comodo to clarify the misleading posts in their forum. It has nothing to do with censorship or bullying. If someone says in public wrong things, it might be an opinion/free-speech, but nevertheless not true, and the offended party can try to get it resolved (either by asking for a public apology or by employing a lawyer), in the hope of getting it sorted out in a professional way. However, instead of getting back to us, they preferred to continue with the campaign.
If not bullying, what is up with that e-mail threatening comodo?
We have told Comodo in the past that testing Comodo Internet Security showed it to be very noisy (very many user-dependent pop-ups and lot of false alarms) and that we could not see it working well enough to test it with the automation we had at that time. This was a nice way to say that their software needed improvements. We even suggested how they could word it on their website, without saying that it was due to their scores; they didn't do that at the time, but kept quiet.
what you said is that because of the user dependent pop-ups you are not capable of testing CIS, not that it needed improvement. Having popups does not mean that it could not protect you, actually looking at the trends form other manufactuers they are showing more popups now then ever before because they realize that total automation in today’s world is impossible while still offering better user security.
Melih stated that we can only test for detection and not for protection. This is not true. We test for protection in our Whole-Product-Dynamic Test (which is part of our public main test-series). Unfortunately, Comodo seems not to be interested in joining that test, as the results would be published. At the beginning of the year we wrote in our survey report that although many users would like to see Comodo tested, Comodo did not apply for inclusion in the test.
You actually said you can’t test the automatic sandbox, not him. Two things, would you be able to test CIS in the WPDT with the autosandbox or would not be able to? And how much does it cost to test, how many thousands and how much more to publish the results?
We give missed samples AFTER most tests to the publicly tested vendors. Comodo currently pays various other sources for samples and would like to get samples from us for free (they tried to persuade us, using public pressure, some years ago). We do not sell samples.
Yes you do, for the people to get the samples that you tested with they have to pay extra. If that is not selling samples I don’t know what is.
Anyway, it is sad to see a vendor trying to publicly bully a testing organization by acting this way: making insinuations in their forum; criticising testing organizations if the results in commissioned tests are not as the vendor would like; publishing confidential/internal email correspondence without permission. This is more the behaviour we would expect from the CEO of a rogue AV vendor, not from a serious company like Comodo. Unfortunately, by acting this way Comodo we probably might not test Comodo’s products in any way for the foreseeable future. The reason is simple: if Comodo were to score badly, they would then state that they got poor results because of this "dispute" (and some Comodo fans might even believe that).The only reason why we will not ask a lawyer to deal with this case is that we prefer to put money into improving tests rather than playing this game with Comodo.
Actually what is sad is a testing organization not having any ethics. Trying to blackmail the other companies to pay more to do anything. I think the only reason you will not go to a lawyer is because you don’t have a case because you tired to bully a company in censoring it’s users and they exposed you for what you are.
The above is my opinion only and does not represent comodo in anyway.
The gloves are off >:-D
No wonder AV-Comparatives sent that mind-numbingly ill-conceived, extortive flavoured, email… they considered it part of some sort of game.
I don’t think they have to public their prices. And they (the prices) should not interfere.
Different methodologies, samples… This will always be this way. Your tests will have the seriousness you can achieve. I’m not criticizing you or your test, just your posts.
If there were an agreement, no part should have publish the results of a confidencial test. But we need to know the details (exactly words) of the agreement.
I think they should publish prices or at least that they are getting paid to do the tests. They don’t mention this anywhere on the website that I can see.
on your second thing, what? I don’t understand what you are saying.
Thirdly, comodo would not have published anything until AVC started threatening they would just to get back at them.
AV-comparatives said: by acting this way Comodo we probably might not test Comodo’s products in any way for the foreseeable future
Seriously???
They can’t test CIS anyway, so what the hell are they talking about???
Here is the email from Peter saying with Automatic Sandboxing they can’t test CIS with their Automated systems:
On 2/3/2011 9:45 AM, Peter Stelzhammer wrote:
Hi Guys,
cool to hear you again. As promised we can test you in our on-demand test separately in a Single Product Test. Fee is Euro 2000. The test will start on 17th of Feb. 2011. Just let us know.
About the whole product test, we would like to see you in, but, and this is an official statement, which you can post in your blog, your product with the current sandbox and the current way it is working, is not compatible with our automation system and as your sandbox does restrict often also the functionality of clean applications, we do not see it fit in our test system right now. So it is not possible to put you in a single whole product test, in the main test it would not be possible anyway as we restrict the number of participants.
Schöne Grüsse / Regards
Peter Stelzhammer
Phone: +43 (512)
Mobile: +43 (664)
——————————————-
Agree, they should publish somewhere how the things work. But not the price itself imho.
I’m just saying that different tests and different methodologies will give you different results (due to your phrase: So how come in your tests other av companies do so well when in my tests most fail miserably?).
its a non for profit organisation they claim, so why do they have such a big issue with the fact that they get paid from AntiVirus vendors and how much they get? Doesn’t make sense.
Anyone who is wondering how Comodo did in these totally irrelevant dead malware testing test, here is what Andreas said:
From: Andreas Clementi [mailto:a@av-comparatives.org]
Sent: Monday, April 11, 2011 11:11 AM
To: ‘Egemen Tas’; ‘Umesh Kumar Gupta’
Cc: p@av-comparatives.org
Subject: AW: AW: AW: AW: Main Test Feb-2011
Hi,
btw, do not be unhappy about the „other malware/viruses“ %s, you will see in the report that also other vendors score low there (mainly because that category contains non-PE malware). Furthermore, your ~90% is higher than what some few other products scored and similar to what other well-known products reached. You will see
regards,
andreas
AV-Comparatives e.V.
http://www.av-comparatives.org
Tel.: +43 512
Mobile: +43 676
Innsbruck | Austria | Europe
“Furthermore, your ~90% is higher than what some few other products scored and similar to what other well-known products reached”
right different tests, seeing as they use more malware then me and seeing as statistics says the bigger the sample base the more accurate your results, they should easily find malware that bypasses most security out there. If I can find some with my small sample set they should have no problem. But yet all of their security tests show that most AV’s do much much better then what they should really do.
Testing methodology at least should basically be the same. It’s really simple, run the malware and see if it does any damage to the system. This is what their “dynamic” test should be.
agreed 10000%!!!
The reality is that in the real world when real users test the real products, we see a different picture than what is being painted by av-comparatives. The end users who do these tests are NOT paid by anyone so they have no reason not to report the reality.
Also it is such a shame to bring your “competitors” into this discussion in an effort to deflect from their issues by referring to other testing organisations in their Blog by AV-Comparatives. I can categorically state that we have found every other testing organisation we worked with to be very professional, honest, open and hard working.
Melih
I have an idea about how to fix the money issue with the testing.
Basically for each test you perform you have a company pay for the test covering everyone else but they are not tested. The next test another one pays and they don’t get tested but the first one does. I know it will be a much more expensive proposition for the companies but they are paying for the testing up front.
Lets say each year you will be doing 10 tests and you have 20 companies. So for test one company 1 and 2 pay for the testing and companies 3 - 20 are tested bot not 1 and 2. The next test companies 3 and 4 pay but 1 and 2 and 5 -20 are tested.
You can see that results from their “competitors” also have some flaws. Just take a look at this:
Microsoft Security Essentials 2.0 CERTIFIED
2.5 Protection
3.5 Repair
5.0 Usability
Total Defense: Internet Security Suite 2011 NOT CERTIFIED
2.5 Protection
3.5 Repair
3.0 Usability
Source: http://www.av-test.org/en/tests/test-reports/julaug-2011/
Both AV got exactly the same VERY LOW protection rate.
Both AV got exavtly the same AVERAGE repair rate.
The only difference its that one got a higher Usability rate.
And just because a higher rate in Usability, it gets Certified. ???
So you have a certified product with very low protection and repair rates (in other words, not good).
And it gets certified next no other good protection products, only because its Usability.
if comodo had scored more than 90% then what was the problem for comodo to release it to public???.. anyway it was good result…
"Melih continuously stated in his forum that we get paid by vendors, and insinuated that the payments have an influence on the results. "
I stated AV-Comparatives gets paid by AntiVirus Vendors…now after my blog post many know this to be a fact…(and this fact should be in bold in their website)
But where have I “insinuated” that the payments have an influence in the posts that Peter refers to? I haven’t…this is more of a case of a “chip on the shoulder” scenario it seems…
http://commentarista.com/wp-content/uploads/2011/03/shoulder370.jpg