SAS & MBAM are very low on that list…
so does an AV have to have an 85%+ detection rate in order for AV comparatives to send them missed samples?
92% for CIS is a good start for one of its first tests.
It is good start - but we don’t know what score it will have with AV Comparatives yet. This 92% is different testing vendor.
My thoughts on this subject I have read Andreas post again and all I get from them is in plain English.
We set the rules (which I agree is fair) we will not tell you precisely what they are (not fair) we will let you know when you reach them.
Please do not think it is my intention to put pressure on your organization.
Please please if you read this topic could you give us a better insight of your rules and when you feel CIS might reach them.
Best Wishes
Dennis
Have you also read http://www.av-comparatives.org/seiten/ergebnisse/methodology.pdf . I thought it was a pretty good discussion of their methodology and a summary of their rules for participation and awards. Looks like 87% is their threshold for consideration, and they use a 3 year sample set split into old and new, and … Comodo has managed to come up their own samples just like all the other vendors. Maybe time to start attending the anti-virus conferences and participating in testing and methodology development with the rest of the industry. And showing what can be done in addition to the signature based approach. This would certainly raise the confidence of the users who are reluctant to take an AV on faith.
Hi Melih
I really do not know what is all the fuss about AV Comparatives and Andreas this and Andreas that. AV Comparatives does not have the definitive word in AVs testing. Moreover, if I have to buy an AV product I will not base my choice upon the testings of AV comparatives, or any testing for that matter period. I will trial a few products and see which one of them best suit my internet surfing lifestyle. Let’s create our own AV testing for not only detection but also prevention with respect to how an AV product will keep a new system clean.
The way AV comparatives as well as Virus Bulletin do their tests are in no way reflecting real life scenario that will take into account a surfer’s lifestyle. Dr. Web gave up on AV comparatives and VB a long time ago, Trend Micro is gone even now kaspersky start to bark at VB since the guys at kaspersky begins to understand exactly what Dr. Web and Trend Micro were saying.
Let us create our own test, I believe in you Melih, I know you have the brain power as well as the infrastructure to do so; if you could in the process bring along Super Antispyware and Malwarebytes antimalware on board, the better. Once again it is a way forward and it will prevent us from kissing Andreas’ B*** and yet still have our honor and integrity intact.
Peace.
FYI
Everybody please read this with regard to AV comparatives
Peace.
Hi Melih
SSupdater provided a Do it yourself Anti-Malware test, on 13.08.2008. I know it is the forums policy not to post any link to malware, therefore I’m not going to post live malware link. However, such a link could be found on their website. If you were not aware of such a malware sample then take opportunity of it.
Peace.
:THNK
If it isn’t too costly, Comodo should have their CIS suite tested by several AV testing labs. This would give them an idea of where their AV detection is at, and also an idea of how accurate the various AV labs were in their testing, based on the variance of the testing scores. I know CIS is very new, but the only way to get a comparison of how it fares against other AV products is by AV testing. All the discussion about giving away free virus samples and what level has to be attained for score reporting and missed samples is peripheral to the issue. The actual testing is the issue. I have faith in CIS and have used it since Beta 1, but to see how it compares to other AV products requires AV testing, so that comparative results can be viewed. Comodo has put a great deal of resources into creating an Internet Security Suite. Quite a bit of time has been spent on this forum talking about testing. Now it is time to test. Let the testing begin!
Not exactly. Melih just doesn’t fully agree about the way AVs are being tested.
Which av testers does Melih agree with?
Hi Melih,
I can not believe what is going on on this thread. Someone told me that I should look again at this forum, else I would not writing this here today. (I thought this case was already solved and totally forgot about this.)
Melih, you can not expect that someone will keep visiting your forum because you chose to post a question to that person there. Nor did you inform me about this open letter at the beginning, and again you haven’t sent me email to remind me or asking me something. Some people even wrote a blog entry about your forum post, summarizing up some things I wrote but probably were not understood due my broken english and keep-it-short-principle (http://www.eset.com/threat-center/blog/?p=162).
OTOH, by reading some more posts in this thread I noticed that you have published the content of email conversations without asking us first. Here in Europe this is considered… well, let’s say quite rude. Even if the mail conversation is innocuous and therefore not worth serious complaints about breaches of privacy, we are not used to seeing such bad manners. You know yourself that people do not like email conversation being made public (https://forums.comodo.com/general_discussion_off_topic_anything_and_everything/result_of_email_encryption_poll-t26632.0.html), and respect for privacy is written into almost all netiquettes: see, for instance, RFC 1855, which states:
“If you are forwarding or re-posting a message you’ve received, do not change the wording. If the message was a personal message to you and you are re-posting to a group, you should ask permission first.”
You asked why you can not get samples from us and I gave you a concise answer (and explained it a bit better later). We would not send you samples without including you in our tests, and one reason we won’t consider doing that at present is your very low detection rate (your product does not even have heuristics!). You should in any case decrease the false alarm rate, but please do not ask us or Bit9 to send you our sets of clean files. It’s your job/responsibility as a software provider/vendor to do QA and have the resources to provide a “good-enough-product”, and we cannot send you such material without breaching our own agreements with the program owners.
Yet you asked us to remove that reason. How can I discuss the issues if the policies are available and you publicly ask for our reasons, and yet at the same time you ask us not to inform the public about how good or bad your product is. (Btw, that is our job, informing the public e.g. how good or bad products are, based on objective standards. It is not our job to provide samples to vendors or to do their QA job for them). I could give more reasons (e.g. about the quality of your signatures and product), but if I did that, you would probably see it as payback instead of objective reasoning. We may review your product in-depth when you are confident that your product is good enough to take part in various independent tests, when the final version for productive enviroments is released, etc. However, it would not make much sense to test the product in its present unfinished state.
You also asked about what the minimum percentage is: a simple google search would have told you it (and you saw that most users even know that by themself: so far it always was and still is 85%). It is no longer explicitly written in the document because we may change it in future and do not want to rewrite the public document again. Anyway this number is pointless for you, as a) your product is not publicly tested, b) your product does by far not reach 85% c) that TOS belongs to the main tests and not internal/single tests, d) also other point in that TOS would not be reached by you.
We heard that you asked several other companies and testers for samples before asking us , and even if they also declined, you wrote this “open letter” pointing only to us instead of writing it to the research community in general. I hope this is not because we tested your product in past and published the results (but at that time we still thought it was because it was new/beta and it was likely to improve significantly over the years).
You wrote you would like to improve test methods etc. There is an international organization consisting of almost all the major vendors, testers, reviewers etc. It is called AMTSO (www.amtso.org). If you are willing to contribute to it and improve tests, I suggest you to join it as it is a good platform for such discussions. It may also be of help to you as regards getting in contact with other vendors by meeting some representatives in person.
BTW: CIS is not the only/first completely free internet security suite. They have already existed for some years, but even if they have much higher detection rates than CIS currently has, they are not well-known. In the past, I liked your firewall quite a lot: why not bundle your firewall with an opensource AV, or license some third party engine? That would give to your free users a free internet suite with a lot better malware detection rate.
This thread is closed as far as I’m concerned, as I can’t see it leading anywhere useful. You and everyone else already know our TOS and methodology. Others may have a TOS of more interest to you, but you will not usually be able to see them until you sign a NDA.
If you have any further questions, please write us an email, because we have not enough time to check all forums with threads about www.av-comparatives.org
Maybe we will test your product in 2009, but you will have to improve it first.
We like the way that you plan to give CIS freely to the community and we wish you good luck in our project.
Best regards,
Andreas
Andreas
Thank you very much for responding.
I am surprised that you did not remember your own promise.
I would like to remind you about your own post where you were the first one who started making a public response to my email that was sent to you in private. Without responding to my email you responded to it in public! Wondering if you call this rude in Europe? 
Also: I do not know where you get your information but I can categorically state that you are wrong that I asked and was refused collabroation by other people! Contrary to your false claim, we had an amazing success from people who really care about Internet security! Please you care to enlighten us with who would be so ignorant as not to care about internet security to share their malware library?
AMTSO: Even though I respect the intention, where likes of wildlist etc failed, yet another testing organisation is not what this industry needs!
We would love to find out the other Internet security suites given for free (not as part of a promotion) We could be wrong and more than happy to be standing corrected.
Andreas, I am sure you know it too well that so called testing is very subjective depending on the malware library you have. A vendor, who has access to same malware as you will, have a high detection because they have the same base of malware as you. Can you deny this to be the case?
How can you be sure the sources you get your malware from are not sharing it with certain vendors and skewing their numbers?
How many % of your malware is old malware (over 1yr, 2yr, 3yr old)? (which was a question I had asked in the email and yet to receive an answer to).
Thank you for taking the time and look forward to your explanations.
Melih
Just curious, has www.av-comparatives.org even tested Comodo Antivirus 3.5? I searched your site briefly without finding anything. If it hasn’t been tested, why reject it already as not good enough?
LA
Is that true?
yes. not till early 09 i hear.
!ot!
That’s actually an interesting point.Surely incorporating the ClamAV engine,if only short term,would offer a quick way to increase the detection database?
Of course the size of the database is less important than the quality and relevance to current threats.A fine example would be MBAM which offers high real world detection on a comparitively small sampleset.
this is based on the assumptiom that CAV has lesser detection than ClamAV!!!
Our detection for the latest malware is good and getting better. We have amazing amount of malware being sent to us now (over 20gig a month!). All this will result in better detection very soon as our analysts work tirelessly creating signatures.
Melih
I was meaning more that it’d enable you to pass the AV comparitives criteria and therefore receive the missed samples you were asking for.I did make the point that quality of detection is the most relevant to the end user,rather than just having a huge database of obscure malware.ClamAV doesn’t necessarily have ‘better’ detection it just conforms more to AV comparitive’s requirements at present.