I thought I should be direct and open and hence this open letter. At the end of the day the need to secure end users must override any motive either party might have and in that spirit I write to you openly.
As you might be aware Comodo has invested millions of dollars building products to secure end users and giving those products for free. We want to get the internet to reach the “Trusted Internet” state and without securing people’s computers we are not going to get there.
I appreciate what you are trying to do, even though I don’t agree with some of the methods as I believe there is room for improvement, I still respect the spirit in which it is carried out.
We now have built world’s first FREE Internet Security Suite (CIS) and its AV, although have a good engine, lacks some of the older malware signatures. You seem to have some of the malware and if you were to share those with Comodo, then Comodo would improve its detection for the product it gives for free securing millions of end users! The winners and beneficiaries are the end users!
We ask you kindly to share your malware library with Comodo so that we can better protect our users! (we understand that if you did share your malware library then in tests we would get high %age unfairly, hence we are more than happy not to be part of these tests! Our one and only desire is to protect end users!)
Andreas, we have put our money where our mouth is and continue to do so. We are hopeful that you will come thru and protect the millions of users we have! They will be grateful for your contribution.
We, as Comodo and Comodo users, hope to hear from you very soon.
On-demand comparative that address existing know virus
Retrospective/ProActive Test That address only new virus in a timefreame
Since Retrospective/ProActive Tests pertain only new samples I would like to be reassured that the open letter doesn’t address them. (B)
Reading their methodological paper it looks like they receive sample from many sources including other AV vendors. It looks like they also send missed samples not covered by NDAs (in such case they send a CRC32 list of those NDA covered samples).
[i]...snip...[/i]
Test Lab Security …snip…
AV‐Comparatives sends (missed) samples only AFTER the main tests in February and August to trusted representatives of vendors whose products were publicly tested. We do not send any samples to unknown/untrusted vendors/individuals, no matter what they say or offer.
…snip…
Sources of samples
AV‐Comparatives have various sources from which it obtains samples. Like anti‐virus vendors, we also use various traps and a large quantity of honeypots from all over the world, as well as samples downloaded from malware downloaders and infected websites. Furthermore, we get samples from the field which were collected by us or our partner companies (e.g. computer repair/cleaning services) on infected PC’s belonging to home users and/or small/medium business companies. We also get samples from various online scanning services and (single and large) submissions from visitors to our website, as well as various organizations that collect malware (internal and public security forums, honeypot projects, anti‐malware initiatives, and so on). In order to have a test‐set that is statistically valid and as large and representative as possible, AV‐Comparatives also accepts samples from (security) vendors. Currently, samples submissions from about a dozen vendors are included in our tests and nearly dozen more vendors which are not included in our tests also contribute.
…snip…
Conditions for participation
Which products are to be tested is decided by the board of AV‐Comparatives e.V. ‐ AV‐Comparatives prefers to include in its tests only anti‐virus products with good detection rates. The product must use its own or licensed engines. Products using the same engines are not tested twice.
The product must be able to finish a scan of the full database using the most secure possible detection settings within a reasonable time,
without crashing or causing major problems. Products must be able to scan a subdirectory tree (depending from the type of test).
The scanner should not move or change in any way the files or the system during the scan when running in report‐only mode.
The product should be a well‐known anti‐virus product used worldwide and anti‐virus and should not produce too many false positives.
Additionally, the following Terms of Service agreement has to be accepted and signed:
Terms of Service for Anti‐Malware Software Testing
…snip…
4. FEES. The Vendor (or a third party) has to pay a fee for the various services provided (e.g. usage of logo in marketing material and time/work spent in providing the various services, etc.). The fee has to be paid quarterly after the tests are finished and already published.
…snip…
6. RESTRICTED DISTRIBUTION OF SAMPLES. The Vendor may request that the Tester restricts distribution of samples to certain other vendo(s) where there is an issue of trust.
The Vendor is obliged to identify clearly the other vendor(s) to which the Vendor wishes such a restriction to apply.
The Tester will review such request individually, and after review by the Tester the Vendor will be informed as to whether the restricted distribution of samples will be applied.
The Tester suggests that in such case the Vendor does not submit further samples for the duration of the review period.
If the Vendor is not satisfied with the outcome, the Vendor may decide to do not send samples or to discontinue sending samples to the Tester.
In cases where an issue of trust arises against the Vendor and a review by the Tester shows the concern to be valid,
the Tester will provide a limited number of missed samples from any test, at the sole discretion of the Tester.
7. MISSED SAMPLES. The Vendor must have an established virus lab in order to be entitled to receive missed samples after the on-demand tests.
The Tester will provide the missed samples to the Vendor only if the Vendor’s product is successfully able to identify a given minimum of the Tester’s actual full test set during an on‐demand scan with the most secure settings.
The Tester will provide missed samples to the Vendor so that the Vendor can verify the validity of the test results.
The Tester will send samples missed by the Vendor’s product, unless the Vendor is subject to a restricted distribution of samples as described in the section above (RESTRICTED DISTRIBUTION OF SAMPLES).
If a distribution restriction has been applied, the Vendor will receive a limited number of samples selected by the Test center together with a list of missed samples in form of log
with CRC32 checksums and ‐ where possible ‐ detailed reasons on why the restriction had to be applied, in order that the remaining missed samples
can be requested from other vendors or located among samples in the Vendor’s own lab.
From what I read it looks like any legitimate vendor can participate to AV‐Comparatives to have their product tested and cooperate to improve the testing accuracy. :-TU
The sharing of undetected samples also grant an improved detection rate of all vendors who endorsed that organization anyway it looks that those services are carried for a fee.
Even though if they did it for free would have been better I guess there is no way to sustain such organization without 3rd-party funding.
That was a very rude response blatantly against their TOS. Any AV brand that is eligible to have their AV tested will receive missed samples.
I guess that AV-comparative CEO could be interested to know that such a reply was provided in order to take appropriate measures. >:(
Thanks for sharing this relply it could prove useful to launch an online petition with the support of all froum members in order to ask Andreas Clementi if this was an undesirable misunderstanding or it is AV comparative official standpoint >:(
This whole thread is very unfair to AV Comparatives. Their TOS says that if Comodo participates in their program, and scores above a threshold level in AV performance, then they will provide copies of the missed samples that are not restricted by an NDA. They are not in the business of providing Comodo (or anyone else) with a comprehensive AV library that might provide a competitive advantage, save them R&D costs, or level the playing field. Comodo’s competitors have also spent a lot of money developing their AV products and databases, and depend on them to support their sales. And they pay for the testing, as well as providing their data. Even though Comodo is free, convincing a user to use the free product instead of the paid product of another vendor takes money out of the vendors pocket. And Comodo wants all of it for free or a small fee so they can use their deep pockets to put the other AVs out of business? (when AVs are deemed unnecessary by Comodo anyway in various threads). Was there more information than the brief quote out of context? Since most of the competition has joined, why doesn’t Comodo do that and reap the benefits per the TOS?
It was Andreas’s response (the CEO). He did point out some other AV firms that might be interested in sample exchange but we already are in touch with them anyway.
So none of the other AV companies are receiving copies of the missed samples, and no one is complaining about that violating the TOS? Even big companies like Symantec? Are you sure? That certainly wasn’t the question you asked him, so out of context it is hard to understand the brief response to your open letter as saying that.
(J)