For those interested here it is the test methodology.
About the testing of anti-viruses for the level of detection of malicious software performed by VirusInfo
(by Nick Golovko)
Since 2005 VirusInfo performs the testing of anti-virus software. Any member of the project's forum can participate in it. The results of this testing are regularly processed and represented as a graph. This graph demonstrates the comparison of anti-virus software by several parameters.
The data presented by VirusInfo are widely used by specialists of various companies dealing with informational security, in order to compare and rate the anti-virus software. Now the latest graph is always available for any visitor of our site.
How we test
The testing of anti-viruses by VirusInfo is powered by free online scanner VirusTotal. Project participants, being practising specialists in the area of computer security, are uploading at VirusTotal the malicious software that they have received from infected machines, and then publish the results of scanning in a special topic on VirusInfo forum. The malicious software should meet the following requirements:
1) The sample should not be detected by the anti-virus software that protects the infected machine.
2) The sample should be found by the consultant him/herself in a real infection case.
3) The sample should not be taken from some other site or from some other collection of malware.
The results of scanning are regularly generalized as a graph of detection level. The graph is prepared in accord with the following principles:
1) The X axis represents the anti-virus software used by VirusTotal at the current moment. The Y axis represents the number of samples uploaded.
2) For each antivirus we mark the number of samples that it has successfully detected using one or another detection method. The graph reflects the general number of detected samples and the each method's part in the general detection.
3) The following detection methods are distinguished:
a) signature detection (detecting already known malware by the signature method)
b) heuristic detection (detecting yet unknown malware by the method of emulation / code analysis / etc. Examples: "Heur.Trojan.Generic"; "a variant of: XXXXX")
c) detection of suspicious file (detecting yet unknown malware by the method of informing the user about suspicious characteristics of a sample under analysis. Examples: "Suspicious file"; "VIPRE: Suspicious")
d) detection of suspicious cryptor / packer (detecting yet unknown malware by the method of informing the user about the unknown / rare / suspicious packer / cryptor or about the fact of multiple packing / crypting. Example: "HEUR/Crypted").
"Heuristic" as represented in that chart refers to "Unclassified Malware" signatures generated by CIMA.
Packer detection for Comodo is absent whereas is featured by some other products providing detection up to 16 samples (14%)
Comodo Heuristic (which includes packer detection) is likely disabled as well.
Total number of samples is approximately 110. The samples were found in the wild specifically by each contributor on infected machines.
Sampleset included also html/js/php samples and Java apps for cellphones (J2ME). Some samples of the same family/variant are also repeated (didn't check their hashes but I guess at least it should be different)
Detection results and sample hashes are available at
http://virusinfo.info/showthread.php?t=33303
Author