Malware Research Group Project#21

why your preoccupation with AMTSO Testing in your own words

I myself, as well as Melih and many others, want to see Comodo AMTSO tested, but you are using this as an opportunity to bash Comodo


The error in the image is corrected. :slight_smile:

How many Samples tested?

How many Samples the top one has missed?

How many Samples do you need for infection?

How many more new viruses/malware out there on top of the Sample set used?

Would be grateful if someone could answer this…thanks


We as users, most of them at least, are quick to take credit for a test result in order to show that CIS is making progress, that’s all fair and good. However, Comodo itself would not recognize such a test as valid. Comodo only recognizes tests that have been reviewed and approved by the AMTSO review board.

In that sense, I do not know if CIS is making progress at all. There is neither Westcoast labs, nor ICSA labs certifications, just plain nothing.

You said that I’m bashing Comodo. Please quote my posts where I made such a statement. I’m a man of my words and there is no doubt that I will be able to own my own statements.


Hi Melih

Would it matter? What difference would it make when the test is not AMTSO approved? Or is it? ;D


How is it that I’m bashing when I’m pointing out something factual? Do facts hurt you this much?


554 891

How many Samples the top one has missed?
≈ 1 110
How many Samples do you need for infection?
1 :P
How many more new viruses/malware out there on top of the Sample set used?
Unknown… many.
Would be grateful if someone could answer this..thanks


Tried. :smiley:

Comodo may not recognize these tests…but that doesn’t mean we shouldn’t. Although not totally accurate, they can still be an insight. I certainly wouldn’t choose Norton over McAfee just if Norton got 98.99% on some test and McAfee 98.98%.

But if Norton got 98.99% and BesterestAntijunk got 12% … I would have a higher opinion of Norton than BesterestAntijunk.

I do not feel like Jaki is “bashing”.

I’m not sure how this was meant…but AMTSO is not the one who decides if a test is absolutely wonderful or worthless.

Hey…languy’s reviews are not “approved” by AMTSO, but I enjoy them, and feel like I gain from them.

Worthless, in terms of its importance. The AMTSO review board must review the test in order for the testing organization to get the AMTSO nod or seal of approval, sort of. Please do not believe me, believe Melih. He is the one who brought that fact to my attention. What can I say, he was right.


Insight to where. Only an AMTSO test that is approved by its review board would divulge the efficacy of a security product like CIS for example.


Bingo… :slight_smile:

so even missing 1 is not acceptable if you want protection!



IMHO, this has highlighted a serious flaw in the AMTSO process.

If AMTSO was set up to produce guidelines on how to dynamically test antimalware products in a real-world environment, they should have included the condition that the testing organisation should have to submit their testing process to AMTSO for review and certification PRIOR to running the test and announcing results.

Membership in an organisation does not guarantee conformity to the published principles and intent of said organisation. Review and certification are essential in safeguarding the integrity of any results.

Ewen :slight_smile:

I could not have agreed more. So has MRG test been reviewed and certified?


from Amount of samples used in this test: 554.891

Malware categories used in this test and the amount of samples in each category:

Trojans/Backdoors- 398.951
Windows Viruses- 8.864
Worms- 61.928
Adware/Spyware- 48.552
Rootkits/Exploits- 10.736
Other Malware- 25.860

How many “false positives” ? Making this review we used the latest version of COMODO Internet Security (3.12.111745.560), Database Version: 2470.

Reviewing process had three stages:

  1. On Demand scan on 50.000 samples of malware (June,July,August and September)

  2. Self Protection test where we used various tools with which we tried to disable COMODO Internet Security and its services.

  3. System Protection test – COMODO Internet Security was tested in Real Time against various most dangerous malware samples (better known as System Killers)

Result of our reviewing process:

  1. On Demand scan test – COMODO Internet Security failed to detect 153 samples of malware out of 50.000, scoring a detection rate of 99.69%.

  2. Self Protection Test – COMODO Internet Security successfully blocked all 10 attempts to disable it and its services.

  3. System Protection Test – COMODO Internet Security successfully detected and blocked all 15 System Killers leaving the system unharmed and fully operational.


COMODO Internet Security offers outstanding level of protection…

No. But then again, they’re not claiming to be AMTSO compliant. This is a detection test with the added ding-■■■■ of termination protection testing (which all tests should do BTW, IMHO, OK ;)).

Current AV testing methods and AMTSO testing methods are apples and oranges - it’s hard to draw a comparison between them. One starts from a position of “assumed dirty” and the other start from a position of “assumed clean”.

Ewen :slight_smile:

P.S. Nice result, though. :slight_smile:

So if they are not claiming to be AMTSO compliant it is even worst that I thought. Show me an AMTSO review board approved test as well as Westcoast and ICSA labs certifications, then and only then I would say nice result. These kind of tests like this one being discussed are just meaningless, pointless, worthless, useless, rubbish, you name it, I cannot find enough epithets to describe it, well to me at least.

AMTSO is the bread and butter of testing, when I reminisce your apple and oranges allegory, period. ;D



so even missing 1 is not acceptable if you want protection!


That is why I use CIS. :wink: To get the strongest proactive protection, that can block any threat, known or unknown. :slight_smile:
So, is this test uninteresting, if D+ an FW can block everything, why care about AV? Obviously reactive protection can not be as strong as proactive, and CIS would not need the AV part to protect. But CIS has AV, and that will be more useful the more it detects, making CIS easier to use. When I started to use CIS (3.5 beta 2), the AV really was not very good. But it is a lot better now, and MRG shows it:

3.5: 91.4%/90.0% (malware/Adware, Spyware)
3.8: 96.2%
3.9: 97.1%
3.12: 98,1%

Looks good. :-TU And I know you will not stop there… :slight_smile:

Very much so… the AV component is just making D+ more usable. I think we can all agree that CIS now is a lot quiter than it was before. And like you rightly said, it won’t stop there… it will continue and with v4 it will be the product that can be used by the most novice computer users.


it will continue and with v4 it will be the product that can be used by the most novice computer users.


If you manage that and the bugs and the updates I’ll fly to NJ just to buy you a coffee.


start picking your airlines :slight_smile: