Malware Research Group Project#21

JoWa · October 17, 2009, 6:39pm

554 891

How many Samples the top one has missed?

≈ 1 110

How many Samples do you need for infection?

1 :P

How many more new viruses/malware out there on top of the Sample set used?

Unknown… many.

Would be grateful if someone could answer this..thanks
Melih

Tried.

jeremysbost · October 17, 2009, 6:40pm

Comodo may not recognize these tests…but that doesn’t mean we shouldn’t. Although not totally accurate, they can still be an insight. I certainly wouldn’t choose Norton over McAfee just if Norton got 98.99% on some test and McAfee 98.98%.

But if Norton got 98.99% and BesterestAntijunk got 12% … I would have a higher opinion of Norton than BesterestAntijunk.

I do not feel like Jaki is “bashing”.

I’m not sure how this was meant…but AMTSO is not the one who decides if a test is absolutely wonderful or worthless.

Hey…languy’s reviews are not “approved” by AMTSO, but I enjoy them, and feel like I gain from them.

Jaki · October 17, 2009, 6:51pm

Worthless, in terms of its importance. The AMTSO review board must review the test in order for the testing organization to get the AMTSO nod or seal of approval, sort of. Please do not believe me, believe Melih. He is the one who brought that fact to my attention. What can I say, he was right.

Peace.

Jaki · October 17, 2009, 8:14pm

Insight to where. Only an AMTSO test that is approved by its review board would divulge the efficacy of a security product like CIS for example.

Peace.

Melih · October 17, 2009, 9:07pm

Bingo…

so even missing 1 is not acceptable if you want protection!

Melih

panic · October 17, 2009, 9:47pm

G’day,

IMHO, this has highlighted a serious flaw in the AMTSO process.

If AMTSO was set up to produce guidelines on how to dynamically test antimalware products in a real-world environment, they should have included the condition that the testing organisation should have to submit their testing process to AMTSO for review and certification PRIOR to running the test and announcing results.

Membership in an organisation does not guarantee conformity to the published principles and intent of said organisation. Review and certification are essential in safeguarding the integrity of any results.

Ewen

Jaki · October 17, 2009, 10:25pm

I could not have agreed more. So has MRG test been reviewed and certified?

Peace.

Regression · October 17, 2009, 11:00pm

from http://malwareresearchgroup.com/?page_id=2 Amount of samples used in this test: 554.891
Malware categories used in this test and the amount of samples in each category:

Trojans/Backdoors- 398.951
Windows Viruses- 8.864
Worms- 61.928
Adware/Spyware- 48.552
Rootkits/Exploits- 10.736
Other Malware- 25.860

How many “false positives” ?

http://malwareresearchgroup.com/?cat=3 Making this review we used the latest version of COMODO Internet Security (3.12.111745.560), Database Version: 2470.
Reviewing process had three stages:

On Demand scan on 50.000 samples of malware (June,July,August and September)

Self Protection test where we used various tools with which we tried to disable COMODO Internet Security and its services.

System Protection test – COMODO Internet Security was tested in Real Time against various most dangerous malware samples (better known as System Killers)

Result of our reviewing process:

On Demand scan test – COMODO Internet Security failed to detect 153 samples of malware out of 50.000, scoring a detection rate of 99.69%.

Self Protection Test – COMODO Internet Security successfully blocked all 10 attempts to disable it and its services.

System Protection Test – COMODO Internet Security successfully detected and blocked all 15 System Killers leaving the system unharmed and fully operational.

Conclusion:

COMODO Internet Security offers outstanding level of protection…

panic · October 17, 2009, 11:12pm

No. But then again, they’re not claiming to be AMTSO compliant. This is a detection test with the added ding-■■■■ of termination protection testing (which all tests should do BTW, IMHO, OK ;)).

Current AV testing methods and AMTSO testing methods are apples and oranges - it’s hard to draw a comparison between them. One starts from a position of “assumed dirty” and the other start from a position of “assumed clean”.

Cheers,
Ewen

P.S. Nice result, though.

Jaki · October 18, 2009, 1:45am

So if they are not claiming to be AMTSO compliant it is even worst that I thought. Show me an AMTSO review board approved test as well as Westcoast and ICSA labs certifications, then and only then I would say nice result. These kind of tests like this one being discussed are just meaningless, pointless, worthless, useless, rubbish, you name it, I cannot find enough epithets to describe it, well to me at least.

AMTSO is the bread and butter of testing, when I reminisce your apple and oranges allegory, period. ;D

Peace.

JoWa · October 18, 2009, 6:34am

so even missing 1 is not acceptable if you want protection!
Melih

That is why I use CIS.

To get the strongest proactive protection, that can block any threat, known or unknown.

So, is this test uninteresting, if D+ an FW can block everything, why care about AV? Obviously reactive protection can not be as strong as proactive, and CIS would not need the AV part to protect. But CIS has AV, and that will be more useful the more it detects, making CIS easier to use. When I started to use CIS (3.5 beta 2), the AV really was not very good. But it is a lot better now, and MRG shows it:

3.5: 91.4%/90.0% (malware/Adware, Spyware)
3.8: 96.2%
3.9: 97.1%
3.12: 98,1%

Looks good. :-TU And I know you will not stop there…

Melih · October 18, 2009, 2:02pm

Very much so… the AV component is just making D+ more usable. I think we can all agree that CIS now is a lot quiter than it was before. And like you rightly said, it won’t stop there… it will continue and with v4 it will be the product that can be used by the most novice computer users.

Melih

system · October 18, 2009, 3:37pm

it will continue and with v4 it will be the product that can be used by the most novice computer users.
Melih

If you manage that and the bugs and the updates I’ll fly to NJ just to buy you a coffee.

Jose.

Melih · October 18, 2009, 7:13pm

start picking your airlines

Melih

Chiron494 · October 20, 2009, 12:07am

An AV will always be a critical part of an Internet Security package. Novice users will not be able to make the correct decisions about whether to allow or deny. Also, even expert users sometimes make mistakes.

Opus_Dei · October 20, 2009, 3:31am

Agreed

X

Jaki · October 20, 2009, 2:04pm

JoWa where did you get that data in order for you to conclude that CIS or any other security product for that matter is looking good?

Peace.

JoWa · October 20, 2009, 3:41pm

I didn’t say that CIS (CAV to be correct) looks good. It’s the improvement that looks good, in my opinion. The data (detection rates) is from MRG Test Archive + their latest test.

Pax.

Jaki · October 20, 2009, 4:19pm

Yes, but we are talking about CAV right. However, the source of that data is not what Melih wants. Please refer to: https://forums.comodo.com/empty-t39829.0.html

Consequently, in all fairness, CAV has not made progress that the current MRG Test indirectly claimed it has based upon the results of such a test.

Peace.

Citizen_K · October 20, 2009, 5:43pm

Comodo most definitely needs a strong AV as well because CIS has the possibility to consider the computer clean, Clean PC mode. So less experienced people starting to use CIS need to be sure that the scan during installation does a thorough job. Melih realises this:

But to get a real idea about the protection capabilities of HIPS based firewall solutions like CIS there are of course different ways of testing required. Its neither black nor white.