SSUpdater's Anti-Malware test [CLOSED]

Comodo Internet Security - CIS News / Announcements / Feedback - CIS
1

Welcome to SSUpdater.com Anti-Malware test, the only independent malware test online!!!
In total 30 programs have been tested, which were picked by our members, all the programs tested used maximum settings including heuristic behavior and riskware defense.
The test was conducted in virtual environment using Microsoft’s Windows XP SP3 with all the latest updates.
The main goal of this test is to make a comparison to VirusBulletin’s VB100 test and therefore we did NOT use any of the samples from their test, another goal is to compare our results with always questionable results form AV Comparatives.
In this case we used a malware sample containing 800.025 items, in the folowing categories:
Windows Viruses, Trojans, Backdoors, Worms, Spyware, Adware, Rootkits, Exploits, Keyloggers, Hacking Tools, Malicious Scripts and other malware
In other words we used 99.5% of the known malware from the last 4 years including this year, we also used samples which are a bit older but only the ones that pose a threat of possible outbreak/infection.
As always we used a collection of unknown malware, those samples are custom made and are NOT for mass distribution, in total there were 18 255 samples in that collection.

This test is brought to you by:

http://img530.imageshack.us/img530/452/01in9.jpg

The Results
(Name/Detection Rate)

1. a-squared Anti-Malware - 99.12%
2. AntiVir PE Premium - 98.88%
3. Avast! - 98.75%
4. Norton Antivirus - 98.69%
5. Ikarus Virus Utilities - 98.48%
6. Kaspersky Antivirus - 97.83%
7. BitDefender Antivirus-97.79%
8. F-Secure -97.21%
9. Zone Alarm-96.91%
10. Spy Emergency-95.64%
11. McAfee VirusScan Plus - 94.18%
12. Comodo Internet Security - 92.53%
13. Twister Anti-TrojanVirus - 92.49%
14. Rising Antivirus - 92.27%
15. Dr.Web- 92.04%
16. Norman Antivirus & Antispyware - 91.78%
17. Blink Personal Edition - 90.93%
18. PC Tools Internet Security - 89.77%
19. Nod32- 88.35%
20. AVG Antivirus - 87.45%
21. Microsoft Windows Live OneCare - 87.12%
22. MoonSecure Antivirus - 86.43%
23. Spy Sweeper with Antivirus - 84.64%
24. TrendMicro Internet Security- 84.13%
25. Spyware Terminator with Antivirus- 61.38%
26. VIPRE- 60.97%
27. 123 Spyware Free - 57.68%
28. Ad Aware - 28.04%
29. SuperAntiSpyware - 07.24%
30. Malwarebytes Anti-Malware - 02.66%

Speed and VB100 Award

1. a-squared Anti-Malware - average - N/A
2. AntiVir PE Premium - fast - Passed
3. Avast! - average - Passed
4. Norton Antivirus - average - Passed
5. Ikarus Virus Utilities - average - Failed(37 wildlist misses, 8 false positives)
6. Kaspersky Antivirus - average - Passed
7. BitDefender Antivirus - average - Failed(4 false positives)
8. F-Secure - average - Passed
9. Zone Alarm - average - N/A
10. Spy Emergency - very slow - N/A
11. McAfee VirusScan Plus - slow - Passed
12. Comodo Internet Security - fast - N/A
13. Twister Anti-TrojanVirus - average - N/A
14. Rising Antivirus - average - Passed
15. Dr.Web - average - N/A
16. Norman Antivirus & Antispyware - slow - Passed
17. Blink Personal Edition - average - N/A
18. PC Tools Internet Security - average - Passed
19. Nod32 - fast - Passed
20. AVG Antivirus - average - Passed
21. Microsoft Windows Live OneCare - average - N/A
22. MoonSecure Antivirus - very slow - N/A
23. Spy Sweeper with Antivirus - slow - Passed
24. TrendMicro Internet Security - average - Failed(3 wildlist misses, 2 false positives)
25. Spyware Terminator with Antivirus - slow - N/A
26. VIPRE - average - N/A
27. 123 Spyware Free - average - N/A
28. Ad Aware - average - N/A
29. SuperAntiSpyware - average - N/A
30. Malwarebytes Anti-Malware - average - N/A

Our Rating and User Experience Advised

1. a-squared Anti-Malware - Good - Expert
2. AntiVir PE Premium - Excellent - Average+
3. Avast! - Excellent - Average+
4. Norton Antivirus - Excellent - Novice
5. Ikarus Virus Utilities - Good - Expert
6. Kaspersky Antivirus - Very Good - Average
7. BitDefender Antivirus - Very Good - Average
8. F-Secure - Very Good - Novice
9. Zone Alarm - Good - Average
10. Spy Emergency - Very Good - Average
11. McAfee VirusScan Plus - Good - Novice
12. Comodo Internet Security - Good - Average
13. Twister Anti-TrojanVirus - Good - Average+
14. Rising Antivirus - Average - Average
15. Dr.Web - Average - Average
16. Norman Antivirus & Antispyware - Average - Novice
17. Blink Personal Edition - Average - Average+
18. PC Tools Internet Security - Average - Average
19. Nod32 - Average - Novice
20. AVG Antivirus - Average - Average
21. Microsoft Windows Live OneCare - Average - Average+
22. MoonSecure Antivirus - Average - Average+
23. Spy Sweeper with Antivirus - Average - Average
24. TrendMicro Internet Security - Average - Average
25. Spyware Terminator with Antivirus - Below Average - Average
26. VIPRE - Below Average - Average
27. 123 Spyware Free - Below Average - Average
28. Ad Aware - Poor - Average
29. SuperAntiSpyware - Poor - Novice
30. Malwarebytes Anti-Malware - Poor* - Novice

* Note: Since this test, like all the other ones that are made by the SSUpdater Crew, will be shown on various websites,forums,support forums… and will be commented by various “security experts” which favor certain programs, I call on them to come here at http://ssupdater.com , and give their opinion here where they can get answers from people who are responsible for this test.
Unlike others WE HAVE NOTHING TO HIDE!!!

WE DEDICATE THIS TEST TO ALL THOSE WHO SEEK THE TRUTH!!!

2

Thank you for posting this here and your willingness to be open. I hope you are ready for the questions as we have many! :slight_smile:

Our members have many questions that we would like to throw at you. I hope you can shed a light to them all.

Thank you and I will now allow our members to start asking questions…

but few from me first:)

  1. Where do you get your malware database and why do you think that represents the current threats?
  2. Why does your numbers vary from likes of av comparitives?
  3. How recent are your malware?
  4. Why do you attack likes of av-comparitive in your posts? What is it that you don’t like about their testing methodology?
  5. Will you submit samples that AV vendors don’t catch to them? if so do you have any rules that would stop you from doing that?

I am sure i will have many more soon :slight_smile: but this is a good enough to start with.

Thank you for your time state.of.security.

Melih

3

Welcome to the Comodo Forums. :slight_smile: I think it’s great that another group of individuals are doing unbiased testing of security software.

I am guessing that “State of Security” is what the “SS” part of ssupdater.com means. :stuck_out_tongue:

Anyways, just a few questions.
I noticed that on your site, there are Anti-Virus testing as well. The last one was October 7th, I think. Comodo Antivirus 2.x was used for that and even Comodo was not happy with that program, hence they went straight to version 3. Do you know when the next round of testing will be for that and will you use Comodo Internet Security in that testing now that it is final? The antivirus part of that is the new Comodo Antivirus 3 and I don’t think a separate version will be made available since the installer of CIS gives the users the option to not include the firewall.
Also, for both Anti-Virus and Anti-Spyware tests, do you provide all the missed samples to the Company so they can research why those samples were not picked up by their program?
Will the next round of tests include the same samples as this on plus new ones, or a completely new set of malwares?

Cheers.

EDIT: Melih beat me to some of my questions. :stuck_out_tongue:
EDIT2: Sorry, another question. Do you think the detection rates for CIS are really good for a new program that has just been released? And do you have any suggestions for improvements. :slight_smile:

4

Hello, Great to have you here :slight_smile: I appreciate your testing.

  1. Where do you get your samples?
  2. Do you submit all the samples your find? If not, How and why are they filtered?
5

I would like to answer both Melih and Star Shadow’s questions at the same time.

First of all I think that the most important question is missing and that is why we have started with these tests?
Simple, if you take a look at other testing sites you will see the need for independence in this area, we felt that our members deserve realistic testing and not the kind that is orchestrated/fixed…
In the beginning we had Anti-Virus, Anti-Trojan and Anti-Spyware tests, and the first test was done using only 19.000+ samples. In time we decided to start doing only 2 kinds of tests, Anti-Malware and Freeware programs, we have some other plans for the future too…
Where do I get my samples? I have 14 people who are my main suppliers, they are catching new malware and sharing them with me, so I can say that I’m pretty up do date when it comes to new samples, the other part comes from pure VX research, I basically collected every piece of malware that was made in the last 10 years, and the third part is sharing with other malware collectors/testers.
AV Comparatives testing results are highly questionable, they clearly favor certain programs which is more then obvious (take a look at Nod32 and AVG for example), also their malware database Vs. results is pure fiction in my opinion because some programs can’t deal with that amount of malware and get 95+% score. Also we can’t talk about their testing methodology, because by their results…something is out of place.
I am willing to provide help only to the vendors who provide help to their users, meaning freeware products,why?
Maybe many people don’t know this but there is a large number of malware that came out of labs form the most popular AV products, so why help them???
The next test on our site will be a Free Edition test, and since Comodo products are free, yes it will me included as it was before, the samples will include all the new malware that came out in the time between the two tests, and also will include all the malware that is in the wild and the samples that could hit the wild in the following months (unknown malware).
The method we use in our test is simple and effective, fresh install of Windows XP SP3 in the virtual environment, we place our malware sample inside, we make full clones on which we install programs, update them, configure them to their maximum capabilities, disconnect from the internet so no new updates are possible, run on demand scans and remove what is detected, and what is left is…well you can figure out the rest

6

pretty impressed since comodo has only been out for like a month! & i think there are still things to be integrated as well (heuristics, i think, boclean, threatcast, cmf) so the only way cis can go is up the list

7

Thanks for taking time to answer our questions. You truely do providea great service and we thank you. I will look forward to teh freeware test to see how Comodo stands. 14 people is a lot of people to provide you malware, so we can really be certain that a fair sampling of what is out there is included in these tests. That is very important. I hope Comodo and you can work together to help make the web a safer place. I am a user and from my point of view, Comodo really cares about it’s users. The CEO even talks to us, since when do you see the CEO of Norton on their forums? :stuck_out_tongue:

state.of.security: By my understanding of what you said, the tests you do are detection and removal tests only, not preventive tests, right? Some AVs and HIPS work to prevent the malware from being installed in the first place. Are there plans to do this type of testing in the future? To see how well certain programs act to prevent the malware?

8

Of course, everything is evolving, programs, malwares and it is only natural that the testing follows that pattern :wink:

9

Welcome to Comodo forum :wink:

I know that the programs you tested were chosen by your forum users, and they chose them well, but I was wondering, how would score ClamWin. I think it is evolving quite well. What’s your opinion?

I don’t know if it would be of any use, but, perhaps in the future you guys could also just make comparisons between Internet Security Suites, though this test already reveals how a few of them scored.

If I am not mistaken, in vbulettin from July or August (something like that), Outpost Internet Security scored quite well. It would be interesting, as I mentioned above, to see tests just between Internet Security Suites. It’s just a thought.

Anyway, congrats for your great and honest! project.

10

ClamWin is getting better, no doubt about that, but there are a few things they need to improve to catch up with their rivals in the freeware zone.
I was expecting that somebody will ask me about Outpost, the truth is that we planed to test 30 programs, and the votes left Outpost out of the top 30, I will keep an eye on it to see how good it really is.

11

:■■■■

12

Oh, another thing I like about your test is that you give the speed and the user experience as well. Most tests out there don’t do speed.

I am still amazed at the results though. Comodo is high up there at number 12 beating out the old timers like Trend and others. That is a huge accomplishment. I am really surprised about Vipre though. I thought that was going to do really well, but it’s towards the bottom. Okay. I’m done now.

13

Yeah I was surprised too that CIS did that well. Well thats realy good news.

V7chy

14

Thanks for the results!

15

With all the great support we are getting, we do catch a good majority of the latest malware!!! :wink:

Melih

16

It’s just detected a big lump of my samples from the 470 update. ;D Emailing some more now.

17

Welcome to the forum!!

Great Test. Will be waiting for a preventive-HPIS-Proactive-kinda-like test!!! This is where CIS will be #1 (R) (R)

Again, nice results. I like the speed results also.

Keep up the good work!

18

Maybe Comodo DiskShield could help you here.

Install XP and the malware samples and then shield the drive. Once shielded you could install the first test app, reboot and run your analyses. Once completed, unshield the drive. This will leave you back at a clean install with the samples untouched, ready to be re-shielded and have the second test app installed.

Hope this helps,
Ewen :slight_smile:

19

State.of.Security

Is it fair to say that you test with very current malware that is out there in the wild and not with malware left over from 1995? :slight_smile:

And this is the reason Comodo does well because we do catch the latest? (our problem is not the latest malware which we get plenty of thanks to our users and contributors and organisations we work with. It is the malware from old old times that other testing orgs test with! that is why when tested for what matters, you can see what CIS can do)

Melih

20

FYI - The user “panic” on the SSUpdater site is NOT the Comodo moderator “panic”.

I call “Shenannigans”! :slight_smile:

Ewen :slight_smile: