Test of CIS 5.9

Comodo Internet Security - CIS News / Announcements / Feedback - CIS
1

If you do not want to read the details of the test, skip to the end of the post for my conclusions.

I did an unofficial test of CIS 5.9 (mainly to investigate it’s performance for myself).

Test set up
All operating systems, applications, and databases listed below were up-to-date as of 12/31/11 at 9:41AM my time zone.

  • Ran tests on VMware Player (version 4.0.1 build-528992)
  • Operating system on WMware: Windows XP Professional (32-bit, Version 5.1.2600), Service pack 3, all updates installed.
  • CIS - Comodo Internet Security 5.9.219863.2196 (AV database version 11153)
  • HMP - Hitman pro 3.6.0 (build 138)
  • MWB - Malwarebytes 1.60.0.1800 free version (Database version v2011.12.41.04)
  • TDSSK -Kaspersky TDSSKiller version 2.6.25.0
  • Internet Explorer 8.0.6001.18702
  • Other security software installed: CCE 2.3.22892.176, Killswitch 2.4.219500.176, Comodo Autorun Analyzer 2.3.219500.176
  • Windows firewall was disabled.

CIS settings (note that many of these settings are NOT default):

  • AV Realtime: disabled

  • AV manual: scan memory on start, scan archives, enabled cloud scanning, enable rootkilt scanning, heuristics low, do not scan files > 40 MB

  • Defense+: safe mode, enable enhanced protection

  • Execution contol: enabled, treat unrecognized files as restricted, scan unrecognized files in the cloud, detect shell code injections

  • Sandbox: Enabled (all check boxes on the sandbox options window were selected)

  • Monitor settings: all items selected.

  • Firewall: safe mode, advanced settings all UNselected, stealth port wizard = block all incoming and stealth ports to everyone

  • Configuration: proactive

Baseline Testing
After clean install of Windows XP on VMware, I ran CCleaner. Then I scanned the system to make sure the scanners detected nothing at baseline:

  • CIS complete scan = no items detected
  • MWB full scan = no items detected
  • HMP default scan = no items detected
  • TDSSK default scan = no items detected
  • Comodo Killswitch = all active processes were safe; quick repair showed everything okay.
  • Comodo Autoruns = everything was safe.

On Demand Test
I put 5 safe files and 10 malware files in a desktop folder called “Testing”. Malware was obtained from posts in Malware domain list and Malcode Database over the last 2 days. The “Testing” folder was then scanned on-demand with MWB, HMP, and CIS (detection rates are in the attached table).

Execution Test
I executed each file with CIS defense+ and firewall active (AV was disabled). The results are in the attached table.

What Was Left Behind After Execution Test?
Then, I rebooted the computer, ran CCleaner, and then did the following scans in the order listed:

  • Comodo Killswitch = all active processes were safe; quick repair showed everything okay.
  • Comodo Autoruns = could not get it to run - it stopped responding before it initiated the scan or after scanning only 7 items, or it spontaneously closed without scanning.
  • CIS on demand: detected a dropped file “pfi.exe” as malware in “C:documents and settings/adminsitrator/local settings/application data” folder (this file was sandboxed during the execution test), also detected “a200” and “a1000” in the “Testing” folder on the desktop (not executable and so were not removed by CIS during execution test) - all were cleaned.
  • HMP default scan: detected “Appinstaller.exe” in the “Testing” folder on the desktop (this file was sandboxed by CIS during the execution test, so it cannot harm the computer if executed with sandbox enabled).
  • MWB full scan: detect “zq1.exe” and “zq2.exe” as “trojans” in the “C:documents and settings/adminsitrator/local settings/application data” folder. Both of these are zero byte files (and therefore could not be uploaded to virus total). Neither of these files could be executed (not valid win 32 applications), but CIS gave an auto sandbox alert for both of them). I am not sure what program dropped these files. These files were removed. MWB also detected all of the files in CIS quarantine.
  • TDSSK default scan = no items detected
  • CIS Autoruns (results are from after deleting all installed safe programs - see conclusions) = all items safe

Conclusions from this test:

  1. Comodo needs to minimize the number of alerts per given malware file. For a single executed malware file, I was usually presented with 3 separate alerts. These multiple alerts are confusing. Comodo needs to find a way to prioritize the alerts, so that you get one alert for the detection and then one confirming quarantine (or other action that was taken).
  2. Full virtualization of the auto sandbox will help prevent dropped files (such as “pfi.exe” discussed above). Technically, this file did not harm the computer (and could not harm the computer if CIS remains enabled), but it is still present on the hard drive. Conceptually, it would be nice to eliminate all traces of malware from the computer on reboot (threatening or not).
  3. ? Compatibility Problem with Autoruns? - To trouble shoot autoruns, I downloaded another copy and tried to run the new copy: it crashed. I tried changing the file name of Autoruns, it crashed. I tried rebooting and then running Autoruns, it crashed. I then started deleting each of the safe programs that were installed. Autoruns seemed to work only after “law of attraction” was removed. I was able to reproduce the crash by re-installing “law of attraction”, and then confirm that Autoruns worked again after uninstalling “law of attraction” [“bug” reported here]

Conclusions from my previous test:

  1. Even with the CIS on-demand scanner turned off, the D+ cloud scanner will alert the user to the majority of malware almost immediately. I know it is signature based, but this shows the effectiveness of the cloud scanner in CIS.
  2. Comodo needs to find a way to prevent a sandboxed malware from taking focus (full screen or not). When this happens, it often makes the PC unusable, especially when an undetected malware goes full screen (an undetected malware can take focus even after reboot). In my case, the heuristics detected the malware, and even though I blocked it, it ran in the sandbox and stole focus.
  3. Comodo needs to minimize the number of alerts per given malware file. For a single executed malware file, I often got 2 or 3 cloud AV alerts, a sandbox alert, and 2 or 3 quarantined alerts (5-7 alerts for one file!). In other cases, I would also get multiple cloud AV alerts and a D+ alert for a single file. These multiple alerts are confusing. Comodo needs to find a way to prioritize the alerts, so that you get one alert for the detection and then one confirming quarantine (or other action that was taken).
  4. CIS must be very careful when trusting programs. I am concerned about the integrity of the whitelist, the integrity of the trusted vendors list, and the validity of signatures. An alert that says “application is signed by a trusted vender, but not yet whitelisted - okay to allow” is not an acceptable alert if a file is malware (not sure the file was indeed malware, but it was detected by 16 scanners on virus total). As suggested many times before, Comodo needs a way to let the user control the trusted vendors list.
  5. Hopefully we will see full virtualization of the auto sandbox.

And this follow-up to my previous test:

The alert described as “1 - D+ alert that application was signed, but not yet whitelisted - seemed okay to allow, so I allowed it.” was caused by a file called “stipsetupris” [the alert screenshot is shown in the link above]. You can see why an average user would think that the application is safe to allow…[snip]…

So, the original installer “stipsetupris” is not detected by Comodo AV, but D+ generates an alert (which is good), but the alert [shown in the link above] seems to convey that it is okay for the user to allow this application. Then, when you allow it, it installs “processindex” malware.

It is true that an alert is only as good as the user who allows or denies it; however, I would expect a truly effective and user friendly security program to guide the user more effectively than Comodo did in this example.

Having said that, CIS did give alerts when the “processindex” malware tried to run, and those alerts conveyed more of a sense of caution (but not much more). I know Comodo cannot make every alert perfect in every circumstance, but I hope my example can help improve the utility of the alerts, especially for novice users.

Happy New Year
Whoop

[attachment deleted by admin]

2

great test. CIS is still strong but like you said it needs work with its alerts. Hopefully comodo addresses these issues with v6.

3

Great test Whoop. :-TU

Good to see CIS did improve over the previous version. I do agree the amount of alerts of the AV is over the top and needs to be addressed. Also I would like to see the full path of the virus being detected instead of having to point my mouse pointer to it.

4

maybe in v6 they will change the scan window to look like the one in CCE, it shows a percentage bar, the full file path, and allows the user to see what viruses have been detected while its still scanning

5

this feature i thought it would be implemented in CIS 5.9 but unfortunately it didn’t happen… hope in CIS 6 we can see this…

6

Good sandbox - that’s a dream!

7

Thanks for the information given about the test that you have made.

8

I think there should be multiple threats detection alert i.e if multiple threats are detected there should be multiple threats detected alert & not a single alert for each malware.

Thanx
Naren

9

good test.

lets get Egemen and Igor’s attention to this post pls

thanks

Melih

10

Hi Melih, I PMed Igor and Egemen.

I repeated the test above with 30 new malware samples (taken from various malware posts in the last 48 hours). Again, CIS prevented infection (no actively running malware on reboot), but several inactive malware files were dropped on the hard drive. I executed the dropped files, but CIS prevented infection from the dropped files too. I also experienced the “steal focus” issue again (as I did in my 5.8 test), and was annoyed by the multiple alerts for a single malware again.

So, even though I tested only a few malware files, there are a few consistent findings:

  1. CIS prevented infection in every case (no actively running malware on reboot). :-TU
  2. Too many alerts per given malware file.
  3. Malware running in the sandbox can go full screen and/or take focus, making the PC unusable and forcing a power down to reboot (no way to save things you are working on!). Hopefully, Comodo will find a way to prevent a sandboxed malware from taking focus (full screen or not).
  4. When malware is run in the auto sandbox, CIS allows inactive malware files to be dropped on the hard drive and these files are not removed on reboot. Conceptually, it would be nice to eliminate all traces of malware from the computer on reboot. It would also be nice to have an “empty sandbox” button that would terminate all sandboxed processes and delete the files that were dropped (with no reboot necessary).

Thanks,
Whoop

11

Ok great thanks…

many alerts during an infection could be a good thing actually…but we will also look into it…

thanks

Melih

12

Getting 5-7 alerts is overdoing it though…