Test of CIS 5.8 Beta

Long post…but it’s interesting. If you do not want to read the details of the test, skip to the end of the post for my conclusions.

I did an unofficial test of CIS 5.8.199581.2037 Beta (mainly to investigate it’s performance for myself). In general, I am not concerned about a security program’s signature based malware detection. If you do not have a signature, you are not protected (if a real-time scanner detects 99%, it takes only one malware to slip through and cause major problems). So, I want to see how well a security program can protect me even if the real-time signature based scanner is disabled.

Test set up
I ran the tests on VMware with a clean install of XP SP3 32 bit. Just to be sure, I ran the most up to date versions of CCE, Malwarebytes, Hitman Pro, and Kaspersky TDSS Killer and all showed no infections.

I started with 61 malware files from Malwaredomain list and malcode database (about 50% of the malware was posted in the last 48 hours, and the rest were from within the last month).

Test of On demand scanner
I started with on demand scans using the most recent version and most recent malware database for CIS, HMP (hitman pro), and MWB (Malwarebytes). I scanned the malware folder sequentially with the 3 scanners (each time I scanned in a different order). Here are the results:

61 files: CIS removed 52 files → MWB removed 7 more → HMP removed 2 more
61 files: MWB removed 57 files → CIS removed 2 more → HMP removed 2 more
61 files: HMP removed 55 files → CIS removed 4 more → MWB removed 2 more

This is a good demonstration that using only signature based protection will expose you to malware that is unknown to your scanner (and this is why I think the signature based detection rate is not very important). The results above also show the importance of using more than one on-demand scanner to check a PC for malware.

Test of CIS with real-time antimalware scanner disabled
I then ran the 61 malware files with CIS 5.8 Beta installed (no other antimalware or security programs were installed). The CIS settings were:
Antivirus - DISABLED
Configuration = proactive
All alert options were set to allow me to chose what to do (no autoblocking). I responded to alerts as follows: Virus alert response was “clean”. If the D+ alert indicated that I should block or sandbox, I did.
Rest of settings were default (firewall safe, D+ safe, sandbox enabled)

Here is what happened (the numbers represent the # of malware files that generated a given result):

  • 40 - Almost immediate Cloud AV scanner alert and sandbox alert.

  • 8 - Almost immediate Cloud AV scanner alert and D+ alert saying block this known malware.

  • 7 - Sandbox alert only.

  • 2 - Sandbox alert, but files would not run (not a valid win32 application error)

  • 2 - D+ alert alone, and text in alert was clear enough to recommend blocking, even to a noob.

  • 1 - D+ alert that application was signed, but not yet whitelisted - seemed okay to allow, so I allowed it. Virustotal detected this as “Gen.Variant.Kazy” malware on 16/43 scanners - not sure if this is actually malware (file submitted to comodo)

  • 1 - ran without any alerts, trusted by comodo. Virus total scan showed malware on 1/43 scanners - may not be malware.

Ran CCleaner, then rebooted. Then I got the following D+ warnings about 30 seconds after reboot (see pics below), but no other alerts. Personally, I think these warnings are not very helpful, and many users will likely allow these requests since the alert is rather neutral. I blocked these alerts. Then, I checked the files that generated these alerts with virus total:

processindex.exe -17/43 scanners detected it as Gen:Variant.Kazy. (possible malware).
stpp.exe - 1/43 scanners detected it as adware (likely not malware).

2 other files were dropped in the same folder as stpp.exe (program files>stip)
rmip.exe - 1/43 virustotal scanners detected it as adware (likely not malware).
stip.exe - 0/43 virustotal scanners detected it (likely not malware).

Then I scanned with MWB, and it showed 8 malicious items: 6 executable files, 1 folder, and 1 registry entry. No malware was running in memory. Also, the files listed above (processindex, stpp, rmip, and stip) were not detected as malware. These INACTIVE files were likely left over because of incomplete virtualization of the automatic sandbox.

I then ran the 6 executable files that MWB detected and CIS gave the following alerts:

  • 1 - Almost immediate Cloud AV scanner alert and D+ alert saying block this known malware.

  • 1 - Almost immediate Cloud AV scanner alert and D+ alert saying block this known malware. But, when I selected block on the D+ alert, the file ran in the sandbox, took focus (window remained in front of everything else), and I could not bring the AV alert to the front. To access the AV alert, I had to open the task manager and terminate the malware process. Comodo needs to find a way to prevent a sandboxed malware from taking focus (full screen or not)!

  • 2 - D+ alert saying block because heuristics detects as malware. But, when I selected block on the D+ alert, the file ran in the sandbox, took focus (window remained in front of everything else), and I could not bring the AV alert to the front. To access the AV alert, I had to open the task manager and terminate the malware process.

  • 1 - Sandbox alert

  • 1 - could not find the file despite viewing hidden files (could not find the folder it was in either)

Scanned next with Hitman Pro. It found 5 additional executable malware files that were not detected by MWB (but nothing malicious was running in memory). I ran the 5 executable files that HMP detected and CIS alerts were:

  • 2 - Almost immediate Cloud AV scanner alert and sandbox alert.
  • 3 - sandbox only (these files were detected by virus total 14/43, 18/43, and 16/43).

My conclusions:

  1. Even with the CIS on-demand scanner turned off, the D+ cloud scanner will alert the user to the majority of malware almost immediately. I know it is signature based, but this shows the effectiveness of the cloud scanner in CIS.
  2. Comodo needs to find a way to prevent a sandboxed malware from taking focus (full screen or not). When this happens, it often makes the PC unusable, especially when an undetected malware goes full screen (an undetected malware can take focus even after reboot). In my case, the heuristics detected the malware, and even though I blocked it, it ran in the sandbox and stole focus.
  3. Comodo needs to minimize the number of alerts per given malware file. For a single executed malware file, I often got 2 or 3 cloud AV alerts, a sandbox alert, and 2 or 3 quarantined alerts (5-7 alerts for one file!). In other cases, I would also get multiple cloud AV alerts and a D+ alert for a single file. These multiple alerts are confusing. Comodo needs to find a way to prioritize the alerts, so that you get one alert for the detection and then one confirming quarantine (or other action that was taken).
  4. CIS must be very careful when trusting programs. I am concerned about the integrity of the whitelist, the integrity of the trusted vendors list, and the validity of signatures. An alert that says “application is signed by a trusted vender, but not yet whitelisted - okay to allow” is not an acceptable alert if a file is malware (not sure the file was indeed malware, but it was detected by 16 scanners on virus total). As suggested many times before, Comodo needs a way to let the user control the trusted vendors list.
  5. Hopefully we will see full virtualization of the auto sandbox.

Sorry for the long post - I do not know how to make a video!

Whoop

[attachment deleted by admin]

Yes, as you mentioned its an interesting read & test. I agree with your points to be improved in CIS. And I liked the way you tested 3 scanners in different way & showed the signature protection. Similarly I liked you tried running the detected malware by MBAM & HP & showed how CIS reacted.

Excellent Test.

Thanxx
Naren

I MUST agree with you, especially in this point:

2) Comodo needs to find a way to prevent a sandboxed malware form taking focus (full screen or not). When this happens, it often makes the PC unusable, especially when an undetected malware goes full screen. In my case, the heuristics detected the malware, and even though I blocked it, it ran in the sandbox and stole focus.

I’ve submitted samples to Egemen which are covering all the user’s screen and we were discussing about this.

He said that it would be too complicated for user (this alert) that some application wants to get full screen access. I can’t agree.

I hope that devs will read this carefully and do something about it, because many malware (right know, ransomware) cover all the screen preventing user to stop this.

Today protection against this action is essential. Really, just look at malware from MDLfrom the last few months. 80% is Ransomware, 50% of them are covering users screen and comodo user is forced to restart the PC because of a no protection from this.

In other points i agree fully as well.

Please, really take it into consideration, i hope it’s not a lot of work and can help a lot. Users shouldn’t be harmed about this, many of them is requesting this D+ ability.

Thank you very much Whoop. That is a great test with clear observations and suggestions for the Comodo devs.

I don’t agree with egmen here either. Outpost has a full screen warning for a long time and I tend to think that the users may get more distressed by the ransom ware take over than by the full screen alert.

On a side note. Would it be possible to provide an option to force a program that is requesting full screen to open in non full screen?

Fully agree Whoop. If we want good protection we MUST fix those problems. Again well done Whoop :-TU

Hmm great test Whoop:

Re: An alert that says “application is signed by a trusted vender, but not yet whitelisted - okay to allow”

Didn’t know there was such an alert, thought, valiod signature from trusted vendor = automatically trusted. Do you have to set a speial setting for this?

Maybe it should say ‘vendor trusted by Comodo’ then ’ if YOU trust this vendor or cOmodo’s list OK to allow’

:slight_smile:

Is it new to 5.8 do you think?

May lead to lots of alerts…

Mouse

Well done. (:CLP) (:CLP) (:CLP)

It really shows how CIS 5.8 (or the other version too) reacting to real life malwares, CIS’s weaknesses are pointed out in the test. I fully agree with your conclusions.

mouse, this pop-up shows when you execute digitally signed malware or adware which are not in Comodo TVL for obvious reasons…
Also, a number of companies that sign their products don’t apply to Comodo policy yet to become member of TVL.

Hi Mouse,
I cannot member the exact wording of the alert. Simply getting an alert is telling the user to be cautious. However, the alert stated that the file was signed (presumably from a TRUSTED vendor) but had not been added to the white-list. This statement implies that the file is safe (and that Comodo simply has not added it to the white-list yet). The wording did not convey a sense of caution to me. It seemed more like a notification than an alert. I think the average user would allow this alert.
I think a bigger concern is that a probable malware generated such a benign alert. The AV reduces the number of D+ alerts, but if the few alerts we get are one’s like this, then users may get infected. If the file is truly malware, then something needs to be changed to make the alert more of a “red light” than a “green light”.

Hello Whoop-dee-doo.

Your findings are very thorough and informative and thank you very much for your
great effort.

In addition to your findings with Defense+ and full screen applications i found that
many times when CIS has not had new or old video games in their trusted vendors
list that when i try to start a full screen video game CIS blocks the application,
which it should by design but CIS does not display pop up notification
for the user to decide to select allow or deny or treat as.

Since i can’t respond to alerts or use the keyboard to exit the application the only
way is to press the reset button on the computer case or the shut down button.

I know that i can manually make the Defense+ rule but i some times i forget to do that
and sometimes the application is accessing more access rights and i can’t respond
to the alerts when the application is in full screen mode.

Do you know or anyone if the developers will enable the alerts to be visible
in any full screen video games and applications?

Update on the test:

The alert described as “1 - D+ alert that application was signed, but not yet whitelisted - seemed okay to allow, so I allowed it.” was caused by a file called “stipsetupris” (the alert is shown below). You can see why an average user would think that the application is safe to allow.

As of today, Comodo real-time AV and cloud AV does not detect “stipsetupris” and now Virustotal detects this as “Gen.Variant.Kazy” malware on 17/43 scanners.

“stipsetupris” installs several files, including “processindex” in “Documents and Settings\All Users\Application Data”. “processindex” is now detected by Virustotal as “Gen.Variant.Kazy” malware on 22/45 scanners, and it is now detected by Comodo AV.

So, the original installer “stipsetupris” is not detected by Comodo AV, but D+ generates an alert (which is good), but the alert (shown below) seems to convey that it is okay for the user to allow this application. Then, when you allow it, it installs “processindex” malware.

It is true that an alert is only as good as the user who allows or denies it; however, I would expect a truly effective and user friendly security program to guide the user more effectively than Comodo did in this example.

Having said that, CIS did give alerts when the “processindex” malware tried to run, and those alerts (shown in my first post) conveyed more of a sense of caution (but not much more). I know Comodo cannot make every alert perfect in every circumstance, but I hope my example can help improve the utility of the alerts, especially for novice users.

[attachment deleted by admin]

Ah now I understand. The vendor is not whitelisted - so it’s a UA alert - not a new alert type.

Ta Whoop. As you say sounds too reassuring, had not noticed that before.

Mouse

Yesterday I tested free AV’s & CIS with zeroday malware from MDL. There were 2 digitally signed malware.
These malware were only detected by AVG & CIS gave Unlimited Rights Alert for both mentioning Digitally Signed but not Whitelisted by Comodo. If you are updating/installing this app you can allow it.

The reason I am mentioning this here is coz of AVG’s detection.

AVG detected both the malware. When I checked the AVG Quarantine, the further info about both the malware was “Certified by Untrusted Certifier”.

I think Comodo should do a check like this for the digitally signed apps not whitelisted by them.

Thanxx
Naren

+1

I totally agree with you right now I think that this is one of the weakest points of CIS.