Seeing how they made a huge hash of avast! kill test, I hold zero credibility for any other test they’ve made.
Hint: They killed a GUI portion of avast! and called it a successful program termination… If you kill GUI, scan/protection service continues to operate. Knowing how Comodo also has separate GUI and protection processes, I wouldn’t be surprised if they messed it up the same. There is no termination video for Comodo for some reason, just keylogger one.
Where I wonder if the guy testing this even understands how sandboxing tech even works and what should or shouldn’t be considered a pass or fail…
Could someone tell me where I can get “Schedtest3” and “Bitstest”? (without having to compile myself)
Or are they already included in the bin folder in ssts64? If so, how does one properly run them?
If the test utility can permanently terminate Comodo GUI, then it is very likely that it could be coded to disable all of Comodo’s protection modules. The test utility is intended only to disable the GUIs as a demonstration of an AV’s self-protection capabilities. Sort of a “Proof-of-Concept” utility…
Schedtest3 and BITStest are part of Level 11 test utilities; they are included in the Matousec Test Suite (SSTS64).
Here is download page for SSTS64 from Matousec\Stanford University:
Unless, I’m not misunderstanding your post and from what I remember ~
basically, you create a config file by utilizing configurator.exe -x snapfilename
& open config file (from bin) > search for agreement line and modify it (read instructions from above line);
& copy config file in folder level, for example ;
& run desired test.
note: (1) try cmd prompt ; (2) it’s recommended to read instructions from config file for in-depth testing.
Hope it helps, Sanya.
Why bother protecting GUI when GUI is not a necessity for protection? avast! from what I know doesn’t even bother to protect the GUI process…
It is only a demonstration that the utility can disable parts of AVs = poor self-protection even if it is only the GUI.
I have found the issue with these test, some files are considered trusted by Comodo, including Schedtest3.exe and bits
"2015-08-23 15:34:42 C:\Users\sanya\Downloads\bin\Level-09\Schedtest3.exe Scanned and Found Safe "
Which means these test won’t work when HIPS is set to Safe Mode which it was for his tests, and when HIPS was disabled the auto-sandbox should have gotten them but didn’t because they were seen as trusted.
So in conclusion, some of these tests are treated as trusted files by CIS which is a huge issue since CIS didn’t fail scheduled task, it failed in that the file was trusted, if the file would be modified so it’s no longer trusted then HIPS would be showing alerts.
Edit: The only reason it beat kill5.exe with HIPS on is because CIS doesn’t allow just any trusted applications to kill its processes, kill5.exe was still rated as Trusted though.
Edit 2: Tested Schedtask3.exe with cloud-lookup off and HIPS alerts showed, I selected block and passed the test.
Edit 3: The fact that these files are considered trusted may in and of itself be an issue, seeing as applications using these methods probably shouldn’t be trusted… Can’t fathom why they are trusted? I assume they were added by an automatic system…
[attachment deleted by admin]
From my observations, these are checked manually.
That keylogger thing should be covered though. Sandboxed apps should not be able to grab your keystrokes. It should either block access to the keyboard or install it own keyboard driver and encrypt the input. Default config for CIS (fully virtualized without restrictions to keyboard access and allowing all outgoing requests for firewall) makes stealing valuables possible.
I knew this was going to be posted here, so I would like to clear things up about this. First as stated earlier ALL of ssts64 tests are trusted by comodo cloud lookup, this obviously will produce incorrect results. So in order to properly test CIS with these tests, you must either turn off online lookup or import the test folder to the file list and select unrecognized for file rating. Another step you must take to run these tests is to run them from an elevated command prompt by right-clicking the start menu and selecting Command Prompt (Admin), even if your account type is administrator and have disabled UAC. These test should be run outside the sandbox and be tested against the HIPS and firewall components of CIS only so disable the AV and auto-sandbox.
I have ran all of ssts64 tests both with and without the -u argument and comodo fails only DNStester and kill3f when executed with the -u parameter, the tests failed without -u are: echotest2, keylog3, sss2, vbstest, keylog5, sss3, dnstester, keylog1, cliplog, inject1, keylog2, kill3e, kill3f, kill5.
I have alread reported a bug with dnstester here:https://forums.comodo.com/bug-reports-cis/access-to-dnsrpc-client-service-no-longer-monintored-on-windows-10-t112528.0.html as it appears that any access attempt to Windows DNS/RPC Client Service is not detected.
That’s kind of what I’ve been saying. The tester has no idea what he’s doing. Comodo whitelisted the files because they were clean (despite the fact they are test files). The reason behind it is most likely the fact that these test aren’t publicly accepted like EICAR for example so there is no point in obeying their “test” status. If they are clean, they are whitelisted by their whitelisting system.
Which in the end explains why it failed…
Much thanks Futuretech…
Which seems hypocritical since Comodo and especially Melih are quick to point to the matousec test results with pride.
I dont know if it’s true but it’s scary if it is!
Hypocritical Melih…Never!..Who remembers Kevin?..
Perhaps it is nothing but an over-sight on the white-listing technician’s part. Is that not the more likely answer rather than the whole hypocrisy theory ?
Someone just needs to submit the SSTS64 test pack on the false negative submission thread along with a full explanation. The files will then be re-rated to Unrecognized and all will be right in the Universe again… order shall be returned to the Force.
I merged your topic with an already existing topic.
What Kevin? Are you referring to? When referring to Kevin McAleavey? In that case tread carefully as controversy is likely not far away.
Yeah that’s who I was talking about. Sorry but it leaves a bitter taste in my mouth when I think about it.
Thanks for the warning and yep I will stop now. That was my thoughts and feelings and not to be confused with anybody else’s thoughts or feelings on any other matter ever.