Let me make it clear that I am not knocking the product as it performs well in many other tests as is as well as for me and the clients I’ve offered this to, but I ask that the staff of Comodo hears me out for a brief moment.
I’ve been lurking on here on the CIS part of the board for a while and I’ve been a proud supporter of Comodo since I’ve first taken notice of it early this year. However, I’ve grown weary of this constant delay in proving the worth of your security product against other established products on the scene in tests that for the most part have been trusted in the industry for a respectable amount of time. No where does it state that building trust involves disregarding the trusts that other people hold for previously established standards.
I know it’s a “better” test you want, but why is this such a key ideal? Is it that costly to participate in all of these other non-AMTSO (or non-“whatever it is you feel is worthwhile”) compliant tests? Will it somehow be such a perfect test to prove that Comodo is on top that so no one will question it? Is there such thing as a perfect test in the first place? Does it matter really if Comodo doesn’t come out on top in lesser tests? Isn’t coming short in a lesser test an indication of some failing in performance, regardless?
Real world performance, theoretical performance, whatever something tests for or doesn’t test for; I don’t see how keeping Comodo out of the popular and possibly tainted tests is really anything but silly pragmatism. It’s not as if your lack of participation in such tests forces them to evaluate their standards, nor (if your confidence is truly justified in this product) will consumer confidence in the product be shaken in any way. If your product truly performs as advertised then put your money where your mouth is even if the competition is stacked. It’s like restaurants attempting a Zagat rating regardless of how corrupt it is, just to see how it would turn out, because once in a while, a deserving restaurant from out of no where can actually break through all the politics and underhanded dealings to gain some recognition. Besides, the logical people would see the value of any test anyways and take it with a grain of salt, AMTSO compliant or not.
I hope that this personal statement of mine reaches some people not as an angry rant, but as a logical plea for Comodo to reconsider what I personally consider to be a ridiculously staunch position in their testing participation. I know that I am not alone in this, but I will not be so presumptuous as to speak for anyone but myself. This is my own view and if any of you share it, so be it. I also know that this has been brought up before and debated over, but I ask now that we re-evaluate this as an AMTSO compliant test is taking its sweet time in its formation and that V4 is hotly anticipated (by myself included) and will be ripe for proving as soon as the initial release hurdles overcome the major bugs that are to be expected with any major upgrade of any program (especially with an overhaul like this). A great product, just like a great boxer, deserves to be put in the ring and I ask you all, is this not a great product?
I agree completely. CIS needs to be tested, even if the test itself isn’t perfect. The people deserve to see how CIS stands up against the competition.
First of all, the above statement is the very problem. There is NO standard, there never was for testing. AMTSO is the first one. The rest of the tests people do complies with no standards.
Now, lets be honest here, its not as if people haven’t tested CIS From matousec to individuals who are running av testing sites, they all have and we have done pretty well in all of them. We even have provided testing tools ourselves! We haven’t gone out to these entities and said, pls test us… Now thats where the issue is: people want us to us av comparitives site. Now these folks do NOT do AMTSO compliant dynamic testing. And that is what you need to test CIS. to show its true strength. They claim they will do it soon, but i don’t know when and as soon as they are, we will be one of the first ones, if not the first. So the issue is really should CIS be tested with 1980s method of testing which does not test CIS AT ALL. Just like testing if you can speak english using chinese language! Thats how much sense it makes to test CIS’s true power with these static 1980s tests which are totally outdated. That is why AMTSO was created. Because new security products have different security methods and current testing methods are not capable of testing them, simple as that. We are NOT stopping people from testing CIS…just like others have tested…anyone is welcome to test it…we are not going to add credibility to any non AMTSO compliant organisation by request a test by them. (on the other hand there are some organisations we are working with for internal purposes).
As Melih has stated, CIS has been tested repeatedly by eg. Matousec (and it currently has the top score).
Now even though it hasn’t been stated specifically, I suspect what you guys are after is for the Antivirus component of CIS to be tested against others out there - and ideally, you’d like AV-Comparatives to do it. If ONLY this Antivirus component is tested, my guess is that it would do very poorly, with only above average detection and numerous false positives, thus scoring poorly on AV-comparatives formula scale.
Now this would make Comodo appear bad. The fact is that Comodo’s Antivirus component alone would be very weak protection. Where Comodo strives is when this Antivirus component is combined with the powerful Defense+ (and Firewall) components.
I personally don’t even use a real-time Antivirus component, but that might be because I’m an “above average” user who knows what I’m doing (also the fact that I’m using a properly configured Sandboxie with LUA + SRP etc).
The fact is that CIS when tested in its entirety (including the powerful Defense+ component) will almost always score 100% in any test out there. The only problem (for the noob user) is when to answer “Yes”. For the “above average” user, the only problem would be coping with the numerous pop-ups whenever updating or installing programs.
However, we’ll see how CIS version 4 deals with these problems.
Reactive - traditional AV (with/without heuristics, doesn’t matter). They scan for files and match them with the signatures in their database. The larger the database the higher the detection rate. Simple-easy. But not effective. WHY? coz it takes time to update the database with the sigs and getting the sigs in also time consuming. Any delay is suicidal as it’ll make the user vulnerable.
Pro-Active - New Default Deny type of systems. They alert user on every action and the user must choose. There is no delay here but the decision is with the user.
CIS uses both approaches. And there in lies the problem. The AV without D+ is type 1 whereas D+ without AV is type 2. A program which uses both approaches must be tested as such. You cannot use a different benchmark as the functions cannot be isolated as they complement each other.
Why Comodo Inc insists on AMTO (whatever) is that those tests will mimic the design of CIS closely and it will/might do well.
Of course, it dosent hurt that Comodo Inc is buying time with this argument and making it ready for Av-Comparatives… say june 2010?? ;D (i can lay a friendly bet on this… ;D ;D )
Now the questions which counter your points comes out to this: Why SHOULDN’T CIS be on par with other scanners in on demand scans and traditional virus detection? Is this an admission that CIS falls short for securing a computer that’s already infected? Must we turn somewhere else for another anti-virus for what anti-viruses have been expected to do since they were first introduced: clean viruses out of a computer?
If Comodo is to be THAT specific with the role of its program as merely an infection and intrusion prevention mechanism and not as a detection and removal tool for a pre-existing condition of infection (considering such functionality is inclusive under the term and genre of “anti-virus”) then it should be clear in differentiating itself from the traditional label as well to remove the expectations the industry and the consumer base has of a product carrying such a title.
Either test CIS’s Anti-Virus component as an Anti-virus or stop calling it such to avoid this confusion.
The definition of “Anti-virus” is changing and/or evolving over time. Some might even argue that CIS in its entirety is an “Anti-virus”.
The way I see it? The Anti-virus component of CIS is an attempt to reduce the number of “Yes/No” pop-ups (from Defense+), and thus is an attempt of making CIS more “noob-friendly”. Don’t forget that CIS contains a very powerful Classical HIPS (Defense+), which 99.99% of people out there generally avoid like the plague.
That doesn’t remove the need for an anti-virus that does the traditional work of removing viruses that have already made it onto a certain target system whether it is before Comodo is installed or if it happens somehow after. The existence of the virus needs to be detected to prove that CIS can do its job as traditional anti-viruses already can. Are we to say that we need to take a step back in the reactive capabilities of an anti-virus to have an effective proactive anti-virus? Nothing discounts the negative of not having the ability to thoroughly clean up an already infected computer. Failure in those tests means that there’s an area it can improve upon within that genre, regardless of whether or not you want to try and redefine the genre. Nothing says it’s fair to neglect the need for traditional functionality.
But what is the goal? The way I see it, CIS is all about prevention. If a system is already infected, that’s a different story. Some scanners will detect, some scanners won’t. Some scanners will remove, some scanners won’t. It’s all a roll of the dice.
The way I see it, CIS’s goal isn’t to detect and/or remove malware on an already infected system. The goal is much more important - to prevent getting infected in the first place.
Remember that some don’t have the luxury of starting from that position. It’s possible to get an infection on your PC that your current AV can’t detect or safely remove. The detection and removal of malware is important.
I think the bone of contention here is different people have differing opinions on how important it is.
But the more you prevent it to come to your PC in the first place the less you will need cleaning. Also you can always reformat to get rid off infections…but there is no easy solution to preventing it coming into your PC.
Prevention is of course better than cure. Simple as that.
However, if you’re already infected, then it’s probably best to start over again, and focus on prevention. And CIS has very powerful prevention, especially if you know how to use Defense+ well.
Even now, the best way to clean an infected computer is to run more than one scanner. You just can’t rely on one scanner to do a decent job. This in itself already implies why prevention is so much more important.
Anyway, if you’re already infected, I’d probably use the following combination of scanners (as a minimum):
Melih got support from a Norton employee, that seems to think that todays tests are irrelevant as well…
(I have shortened it down some, Read the full post here).
We are trying to address the problem that the major labs - av-comparatives, av-test, VB, ICSA, West Coast - none of them test what we consider "real-world" scenarios. Most of these tests are of zoos of malware sitting on hard disk. This simply isn't how most users encounter viruses. So we asked Dennis Labs to identify malware infected sites and to surf those sites with 10 different internet security suites installed - and to record the full experience.
The results are interesting in that where most products score at near 100% detection on the zoo tests - more then half scored 75% or below on the Dennis Labs results.
Basically it looks as if he is talking for Norton and is saying that todays (static) tests are misleading and they want “real world” tests.
Again, starting over is a time consuming luxury not everyone can afford. Dealing with the here and now and being back up and running is a huge point of contention for everyone. Even if you have a backup or restore a system to a previous state, you still have to recover certain things and many times, with a deadline constraint, things get lost in the process and productivity is lost as a result.
The crux of this debate lies in the necessity to cover the scenarios that traditional anti-virus programs already cover when trying to improve upon the model. Leaving behind a level of functionality in some arbitrary decision that prevention would render traditional scanning obsolete is presumptuous. There will never be a time in the world when crackers and malicious programmers would stop finding ways around any type of security. Having scanner technology developed for infections that made their ways to computers before the proactive security technology can be improved to catch up to a new type of attack would reduce downtime and losses.
You’ve basically agreed with me that Prevention is better than cure in that first paragragh.
You state “There will never be a time in the world when crackers and malicious programmers would stop finding ways around any type of security”. I suppose you’re going along the lines that everything is able to be bypassed in theory. So therefore we should talk about probabilities, and I’ll come to that in a moment.
First though, I think we need to address the fact that black-lister/behaviour-blocker technology is not a good way to keep a computer clean:
Your antivirus (or black-lister scanner) detects 99.99% of all known malware (keeping in mind that 99.99% is being very generous)
Let’s say 10,000 malware are released every day
That means 100 malware are undetected by your scanner
And keep in mind it only takes ONE piece of malware to get infected badly
Also what about unknown malware? How far will heuristics and behaviour-technology take you?
As you can see, traditional “Anti-virus” technology is fairly hopeless in preventing a system from getting infected. It’s all a roll of the dice. Whenever you come across a piece of malware, all you can do is hope that your scanner will detect it - roll the dice, and roll the dice again.
However, other technology like Defense+ and Malware Defender (that is, classical HIPS) would pick up everything - let’s say 99.99999% of both known and unknown malware. If you further configure your classical HIPS properly, it will be “100%”. Of course, the argument against this is that the “noob” user wouldn’t know how to setup/control and answer the classical HIPS properly. And that is a fair point, but it isn’t really relevant to the overall point I am trying to make.
Of course, ideally, the “Anti-virus” component should be up there with Avira or whatever you think is good. Ideally, there shouldn’t be any malware in this world too. The fact is that CIS already has “100%” detection and prevention. You just need to learn how to use classical HIPS properly.
And therefore, ultimately, user knowledge and common sense is perhaps the most powerful form of prevention.
Except the whole detection part is what we’re asking to have tested.
The rare chance an infection or vulnerability is exploited to bypass a Proactive measure will be the only chance it needs to sit and sit quietly on a system where it can deconstruct the protection you were expecting to operate under altogether. Having a powerful scanner that can be used to sweep something after taking the system offline and perhaps doing it in safe mode or some other format would clear you of an infection after the fact that it’s made its way in and the security community has recognized it would be beneficial to those who would rather not have to start over from their most recent backup or even from the OEM’s set up.
An infection these days usually operates in some variation of this format:
Exploit security hole and place itself upon system.
Unleash debilitating payload to prevent security from recovering or operating as normal.
Spread throughout the system using any number of mechanisms and make complete detection and removal as difficult as possible.
In which case, you must take a system offline and attempt removal without activating the infection’s own self defense and preservation mechanisms.
Traditional file and registry scanning is more effective in this case.
Even if the infection does not deliver a payload as extreme as those in the league of the Conficker worm, it’s good to have something that can detect an infection after the fact it’s gotten on your system. Weekly scans and constant updates to the definitions files are that which ensures that what might have gotten on your system doesn’t stay that way.
Test the scanner for that capability (i.e. traditional AV testing) and prove it performs and we’ll be happy.
Prevention does not supplant the need for effective cures.
No arguments from me there. All I was saying is that CIS is focussed on prevention, which is the most important. There are plenty of other programs out there (free ones too) which are more focussed on cure/cleaning. I’d be very surprised if Comodo can come up with a top “Antivirus” component in such a short time. Give them another few years perhaps.
EDIT: also the probability of bypassing Defense+ is so low, it’s negligible. All I’ve heard of in recent times are questionable POCs that are able to bypass it. Combine Defense+ with a limited user account and you won’t really need to worry about getting infected. Better still, learn how to use Sandboxie and implement SRP, and you’ll realise Defense+ isn’t needed!
Testing provides information on how one might improve. It’s not just for bragging rights. Comodo should take those tests and figure out exactly how it can pull ahead.
Yes indeed. However, I think Comodo should be able to have enough resources to test their Antivirus component fairly comprehensively. And my guess is that their tests have revealed it’s not quite up there with the top competition just yet. However, Comodo Antivirus component seems to be detecting stuff that other top scanners are missing, so it seems they are getting fairly decent at rolling the dice and getting the right number!
I don’t find anything to be a valid reason to withhold CAV from traditional testing (the ones that you have to pay for). The best thing they could tout is the fact that they have a completely fresh scanning engine with CIS V4 coming out soon. After that comes out, it should be open season for tests.
From matousec to individuals who are running av testing sites, they all have and we have done pretty well in all of them. We even have provided testing tools ourselves! We haven’t gone out to these entities and said, pls test us… Now thats where the issue is: people want us to us av comparitives site. Now these folks do NOT do AMTSO compliant dynamic testing. And that is what you need to test CIS. to show its true strength. They claim they will do it soon, but i don’t know when and as soon as they are, we will be one of the first ones, if not the first. So the issue is really should CIS be tested with 1980s method of testing which does not test CIS AT ALL. Just like testing if you can speak english using chinese language! Thats how much sense it makes to test CIS’s true power with these static 1980s tests which are totally outdated. That is why AMTSO was created. Because new security products have different security methods and current testing methods are not capable of testing them, simple as that. We are NOT stopping people from testing CIS…just like others have tested…anyone is welcome to test it…we are not going to add credibility to any non AMTSO compliant organisation by request a test by them. (on the other hand there are some organisations we are working with for internal purposes).