The way the AV Test’s are done: Are they really of benefit to users?
Click to read and please let me have your say on this.
thanks
That is a good point, Melih. Well spoken and directed towards the people who can help the most - those with viruses unknown to the security companies.
Indeed.
They should release all malware to all AV companies asap (if they have them, as noone has been able to confirm that they do have what they say but lets assume they do).
And then test the “Capability” of the AV!
Capability could be: Types of malware caught, speed of the AV on user’s machine, speed of signature creation and so on. these are the things that could differentiate AV products.
Melih
This is a big what if … what if the AV companies do have the same samples as the testers already, but the product’s scanning engine is incapable of detecting it for some reason or another? It’s one thing to have the sig of a virus, but it’s another thing to detect it buried deep in an executable. There is a lot of malware out there, but I am sure that AV companies have a good sampling of them, and I am sure that most AV testers would have all the samples they can find, and if they can find it then a large AV company with thousands of users would have said malware submitted to them.
But, you bring up good points. The quality of the AV product is how fast it can run and how many baddies it can find. Since there are a lot of malware out there that is unknown, AVs need to be really smart about how an application is behaving. It needs to have some form of HIPS and some other mechanisms that know what is bad and what is good. Since not all the malware out there is known.
Then again, maybe some companies have a smaller set of malware to work with, but I think some just don’t have the capability of detecting the malware. So, to fully see if this is the case, one needs to compare the list of malware sigs in each AV program to each other to see how they differ and then those lists need to be compared to the list the AV testers use. Only then will we see who has the greater list of malware and we can see if the tester uses malware that the company does not have. However, if the tester is using malware that is in each AV program’s database, then it really is a true test of how well the program can actually detect what it knows is to be bad. 
Cheers.
Hi Melih, BTW really nice work on CIS, I still believe the Whitelist & block all others pending verification is the way to go especially if you reintegrate Threatcast to CIS, you will facilitate this for yourselves & all the users’ of CIS because strength in numbers is a real fact of life and you already have THE Firewall everybody else wished they could have developed but couldn’t, so in short reintroduce Threatcast along with heuristics in CIS & fly high forever, always remember your set goal of prevention over detection…
Cheers to you and the guys :■■■■ :BNC :-TU :-TU :-TU
Xman (:KWL)
I plenty agree.
If possible I wish Comodo to Launch some AV alliance Consortium and actively promote this joint-operation to the fullest. (L)
Hi Gibran, agreed and stupid to say with Rising in China which I had been using till CIS release & miss somehow since 30.6% of all malware created originates from China, www.threatexpert.com
Would be a great collaboration IMO! :-TU PS: Never caught an infection during its’ use in last 4 months prior to currrent roadtest of CIS now and with CIS still not infected to date, I do however sorely miss some of the great features in Risings’ AV, like time remainng to scan, a progress bar, custom scan configurations, flexible virus definitions update scheduling, etc…
Cheers :■■■■
Regards
Xman (:KWL)
Thanks Xman…yep… thats the next CIS version! 
Threatcast plus few other goodies watch this space…
Melih
Hi Melih!, good stuff! read my prior post moments ago to Gibran, I think it’s important…
Xman & cheers :■■■■
(:KWL)
Would Dec 2008-Jan 2009 be optimistic?
Regards
Xman :■■■■ (:KWL)
Yep.
Josh
I 100% Agree with your Blog, Melih.
Selfishness of AV Testers is a big problem.
Poll Made: AV Testing: Service or Selfishness?
Josh
Melih,
You have a security testing site for firewalls and HIPS (http://www.testmypcsecurity.com/). The name of that address doesn’t specifically say firewall, so are you going to give thought to adding a AV section to that site since Comodo now has a a great AV technology that will become great?
My thoughts are, like I mentioned earlier in this thread, that if all the AVs are tested against known malware and their malware databases are compared to others to see if they contain the same items, then it is a fair test. The AV part of the site would be just like the Firewall part and have all tests freely available so that all AV companies, if they decide to, can download the samples and improve upon their own product. The site would have all known baddies for the samples in multiple forms, like some embedded in pictures and some embedded in exes and so forth. This would provide fairness.
Some things the results needs to show for the AV test are as follows: Speed of the scan, processor and memory usage, detection rate, a list of all baddies in the test that are not in the sigs for that AV, the percentage of how many baddies are found that are in the AVs database – in other words, how good the engine is to pick up a baddie that it is suppose to know about. Maybe some other infos that I thought of. The engine speed and quality tests are probably the most important, because if the engine is really slow and only picks up a few baddies, then the AV is useless, but if the Engine is really fast and it actually detects all the baddies that it has definitions for, then that is a great AV, so a ratio of speed to known baddie detection rate is good way to see the quality of the product. If the know baddie list is small, but it detects 100% of the malware, then that company should be notified and they can ■■■■ the complete known baddie file from the site and add the definitions.
This would help other AVs, and your competition, but on the other hand, people tend to be stuck on certain products and will not budge from using them no matter what anyone says, so if the companies play ball and get the data from your site, then that helps out everyone, thus making the web safer for everyone.
Also, the effectiveness of complete security suites should be listed as well, like with the individual AVs and Firewalls, so then people can see how much resources some suites use and how well they protect you as a whole. Matousec doesn’t do suites and that is something that should be done since I have read comments from some vendors saying that their product works best as a suite. This would test their claim. Test my PC security should be all product ranges. It would be the ultimate one stop place.
Is this an idea?
Yes indeed.
We have been thinking about how to test if a product “protect” you or not. Its not about detection its about “protection”. What we want to do is to protect ourselves from baddies and we don’t really care how its done as long as its done. Hence there are different types of methods that we should be able to test. Lets see…
Melih
I went to that site, read the pages, and attempted to DL the all_tests.zip file. Both CIS and Avast (seperately and on different parts) claim it contains harmful viruses (CIS wanted to block the file, Avast asked to terminate the connection).
I am asking to confirm these are actually clean and not false alarms. Thank you.
Pleasant dream. Why not start it yourself (it is your idea)?
The problem is that almost all the anti virus companies out there are commercial and do have a lot of paying customers… There is a huge market for solutions against malware and it is expanding, growing, becoming bigger and bigger since it started 25 years ago! So there is competition!
Because of this competition av vendors need a good reputation, av testing companies do give this opportunity for av vendors to keep a good reputation…
So the idea of sharing malware/selling malware to each other (maybe also real time) is not of this world… because it does injustice to the competition/ the existing market we know today!
Melih, in order to achieve your goal, the system you really want to have (read: real time malware sharing), there has to be created a new kind of market!!! A new business model!!! A totally new system! and all the existing vendors must step in it!!!
This means: the destruction or partly destruction of 25 year old eco system!
Do you see this happening ??? I do not see it happen at the moment… But who knows what the future brings…
As for me, I am a supporter of this new system you want to implement and when it starts it will shake the hell out of the current system… (:WIN)
I mostly agree with Fake vegeta :-TU Some of my points of view without facts:
-
AV Testing organisations like any other commercial organisations (I suppose most of them are commercial) strive for profit (the bigger the better). So if they would share all samples they have that might break their business due to wich they earn money. To “adopt new and better ways to serve the users” they are required either to drop their business and profit (…is that possible without external force? …like government etc.) or change their business model (which requires hard work to be done and success is not guaranteed).
-
how do av vendors benefit from sharing their db’s contents with other av vendors in terms of gaining more profit ? It seems they won’t win much (at least they might think so), because if they do they would have done this already. If they won’t gain extra profit why should they bother?
In most cases “end users” and their safety is a third-rate subject when we talk about big business and its profits. It seems the driving force of most av/firewall/…/ vendors to enhance their products is not a concern about “end users” but a desire to increase sales (marketshare) by surpassing competitors… and their products.
Not many people out there like Melih who can develop different business model that really takes care about end users (OK, providing HIPS and outbound protection that don’t leak for free IS a concern about end users).
IMO…
The last two posters do bring up good points, but there is counter to what they say. It doesn’t matter if an AV has every single possible piece of baddie out there in their DB, what matters is how well the AV engine can find and clean the infection out of files. It’s a whole other ballgame in trying to just detect the baddie to begin with. Then it’s another to actually remove the infection from an important system file. So, even if all the AV companies shared their DBs, it would still not really hurt them if they have a better and faster scan engine that is cable of detecting the the malware and also able to actually remove it. Users will buy products based on those stats alone: Detection and removal. They don’t care how many samples are in the DB. So, an AV company has nothing to lose if they are better at detection than the competition. Having a shared DB will only make AV companies work harder to improve their detection and healing capabilities, and also make their programs smaller and faster than others.
Thoughts to chew on for awhile.
Even when companies have the samples, they don’t always achieve 100% detection on the samples they have.
Besides, I find this whole thread quite pointless. Testing organizations DO share samples with vendors that manage to meet their criteria - that’s a known fact, and this has been going on for years. In fact, it’s even a two-way process where testers sometimes accept submissions from different vendors, and then redistribute those samples back among the vendors that don’t have them. What Melih is trying to do is simply to pressure testing organizations into handing samples over to Comodo even though CAV does NOT meet those minimum criteria, by creating negative PR for the testers and deliberately encouraging the misconception that testers are elitist and do not share samples at all.