Service to human race or fame seeking selfishness?

[Guest]

[Melih]

The way the AV Test's are done: Are they really of benefit to users?
Click to read and please let me have your say on this.

thanks

[John Buchanan]

That is a good point, Melih.  Well spoken and directed towards the people who can help the most - those with viruses unknown to the security companies.

[Melih]

That is a good point, Melih.  Well spoken and directed towards the people who can help the most - those with viruses unknown to the security companies.

Indeed.

They should release all malware to all AV companies asap (if they have them, as noone has been able to confirm that they do have what they say but lets assume they do).

And then test the "Capability" of the AV!

Capability could be: Types of malware caught, speed of the AV on user's machine, speed of signature creation and so on. these are the things that could differentiate AV products.

Melih

[Star Shadow]

This is a big what if ... what if the AV companies do have the same samples as the testers already, but the product's scanning engine is incapable of detecting it for some reason or another? It's one thing to have the sig of a virus, but it's another thing to detect it buried deep in an executable. There is a lot of malware out there, but I am sure that AV companies have a good sampling of them, and I am sure that most AV testers would have all the samples they can find, and if they can find it then a large AV company with thousands of users would have said malware submitted to them.

But, you bring up good points. The quality of the AV product is how fast it can run and how many baddies it can find. Since there are a lot of malware out there that is unknown, AVs need to be really smart about how an application is behaving. It needs to have some form of HIPS and some other mechanisms that know what is bad and what is good. Since not all the malware out there is known.

Then again, maybe some companies have a smaller set of malware to work with, but I think some just don't have the capability of detecting the malware. So, to fully see if this is the case, one needs to compare the list of malware sigs in each AV program to each other to see how they differ and then those lists need to be compared to the list the AV testers use. Only then will we see who has the greater list of malware and we can see if the tester uses malware that the company does not have. However, if the tester is using malware that is in each AV program's database, then it really is a true test of how well the program can actually detect what it knows is to be bad.

Cheers.

[Xman]

Hi Melih, BTW really nice work on CIS, I still believe the Whitelist & block all others pending verification is the way to go especially if you reintegrate Threatcast to CIS, you will facilitate this for yourselves & all the users' of CIS because strength in numbers is a real fact of life and you already have THE Firewall everybody else wished they could have developed but couldn't, so in short reintroduce Threatcast along with heuristics in CIS & fly high forever, always remember your set goal of prevention over detection...
Cheers to you and the guys

Xman

[gibran]

I plenty agree.

I would like to add also one joint sample gathering organization
There will be obviously difference in AV engines but having all brands excange new samples will improve the current situaltion a lot and spare us the need to use online multi AV scan sites 

If possible I wish Comodo to Launch some AV alliance Consortium and actively promote this joint-operation to the fullest. (L)

[Xman]

Hi Gibran, agreed and stupid to say with Rising in China which I had been using till CIS release & miss somehow since 30.6% of all malware created originates from China, www.threatexpert.com
Would be a great collaboration IMO!    PS: Never caught an infection during its' use in last 4 months prior to currrent roadtest of CIS now and with CIS still not infected to date, I do however sorely miss some of the great features in Risings' AV, like time remainng to scan, a progress bar, custom scan configurations, flexible virus definitions update scheduling, etc...
Cheers
Regards
Xman 

[Melih]

Hi Melih, BTW really nice work on CIS, I still believe the Whitelist & block all others pending verification is the way to go especially if you reintegrate Threatcast to CIS, you will facilitate this for yourselves & all the users' of CIS because strength in numbers is a real fact of life and you already have THE Firewall everybody else wished they could have developed but couldn't, so in short reintroduce Threatcast along with heuristics in CIS & fly high forever, always remember your set goal of prevention over detection...
Cheers to you and the guys

Xman

Thanks Xman...yep.. thats the next CIS version! Threatcast plus few other goodies watch this space..

Melih

[Xman]

Hi Melih!, good stuff! read my prior post moments ago to Gibran, I think it's important...
Xman & cheers
 

[Xman]

Thanks Xman...yep.. thats the next CIS version! Threatcast plus few other goodies watch this space..

Melih

Would Dec 2008-Jan 2009 be optimistic?
Regards
Xman

[3xist]

Would Dec 2008-Jan 2009 be optimistic?
Regards
Xman

Yep.

Josh

[3xist]

I 100% Agree with your Blog, Melih.

Selfishness of AV Testers is a big problem.

Poll Made: AV Testing: Service or Selfishness?

Josh

[Star Shadow]

Melih,

You have a security testing site for firewalls and HIPS (http://www.testmypcsecurity.com/). The name of that address doesn't specifically say firewall, so are you going to give thought to adding a AV section to that site since Comodo now has a a great AV technology that will become great?

My thoughts are, like I mentioned earlier in this thread, that if all the AVs are tested against known malware and their malware databases are compared to others to see if they contain the same items, then it is a fair test. The AV part of the site would be just like the Firewall part and have all tests freely available so that all AV companies, if they decide to, can download the samples and improve upon their own product. The site would have all known baddies for the samples in multiple forms, like some embedded in pictures and some embedded in exes and so forth. This would provide fairness.

Some things the results needs to show for the AV test are as follows: Speed of the scan, processor and memory usage, detection rate, a list of all baddies in the test that are not in the sigs for that AV, the percentage of how many baddies are found that are in the AVs database -- in other words, how good the engine is to pick up a baddie that it is suppose to know about. Maybe some other infos that I thought of. The engine speed and quality tests are probably the most important, because if the engine is really slow and only picks up a few baddies, then the AV is useless, but if the Engine is really fast and it actually detects all the baddies that it has definitions for, then that is a great AV, so a ratio of speed to known baddie detection rate is good way to see the quality of the product. If the know baddie list is small, but it detects 100% of the malware, then that company should be notified and they can crap the complete known baddie file from the site and add the definitions.

This would help other AVs, and your competition, but on the other hand, people tend to be stuck on certain products and will not budge from using them no matter what anyone says, so if the companies play ball and get the data from your site, then that helps out everyone, thus making the web safer for everyone.

Also, the effectiveness of complete security suites should be listed as well, like with the individual AVs and Firewalls, so then people can see how much resources some suites use and how well they protect you as a whole. Matousec doesn't do suites and that is something that should be done since I have read comments from some vendors saying that their product works best as a suite. This would test their claim. Test my PC security should be all product ranges. It would be the ultimate one stop place.

Is this an idea?

[Melih]

Yes indeed.
We have been thinking about how to test if a product "protect" you or not. Its not about detection its about "protection". What we want to do is to protect ourselves from baddies and we don't really care how its done as long as its done. Hence there are different types of methods that we should be able to test. Lets see..
Melih

[John Buchanan]

Melih,

You have a security testing site for firewalls and HIPS (http://www.testmypcsecurity.com/).

I went to that site, read the pages, and attempted to DL the all_tests.zip file.  Both CIS and Avast (seperately and on different parts) claim it contains harmful viruses (CIS wanted to block the file, Avast asked to terminate the connection).
I am asking to confirm these are actually clean and not false alarms.  Thank you.

[Tags:]