CAV3 detection rate test [2008.10.05] - 10.26% [CLOSED]

[Guest]

[darcjrt]

This thread is getting veeery long, and the only thing in it that really matters is whether Comodo ads submitted samples to the database in a reasonable time or not, because anything else is unverifiable as things stand.

I'm glad to say the two samples I submitted a few days ago were added with the latest update. It took 5 to 7 days (which may be a bit too long), but the system does work. It can only get better.  

That is great news. How did you submit the samples? CIMA? Email? Comodo file Submitter?

[fazio93]

This thread is getting veeery long, and the only thing in it that really matters is whether Comodo ads submitted samples to the database in a reasonable time or not, because anything else is unverifiable as things stand.

I'm glad to say the two samples I submitted a few days ago were added with the latest update. It took 5 to 7 days (which may be a bit too long), but the system does work. It can only get better.   

I believe Melih said his goal was to get the sig into the database in less than an hour from submitting it.
 

 (V)

[Star Shadow]

Wow! So much fighting in this tread. :| My solution to this whole mess.

solcroft said he will post the proof at the end of the month that he submitted the files to Comodo, so let's all just assume that he will do just that. He will post his daily results. However, to make things fair to others, I suggest that his results are looked at, but you do not need to believe them until the end of the month: if solcroft does in fact post the proof, then then all the results are taken seriously, but if the proof is not posted, then then all the results are not believed by anyone and all the nay-sayers of solcroft will be proven right.

Soooooo let's hold off the harsh words and fighting until the end of the month. Can we wait that long? If solcroft does not post the proof, then you all can say whatever you want about him, however if he does provide proof at the end of the month, I think some people should at least apologize to him.

Is this a fair settlement to all this bickering? So, let's all calm down please.

[sded]

Perhaps to avoid driving away most of the potential CIS/CAVS3 beta users who might otherwise use the product and help to improve it, maybe we can summarize how all of this fits in, since it certainly may not be obvious to the newb.  My understanding of it:
1.  CAVS3 is a beta product, not intended to meet any particular performance criteria.  It is intended to test and incorporate feed back from the users on functionality, utility, etc.  It is also the next phase (after CAVS2, which was never released) of collection of the appropriate databases for a fully capable AV over the next 12 months or so.  And a platform to incorporate new Comodo ideas for further testing by the users.
2.  The prototype on demand scanner previously shown in CFP3 exhibited many false positives, less than satisfactory detection performance.  Similar reports occur for the CAVS3/CIS beta.  These can be dangerous to your system unless you are careful about backups-I use Acronis True Image frequently.  I repeat, THIS IS A BETA.
3.  Users of the beta need to understand that is it a beta, and that the key heuristics features (based on D+) are not yet included.  The malware data base is also quite preliminary.  Those who are comfortable with observing D+ alerts to detect malware should be comfortable with beta testing the product.  Others may want to wait for a later release and install only the CIS firewall.
4.  Comodo is working to bring the product up to excellent performance, not dependent on pressuring other organizations to give up their proprietary work products to help Comodo leapfrog their current position in the AV business.
5.  I am just another user, no connection with Comodo except to volunteer support to users of some of their products.  But I am also not a proponent of faith based security protection, and think users need to understand better what they are getting into.  And look forward to seeing data presented by both sides of the current discussions to help that understanding.

[foxman]

And what makes you think I haven't done that?

Simple, PROVE IT. Not just cheap talk.

[andyman35]

Looking at the results, I don't think so. Do you honestly believe that there are only ~700 pieces of malware in circulation for the last 2 years or so, when typical daily updates from some vendors easily reach more than twice that number?

By elevating yourself above virtually the whole of the AV industry you lose credibility.Nobody is suggesting that there are only 700 pieces of malware,but since the vast majority of malware are just variants of a limited number of unique originals,this subset is supposedly what is infecting the highest percentage of users.If you know different then please show so and I'll certainly take my hat off to you >>>>>

Secondly I can't find details of your testing methodology on skimming through the vast number of posts in this thread.Do you use a dedicated system or a VM? Are the samples dormant or running?

You do make some valid points on the efficacy of D+ being dependant upon the technical knowledge of the user,I've made the same point myself numerous times.It's been said that this will be addressed in future releases and that dumb HIPS will learn some smarts.

[gibran]

I take this chance for thanking solcroft about the wildlist articles he cited (it was a long reading)

http://www.people.frisk-software.com/~bontchev/papers/wildlist.html dating back around '00
http://www.sunbelt-software.com/ihs/alex/vb_2007_wildlist_paper.pdf
http://sunbeltblog.blogspot.com/2008/06/wildlist-battles.html

Those paper were written by AV researchers and also Wildlist org members.

The doubts about Wildlist.org's wildlist even address the sample selection requirements although they are not limited to only that:

The WildList only contains intentionally malicious software which is able to self-replicate by infecting other files (viruses) and PCs in a network environment (worms). After someinternal discussions, the WildList coverage was slightly extended to include some known bots, but only those that areable to spread by themselves, excluding the ones that fall more into the backdoor category.

Trojan, dialers(now a rare occurence), potentially unsafe apps don't meet the requirements for Wildlist inclusion. Another important element is the observation that current threats are targeted, regional, web-based and financially motivated.

Most of these articles express  doubts  about the Wildlist Org methodology and other valid concerns but the idea of a properly compiled wildlist is not rejected per se.

Since the overall focus further restricted on much specific aspects intead of a more general approach I guess this is my last post in this topic.
Even though I would like to ask everyone to moderate their tones from now on.

[Toxteth O'Grady]

That is great news. How did you submit the samples? CIMA? Email? Comodo file Submitter?

CIMA and email. Just to make sure. 

I have to correct myself. The samples may have been added anywhere between 3 to 7 days after submitting them. I forgot to check for a few days. 

[Toxteth O'Grady]

Just found two more pieces of malware embedded in websites. I scanned both files over at Jotti.

- The first file had been scanned before, and yet not all AVs detected it (including some of the big names).
http://img375.imageshack.us/my.php?image=pdfpq5.gif


- The second file had not been scanned before. Again, look at how many scanners missed it:
http://img375.imageshack.us/my.php?image=trojanos9.gif

Comodo AV also missed it, but this is what D+ did:
http://img375.imageshack.us/my.php?image=comodomf9.gif

[3xist]

Just found two more pieces of malware embedded in websites. I scanned both files over at Jotti.

- The first file had been scanned before, and yet not all scanners detected it (including some of the big names).
http://img375.imageshack.us/my.php?image=pdfpq5.gif


- The second file had not been scanned before. Again, look at how many scanners missed it:
http://img375.imageshack.us/my.php?image=trojanos9.gif

Comodo AV also missed it, but this is what D+ did:
http://img375.imageshack.us/my.php?image=comodomf9.gif

Hey Toxteth O'Grady

Nice image of D+! As you can see... Prevention should be your first line of Defense! Do you mind sending me this malware?

Thanks
Josh

[Toxteth O'Grady]

Hey Toxteth O'Grady

Nice image of D+! As you can see... Prevention should be your first line of Defense! Do you mind sending me this malware?

Thanks
Josh

The last link to uploaded malware I posted in the forum was deleted.
I'll PM a link, wait a few minutes.


Edit:
PM sent (at least, I hope it worked). Three samples.

[solcroft]

By elevating yourself above virtually the whole of the AV industry you lose credibility.

Well, that's a pretty amazing accusation, since if you actually read the links I provided you'll find out it's the AV industry who's pointing out that the WildList is obsolete. You make it sound like as though the industry is singing praises about the WildList, but I know better than them.

Secondly I can't find details of your testing methodology on skimming through the vast number of posts in this thread.Do you use a dedicated system or a VM? Are the samples dormant or running?

Live system, samples are not executed. Your first question makes no difference to the results, and the second is kind of a no-brainer.

Most of these articles express  doubts  about the Wildlist Org methodology and other valid concerns but the idea of a properly compiled wildlist is not rejected per se.

As I previously said, it's the results the WildList produces that are out of touch with reality. The intent is good, but to properly carry out that intent with any measure of effectiveness will most probably require a complete overhaul of the methodology.

As you can see... Prevention should be your first line of Defense!

So why is D+ using detection instead of prevention in that image instead?

[3xist]

So why is D+ using detection instead of prevention in that image instead?

Because there are more technologies then you think in D+. It's not a classical HIPS, An Above average developer can make a classical HIPS in 2 weeks, D+ took time. "Detecting" something based on prevention rule-based tech, and other tech in D+ is normal behavior.

Josh

[Kyle]

D+ has hueristics that help you decide whether it is bad or not. They are rarely wrong +
That was a firewall alert

[3xist]

D+ has hueristics that

That's one technology.

By the way solcroft, You can read CFP 3's benefits too.

Josh

[Tags:]