Should I Be Worried? Because I AM...

[Guest]

[kail]

Good grief, no speculation is required (I think Cassette was talking about the other vendors products). They are different tests, read the Matousec web site (again) & the two different PDF reports for CIS from each of tests (32bit PDF CIS report & 64bit PDF CIS report) yourself.

PS Virus definitions have nothing to do with these Matousec tests.

[BTJustice]

Good grief, no speculation is required (I think Cassette was talking about the other vendors products). They are different tests, read the Matousec web site (again) & the two different PDF reports for CIS from each of tests (32bit PDF CIS report & 64bit PDF CIS report) yourself.

It is speculation.  You are comparing two different tests with two different versions of CIS which used two different versions of the virus database.  The only way to get a definitive answer would be to run the old test on the new CIS or run the new test on the old version of CIS used in the old test.

I am tired of this back and forth.  I simply want to see CIS start winning at the other test sites I listed earlier where it is either sucking at or not being tested at.

[kail]

It is speculation.  You are comparing two different tests with two different versions of CIS which used two different versions of the virus database.  The only way to get a definitive answer would be to run the old test on the new CIS or run the new test on the old version of CIS used in the old test.

I am tired of this back and forth.  I simply want to see CIS start winning at the other test sites I listed earlier where it is either sucking at or not being tested at.

Unbelievable. You compared the tests, not me. I said they were different. And, since you obviously still have no idea what Matousec are actually testing (hint: virus definitions & the AV component are irrelevant), you would do well to stop now.

[BTJustice]

Unbelievable. You compared the tests, not me. I said they were different. And, since you obviously still have no idea what Matousec are actually testing (hint: virus definitions & the AV component are irrelevant), you would do well to stop now.

And you said the older test was still valid even though it has been replaced.

I had to research Matousec as the first time I ever heard of it was in this thread.  As far as I can tell, they test firewalls but make more demands of firewalls that are kind of silly such as key logging.  A firewall should protect against and stop leaks.  The anti-malware part of the program should protect against and remove malware.  Yet Matousec tests with the anti-malware portion turned off.

It also appears vendors can pay Matousec to retest giving the vendors the right to either publish the results or not publish them.  This is a mafia-style extortion tactic...

"Hey your product didn't score well in out test.  If you don't want the results published, pay up and we will retest it and if the results are still not good then we won't publish any results.  This is an offer you can't refuse!"

And apparently they score the results of 10 tests across 11 levels even if they all are not run.  Take ESET for example...

http://www.matousec.com/projects/proactive-security-challenge-64/reports/PSC64%20report%20-%20ESET%20Smart%20Security%205.0.94.0.pdf

In the latest test it got 33% but only ran in 6 tests.  Take the total product percentage of 360 and divide it by 6 and it is 60%.  All tested products need to be ran across all tests.

I found a lot of criticisms against Matousec...

http://www.thetechherald.com/articles/Security-vendors-respond-to-Matousec-research/10138/

http://smokeys.wordpress.com/2008/04/20/matousecs-firewall-challenge-wrinkle-conflict-of-interests/

http://www.techsupportalert.com/content/matousec-personal-firewall-tests-analyzed.htm

http://answers.yahoo.com/question/index?qid=20101003115113AATE4G4

http://www.wilderssecurity.com/showthread.php?t=212594

http://www.wilderssecurity.com/showthread.php?p=189909

http://www.dslreports.com/forum/r20905485-firewall-tests

So in taking all of this into account, Matousec is meaningless to me.  I still want to see CIS kick butt at real tests like the VB100, AV-TEST, AV-Comparative, ICSA, and Checkmark.  It is not the responsibility of the end user to independently test security software to find the best one.

[kail]

And you said the older test was still valid even though it has been replaced. ..

Utter rubbish, where did I say that the "older test was still valid"? I said the the 32bit results/score were still valid and they are. You compared the CIS results of the different tests, I didn't. Please stop trying to pin your own misconceptions and misunderstandings on me. If I was wrong, I would say so.

.. I found a lot of criticisms against Matousec... ..

You can find a lot of criticism for just about anything on the Internet. But, I note that didn't look for, or cite, any positive views though (very telling).

However, since you obviously explicitly trust those sources that you found.. you should probably address you concerns there. After all, you're not apparently believing much of what is being said here.

[BTJustice]

Utter rubbish, where did I say that the "older test was still valid"? I said the the 32bit results/score were still valid and they are. You compared the CIS results of the different tests, I didn't. Please stop trying to pin your own misconceptions and misunderstandings on me. If I was wrong, I would say so.
You can find a lot of criticism for just about anything on the Internet. But, I note that didn't look for, or cite, any positive views though (very telling).

You said..

The 32bit test results are still valid because not all firewalls (excluding CIS) support 64bit yet.

However, since you obviously explicitly trust those sources that you found.. you should probably address you concerns there. After all, you're not apparently believing much of what is being said here.

Sorry you disagree.  But the only test where CIS is doing well is the Matousec test which has criticism against it.  CIS is sucking in the tests most vendors want to do well in.

The ONE time CIS won the VB100, COMODO was proud of the accomplishment...

http://www.comodo.com/news/press_releases/2011/04/Comodo-Internet-Security-Earns-VB100-Virus-Certification.html

[kail]

Yes, of course, you fail to prove your assertion that I said that Matousec 32bit tests were still valid* and even cite where I said "results" (underlined!) were still valid and then quickly divert to something else.

If you believe CIS is "sucking in the tests most vendors want to do well in" and that is a problem for you, then use something else. That is your choice and it should be a fairly straight forward one, given your current assertions and conclusions.

*Actually I don't know of anything that says that Matousec's 32bit tests are invalid (not valid), Matousec have said that they have been replaced. Either way, I didn't comment on the actual tests other than to say they were different when you tried to compare them.

[BTJustice]

Yes, of course, you fail to prove your assertion that I said that Matousec 32bit tests were still valid* and even cite where I said "results" (underlined!) were still valid and then quickly divert to something else.

If you believe CIS is "sucking in the tests most vendors want to do well in" and that is a problem for you, then use something else. That is your choice and it should be a fairly straight forward one, given your current assertions and conclusions.

*Actually I don't know of anything that says that Matousec's 32bit tests are invalid (not valid), Matousec have said that they have been replaced. Either way, I didn't comment on the actual tests other than to say they were different when you tried to compare them.

Damn man!  What is all of this hostility?

I am simply questioning why CIS isn't doing well in the other tests and am hoping, as a current CIS user, that it will start to shine in them.

[kail]

Hostility? That is some negative spin there.. you keep saying I said things, when I didn't and now I'm hostile? Seriously? I would normally ask for you to show me where I'm being hostile, but given your previous efforts at showing me what I said previously, I think I'll skip doing that this time.

To recap, when you said: "And in the latest test, CIS dropped to 94%" and replied "CIS has not dropped.. I believe that's CIS's initial test score".. you probably should have said "Oh yea, OK". But, you chose a different approach.. ending with "Matousec is meaningless to me".

If you are indeed "simply questioning why CIS isn't doing well in the other tests" with an aim having CIS "shine", then you are going at it in a very strange way (with added hostility & negativity I might add). Given your simple aims, I recommend a change in approach.

[BTJustice]

I am not getting back into arguing.  You did say the old results (which come for a test) are valid which I quoted.

And Matousec has a lot of criticism against its methods as I posted numerous links to.  As I said, this thread is the first time I had ever heard of Matousec.

As for your recommendation of switching to something else, I may do that, but I like CIS and simply want it to do well in the other tests which currently it is not.

So the question remains.  Why is CIS not doing well in the other tests?

[EricJH]

The way I see it is that Matousec raised the bar some but that does not make top notch products like CIS and Outpost less secure. I expect CIS to improve in the Matousec test in the future.

If you mean with other tests detection based tests. To be honest I don't care that much about those. With CIS you can stop malware from infecting your system without the use of an AV. During testing of CIS v5 egemen gave us mods a heads up on developments and told that they had tested the new sandbox against 15,000 malwares and none of them was able to infect the system.

All you want is CIS to be the top dog in detection tests and since that is not happening you start to aggressively ask for an explanation. If you want to use another AV that does well in the tests you hold high please do so.....

[BTJustice]

I just had an interesting development.  I ran a full scan earlier today and CIS detected no threats.  I had been wanting to try the latest avast! as I am using it on my Android phone so I installed it and let it run a scan while I was away.  I just got back and it found 6 threats that CIS did not.

I am not impressed with Matousec.  Come on CIS!  Pass real tests!

EDIT:  I found out some more about Matousec.  They test HIPS applications and not true firewall programs.  Security products including a firewall rely on their ant-mailware part to catch malware.  No wonder a lot of companies are doing badly in the Matousec test.  I have been lurking on the avast! forums.  Matousec is a JOKE!

Go to http://forum.avast.com/index.php and search for matousec... ENJOY!

[Dennis2]

Sorry but that is where you fail CIS is a HIPS application.

Just do a search for Hips firewall you should find quite a few entries.

As far as what you call real tests, Comodo has always been more interested in protection of their users than passing what you say are real tests.

Thank you

Dennis

[TJ888]

Not trying to butt in, but it would be nice to see if those 6 avast detection were files which you have trusted in Comodo or if they are f/p if so they don't count.

You should have scanned them with malwarebytes or something that's a strong secondary opinion scanner to see if they not just traces or temporary. If so they can't do anything.

Also, I agree with Eric, a Detection test is nothing compared to prevention. If you can have the best hips and sandbox out there and nothing can hurt your system, who cares about the detection tests if the av doesn't catch it, the sandbox, or better yet the new sandbox in 6.0 will stop it.

[BTJustice]

Sorry but that is where you fail CIS is a HIPS application.

Just do a search for Hips firewall you should find quite a few entries.

As far as what you call real tests, Comodo has always been more interested in protection of their users than passing what you say are real tests.

Thank you

Dennis

And that is where you fail.  Protection is low in the tests I mentioned.

It's puzzling to me how people have the attitude that, when they like something, that a test that says it is not good is a bad test.

[Tags:]