New Matousec Firewall Challenge

[Guest]

[TerryWood]

Hi All

Despite my previous post Melih still does not get it!!!!

If Comodos quality control had been as good as its marketing this thread would not have been necessary.

Melih wants to consort with God and the devil. The net result is that a so called "strong brand" organisation is brought down to the level of in fighting with a partner organisation. Where's the professionalism....?

I remember the "spat" with Online Armor when somebody shot his mouth off.....

Mind you if if it were not for issues like this we would not need the number of fora which are in overdrive at the moment.

Terry

[Dennis2]

Can you please edit you first post this Quote "Comodo cannot get near pole position in the Matousec tests?" End Quote as I class second as near pole.
Thank You
Dennis

[Melih]

Terry

We continually increase our spending on QA.

We also run huge Beta runs for our software as you know. Millions of users use our s/w and found it to be very stable. Bugs found by Matousec are very specific and we haven't heard this bug from millions of users we have.

You simply cannot provide a bug free software, period! So if you will come here and shout saying hey I told you you should have QA for every bug discovered, we will give you your own board!

Like you, we too are questioning Matousec! He started well with good intentions and did some decent work initially and contributed with finding new attack vectors. But lately he seems to have some questionable acts.

Did you read Gibran's post? It gives an example of many flaws in the methodology.

If you expect us to sit and take it on the chin from some testing organisation while they benefit from mis reporting on Comodo, pls think again!

Melih

[Xman]

Or we could simply let Terry write all the code involved in CIS, that would overwhelm him, nothing is perfect but striving in that direction is more than honorary
Don't be influenced and especially remember the goal at heart Melih
Regards to you and the staff
Xman

[Melih]

Terry

We continually increase our spending on QA.

We also run huge Beta runs for our software as you know. Millions of users use our s/w and found it to be very stable. Bugs found by Matousec are very specific and we haven't heard this bug from millions of users we have.

You simply cannot provide a bug free software, period! So if you will come here and shout saying hey I told you you should have QA for every bug discovered, we will give you your own board!

Like you, we too are questioning Matousec! He started well with good intentions and did some decent work initially and contributed with finding new attack vectors. But lately he seems to have some questionable acts.

Did you read Gibran's post? It gives an example of many flaws in the methodology.

If you expect us to sit and take it on the chin from some testing organisation while they benefit from mis reporting on Comodo, pls think again!

Melih

PS: Terry, your statement of "If Comodos quality control had been as good as its marketing this thread would not have been necessary" is totally incorrect. 

We are here not because of our bug, but the error that Matousec did! What grabbed our attention is not that we are 1st or 2nd (as we really do not care) but we were way below. And that was due to Matousec error which is now corrected.

[Xman]

But we didn't pay for this test in the first place!
And he made an error when reporting our results!

Why did he include CIS if we didn't pay for it?
Why is he reluctant to retest knowing that CIS will get no 1 position? He corrected his mistake on his website, so he must have retested it and modified the results without us paying, why not do it with the fixed version?

Melih

So Melih, although I believe in the case at hand this seems to be for David anyhow now a case of put your money where your mouth is unfortunately, your call...
As Nike would say, just do it!
Regards
Xman

[MagisDing]

Well, about the tests named suspend1&2, kill12 that CIS failed to pass, what do they simulate? Will the next version fix them?

[seed.helper]

Just to re-iterate the point:

We always had the security of the user in mind and CIS does have all the protection code built in to protect the user! That is our no 1 priority, always has been, always will be! We invented User Security in the world of false sense of security with AV companies giving products that does not protect end users from malware they can't detect, and we are the guys who put prevention as first line of defense into a mass product!

We will continue to protect the users and give them the best possible security solution, period!

We will NOT be guided by some leak test website for our development nor will be pushed into developing code just to pass these leak tests if we don't feel it will offer any benefit to our users!

To summarise: You get the best possible protection with CIS! You always had, you always will!

Melih

that is why i will use comodo allways and only see this tests for curiosity... dont care 

but is good to have this tests, or firewalls couldnt be compared or properly tested and this bug in comodo firewall woudnt have been discovered so quicly i think

thanks comodo team and Melih

[hullboy]

Now I'm a bit confused
Do we have to expect a program update (as egemen said at a first time) or the bug has to be considered fixed without anything to do but choosing Proactive Security? (R)

Hi Guys,

We have reviewed the results. Dont worry. There are 3 issues to be highlighted here.

1 - There is a single bug in CIS/CFP which affects the previsous versions too.
This bug allows CFP/CIS to fail to detect a special type of handle duplication operation. And thats why it cause CFP/CIS to fails the following tests:

Kill1.exe kill2.exe:  kill9.exe kill12.exe crash1.exe crash2.exe crash3.exe crash4.exe crash5.exe crash6.exe

Do not worry. We have fixed the bug and you will be receiving the update on Tuesday(2/12/2008).

2 - There are errors in the test report:

CFP/CIS does not fail the following tests: kill3b.exe kill3f.exe kill3e.exe kill5.exe SSS2.exe SSS3.exe

It is unclear why CFP is reported as failed in these tests. You might try yourself and see. Attachment contains ssts.conf file with which you can test CFP/CIS.

CFP/CIS intercepts system shutdown privilege elevation requests and hence effective blocks sss2.exe and sss3.exe tests.

CFP/CIS can not be terminated by any of those kill tests.
3 - There are some insignificant tests that do not pose any real threat and hence we will not do anything about them.

--------------
CFP/CIS is marked as failed in the following tests:   SSS.exe i.e. System Shutdown Simulation tests.
--------------
It has been scored 50% because CFP/CIS does not intercept system shutdown requests. This is the testing methodology of the tester.

System shutdown poses no real threat. The malware waits for system shutdown to perform its harmful actions. So whether you intercept or not, it can attack the user when the user manually logs out. Original System Shutdown Simulation tests do care about this fact.
So we do not plan to add this redundant protection to pass any tests.

--------------
socksnif.exe: This test is designed to test if a malware can snif your network connection or not.
--------------

If "\Device\Afd\EndPoint" is added to the Defense+ My Protected Files list, this can be easily intercepted. Adding this entry to the protected files means, making Defense+ to alert you for each and every application which tries to access Windows Sockets.

However, we do not plan to add this to our default protected files. Because

* It poses no real threat. Malware can not sniff your everyday bank transactions because everything is already SSL encrypted,
* This is basically no different than sniffing your network traffic from another computer,
* It will increase the number of popups unnecessarily,

--------------
crash7.exe: This test tries to allocate all the memory of the computer to crash applications including the security software
--------------

It might be possible for an application to crash if there is no more computer memory available. This is usually a random case. We do not plan to make any changes to pass this test because

* The crash can be random, intermittent and ubiquitous
* Assuming CFP/CIS processes also crashed, there is no real threat to the system because by terminating CFP/CIS, malware will not gain any advantage for byapssing Defense+.

So in summary: by terminating CFP/CIS, Defense+ will not be able to be bypassed.

[gibran]

Now I'm a bit confused
Do we have to expect a program update (as egemen said at a first time) or the bug has to be considered fixed without anything to do but choosing Proactive Security? (R)

The incorrect procedure used to test CIS has nothing to do with the bugfix but it was nevertheless one of the aspects triggered by the feedback posted in this topic.

As Egemen didn't post otherwise there is no reason to doubt that a new update will be released.

As the most relevant aspects have been already addressed the topic will be closed, and locked for reference.

If it needs to be reopened, please send a PM to any active moderator.

[Tags:]