New Matousec Firewall Challenge

[Guest]

[Kyle]

Hey, I thought that Comodo weren't going to participate in those tests because some of the tests were irelevant in real world situations, Has Melih changed his mind or as Matousec changed his methods?

[Goose19]

Well seeing how alot of people read these "tests" and decide what they will use by these tests only I could see why Comodo would like to be #1 or the top 3 like they have been for so long. 

[3xist]

There is a reason... But 

Josh

[MagisDing]

There is a reason... But 

Josh

what reason?

[hullboy]

Hey, I thought that Comodo weren't going to participate in those tests because some of the tests were irelevant in real world situations, Has Melih changed his mind or as Matousec changed his methods?

As we have seen, participating in this kind of tests is very useful otherwise the bug wouldn't have come out (B)

[TerryWood]

Hi All

Being a long time user of Comodo Firewall, and supporter of Comodo, it is both disappointing and disturbing that Comodo is well down the lists in the latest Matousec challenge.

Some of the small organisations appear to be running rings round Comodo, in that their submissions appear to be fully checked for workability and compliance.

We have the strange situation where Comodo is developing a suite of leak tests that are supposedly onerous and yet Comodo cannot get near pole position in the Matousec tests? A bug we are told is the problem (pretty quickly after Matousecs tests were released). Its a pity that this was not determined before the test.

If the Comodo brand (Yes it is a brand, and potentially a strong one at that) is to mean anything to the loyal users who use it and create the brand strength, then the brand has to have credibility. I am surprised that Melih is not spitting blood just as Comodo's competitors will be laughing up their sleeves.

It really is not good enough, it is sloppiness of a high order, and does not bode well for Comodos commercial aspirations.

Yes I am sure there will be plenty of forgiveness about in this forum for those closer to the Comodo organisation. I am not sure that is what is needed though. 

Terry

[nick55555]

The latest Firewall Challenge tests on matousec.com are a bit disappointing. CIS fails on many crash and termination tests...   

[Kyle]

The latest Firewall Challenge tests on matousec.com are a bit disappointing. CIS fails on many crash and termination tests...   

Hey Nick, Please do a little bit of reading on this thread - it will explain that there was a bug.

[Star Shadow]

Hi All

Being a long time user of Comodo Firewall, and supporter of Comodo, it is both disappointing and disturbing that Comodo is well down the lists in the latest Matousec challenge.

Some of the small organisations appear to be running rings round Comodo, in that their submissions appear to be fully checked for workability and compliance.

We have the strange situation where Comodo is developing a suite of leak tests that are supposedly onerous and yet Comodo cannot get near pole position in the Matousec tests? A bug we are told is the problem (pretty quickly after Matousecs tests were released). Its a pity that this was not determined before the test.

If the Comodo brand (Yes it is a brand, and potentially a strong one at that) is to mean anything to the loyal users who use it and create the brand strength, then the brand has to have credibility. I am surprised that Melih is not spitting blood just as Comodo's competitors will be laughing up their sleeves.

It really is not good enough, it is sloppiness of a high order, and does not bode well for Comodos commercial aspirations.

Yes I am sure there will be plenty of forgiveness about in this forum for those closer to the Comodo organisation. I am not sure that is what is needed though. 

Terry

There are more tests in the Matousec tests. Some of them actually can damage the system if I read it correctly. So, I don't think people want to use those tests.

[larsson]

I notice that this test is not a complete new test. Online Armor´s free FW result is dated 15 of May and thus made with previous test setup...

I think this test should be done with the same test setup as ranking otherwise might not be correct

[egemen]

Hi Guys,

We have reviewed the results. Dont worry. There are 3 issues to be highlighted here.

1 - There is a single bug in CIS/CFP which affects the previsous versions too.
This bug allows CFP/CIS to fail to detect a special type of handle duplication operation. And thats why it cause CFP/CIS to fails the following tests:

Kill1.exe kill2.exe:  kill9.exe kill12.exe crash1.exe crash2.exe crash3.exe crash4.exe crash5.exe crash6.exe

Do not worry. We have fixed the bug and you will be receiving the update on Tuesday(2/12/2008).

2 - There are errors in the test report:

CFP/CIS does not fail the following tests: kill3b.exe kill3f.exe kill3e.exe kill5.exe SSS2.exe SSS3.exe

It is unclear why CFP is reported as failed in these tests. You might try yourself and see. Attachment contains ssts.conf file with which you can test CFP/CIS.

CFP/CIS intercepts system shutdown privilege elevation requests and hence effective blocks sss2.exe and sss3.exe tests.

CFP/CIS can not be terminated by any of those kill tests.
3 - There are some insignificant tests that do not pose any real threat and hence we will not do anything about them.

--------------
CFP/CIS is marked as failed in the following tests:   SSS.exe i.e. System Shutdown Simulation tests.
--------------
It has been scored 50% because CFP/CIS does not intercept system shutdown requests. This is the testing methodology of the tester.

System shutdown poses no real threat. The malware waits for system shutdown to perform its harmful actions. So whether you intercept or not, it can attack the user when the user manually logs out. Original System Shutdown Simulation tests do care about this fact.
So we do not plan to add this redundant protection to pass any tests.

--------------
socksnif.exe: This test is designed to test if a malware can snif your network connection or not.
--------------

If "\Device\Afd\EndPoint" is added to the Defense+ My Protected Files list, this can be easily intercepted. Adding this entry to the protected files means, making Defense+ to alert you for each and every application which tries to access Windows Sockets.

However, we do not plan to add this to our default protected files. Because

* It poses no real threat. Malware can not sniff your everyday bank transactions because everything is already SSL encrypted,
* This is basically no different than sniffing your network traffic from another computer,
* It will increase the number of popups unnecessarily,

--------------
crash7.exe: This test tries to allocate all the memory of the computer to crash applications including the security software
--------------

It might be possible for an application to crash if there is no more computer memory available. This is usually a random case. We do not plan to make any changes to pass this test because

* The crash can be random, intermittent and ubiquitous
* Assuming CFP/CIS processes also crashed, there is no real threat to the system because by terminating CFP/CIS, malware will not gain any advantage for byapssing Defense+.

So in summary: by terminating CFP/CIS, Defense+ will not be able to be bypassed.

[lordraiden]

You can reply exactly the text of your post to Matousec for improve the Suite Test and the methology

[Melih]

We hope David/Matousec will retest and update his results with our new version with the bugs fixed.

It would be very suspicious if he chose not to retest the bug fixed version, knowing that he made an error on his tests. It would show a goodwill from his side, we'll see if he does that or not...

lets watch out..


Melih

[Melih]

Just to re-iterate the point:

We always had the security of the user in mind and CIS does have all the protection code built in to protect the user! That is our no 1 priority, always has been, always will be! We invented User Security in the world of false sense of security with AV companies giving products that does not protect end users from malware they can't detect, and we are the guys who put prevention as first line of defense into a mass product!

We will continue to protect the users and give them the best possible security solution, period!

We will NOT be guided by some leak test website for our development nor will be pushed into developing code just to pass these leak tests if we don't feel it will offer any benefit to our users!

To summarise: You get the best possible protection with CIS! You always had, you always will!

Melih

[Star Shadow]

Melih, thank you for keeping us informed as to what is going on. I know you don't let testing sites guide your development, but I was wondering if you have been in contact with David/Matousec and team to see why they labeled failures when you said that CFP does not fail those tests. I think it is important for testing sites to at least have accurate results posted, so it shows that they are not unbiased. Even if they do not retest the bug fix version and amend the results for the version they did test, it would at least put Comodo higher in the list, and thus users that do not understand anything about testing sites will see Comodo as a better product, and for the vendor comments section, you can post that there is a big fixed version and also do your own testing with the Matousec test suite and state what the score is with the bug fixed version for the comment section as well. That would show other people how serious you are about security. We all know as forum members, but other people that do not visit these forums do not know, so that comment section is the place where you really need to show it. I know a lot of people that base everything on testing sites, so the comments are very important. I think egemen's post is really good and would benefit a lot of people being posted on the Matousec site since it shows how fast bugs are taken care of and how committed everyone here is.

Keep up the great work all.

[Tags:]