Matousec Firewall Test Results - 2008


There is no doubt that you run a very competent operation. Otherwise how would Comodo have achieved the level of excellence that it has!

There is an element of volatilty in the organisation (lets not pretend, you) BUt this is what you get with entrpreneurs/geniuses.

No I don’t see your point of view. You, in my opinion misjudge many of your users and fans, but, I accept that you cannot satisfy all of the people all of the time.

My point is, Comodo has such a wealth of good willl, because of where it has come from and what it has achieved that, in my opinion there is no need to “bleed” in public in the way that you are doing. You are good at what you do. If you foul up, you will put it right, why give succour to to the minority and to an incompetent testing organisation. Outbursts such as have been displayed by you have given support to the opposition.

Look at the forums in On Line Armor if you don’t believe me. SILENCE IS GOLDEN.

I cannot and never will understand the logic of your response. But I recognise the strength of the Comodo Brand and the “trust” built up on it. Do not sully it by injudicious comments and responding to charlatans masquerading as test sites.

This new found friendship with “David” of Matousec is bewildering after what has happened.

Thank you for allowing my comments AND don’t underestimate the strength of the Comodo Brand. The competion ought to be way behind you…?



You are very wise and a gentleman!

Thank you for your comments indeed.


I’m not about to sink into this lake of drama, however, isn’t the Firewall Challenge exactly what it’s supposed to be - a test about how sturdy a firewall is built? And aren’t they offering to do paid testing? There’s nothing illegal here. The results of that test mirror quirks more or less serious in these products. How relevant are they to the average user? I don’t care. They’re just some test site, and Comodo wants to pay them for a retest after patching things up. Charlatans? maybe, maybe not. But their testing suite is open-source you can tell for yourselves. And if they are, what is Comodo really worth being lectured for? It’s just a test, and they wanted to do it.

Reguarding this
Look at the forums in On Line Armor if you don’t believe me. SILENCE IS GOLDEN.
If you tried to post what you have just posted above on the forums in On Line Armor it would not be there very long.
Please read all the post in this thread and the posts in Firewall Board threads that is why SILENCE IS GOLDEN
This forum is open to all comments good or bad.
Thanks to Melih

very good point Dennis! thank you!


I actually commend Comodo for funding a public testing site. It is very risky to allow one person or organization to gain a monopoly over testing. In testing laboratories (material testing like strength, corrosion resistance, etc) there is one company that approves you for testing. They are basically known as a mafia because they can adjust their prices to whatever they want, come to your company and demand to see things unrelated to their audit, or impose certain rules on you just because they feel like it. If you object, then they don’t approve you and you are almost out of business.

If you are looking for firewall reviews other than those in PC magazines, Matousec is basically your only source. For anti-viruses, you have at least 3 or 4. The audience that puts in the effort to find these types of reviews is probably the same audience as Comodo’s. Since there is only one tester, Melih was compelled to pay for a re-test and support a potential trend of vendors paying for re-tests thereby increasing costs. It’s worse if you find something wrong with Matousec’s concept of “re-testing”.

Funding a public firewall testing site is a way to prevent this situation. Whether testmypcsecurity succeeds depends upon transparency and marketing I suppose. At the same time, it is a way for people interested in testing firewalls but with no capital to support a website to gather and work together. This alone could give rise an individual or group that could go off and create the next Matousec.

I agree with you in some part, but at it us You who does the tests or any other users, not Comodo. Comodo is only hosting it, I don’t see why people think it could be biased because of this. If somebody disagrees with the results they are welcome to do the tests and post their results, or to discuss it with other testers.

Hi Melih

You have now blocked two posts of mine, three if this one does not get through


eh? how did i do that?


So much for that theory. :wink:

Only kidding. terry, please check your PM’s.

Ewen :slight_smile:

what we’re trying to do in our testing team is to post the best results we can and minimize errors the best we can to provide to users results with the less errors possible.
we’re not here to show that comodo is the best firewall but to inform customers about products the best we can.
now if people dont trust our work, and pretend we post false results cause we’re just comodo fans, they insult the work we do as it took me like a week to test completly OA 95, jetico 2 and comodo 3(not the last build).
first u need to learn what the exploit is trying to do, how the firewall is reacting to exploits, then u need to retest your results where u’re not sure of u, and u got to look at any rules then be sure u deleted all rules that affect the exploit to retest it. u need other testers to alert u about an error u made with some exploit, like running comodo in clean mode when u’re testing an exploit so u got no alert and post wrong result.
so to release a good test about a firewall it takes time, and testers of comodo are not paid to do that.
so when i see tests posted fast on some sites as we find different results cause we take more time to test a product to be sure of our results, i wonder if it’s not time to stop believing results eyes closed and wake up to ask right questions to yourselves. where is the interest for comodo to try to shout an other firewall team by lying on our tests as comodo is a freeware? there’s no money interest, people got to trust me when i say that comodo testing group just want to give customers the right infos for any product. we’re not protected against mistake, it can happen to anyone but those last days, we threw a paving in the pond and all would be settle if people concerned admitted their errors instead of trying to drown the fish.
it seems the story is not over yet as we disagree again on some points.
so i think we should open a topic on any prob we face with some other testing group to let people react about it, that way we’ll have the thoughts of people that dont belong to any team, so comodo testing team impartiality about their testing work will not be put in doubt, and our work got his place now. i’m proud to be part of this group and the job that is done. now there’s an alternative for users, as firewalls need to be tested by others with the last events of these days. i invit people to join the site to post their own results about their firewalls or they can disagree our results with some real prooves and not just a blablabla to minimize the level of comodo testing team in ability to bring serious results when we test a product.
there are facts now, so our work cannot be ignored anymore. we got members that know what they’re talking about, and when they alert me on some problem they detected with a test on a firewall, i know that i can trust them as they got a serious knowledge about security, way better than me.
so before taking this team as a some little team made of incompetent members in term of security, some people should think again about the credibility of our testing group.
i don’t say that we’re the best team in the world but we will no more inform others about our work to help them as we see how it is now, and reactions we received that we dont share about how a test should be done, like bugs found are not same thing as security failure if the firewall cant protect the user in some test.
except that a bug can be so important that u’re not safe at all.
imagine your pc is running 24h/24, when u sleep, u got an alert but u’re not here to set the rule so the attack runs on your pc cause of a bug that doesnt block the alert if there’s no answer from you. so your machine is not protected at all if u’re not in front of your screen 24h/24.
if u answer to the alert, the attack is blocked so the firewall is able to protect u in this case.
so as this problem is a bug, and when u’re testing it if u answer to all alerts, the firewall can block the attacks, the fact that if u’re not in front of your screen when u got the alert so after maybe less than 1 minute your firewall lets the attack run, for some testing people this bug is not integrated in their results as they’re not here to find bugs but launch exploits to try to bypass the firewall protection.
when u read that, u seriously wonder if the man is not taking u for an idiot or if he’s trying to find a way to escape from a real failure that would have a big impact on his reputation.
instead to admit some facts, he found an answer to skirt around the problem.
so he says that the problem doesnt affect the product in most of tests.
so u list all the tests that affect the product and send this to him. at this time, what he said about the impact on tests became just totaly false.
cause, yes mister, we tested all again to know which test is affected or not. and 8 more tests are affected as for u, when we read your results, there’s no problem and the product passed most of tests.
but, ok, now your new tests made with updated product show that the team fixed this prob, what is a good new, but a member of comodo found a new prob with some test showing u can bypass the product.
but for u as the product pass the test in most of cases, the result is 100%.
how can it be 100% if the product pass “most” of tests?
100% means ALL of tests.
so i wonder about the credibility of your method tests like said a member of our team that found this prob.
anyway, i already imagine the answers about the fact that we’re not impartial about our tests and our credibility.
or that we’re just trying to looking for troubles as we’re just doing our job in a logical way.

If you say that the problem with NtcreatePort and the Kill5 method are not a threat to the user then why the problems were fixed rather than simply stating that they are not a problem? That tells me either that they were actual problems and comodo is giving excuses acting as a victim or that CFP development is driven by marketing. I don’t think that protecting users from “marketing hype” is the reason (BTW if the user is not smart enough to check the vendor comments then choosing the “wrong” software is the smallest of his problems).

Anyway I don’t know why there is so much commotion because a product was retested and got 2% more than comodo.

Thanks horrified, but I can make a big enough git of myself without your help.

You said

as Panic puts it, the ugliness of Melih that I object to
and then immediately retracted it.

If you are referring to a statement I made in a public post, then I’d rather you simply include a link to my statement. If you are referring to anything I’ve said in a PM (and I can’t find one in the last three months) then you should have both the nuts and the courtesy to tell me that you were going to quote off-forum data before doing so.

I imagine you are now going to bleat

"Ok, that was a bit twisty. Sorry in advance, Panic"

If you were going to immediately retract your statement before issue was taken with it, then why in God’s name bother to make the statement in the first place.

As I said previously, I make enough mistakes here without anyone else’s assistance. I do not need you to make false quotations and attribute them to me.

I hang around here to help.

Why the #&$% are you here?

Please, either accurately QUOTE me, link to me and do nothing at all.

Thank you and hopefully goodbye.

Ewen :slight_smile:

P.S. check your PM

I agree on this. That’s why I linked your profile posts in my previous reply. All member should acknowledge your contribution to this forum and form an opinion about that if they have time to spare.

As I don’t consider Wilders a reference site for Moderation Best Practices there is only one thing that coud be inferred from that. Wilders policies are way more strict than the ones we have here. That’s why we attempt attempt to create a dialogue with misbehaving members and limit the censor to spamming posts or foul words. What is staggering is to find someone that is able to neglect even a loose policy like this.

I expected that much from you. Evading those tenuous argument and mentioning misinformation. It’s easy to carve words without a context as many reader will not bother to track down what I wrote.

Do you think that leveraging on a generic sentiment of fear to express an opinion can be regarded as Misinformation, don’t you?

I don’t have to believe. I’m not a believer.
If you even bothered to consider those tenuous points I made in my previous post you got already a reply.
I’ll post a new one here for those who are genuinely interested about this topic.

Having a bunch of test results means nothing without reading about the methodologies followed to take these tests.
It would be the same thing for an individual expressing himself using a series of arbitrary postulates.
However when a suite of tests is public and the same holds for the test methodologies everyone can check the results or spot methodological weakness.

Everyone has to be responsible about his information or they’ll have no way to tell the difference from a tenuous argument and a solid one.
Having a public test suite and a public set of methods and letting everyone join (like testmypcsecurity) make even more easy to improve and find flaws.
No one is granted to be infallible and honest questioning is always welcomed.

As is way too easy to spot deceitful questioning, even such contribution are allowed here.

Anyway as long you follow the forum policies and respect personal boundaries enjoy your stay.

…that’s all? I’m surprised none of you have commented on this yet Personally I don’t care one bit wether it was free or paid. Good job Comodo, and thumbs up.

Thank you for that Searinox.

however, it was a quick job for us to do that (around 40 min to add that the code) and it was done purely for marketing gains (:SAD), because we were put in that position. It was first and last time we will do such a thing and we don’t like doing things like that. We care about building good and solid security products. Good people will appreciate and spread the word!

We have architected our software from ground up to be secure rather than trying to bolt on security to provide leak protection afterwards!



Melih don’t think of it as a marketing gain too much okay? Afterall it WAS a flaw! 40 mins to patch up a security flaw in the firewall, even if it was for marketing reasons, is a good thing.

Think of the guy who sells bread at the local store - he doesn’t do it because he feels it’s his duty to keep people from starving, he’s doing it because he needs the money. That’s economy for ya. Ethically speaking the reasons WHY you’re doing it may be less noble, but the act itself remains noble nonetheless. People need to understand that. And you need to come to terms with it: you did nothing wrong. Great job again.

the difference though: It was NOT a practical security flaw! Failing that test did not make cpf less secure!

Don’t you think we would have fixed it if we thought it was insecurity!!! Of course we would have!


Hmm… you got a point. I guess that does indeed make it nothing more than a marketing ploy. Oh well, so what if it is? I don’t mind it as much as to start a crusade for it. I guess even the best free products need good publicity huh? And that costs. It doesn’t really matter if it’s in the form of paying to have banners put up on a site or paying a test site to rate your program, it’s still the same thing in essence, even though the latter costs much more. As long as the facts are genuine I don’t see anything wrong with it, and CFP’s genuine facts are full of praise. 40 minutes for a perfect score? Certainly worth it.

Thanks Searinox! :slight_smile: