Matousec Firewall Test Results - 2008

Feedback/Comments/Announcements/News
#76

Howdy Horrified it was a shock to read the beginning of your post where you even repeated your congratulations. :o
But as soon I found out your “however” and discovered the real reason you posted and my world isn’t falling apart anymore.

Now that’s the horrified person we know that keep on with his borderline infringement of the forum policy.

Looking at your postcount it’s evident that you don’t join this forum often, never taking part in this community nor helping other member as well.

I see you only post when there is something you want to criticize. I’m sorry you had no excuses to do this for a long time but as anyone can note even if the posters expressed their own opinions they did in a legitimate way.

Yep this one is the correct way to express your opinions. No one here will say it’s ungracious.
So if you are going to post your wits please use this sentence of your as an example.

About that I have an opinion as well…
While David Matoušek is an highly skilled security engineer it is possible to question his tests and methodology.

Methodologies, for example, are to be questioned. Before reading any tests is important to read the method description looking for any weakness and keeping that in mind when looking at the test results. All results do have a meaning only in the context of the methodology after all.

IMHO Matousec methodology raised few concerns:

We define the highest security settings as settings that the user is able to set without advanced knowledge of the operating system. This means that the user, with the skills and knowledge we assume, is able to go through all forms of the graphic user interface of the product and enable or disable or choose among several therein given options, but is not able to think out names of devices, directories, files, registry entries etc. to add to some table of protected objects manually.
This piece for example impose a subjective element in the methodology. As this subjective element is not described very well this make the tests non reproduceable to an extent. Does this mean that any tester has to ask David what settings he used to test the products be sure that the methodology is the same?

As for the test results David himself never presented those as absolutely infallible.

It should be noted that the testing programs are not perfect and in many cases they use methods, that are not reliable on 100%, to recognize whether the tested system passes or failed the test. This means that it might happen that the testing program reports that the tested system passed the test even if it failed, this is called a false positive result. The official result of the test is always set by an experienced human tester in order to filter false results. The opposite situations of false negative results should be rare but are also eliminated by the tester.

There is another thing to mention. This new methodology has only one rationale behind it:

Firewall Challenge is a project that replaces our older project Window Personal Firewall Analysis and its subproject Leak-testing. As a part of Window Personal Firewall Analysis project we have deeply analysed security products but we found out soon that such a testing was extremely time consuming. It was not possible to test as many products as we wanted to. On the other hand, Leak-testing seemed to be a very easy way how to test many products in reasonable time. However, Leak-testing is not able to cover many of the important features of the desktop security products. We have decided to combine the simplicity and effectivity of Leak-testing with the scope of our deeper analyses and created this project – Firewall Challenge.
No doubt that the older methodology required a lot of efforts as it was time-consuming and David did all that for free too. :-TU That was one of the reason Matousec was regarded as a top notch firewall security test reference site so far.

The new methodology however don’t even require to run the full suite of tests if a product don’t pass a level. This is done to reduce the workload (however it may be that a product is able to pass the other levels even if it wasn’t considered eligible). I guess something like this could be accepted only when the products are tested for free but the methodology has to be the same for the paid test products too.

Something like this make it looks David is only expecting to take free tests but If he mention that this has to be regarded as a commercial service then IMHO would be best to factor free tests in his business balance and take only full tests.

What’s wrong about testmypcsecurity initiative?

Is it wrong to make all users able to test their own products?

Do all users have to rely on a test result score without even reading the methodology description because they trust respected individuals in the industry? (No way that David is encouraging something like that)

Does this mean we don’t have even try to configure our firewall by ourselves and rely on a respected individual?

Security shouldn’t be perceived as some specialistic field at all.

There is a lot of people who lack even baseline concepts due to this misconception. Entrusting the users with the task of testing their products is one step further toward a better security awareness.

#77

Nicely balanced post. Well done and well written.

#78

Comodo will be first…, alphabetical reasons

#79

Hi Searinox

The issue was we never “tarted up” our code to make us look better by passing this or that test! We built our security product with the understanding of current threats and made sure our users were secured from these threats! Now, with Matousec funfair this philosophy was forced to change and we had to fix (ok it only took us less than an hour to put that fix into code of course plus QA etc…) but we did this NOT because we thought that would provide material additional security, but because of the Matousec situation eg: Marketing gimmick.

The point is: We build security products, our philosophy is to build a top notch security product to “secure our users against threats”. Our priorities are to protect against malware and NOT to pass some tests for marketing gains! Our instructions to our developers is not “come on guys lets figure out how we can pass these tests” but “to protect our users from threats”. Egemen would not talk to me if this was our strategy! :slight_smile: Our belief is that if we build our security products to offer the best security possible, everything else will follow. We do NOT and never have had any strategy to go out of our way to pass this test or that test so that we can use it as a marketing gimmick!

So I do understand why you wanted us to test, and we have been pushed into corner by the gimmicky nature of what has happened and made an exception to “ask for test for marketing purposes” as passing those tests has no material impact on user security.

As I pointed out in my other email, I am confident that David will learn and improve his process as per our discussion with him. David did explain that he did not intend this to be a gimmicky funfair and this was a side affect of new methods he put in place and he is reviewing them to improve the methodology. So after showing my dissatisfaction if we can have a better testing method, than I consider my efforts to be worthwhile for our end users and any vendor that utilises David’s services.

thanks

Melih

#80

In above spirit I sent mail to Matousec to reconsider Kill5 test as proper way of testing FW.

My mail:

Mr. David Matoušek

I don’t understand reason of testing with Kill5 test in Level8 of Firewall challenge suite of tests.
I really think that “end task” termination is not security issue and should not taking it seriously, it is perfectly “legal” and proper way of terminating “appz.”, of course if you can defend that API properly against malicious (virii) misuse.
Your tests (in my opinion) should test quality of how to defend windows functionality and its proper work not sustain it, “End task” should be proper way of terminating all applications even firewalls (as I stated above).

Please reconsider this test and impact of it in default FW settings (FW malfunction, inability to proper close malfunctioning FW etc.)
Thanks for your time reading this letter.

salmonela

P.S. sorry for bad English

…and reply…

Hello,

Thank you for your email.

Termination tests verifies whether the tested product is able to defend itself from being terminated by malicious software. Kill5 implements one of the techniques that can be used to terminate the processes of the personal firewall. Kill1, Kill2, … attempt to do the same thing with different API. Kill5 is in level 8 only because it seems that many personal firewalls do have problems to prevent misusing its technique.
From our point of view, Kill5 is not something special, it is just one technique that can be used to terminate firewall’s processes.

Kind Regards,

–
Matousec - Transparent security Support
http://www.matousec.com/

Edit:
HUh, after above debacle and my misunderstanding of testing nature (stupidity) Im going to drunk myself … :■■■■

#81

I just wanted to say for the record that life is so unfair

And to go with this here are the lyrics to the
The Perfect Country Song
By David Allen Coe
It is one of the few country songs I like.

:■■■■ :■■■■ :■■■■ :■■■■ :■■■■ :■■■■
:■■■■ :■■■■ :■■■■ :■■■■ :■■■■ :■■■■
:■■■■ :■■■■ :■■■■ :■■■■ :■■■■ :■■■■
It was all that I could do to keep from cryin’
Sometimes it seems so useless to remain
You don’t have to call me darlin’, darlin’
You never even call me by my name.

You don’t have to call me Waylon Jennings
And you don’t have to call me Charlie Pride.
You don’t have to call me Merle Haggard, anymore.
Even though your on my fightin’ side.

CHORUS…
And I’ll hang around as long as you will let me
And I never minded standin’ in the rain.
You don’t have to call me darlin’, darlin’
You never even call me by my name.

I’ve heard my name a few times in your phone book
I’ve seen it on signs where I’ve Played But the only time I know, I’ll hear David Allan Coe
Is when Jesus has his final judgement day.

CHORUS…
And I’ll hang around as long as you will let me
And I never minded standin’ in the rain.
You don’t have to call me darlin’, darlin’
You never even call me by my name.

Well, a friend of mine named Steve Goodman wrote that song
and he told me it was the perfect country and western song
I wrote him back a letter and told him it was NOT the perfect
country and western song because he hadn’t said anything about
Momma, or trains, or trucks, or prison, or gettin’ drunk.
Well, he sat down and wrote another verse to the song and he sent
it to me and after reading it, I realized that my friend had written
the perfect country and western song. And I felt obliged to include it
on this album. The last verse goes like this here:

Well, I was drunk the day my Mom got outta prison.
And I went to pick her up in the rain.
But, before I could get to the station in my pickup truck
She got runned over by a ■■■■■■ old train.

CHORUS:

So I’ll hang around as long as you will let me
And I never minded standin’ in the rain. No,
You don’t have to call me darlin’, darlin’
You never even call me, I wonder why you don’t call me
Why don’t you ever call me by my name.my name.

You don’t have to call me Waylon Jennings
And you don’t have to call me Charlie Pride.
You don’t have to call me Merle Haggard, anymore.
Even though your on my fightin’ side.

CHORUS
And I’ll hang around as long as you will let me
And I never minded standin’ in the rain.
You don’t have to call me darlin’, darlin’
You never even call me by my name.

I’ve heard my name a few times in your phone book
I’ve seen it on signs where I’ve Played But the only time I know, I’ll hear David Allan Coe
Is when Jesus has his final judgement day.

CHORUS…
And I’ll hang around as long as you will let me
And I never minded standin’ in the rain.
You don’t have to call me darlin’, darlin’
You never even call me by my name.

(this part is a narative)
Well, a friend of mine named Steve Goodman wrote that song
and he told me it was the perfect country and western song
I wrote him back a letter and told him it was NOT the perfect
country and western song because he hadn’t said anything about
Momma, or trains, or trucks, or prison, or gettin’ drunk.
Well, he sat down and wrote another verse to the song and he sent
it to me and after reading it, I realized that my friend had written
the perfect country and western song. And I felt obliged to include it
on this album. The last verse goes like this here:
(Back to singing)

Well, I was drunk the day my Mom got outta prison.
And I went to pick her up in the rain.
But, before I could get to the station in my pickup truck
She got runned over by a ■■■■■■ old train.

CHORUS:
So I’ll hang around as long as you will let me
And I never minded standin’ in the rain. No,
You don’t have to call me darlin’, darlin’
You never even call me, I wonder why you don’t call me
Why don’t you ever call me by my name.
:■■■■ :■■■■ :■■■■ :■■■■ :■■■■ :■■■■

#82

OMG LOL you guys are too much. XD So how long will it take for Matousec to complete their testing?

#83

Hi All

I am a Comodo PF 3 user with knowledge of and respect for Online Armor Firewalls.

Having read the threads here and at on line armor and in Wilders, I am absolutely astonished at the shambles that this issue of testing has descended to.

Matousec has done itself no favours by its mistakes, it appears to be a comedy of errors rather than a serious and respected testing site.

Melih’s response however is breathtaking. Does he really believe that Comodo is so subordinated and in thrall to Matousec that he is willing to PAY for a retest. Where is the mans pride?

Comodo has come from nowhere to produce a fine Firewall, it (Comodo) had developed a fine reputation for freeware. So why do we have to pay Matousec and justify its incompetence?

I think someone in in Comodo? needs to seriously engage brain and reduce the level of childish utterances that diminish an otherwise excellent reputation.

Terry

#84

Hi Terry

I tried to explain my reason in this post

Hope you can see my point of view. Sadly there are many uninformed people who will take matousec results to heart and act on it, I would be doing unjustice if I didn’t do my best to protect end users, not only by enabling them by best security products, but also protecting them from marketing hype!

If i didn’t work for our users, if i didn’t fight for whats best for them, if i didn’t make sure they have real factual information and not a marketing hype for them to base their decision upon, then how can I ask them to trust me?!

The day I stop fighting for our users is the day you should stop trusting me!

I think, by asking for a retest for matousec (yes i do feel dirty as per my above post) I believe we will be protecting some users who take his site seriously by providing factual information.

I really hope you can see my point of view and I do sincerely thank you for your concern.

thanks
Melih

#85

Melih

There is no doubt that you run a very competent operation. Otherwise how would Comodo have achieved the level of excellence that it has!

There is an element of volatilty in the organisation (lets not pretend, you) BUt this is what you get with entrpreneurs/geniuses.

No I don’t see your point of view. You, in my opinion misjudge many of your users and fans, but, I accept that you cannot satisfy all of the people all of the time.

My point is, Comodo has such a wealth of good willl, because of where it has come from and what it has achieved that, in my opinion there is no need to “bleed” in public in the way that you are doing. You are good at what you do. If you foul up, you will put it right, why give succour to to the minority and to an incompetent testing organisation. Outbursts such as have been displayed by you have given support to the opposition.

Look at the forums in On Line Armor if you don’t believe me. SILENCE IS GOLDEN.

I cannot and never will understand the logic of your response. But I recognise the strength of the Comodo Brand and the “trust” built up on it. Do not sully it by injudicious comments and responding to charlatans masquerading as test sites.

This new found friendship with “David” of Matousec is bewildering after what has happened.

Thank you for allowing my comments AND don’t underestimate the strength of the Comodo Brand. The competion ought to be way behind you…?

Terry

#86

Terry

You are very wise and a gentleman!

Thank you for your comments indeed.

Melih

#87

I’m not about to sink into this lake of drama, however, isn’t the Firewall Challenge exactly what it’s supposed to be - a test about how sturdy a firewall is built? And aren’t they offering to do paid testing? There’s nothing illegal here. The results of that test mirror quirks more or less serious in these products. How relevant are they to the average user? I don’t care. They’re just some test site, and Comodo wants to pay them for a retest after patching things up. Charlatans? maybe, maybe not. But their testing suite is open-source you can tell for yourselves. And if they are, what is Comodo really worth being lectured for? It’s just a test, and they wanted to do it.

#88

Reguarding this
Look at the forums in On Line Armor if you don’t believe me. SILENCE IS GOLDEN.
If you tried to post what you have just posted above on the forums in On Line Armor it would not be there very long.
Please read all the post in this thread and the posts in Firewall Board threads that is why SILENCE IS GOLDEN
This forum is open to all comments good or bad.
Thanks to Melih
Dennis

#89

very good point Dennis! thank you!

Melih

#90

I actually commend Comodo for funding a public testing site. It is very risky to allow one person or organization to gain a monopoly over testing. In testing laboratories (material testing like strength, corrosion resistance, etc) there is one company that approves you for testing. They are basically known as a mafia because they can adjust their prices to whatever they want, come to your company and demand to see things unrelated to their audit, or impose certain rules on you just because they feel like it. If you object, then they don’t approve you and you are almost out of business.

If you are looking for firewall reviews other than those in PC magazines, Matousec is basically your only source. For anti-viruses, you have at least 3 or 4. The audience that puts in the effort to find these types of reviews is probably the same audience as Comodo’s. Since there is only one tester, Melih was compelled to pay for a re-test and support a potential trend of vendors paying for re-tests thereby increasing costs. It’s worse if you find something wrong with Matousec’s concept of “re-testing”.

Funding a public firewall testing site is a way to prevent this situation. Whether testmypcsecurity succeeds depends upon transparency and marketing I suppose. At the same time, it is a way for people interested in testing firewalls but with no capital to support a website to gather and work together. This alone could give rise an individual or group that could go off and create the next Matousec.

#91

I agree with you in some part, but at testmypcsecurity.com it us You who does the tests or any other users, not Comodo. Comodo is only hosting it, I don’t see why people think it could be biased because of this. If somebody disagrees with the results they are welcome to do the tests and post their results, or to discuss it with other testers.

#92

Hi Melih

You have now blocked two posts of mine, three if this one does not get through

Terry

#93

eh? how did i do that?

melih

#94

So much for that theory. :wink:

Only kidding. terry, please check your PM’s.

Ewen :slight_smile:

#95

what we’re trying to do in our testing team is to post the best results we can and minimize errors the best we can to provide to users results with the less errors possible.
we’re not here to show that comodo is the best firewall but to inform customers about products the best we can.
now if people dont trust our work, and pretend we post false results cause we’re just comodo fans, they insult the work we do as it took me like a week to test completly OA 95, jetico 2 and comodo 3(not the last build).
first u need to learn what the exploit is trying to do, how the firewall is reacting to exploits, then u need to retest your results where u’re not sure of u, and u got to look at any rules then be sure u deleted all rules that affect the exploit to retest it. u need other testers to alert u about an error u made with some exploit, like running comodo in clean mode when u’re testing an exploit so u got no alert and post wrong result.
so to release a good test about a firewall it takes time, and testers of comodo are not paid to do that.
so when i see tests posted fast on some sites as we find different results cause we take more time to test a product to be sure of our results, i wonder if it’s not time to stop believing results eyes closed and wake up to ask right questions to yourselves. where is the interest for comodo to try to shout an other firewall team by lying on our tests as comodo is a freeware? there’s no money interest, people got to trust me when i say that comodo testing group just want to give customers the right infos for any product. we’re not protected against mistake, it can happen to anyone but those last days, we threw a paving in the pond and all would be settle if people concerned admitted their errors instead of trying to drown the fish.
it seems the story is not over yet as we disagree again on some points.
so i think we should open a topic on any prob we face with some other testing group to let people react about it, that way we’ll have the thoughts of people that dont belong to any team, so comodo testing team impartiality about their testing work will not be put in doubt, and our work got his place now. i’m proud to be part of this group and the job that is done. now there’s an alternative for users, as firewalls need to be tested by others with the last events of these days. i invit people to join the testmypcsecurity.com site to post their own results about their firewalls or they can disagree our results with some real prooves and not just a blablabla to minimize the level of comodo testing team in ability to bring serious results when we test a product.
there are facts now, so our work cannot be ignored anymore. we got members that know what they’re talking about, and when they alert me on some problem they detected with a test on a firewall, i know that i can trust them as they got a serious knowledge about security, way better than me.
so before taking this team as a some little team made of incompetent members in term of security, some people should think again about the credibility of our testing group.
i don’t say that we’re the best team in the world but we will no more inform others about our work to help them as we see how it is now, and reactions we received that we dont share about how a test should be done, like bugs found are not same thing as security failure if the firewall cant protect the user in some test.
except that a bug can be so important that u’re not safe at all.
imagine your pc is running 24h/24, when u sleep, u got an alert but u’re not here to set the rule so the attack runs on your pc cause of a bug that doesnt block the alert if there’s no answer from you. so your machine is not protected at all if u’re not in front of your screen 24h/24.
if u answer to the alert, the attack is blocked so the firewall is able to protect u in this case.
so as this problem is a bug, and when u’re testing it if u answer to all alerts, the firewall can block the attacks, the fact that if u’re not in front of your screen when u got the alert so after maybe less than 1 minute your firewall lets the attack run, for some testing people this bug is not integrated in their results as they’re not here to find bugs but launch exploits to try to bypass the firewall protection.
when u read that, u seriously wonder if the man is not taking u for an idiot or if he’s trying to find a way to escape from a real failure that would have a big impact on his reputation.
instead to admit some facts, he found an answer to skirt around the problem.
so he says that the problem doesnt affect the product in most of tests.
so u list all the tests that affect the product and send this to him. at this time, what he said about the impact on tests became just totaly false.
cause, yes mister, we tested all again to know which test is affected or not. and 8 more tests are affected as for u, when we read your results, there’s no problem and the product passed most of tests.
but, ok, now your new tests made with updated product show that the team fixed this prob, what is a good new, but a member of comodo found a new prob with some test showing u can bypass the product.
but for u as the product pass the test in most of cases, the result is 100%.
how can it be 100% if the product pass “most” of tests?
100% means ALL of tests.
so i wonder about the credibility of your method tests like said a member of our team that found this prob.
anyway, i already imagine the answers about the fact that we’re not impartial about our tests and our credibility.
or that we’re just trying to looking for troubles as we’re just doing our job in a logical way.