I would give a big fat minus to Comodo for getting in this Matousec paid testing sh…t, but since Mr. Melih explained why Comodo is doing it I won’t. And it will be quite fun to watch.
BUT NEVER DO IT AGAIN 
I really think that “end task” termination is not security issue and should not taking it serious in COMODO case, it is perfectly “legal” and proper way of terminating apps., of course if you can defend that API properly against malicious (virii) misuse.
BTW. if I remember right there was at least couple of requests and concerning posts earlier here on this forum about end task “issue”
I was VERY adamant about Comodo paying to do another test because I don’t see it fair for Matousec to announce OA as 100% and CFP as 98% until the very next round of free tests. CFP developers were quick to patch up the remaining 2% and I believe that should show to the world. But reading the staff posts, I realise that apparently there are some morality issues which I don’t fully understand, so I’m stepping down with my opinion.
yes the man of matousec seems to be a good guy.
i asked him to change the info about the OA issue.
i asked him to add MaratR with my name and now the info is modified.
MaratR showed that OA failed in most of tests cause of this bug as i just noticed that jumper bypassed OA when u wait on window alert. i didnt wait on other tests to see if i had the same result.
so thanks to david about my request.
If CFP comes out 100% on this re-test, then Comodo should be listed first above OA since Comodo was the best before re-testing. Melih, make sure David understands that point. 
Comodo was the best and OA had that bug for a very long time, from what I hear, and they only fixed it when the test results were not favorable to them. In my view, this make Comodo the best, so it should be listed first. So, tell David all this when he updates the score.
Cheers
Howdy Horrified it was a shock to read the beginning of your post where you even repeated your congratulations. :o
But as soon I found out your “however” and discovered the real reason you posted and my world isn’t falling apart anymore.
Now that’s the horrified person we know that keep on with his borderline infringement of the forum policy.
Looking at your postcount it’s evident that you don’t join this forum often, never taking part in this community nor helping other member as well.
I see you only post when there is something you want to criticize. I’m sorry you had no excuses to do this for a long time but as anyone can note even if the posters expressed their own opinions they did in a legitimate way.
Yep this one is the correct way to express your opinions. No one here will say it’s ungracious.
So if you are going to post your wits please use this sentence of your as an example.
About that I have an opinion as well…
While David Matoušek is an highly skilled security engineer it is possible to question his tests and methodology.
Methodologies, for example, are to be questioned. Before reading any tests is important to read the method description looking for any weakness and keeping that in mind when looking at the test results. All results do have a meaning only in the context of the methodology after all.
IMHO Matousec methodology raised few concerns:
We define the highest security settings as settings that the user is able to set without advanced knowledge of the operating system. This means that the user, with the skills and knowledge we assume, is able to go through all forms of the graphic user interface of the product and enable or disable or choose among several therein given options, but is not able to think out names of devices, directories, files, registry entries etc. to add to some table of protected objects manually.This piece for example impose a subjective element in the methodology. As this subjective element is not described very well this make the tests non reproduceable to an extent. Does this mean that any tester has to ask David what settings he used to test the products be sure that the methodology is the same?
As for the test results David himself never presented those as absolutely infallible.
It should be noted that the testing programs are not perfect and in many cases they use methods, that are not reliable on 100%, to recognize whether the tested system passes or failed the test. This means that it might happen that the testing program reports that the tested system passed the test even if it failed, this is called a false positive result. The official result of the test is always set by an experienced human tester in order to filter false results. The opposite situations of false negative results should be rare but are also eliminated by the tester.
There is another thing to mention. This new methodology has only one rationale behind it:
Firewall Challenge is a project that replaces our older project Window Personal Firewall Analysis and its subproject Leak-testing. As a part of Window Personal Firewall Analysis project we have deeply analysed security products but we found out soon that such a testing was extremely time consuming. It was not possible to test as many products as we wanted to. On the other hand, Leak-testing seemed to be a very easy way how to test many products in reasonable time. However, Leak-testing is not able to cover many of the important features of the desktop security products. We have decided to combine the simplicity and effectivity of Leak-testing with the scope of our deeper analyses and created this project – Firewall Challenge.No doubt that the older methodology required a lot of efforts as it was time-consuming and David did all that for free too. :-TU That was one of the reason Matousec was regarded as a top notch firewall security test reference site so far.
The new methodology however don’t even require to run the full suite of tests if a product don’t pass a level. This is done to reduce the workload (however it may be that a product is able to pass the other levels even if it wasn’t considered eligible). I guess something like this could be accepted only when the products are tested for free but the methodology has to be the same for the paid test products too.
Something like this make it looks David is only expecting to take free tests but If he mention that this has to be regarded as a commercial service then IMHO would be best to factor free tests in his business balance and take only full tests.
What’s wrong about testmypcsecurity initiative?
Is it wrong to make all users able to test their own products?
Do all users have to rely on a test result score without even reading the methodology description because they trust respected individuals in the industry? (No way that David is encouraging something like that)
Does this mean we don’t have even try to configure our firewall by ourselves and rely on a respected individual?
Security shouldn’t be perceived as some specialistic field at all.
There is a lot of people who lack even baseline concepts due to this misconception. Entrusting the users with the task of testing their products is one step further toward a better security awareness.
Nicely balanced post. Well done and well written.
Comodo will be first…, alphabetical reasons
Hi Searinox
The issue was we never “tarted up” our code to make us look better by passing this or that test! We built our security product with the understanding of current threats and made sure our users were secured from these threats! Now, with Matousec funfair this philosophy was forced to change and we had to fix (ok it only took us less than an hour to put that fix into code of course plus QA etc…) but we did this NOT because we thought that would provide material additional security, but because of the Matousec situation eg: Marketing gimmick.
The point is: We build security products, our philosophy is to build a top notch security product to “secure our users against threats”. Our priorities are to protect against malware and NOT to pass some tests for marketing gains! Our instructions to our developers is not “come on guys lets figure out how we can pass these tests” but “to protect our users from threats”. Egemen would not talk to me if this was our strategy! Our belief is that if we build our security products to offer the best security possible, everything else will follow. We do NOT and never have had any strategy to go out of our way to pass this test or that test so that we can use it as a marketing gimmick!
So I do understand why you wanted us to test, and we have been pushed into corner by the gimmicky nature of what has happened and made an exception to “ask for test for marketing purposes” as passing those tests has no material impact on user security.
As I pointed out in my other email, I am confident that David will learn and improve his process as per our discussion with him. David did explain that he did not intend this to be a gimmicky funfair and this was a side affect of new methods he put in place and he is reviewing them to improve the methodology. So after showing my dissatisfaction if we can have a better testing method, than I consider my efforts to be worthwhile for our end users and any vendor that utilises David’s services.
thanks
Melih
In above spirit I sent mail to Matousec to reconsider Kill5 test as proper way of testing FW.
My mail:
Mr. David MatoušekI don’t understand reason of testing with Kill5 test in Level8 of Firewall challenge suite of tests.
I really think that “end task” termination is not security issue and should not taking it seriously, it is perfectly “legal” and proper way of terminating “appz.”, of course if you can defend that API properly against malicious (virii) misuse.
Your tests (in my opinion) should test quality of how to defend windows functionality and its proper work not sustain it, “End task” should be proper way of terminating all applications even firewalls (as I stated above).Please reconsider this test and impact of it in default FW settings (FW malfunction, inability to proper close malfunctioning FW etc.)
Thanks for your time reading this letter.salmonela
P.S. sorry for bad English
…and reply…
Hello,Thank you for your email.
Termination tests verifies whether the tested product is able to defend itself from being terminated by malicious software. Kill5 implements one of the techniques that can be used to terminate the processes of the personal firewall. Kill1, Kill2, … attempt to do the same thing with different API. Kill5 is in level 8 only because it seems that many personal firewalls do have problems to prevent misusing its technique.
From our point of view, Kill5 is not something special, it is just one technique that can be used to terminate firewall’s processes.Kind Regards,
–
Matousec - Transparent security Support
http://www.matousec.com/
Edit:
HUh, after above debacle and my misunderstanding of testing nature (stupidity) Im going to drunk myself … :■■■■
I just wanted to say for the record that life is so unfair
And to go with this here are the lyrics to the
The Perfect Country Song
By David Allen Coe
It is one of the few country songs I like.
:■■■■ :■■■■ :■■■■ :■■■■ :■■■■ :■■■■
:■■■■ :■■■■ :■■■■ :■■■■ :■■■■ :■■■■
:■■■■ :■■■■ :■■■■ :■■■■ :■■■■ :■■■■
It was all that I could do to keep from cryin’
Sometimes it seems so useless to remain
You don’t have to call me darlin’, darlin’
You never even call me by my name.
You don’t have to call me Waylon Jennings
And you don’t have to call me Charlie Pride.
You don’t have to call me Merle Haggard, anymore.
Even though your on my fightin’ side.
CHORUS…
And I’ll hang around as long as you will let me
And I never minded standin’ in the rain.
You don’t have to call me darlin’, darlin’
You never even call me by my name.
I’ve heard my name a few times in your phone book
I’ve seen it on signs where I’ve Played But the only time I know, I’ll hear David Allan Coe
Is when Jesus has his final judgement day.
CHORUS…
And I’ll hang around as long as you will let me
And I never minded standin’ in the rain.
You don’t have to call me darlin’, darlin’
You never even call me by my name.
Well, a friend of mine named Steve Goodman wrote that song
and he told me it was the perfect country and western song
I wrote him back a letter and told him it was NOT the perfect
country and western song because he hadn’t said anything about
Momma, or trains, or trucks, or prison, or gettin’ drunk.
Well, he sat down and wrote another verse to the song and he sent
it to me and after reading it, I realized that my friend had written
the perfect country and western song. And I felt obliged to include it
on this album. The last verse goes like this here:
Well, I was drunk the day my Mom got outta prison.
And I went to pick her up in the rain.
But, before I could get to the station in my pickup truck
She got runned over by a ■■■■■■ old train.
CHORUS:
So I’ll hang around as long as you will let me
And I never minded standin’ in the rain. No,
You don’t have to call me darlin’, darlin’
You never even call me, I wonder why you don’t call me
Why don’t you ever call me by my name.my name.
You don’t have to call me Waylon Jennings
And you don’t have to call me Charlie Pride.
You don’t have to call me Merle Haggard, anymore.
Even though your on my fightin’ side.
CHORUS
And I’ll hang around as long as you will let me
And I never minded standin’ in the rain.
You don’t have to call me darlin’, darlin’
You never even call me by my name.
I’ve heard my name a few times in your phone book
I’ve seen it on signs where I’ve Played But the only time I know, I’ll hear David Allan Coe
Is when Jesus has his final judgement day.
CHORUS…
And I’ll hang around as long as you will let me
And I never minded standin’ in the rain.
You don’t have to call me darlin’, darlin’
You never even call me by my name.
(this part is a narative)
Well, a friend of mine named Steve Goodman wrote that song
and he told me it was the perfect country and western song
I wrote him back a letter and told him it was NOT the perfect
country and western song because he hadn’t said anything about
Momma, or trains, or trucks, or prison, or gettin’ drunk.
Well, he sat down and wrote another verse to the song and he sent
it to me and after reading it, I realized that my friend had written
the perfect country and western song. And I felt obliged to include it
on this album. The last verse goes like this here:
(Back to singing)
Well, I was drunk the day my Mom got outta prison.
And I went to pick her up in the rain.
But, before I could get to the station in my pickup truck
She got runned over by a ■■■■■■ old train.
CHORUS:
So I’ll hang around as long as you will let me
And I never minded standin’ in the rain. No,
You don’t have to call me darlin’, darlin’
You never even call me, I wonder why you don’t call me
Why don’t you ever call me by my name.
:■■■■ :■■■■ :■■■■ :■■■■ :■■■■ :■■■■
OMG LOL you guys are too much. XD So how long will it take for Matousec to complete their testing?
Hi All
I am a Comodo PF 3 user with knowledge of and respect for Online Armor Firewalls.
Having read the threads here and at on line armor and in Wilders, I am absolutely astonished at the shambles that this issue of testing has descended to.
Matousec has done itself no favours by its mistakes, it appears to be a comedy of errors rather than a serious and respected testing site.
Melih’s response however is breathtaking. Does he really believe that Comodo is so subordinated and in thrall to Matousec that he is willing to PAY for a retest. Where is the mans pride?
Comodo has come from nowhere to produce a fine Firewall, it (Comodo) had developed a fine reputation for freeware. So why do we have to pay Matousec and justify its incompetence?
I think someone in in Comodo? needs to seriously engage brain and reduce the level of childish utterances that diminish an otherwise excellent reputation.
Terry
Hi Terry
I tried to explain my reason in this post
Hope you can see my point of view. Sadly there are many uninformed people who will take matousec results to heart and act on it, I would be doing unjustice if I didn’t do my best to protect end users, not only by enabling them by best security products, but also protecting them from marketing hype!
If i didn’t work for our users, if i didn’t fight for whats best for them, if i didn’t make sure they have real factual information and not a marketing hype for them to base their decision upon, then how can I ask them to trust me?!
The day I stop fighting for our users is the day you should stop trusting me!
I think, by asking for a retest for matousec (yes i do feel dirty as per my above post) I believe we will be protecting some users who take his site seriously by providing factual information.
I really hope you can see my point of view and I do sincerely thank you for your concern.
thanks
Melih
Melih
There is no doubt that you run a very competent operation. Otherwise how would Comodo have achieved the level of excellence that it has!
There is an element of volatilty in the organisation (lets not pretend, you) BUt this is what you get with entrpreneurs/geniuses.
No I don’t see your point of view. You, in my opinion misjudge many of your users and fans, but, I accept that you cannot satisfy all of the people all of the time.
My point is, Comodo has such a wealth of good willl, because of where it has come from and what it has achieved that, in my opinion there is no need to “bleed” in public in the way that you are doing. You are good at what you do. If you foul up, you will put it right, why give succour to to the minority and to an incompetent testing organisation. Outbursts such as have been displayed by you have given support to the opposition.
Look at the forums in On Line Armor if you don’t believe me. SILENCE IS GOLDEN.
I cannot and never will understand the logic of your response. But I recognise the strength of the Comodo Brand and the “trust” built up on it. Do not sully it by injudicious comments and responding to charlatans masquerading as test sites.
This new found friendship with “David” of Matousec is bewildering after what has happened.
Thank you for allowing my comments AND don’t underestimate the strength of the Comodo Brand. The competion ought to be way behind you…?
Terry
Terry
You are very wise and a gentleman!
Thank you for your comments indeed.
Melih
I’m not about to sink into this lake of drama, however, isn’t the Firewall Challenge exactly what it’s supposed to be - a test about how sturdy a firewall is built? And aren’t they offering to do paid testing? There’s nothing illegal here. The results of that test mirror quirks more or less serious in these products. How relevant are they to the average user? I don’t care. They’re just some test site, and Comodo wants to pay them for a retest after patching things up. Charlatans? maybe, maybe not. But their testing suite is open-source you can tell for yourselves. And if they are, what is Comodo really worth being lectured for? It’s just a test, and they wanted to do it.
Reguarding this
Look at the forums in On Line Armor if you don’t believe me. SILENCE IS GOLDEN.
If you tried to post what you have just posted above on the forums in On Line Armor it would not be there very long.
Please read all the post in this thread and the posts in Firewall Board threads that is why SILENCE IS GOLDEN
This forum is open to all comments good or bad.
Thanks to Melih
Dennis
very good point Dennis! thank you!
Melih
I actually commend Comodo for funding a public testing site. It is very risky to allow one person or organization to gain a monopoly over testing. In testing laboratories (material testing like strength, corrosion resistance, etc) there is one company that approves you for testing. They are basically known as a mafia because they can adjust their prices to whatever they want, come to your company and demand to see things unrelated to their audit, or impose certain rules on you just because they feel like it. If you object, then they don’t approve you and you are almost out of business.
If you are looking for firewall reviews other than those in PC magazines, Matousec is basically your only source. For anti-viruses, you have at least 3 or 4. The audience that puts in the effort to find these types of reviews is probably the same audience as Comodo’s. Since there is only one tester, Melih was compelled to pay for a re-test and support a potential trend of vendors paying for re-tests thereby increasing costs. It’s worse if you find something wrong with Matousec’s concept of “re-testing”.
Funding a public firewall testing site is a way to prevent this situation. Whether testmypcsecurity succeeds depends upon transparency and marketing I suppose. At the same time, it is a way for people interested in testing firewalls but with no capital to support a website to gather and work together. This alone could give rise an individual or group that could go off and create the next Matousec.