OA must of paid for the retest if they did it perfectly fair this would be per matousec policy and the option of OA. Comodo probably does not want to pay as they want to keep the expenses as reasonable, after all, the only gain Comodo gets is good will an promotion. It is in the best interest of OA as paid firewall to maintain top standing.
I would like to see CFPv3 pass deletevolume and Keylogger. I am sure it will in its own time, and it will regain the top.
In the words of Mike Nash, CEO of Tall Emu makers of OA, on Wilders
Now this is what I call a truly open security forum. Reporting strengths and weaknesses in any product for the benifit of all. Now If Comodo wants to maintain in the top 2 or three the will have to work for it. Also hopefully Paul wont hold any thing back, this means a from the Testmysecurity.com website and eventually our site may gain credibility in the security community.
OD
PS I think if the testers can make the time they should try to test to see the results of unanswered alerts as well
Congrats’ AILEF!
I’m not a fan, football or anything else (just want to learn how to be safe) but IMHO, Matousec cannot afford not to consider the best (Comodo!) in the next test if they want to remain a credible authority. All this doesn’t have to degenerate into money spiral – otherwise the next step would be that vendors would provide their tests, also – for a fee.
Nope, You’re not. Only BLAP are. (:TNG)
I was asking the same questions, and now I understand why I didn’t received the kind of detailed answer I’m used to on this forum - some tests were performed at that time – but no harm done. CFP is SO customizable that only the user can tell what’s his/hers best config.
Matty started here a tread addressing similar concerns. Maybe we should join there.
Regards, Gabi
As someone who tries to keep updated on overall computer security issues, maybe its time that someone tossed some doubt that its even possible to give an absolute test of any firewall in a real world situation. Because in the world I live in, my firewall is just part of my defenses and I don’t expect to be safe from just a firewall alone.
And its my responsibility to know some of my security holes and make them as small as possible. And to have layered defense with back up systems. In the case of the OA defect now band aid fixed with a now default block instead of check, had I been using a previous version of OA, I could have gone to bed and that dll injection would be still blocked by a process control program I also still use come morning…
And because my over security set up is probably different from everyone else, we could even go to testmysecurity.com and get different scores even if we both used the same firewall. I am not arrogant enough to say I would always get the better score, its always an unknowable gamble and just a small snapshot in time.
The fact that the “bad guys” get to shoot at me and I can’t shoot back bothers me. But thats just the reality of the internet and the short reach of international law. And it bothers me that I have to spend time and effort researching security choices. And then have to spend more time and effort keeping them updated. But I can reduce the unit time and effort required to reach a certain level of safety at least in my mind.
At least in my opinion, its no accident that comodo3 and OA occupy the top two spot in the rather biased Matousec tests. Because they represent the firewall trend of the future and incorporate in one application things like hips and other defenses I used to have to go to multiple programs to get. So in that sense, my life gets easier, fewer applications to juggle and update as I can get leaner yet maintain at least the same level of protection. But I sure don’t put all my security eggs in one basket and try to have some backup.
I have decided to stay with comodo3 as my firewall. Rightly or wrongly, its still my less than perfectly informed choice. And a choice I am prepared to take the consequences for. Its always a gamble and sometimes all paths can lead to ruin. Or to put it another way, even if we get burned with the choice I made, I never know if we would have been burned worse by taking the path I did not take.
As if anyone is perfectly informed or can anticipate the future security threats we will have to dodge as the
the “bad guys” keep getting smarter. I see no reason to change due to the results of just one test and a biased one at that. My gamble is that comodo will do better than any firewall now on the planet at protecting me from current and future threats. And that I hope is the best informed choice as of now.
I think I can say comodo3 has more than EARNED my trust. I will not bash OA in any way, its a user choice and a user gamble.
I agree with you 100%, you need a layered protection and even with the best “best of” you will not get perfect protection, however, Firewall and/or HIPS security program tests should not be conducted in a real world environment. In fact for the truest results the tested firewall should be the only security software with the possible exception of a “Pure” AV program. You are testing the Firewall and/or HIPS program not the other software
OD
Edit note: In facgteven testing in a VM could in theory affect results if the VM is not conficured correctly. However for most people/Companies this is the most practical method of testing and I believe most of the test have no problem with a VM eviron.
I believe TC will stay in Beta for at least a little while
Also
Melih has said that TC will be an option so it should not automatically upgrade uo to TC however When they do include TC in the full release the will probably IMO be a way to activate it without a full reinstall (I hope) or at least a way to import your settings from the prev version. I would assume TC to be a major Upgrade. But I d get sick of reconfiguring V3
As you know we did a release today that we believe will address the leak test issues reported in Matousec.
I would like to re-iterate and be frank that this is the FIRST time we have done something in our firewall development that we don’t believe will provide material security benefit to end users today. If passing those two tests were important we would have done it ages ago! The issue was NOT about getting 100% on some tests, it was about “Securing our end users”!
However, the funfair that ensued the method in which Matousec has handled things, left us no choice but to “show” what we are capable of “as we have been last few years”.
We asked Matousec to re-test and, as per their rule, will pay accordingly. But rest assured this is the “LAST” time we will pay unless Matousec changes the way he publishes results willy nilly which causes his credibility to be questioned. We expect a more credible, scheduled testing to be done, which we believe will create a better atmosphere and will benefit end users without creating a bidding war for “$ for top spot” and without distrupting development processes which at the end of the day will affect end users!
Now I feel “dirty” (:SAD) and somewhat “cheap” that I have resulted to this, but I feel it had to be done to set the record straight as this matousec funfair caused much confusion to our users!
On a positive note I have been exchanging emails with David (the chap behind Matousec) and happy to report that he understands my issues, meant no harm and I am sure he will learn from this experience and improve his service.
I do believe David is honest and credible and will do the right thing and improve his services.
Well, you were in a bit of a trap here - if you did nothing, then a lot who read Matousec would go on accepting OA as ‘best’; if you try for parity it’s almost like paying a ■■■■■ (but without the fun!).
From what you say, your [one-off] move might well result in fairer tests, even to the point where if one vendor pays for a retest then other software will be rechecked much soonerafter updates just to ensure an honest comparison (which is we’re asking).
I would give a big fat minus to Comodo for getting in this Matousec paid testing sh…t, but since Mr. Melih explained why Comodo is doing it I won’t. And it will be quite fun to watch. BUT NEVER DO IT AGAIN
I really think that “end task” termination is not security issue and should not taking it serious in COMODO case, it is perfectly “legal” and proper way of terminating apps., of course if you can defend that API properly against malicious (virii) misuse.
BTW. if I remember right there was at least couple of requests and concerning posts earlier here on this forum about end task “issue”
I was VERY adamant about Comodo paying to do another test because I don’t see it fair for Matousec to announce OA as 100% and CFP as 98% until the very next round of free tests. CFP developers were quick to patch up the remaining 2% and I believe that should show to the world. But reading the staff posts, I realise that apparently there are some morality issues which I don’t fully understand, so I’m stepping down with my opinion.
yes the man of matousec seems to be a good guy.
i asked him to change the info about the OA issue.
i asked him to add MaratR with my name and now the info is modified.
MaratR showed that OA failed in most of tests cause of this bug as i just noticed that jumper bypassed OA when u wait on window alert. i didnt wait on other tests to see if i had the same result.
so thanks to david about my request.
If CFP comes out 100% on this re-test, then Comodo should be listed first above OA since Comodo was the best before re-testing. Melih, make sure David understands that point. Comodo was the best and OA had that bug for a very long time, from what I hear, and they only fixed it when the test results were not favorable to them. In my view, this make Comodo the best, so it should be listed first. So, tell David all this when he updates the score.
Howdy Horrified it was a shock to read the beginning of your post where you even repeated your congratulations. :o
But as soon I found out your “however” and discovered the real reason you posted and my world isn’t falling apart anymore.
Now that’s the horrified person we know that keep on with his borderline infringement of the forum policy.
Looking at your postcount it’s evident that you don’t join this forum often, never taking part in this community nor helping other member as well.
I see you only post when there is something you want to criticize. I’m sorry you had no excuses to do this for a long time but as anyone can note even if the posters expressed their own opinions they did in a legitimate way.
Yep this one is the correct way to express your opinions. No one here will say it’s ungracious.
So if you are going to post your wits please use this sentence of your as an example.
About that I have an opinion as well…
While David Matoušek is an highly skilled security engineer it is possible to question his tests and methodology.
Methodologies, for example, are to be questioned. Before reading any tests is important to read the method description looking for any weakness and keeping that in mind when looking at the test results. All results do have a meaning only in the context of the methodology after all.
IMHO Matousec methodology raised few concerns:
We define the highest security settings as settings that the user is able to set without advanced knowledge of the operating system. This means that the user, with the skills and knowledge we assume, is able to go through all forms of the graphic user interface of the product and enable or disable or choose among several therein given options, but is not able to think out names of devices, directories, files, registry entries etc. to add to some table of protected objects manually.
This piece for example impose a subjective element in the methodology.
As this subjective element is not described very well this make the tests non reproduceable to an extent.
Does this mean that any tester has to ask David what settings he used to test the products be sure that the methodology is the same?
As for the test results David himself never presented those as absolutely infallible.
It should be noted that the testing programs are not perfect and in many cases they use methods, that are not reliable on 100%, to recognize whether the tested system passes or failed the test. This means that it might happen that the testing program reports that the tested system passed the test even if it failed, this is called a false positive result. The official result of the test is always set by an experienced human tester in order to filter false results. The opposite situations of false negative results should be rare but are also eliminated by the tester.
There is another thing to mention. This new methodology has only one rationale behind it:
Firewall Challenge is a project that replaces our older project Window Personal Firewall Analysis and its subproject Leak-testing. As a part of Window Personal Firewall Analysis project we have deeply analysed security products but we found out soon that such a testing was extremely time consuming. It was not possible to test as many products as we wanted to. On the other hand, Leak-testing seemed to be a very easy way how to test many products in reasonable time. However, Leak-testing is not able to cover many of the important features of the desktop security products. We have decided to combine the simplicity and effectivity of Leak-testing with the scope of our deeper analyses and created this project – Firewall Challenge.
No doubt that the older methodology required a lot of efforts as it was time-consuming and David did all that for free too. :-TU
That was one of the reason Matousec was regarded as a top notch firewall security test reference site so far.
The new methodology however don’t even require to run the full suite of tests if a product don’t pass a level. This is done to reduce the workload (however it may be that a product is able to pass the other levels even if it wasn’t considered eligible). I guess something like this could be accepted only when the products are tested for free but the methodology has to be the same for the paid test products too.
Something like this make it looks David is only expecting to take free tests but If he mention that this has to be regarded as a commercial service then IMHO would be best to factor free tests in his business balance and take only full tests.
What’s wrong about testmypcsecurity initiative?
Is it wrong to make all users able to test their own products?
Do all users have to rely on a test result score without even reading the methodology description because they trust respected individuals in the industry? (No way that David is encouraging something like that)
Does this mean we don’t have even try to configure our firewall by ourselves and rely on a respected individual?
Security shouldn’t be perceived as some specialistic field at all.
There is a lot of people who lack even baseline concepts due to this misconception. Entrusting the users with the task of testing their products is one step further toward a better security awareness.
The issue was we never “tarted up” our code to make us look better by passing this or that test! We built our security product with the understanding of current threats and made sure our users were secured from these threats! Now, with Matousec funfair this philosophy was forced to change and we had to fix (ok it only took us less than an hour to put that fix into code of course plus QA etc…) but we did this NOT because we thought that would provide material additional security, but because of the Matousec situation eg: Marketing gimmick.
The point is: We build security products, our philosophy is to build a top notch security product to “secure our users against threats”. Our priorities are to protect against malware and NOT to pass some tests for marketing gains! Our instructions to our developers is not “come on guys lets figure out how we can pass these tests” but “to protect our users from threats”. Egemen would not talk to me if this was our strategy! Our belief is that if we build our security products to offer the best security possible, everything else will follow. We do NOT and never have had any strategy to go out of our way to pass this test or that test so that we can use it as a marketing gimmick!
So I do understand why you wanted us to test, and we have been pushed into corner by the gimmicky nature of what has happened and made an exception to “ask for test for marketing purposes” as passing those tests has no material impact on user security.
As I pointed out in my other email, I am confident that David will learn and improve his process as per our discussion with him. David did explain that he did not intend this to be a gimmicky funfair and this was a side affect of new methods he put in place and he is reviewing them to improve the methodology. So after showing my dissatisfaction if we can have a better testing method, than I consider my efforts to be worthwhile for our end users and any vendor that utilises David’s services.
In above spirit I sent mail to Matousec to reconsider Kill5 test as proper way of testing FW.
My mail:
Mr. David Matoušek
I don’t understand reason of testing with Kill5 test in Level8 of Firewall challenge suite of tests.
I really think that “end task” termination is not security issue and should not taking it seriously, it is perfectly “legal” and proper way of terminating “appz.”, of course if you can defend that API properly against malicious (virii) misuse.
Your tests (in my opinion) should test quality of how to defend windows functionality and its proper work not sustain it, “End task” should be proper way of terminating all applications even firewalls (as I stated above).
Please reconsider this test and impact of it in default FW settings (FW malfunction, inability to proper close malfunctioning FW etc.)
Thanks for your time reading this letter.
salmonela
P.S. sorry for bad English
…and reply…
Hello,
Thank you for your email.
Termination tests verifies whether the tested product is able to defend itself from being terminated by malicious software. Kill5 implements one of the techniques that can be used to terminate the processes of the personal firewall. Kill1, Kill2, … attempt to do the same thing with different API. Kill5 is in level 8 only because it seems that many personal firewalls do have problems to prevent misusing its technique.
From our point of view, Kill5 is not something special, it is just one technique that can be used to terminate firewall’s processes.
Comodo was the best and OA had that bug for a very long time, from what I hear, and they only fixed it when the test results were not favorable to them. In my view, this make Comodo the best, so it should be listed first. So, tell David all this when he updates the score.