Who needs av-comparatives.org with their static tests? I really believe that they consider themselves as God’s gift to testing, God forbid, obviously. Here comes Nsslabs with comparative Dynamic testing of security products that follow AMTSO guidelines. Some will be surprised of the results and some will be disappointed or shocked.
Here are the respective links:
Main website:
Dynamic Comparative test:
Click on Anti-Malware products and then click on Consumer Endpoint Protection Products Q3 to read the test results.
Why do you think NSS are AMTSO-compliant? (Yes, I know they’ve been AMTSO members.) There’s something very off-color about a company that will only discuss its methodology with companies that pay them a consultancy fee.
“NSS Labs is an active member of the Anti-Malware Testing Standards Organization (AMTSO). All testing is conducted closely following the AMTSO guidelines.”
Here is what NSS LABS states on page 3 of their consumer test results:
"The objective of these procedures is to provide a thorough, real-world test of the malware protection in a controlled and verifiable manner. Given the speed with which new threats arrive and spread through the Internet, legacy testing techniques are no longer a relevant measure of a product’s capabilities.
•
Tests that rely on Wildlist samples or that presume a 100% score objective are not measuring current threat protection. Malware must be fresh and represent the current distribution on the internet, not the malware family taxonomy.
•
Tests that do not provide access to the vendor reputation systems during testing unfairly disadvantage more advanced products by denying them a key component in protection.
• Static testing or on-demand scanning generally does not enable the most robust detection techniques. And even dynamic testing alone is insufficient given the increasing reliance on real-time, in-the-cloud reputation systems. A combination of reputation/download and execution analysis provides the best analysis of real-world product capabilities."
The bold font is my own emphasis.
PS: The quote from the NSSLABS is purely for educational purpose and not to infringe on NSSLABS copyrights.
You can only thing AV-Comparatives static tests as stupid and obsolete if you don’t understand them. But for others, they are a great source of information.
I think a lot of people can agree with me when I say this: Static tests are greatly exaggerated in their importance. To me since malware is not executed how do I really know that my security apparatus can handle it for real? Thus, in my opinion, Dynamic testing is of a greater value than static testing.
Moreover could you rephrase your statement when you said: “You can only thing AV-Comparatives static tests as stupid and obsolete if you don’t understand them.” what do you mean by “you can only thing”?
My only conclusion is that you did not write what you thought or you did not know how to write it (whatever you were thinking) properly.
Static tests can be revealing.
Dynamic tests are surely more so.
But there’s always the doubt as to how independent the test site is.
We (users) have to make our own unscientific statistics of every thing we read and watch. And then we add our little experiences.
One of my little experiences: about one month ago I went to my local library with a USB pen and brought it back infected. As soon as I attached it to my laptop the resident AV (MSE) found one malicious file trying to get through. It told me it had come from F: so, after it cleaned it I went to C:/F: and immediately it found and cleaned the main infection.
I’m sure there are others like me that stick to this or that AV because of things like this… until we get a bad experience and go for something else.
I find it strange that they rate Trend Micro the highest and Matousec says Trend Micro provides no protection. I guess Matousec only rates the firewall portion of the Trend Micro suite, which has always been it’s weak point, where NSS concentrates on detection which is mainly done by the AV component where Trend Micro has always been strong. Overall, the full suite offered by Trend Micro falls way behind Norton.
I want to see how NSS will rate Comodo, if they ever test it. I want to see the full suite tested though and rated against the other similar offerings.
Isn’t refusing to give information except when paid a consultancy fee a breach of principle 3? And yes, dynamic testing is a better reflection in principle of how current threats work, but a good static test is better than a bad dynamic test.
Well, I wouldn’t trust NSS at all after they claimed that IE8 was the safest browser - see the Opera blog on how the stats given are wildly inaccurate:
Actually, it is the safest browser in terms of malware blocking. Love it or hate it, Microsoft has done a wonderful job in improving Internet Explorer.
Just think of it, and I really do not want to be the devil’s advocate; do you think that Opera would talk of its competitors in a good light? It is funny for me to realize that Opera did not dispute that IE 8 blocks more malware; Opera’s beef has to do with the assertion that NSSLABS manipulate statistics. In the same token, Opera did not prove either that without these “alleged” manipulations the Opera Browser would have came on top.
Consequently, since Opera did not directly deny that IE 8 blocks more malware than it; and moreover, since there was no concrete evidence to the contrary, therefore I have to believe that NSSLABS browser test results were true.
Actually, it is the safest browser in terms of malware blocking. Love it or hate it, Microsoft has done a wonderful job in improving Internet Explorer.
Agreed
since Opera did not directly deny that IE 8 blocks more malware than it; and moreover, since there was no concrete evidence to the contrary, therefore I have to believe that NSSLABS browser test results were true.
Agreed again
Quote from: Dch48 on Today at 02:06:09 AM
I find it strange that they rate Trend Micro the highest and Matousec says Trend Micro provides no protection. I guess Matousec only rates the firewall portion of the Trend Micro suite, which has always been it’s weak point, where NSS concentrates on detection which is mainly done by the AV component where Trend Micro has always been strong. .
Matousec as an entity does not use malware, only penetration techniques known as leak tests.
So that means they test Firewall technologies,as I said.