Community team-based testing and evaluation of security products...

[Guest]

[kagun]

Where's Comodo?
That's interesting. How to help? I ain't any tech. BTW if you use the tests on your website it will give it good plus.

Comodo will be there always and will not be biased...
Well, we need to assemble a team. To see who does what.
Who will handle the testing, uploading to youtube, creating PDF document, maintaining website, linking, making graphics.... This discussion continues here...
from here:
https://forums.comodo.com/empty-t73852.15.html

OK, we need:
1: a website. Someone good with graphics and knowledge how to set up free and clean website, professionally looking.
2: Documents... We need CLEAR and strict guidelines and rules to follow in testing and possibly scoring of a product
I will be checking out AMTSO...
3: Community Manager. He will maintain communication with the community and take notes what people think, what they want/need, how to improve.....
4: Research and Developement department. He will be responsible for hunting for malware [NEW, OLD, DOS, WINDOWS, trojans, rogues, rootkits]...
5: Project Manager. He will be responsible for scheduling of tests so it all goes on time as it should and to organize which product will be tested and deadline requirements to meet
6: Video Manager. I need him to make an intro that looks cool and to edit video into professional looking... Also, he will be responsible for Youtube comments and clean up spam or offensive posts. We will need a dedicated Youtube channel for this
7: Designated Supervisor: A moderator who will make sure my tests are fair and not biased. He will be a sort of Team Leader and to make sure there's no power struggle and everything's according to plan!
His word is stronger than mine and I play by his rules, which, in turn, will have to be in according to Operations Plan that will be compiled as a guideline to testing...
8: Tester: He's a person who..... tests?

Any suggestions, fellas?! Let's do this!  
UPDATES

29.06.2011
Testing methodology draft v1 uploaded....
Thanks Valentin N for making it so nice and readable, you rock!!!  
03.07.2011
Organization Chart Diagram added for better view of organizational structure, roles and responsibilities
05.07.2011
Test Sample Report uploaded, version 1

[Solarlynx]

So make detailed job opportunities and we can choose.

[w-e-v]

COUNT ME IN GAKUNGAK.

Remember I am your third team member. 
First - CIS.FAN
Second - SOLARLYNX (I believe)
Third - W-E-V (me, of course... if solarlynx does not confirm, I become the second )

I can provide technology (servers, domains, design, etc).
And of course run some tests.

Really, count me in.

[kagun]

first post updated!
If you know someone talented, count them in 

[w-e-v]

1: a website. Someone good with graphics and knowledge how to set up free and clean website, professionally looking.

I can provide that. Thats what my company does anyway,

3: Community Manager. He will maintain communication with the community and take notes what people think, what they want/need, how to improve.....

Count me in with that. Research and Customer Care, Marketing and International Communication/Relations.

5: Project Manager. He will be responsible for scheduling of tests so it all goes on time as it should and to organize which product will be tested and deadline requirements to meet

I can do this with someone else. 

[kagun]

Many thanks, w-e-v, this is WAY FASTER than I expected!!! 

I will be updating first post in case I forgot something... 

[Solarlynx]

Many thanks, w-e-v, this is WAY FASTER than I expected!!! 

I will be updating first post in case I forgot something... 

Only mark that you've updated in regular posts. For us to know that.

[kagun]

Only mark that you've updated in regular posts. For us to know that.

Good idea!!!!! 

[w-e-v]

Well, I believe if this is going to be a neutral community team-based testing, CIS.FAN will have to change his name.    J/K

GakunGak, have you thought on any type of certification?
I believe according to the community point of view, its better if no certifications are given (plus the margin of error as we have witnessed on AV-TEST.ORG) because the community its going to test the products and tell people where they are good and where they failed. That way is more neutral, without anyone giving self-opinions.

And if any certification has to be given, I believe the users who visits and review the results are the one who should give the certification (like if a number of good reviews for a testing result is reached).

This is only a suggestion.

[kagun]

If there's going a be a certification, I would want if strict judging is used... I said if.... It is a complex thing and not to be used lightly...
What I tend to like, personally, is the PROS, CONS and Conclussion, like:
Pros: Light on the system and fast on demand scanning, rich selection of options
Cons: dependent on cloud, poor disinfection and cleanup
Conslusion: Good for low-end machines always connected to the internet, bad protection bla bla bla....

What do you think is better?

And if any certification has to be given, I believe the users who visits and review the results are the one who should give the certification (like if a number of good reviews for a testing result is reached).
This is only a suggestion.

Interesting concept, but votes could be faked [voting with proxy addresses, spamming etc....]...
Maybe if there's a captcha or challenge system in place to prevent bots from messing it up 

[w-e-v]

What I tend to like, personally, is the PROS, CONS and Conclussion, like:
Pros: Light on the system and fast on demand scanning, rich selection of options
Cons: dependent on cloud, poor disinfection and cleanup
Conslusion: Good for low-end machines always connected to the internet, bad protection bla bla bla....

What do you think is better?

That would be something like the way PCMAG.COM reviews every year:
http://www.pcmag.com/article2/0,2817,2381733,00.asp
http://www.pcmag.com/article2/0,2817,2368876,00.asp
http://www.pcmag.com/article2/0,2817,2367794,00.asp

I believe we need something different.
No personal concepts, or community addressed concepts.
For example, people want to see if a product failed or not. The testing should be for the whole suite, not leaving something out just because it is an automated test and certain things cannot be included. Thats where human testing comes in handy.

And that if gets a serious certification like you mention, it really have to pull out a good protection, like 100%.
Thats strict judging. 

And a small percentage given by good reviews from home users (not votes).

[kagun]

For example, people want to see if a product failed or not.

So how do we measure this? Failed because of one malware breach, half, more than a half?
For example, Comodo quarantines malware in the sandbox but still lets it run, infecting empty space until a restart is initiated. Some people might interpret this as a breach and some would not as the malware is contained, but still live. Same with Sandboxie....

And that if gets a serious certification like you mention, it really have to pull out a good protection, like 100%.
Thats strict judging.  
And a small percentage given by good reviews from home users (not votes).

How about:
1: Our review: 8/10
2: Readers review: 7/10 [based on xx votes]

[kagun]

Also, what do you think about system hardening tools like EMET, System-Protect, DropMyRights, virtualization like Sandboxie, Shadow Defender etc?

I would also like to test custom built security, like A antivirus and B firewall with bb or hips....
People could suggest what to use? Those fine gentlemen at wilders would like that 

[w-e-v]

So how do we measure this? Failed because of one malware breach, half, more than a half?
For example, Comodo quarantines malware in the sandbox but still lets it run, infecting empty space until a restart is initiated. Some people might interpret this as a breach and some would not as the malware is contained, but still live. Same with Sandboxie....

Thats precisely what is needed.
You cant compare sandbox with cloud protection, of course.
So at the end, is how a suite responded to a test, with all its includes.

In other words, Norton did this because it includes 1 and 2 protection.
Kaspersky did this, because it includes 1, 2 and 3 protection.

The test should include how much a user has to play with the suite.

I think it must aim not only ITs, but home users. People that know they must have protection in their PCs, but know nothing about technical stuffs. I bet that a high percentage of people buying protection, they dont know how it works. They just know what its included (or even sometimes because the trial period of the pre-installed security software expired). 

[w-e-v]

Also, what do you think about system hardening tools like EMET, System-Protect, DropMyRights, virtualization like Sandboxie, Shadow Defender etc?

I would also like to test custom built security, like A antivirus and B firewall with bb or hips....
People could suggest what to use? Those fine gentlemen at wilders would like that 

Obviously there should be a custom test for all other tools.
And compare each one of the corresponding to the same tool.

All I am saying are suggestions, but what really matters is what the users are going to say at the end. 

[Tags:]