Comodo will be there always and will not be biased…
Well, we need to assemble a team. To see who does what.
Who will handle the testing, uploading to youtube, creating PDF document, maintaining website, linking, making graphics… This discussion continues here…
from here: https://forums.comodo.com/empty-t73852.15.html
OK, we need:
1: a website. Someone good with graphics and knowledge how to set up free and clean website, professionally looking.
2: Documents… We need CLEAR and strict guidelines and rules to follow in testing and possibly scoring of a product
I will be checking out AMTSO…
3: Community Manager. He will maintain communication with the community and take notes what people think, what they want/need, how to improve…
4: Research and Developement department. He will be responsible for hunting for malware [NEW, OLD, DOS, WINDOWS, trojans, rogues, rootkits]…
5: Project Manager. He will be responsible for scheduling of tests so it all goes on time as it should and to organize which product will be tested and deadline requirements to meet
6: Video Manager. I need him to make an intro that looks cool and to edit video into professional looking… Also, he will be responsible for Youtube comments and clean up spam or offensive posts. We will need a dedicated Youtube channel for this
7: Designated Supervisor: A moderator who will make sure my tests are fair and not biased. He will be a sort of Team Leader and to make sure there’s no power struggle and everything’s according to plan!
His word is stronger than mine and I play by his rules, which, in turn, will have to be in according to Operations Plan that will be compiled as a guideline to testing…
8: Tester: He’s a person who… tests?
Any suggestions, fellas?! Let’s do this! UPDATES
29.06.2011
Testing methodology draft v1 uploaded…
Thanks Valentin N for making it so nice and readable, you rock!!! :-TU :-TU :-TU
03.07.2011
Organization Chart Diagram added for better view of organizational structure, roles and responsibilities
05.07.2011
Test Sample Report uploaded, version 1
Remember I am your third team member. 8)
First - CIS.FAN
Second - SOLARLYNX (I believe)
Third - W-E-V (me, of course… if solarlynx does not confirm, I become the second )
I can provide technology (servers, domains, design, etc).
And of course run some tests.
Well, I believe if this is going to be a neutral community team-based testing, CIS.FAN will have to change his name. J/K
GakunGak, have you thought on any type of certification?
I believe according to the community point of view, its better if no certifications are given (plus the margin of error as we have witnessed on AV-TEST.ORG) because the community its going to test the products and tell people where they are good and where they failed. That way is more neutral, without anyone giving self-opinions.
And if any certification has to be given, I believe the users who visits and review the results are the one who should give the certification (like if a number of good reviews for a testing result is reached).
If there’s going a be a certification, I would want if strict judging is used… I said if… It is a complex thing and not to be used lightly…
What I tend to like, personally, is the PROS, CONS and Conclussion, like:
Pros: Light on the system and fast on demand scanning, rich selection of options
Cons: dependent on cloud, poor disinfection and cleanup
Conslusion: Good for low-end machines always connected to the internet, bad protection bla bla bla…
What do you think is better?
Interesting concept, but votes could be faked [voting with proxy addresses, spamming etc…]…
Maybe if there’s a captcha or challenge system in place to prevent bots from messing it up
I believe we need something different.
No personal concepts, or community addressed concepts.
For example, people want to see if a product failed or not. The testing should be for the whole suite, not leaving something out just because it is an automated test and certain things cannot be included. Thats where human testing comes in handy.
And that if gets a serious certification like you mention, it really have to pull out a good protection, like 100%.
Thats strict judging.
And a small percentage given by good reviews from home users (not votes).
So how do we measure this? Failed because of one malware breach, half, more than a half?
For example, Comodo quarantines malware in the sandbox but still lets it run, infecting empty space until a restart is initiated. Some people might interpret this as a breach and some would not as the malware is contained, but still live. Same with Sandboxie…
How about:
1: Our review: 8/10
2: Readers review: 7/10 [based on xx votes]
Also, what do you think about system hardening tools like EMET, System-Protect, DropMyRights, virtualization like Sandboxie, Shadow Defender etc?
I would also like to test custom built security, like A antivirus and B firewall with bb or hips…
People could suggest what to use? Those fine gentlemen at wilders would like that
Thats precisely what is needed.
You cant compare sandbox with cloud protection, of course.
So at the end, is how a suite responded to a test, with all its includes.
In other words, Norton did this because it includes 1 and 2 protection.
Kaspersky did this, because it includes 1, 2 and 3 protection.
The test should include how much a user has to play with the suite.
I think it must aim not only ITs, but home users. People that know they must have protection in their PCs, but know nothing about technical stuffs. I bet that a high percentage of people buying protection, they dont know how it works. They just know what its included (or even sometimes because the trial period of the pre-installed security software expired). ???
Very well, and I agree on all…
Shall we include a testing on BETA’s? Obviously a non-finished product cannot be certified or measured, but maybe something like a small review like new features are this and that, and maybe a malware test just for a test drive or something…
I plan on working on test methodology and would need all the help I can get tomorrow about this time…
I will make a draft and set up guidelines and fill details later and present it for final approval from ALL team members…
I guess BETAS would be just fine… for the ITs.
There is always someone who wants to know how its going to work their predilection product.
It is just matter to see how available the beta of the products are. And how much testers are available.
Whatever I can help with, let me know. I will like to support this community for sure.
Tomorrow I start work on the rules and guidelines…
I hope to finish it on Sunday so I can present it to review…
Next week, we start the project for website, brainstorming layout and what to use, making Youtube Channel and the rest…
If all goes well, we could see our first pilot test mid-july…
Then we go ballistic…
For website, we need shoutbox/chat integration, comments section, forum if needs be, social networking integration [share button]…
If the site needs to pay for traffic, maybe we could have some sort of ad support system…
I am a graphic effects ■■■■■ so I would also like to see something like Securelist | Kaspersky’s threat research and reports
and like my website www.xrl.us/gakungak
Many testers switch to Win7. I kinda want to stick to XP for a while because a lot of people
still use it… Or to make a switch to 7? Se7en is highly stable and resistant to viruses and I want malware to ■■■■ the system hard B-)
Thanks, boss! :-TU
I was inspired by your setting-the-testing-standard and AMTSO a while back so I thought, why not make a little initiative to see how it works out…
I have ambitions to make this as big as resources and time allows…
Unlike those organisations where they just provide the PDF document, I want to give:
1: PDF with detailed description of what happened
2: Video, so everyone can see what and how the test went
3: Discussion, where people can say their opinion and bring their criticism to the table towards a program or the tester, also share ideas and how to improve something…
)
J/K