But with all due respect,who will determine compliance?
I believe that AMTSO, who are drafting the guidelines for dynamic testing, also have a compliance testing review procedure.
For a test to be fully AMTSO compliant, the testing methodology needs to conform to the AMTSO guidelines AND the processes used throughout the test can be examined, commented on, approved and certified as compliant by AMTSO itself.
David Hartley (from ESET) makes a telling quote on the ESET site
I’d guess (or hope) that eventually you’ll be able to check on the AMTSO web site as to whether a given tester has completed the self-assessment process (when it actually exists). Even then, since AMTSO is not a certification body (not yet, anyway – who knows what will happen further down the line?), it probably won’t mean that any specific test from that tester or organization is compliant. Unless, of course, an analysis from the Review Analysis Board has determined that it is.
Even if the tester is a member of AMTSO, that doesn’t mean at all that they have the automatic endorsement of the organization for their testing. Indeed, they’re at least as liable as anyone else to have their adherence to the AMTSO principles scrutinized by the Review Analysis Board.
Rather than asking AV companies why they won't submit to this or that test, wouldn't it be smarter to ask why the current crop of testing methods aren't AMTSO compliant?
The following testing organisations are already members of AMTSO, yet none of them have an AMTSO compliant test suite;
Cascadia Labs
hispasec.com
ICSA Labs
NSS Labs
Virus Bulletin
West Coast Labs
These are all exceptionally capable companies and have consistently been seen as reliable testers (in the context of past testing methods). Maybe, creating AMTSO compliant tests is not as simple as we would like to think it is.
Ewen :-)
Author