Petition for CIS to be Tested by NSSLABS

Comodo Internet Security - CIS News / Announcements / Feedback - CIS
21

Does anybody know if any other organization so far provided a test claiming or implying compliance like NSSLABS did?

Because David Harley from AMTSO board of directors later wrote about AMTSO compliance with particular regards to tests claiming compliance or using phrasing that implies compliance, such as “following AMTSO principles”

Though he pointed out that AMTSO hasn’t formally defined yet what “AMTSO compliance” actually is , he made a point that claims of “compliance” doesn’t mean that they are proven to comply or have the blessing of AMTSO. and that Even if the tester is a member of AMTSO, that doesn’t mean at all that they have the automatic endorsement of the organization for their testing. Indeed, they’re at least as liable as anyone else to have their adherence to the AMTSO principles scrutinized by the Review Analysis Board.

Though it appear to be rumors that relegate AMTSO review as an optional requirement it doesn’t appear that claims would be enough.

Obviously it’s up to the whole of AMTSO to come with a formal definition of “compliance” though it would make more sense to validate tests before publication and not thereafter delegating to anyone other than the tester the submission AMTSO boards for peer review.

I would guess if any tester makes a point of claiming that his/her methodology is conformant with the AMTSO guidelines, his/her tests should be reviewed (or pending review) by AMTSO Review Analysis board.

If not it would appear that a test is “compliant” if anybody claims it so even more if no one submitted the test for review. ???
Though anyone can claim compliance it looks that ATM no one is willing to have their test reviewed by AMTSO review board before publication. ???

Nevertheless it appears that “AMTSO compliance” and related procedures are still ongoing debate and there is no telling what the final definition of the whole organization will be.

Guess it ought to be clarified at the end if an entire board that review tests for AMTSO compliance can be superseded by any single tester with few lines of text whereas researchers of any other fields are used to willingly submit (by themselves) their works for Peer review before publication.

In addition to the current result of providing “advices”, as an organization AMTSO would be able to fulfill the above role and have AV tests finally take one step forward by factual procedures.

Whereas such yet to come agreement will undoubtedly reflect what value AMTSO, its boards and its “compliance” is actually going to hold for the time being.

22

Hi Melih

Here is what AV-comparatives, a member of the AMTSO, stated on their forums with respect to AMTSO compliant test:

AMTSO has a document with 9 “fundamental principles of testing”. You can find it on their website. Basically, a test needs to follow those fundamental principles to be “compliant” (though that term has no formal definition at the moment). This means that ANY kind of test, as long as it follows those principles, could be considered in some sense compliant. You will find many testers that already follow those principles. And many tests which do not. Therefore, I advise readers to check out the principles and check if a test is in their own eyes compliant or not.

Now Comodo is also a member of the AMTSO, do you Melih disagree with AV-comparatives in that respect? If you do disagree then let the AMTSO review board know that. Please be proactive here, no more subterfuge, tricks, technicalities, and smoke and mirrors, please.

Reference:

http://www.av-comparatives.org/forum/index.php?page=Thread&threadID=949

Peace.

23

It looks AV-C will have to make their point heard at AMTSO meetings as well…

… as to what sense “AMTSO compliant” is possibly going to have.

If anybody wonders yet it will be “some sense”…

…agreed by AMTSO as a whole and formally specified by means of definitions and related procedures (ATM not yet agreed/provided).

And will probably listed in AMTSO’s Documents and Principles section along with:
AMTSO Fundamental Principles of Testing as approved by the AMTSO meeting held in Oxford 31st October 2008.
AMTSO Best Practices for Dynamic Testing as approved by the AMTSO meeting held in Oxford 31st October 2008
AMTSO Best Practices for validation of samples as approved by the AMTSO meeting held in Budapest 7th May 2009
AMTSO Best Practices for Testing In-the-Cloud Security Products as approved by the AMTSO meeting held in Budapest 7th May 2009
AMTSO Analysis of Reviews Process as approved by the AMTSO meeting held in Budapest 7th May 2009
AMTSO Guidelines for testing Network Based Security Products as approved by the AMTSO meeting held in Prague 13th October 2009
AMTSO Issues involved in the “creation” of samples for testing as approved by the AMTSO meeting held in Prague 13th October 2009

24

Why don’t u ask nsslab all these questions? They too are amtso members and they are the testing organisation. We are not!

melih

25

Au contraire my friend the burden of proof is on you, since you are the one who doubt their claim. To prove your own self right you should be the one who must report them to the AMTSO review board. Moreover, you are the one who talked to them before; what transpired in your conversation with them? Why did you contact them in the first place since you already know that they did not submit their test to the AMTSO. Is that one of your tricks?

You still have not answered my question: “I was told that the AMTSO review board will not initiate any direct review of an AMTSO dynamic test; that test must be reported to it as not fulfilling the AMTSO guidelines, then and only then it will review and eventually approve or reject any reported dynamic test after its findings. Is it true or is it false?”

Please with all due respect Melih answer the question or maybe you already know the answer. I may be wrong about this but I’m thinking that If the AMTSO does not directly on its own review test(s) unless that test is reported to its review board for a review why would you say I’m waiting for an AMSTO review board test if you do not report that test to the AMTSO review board.

In a nutshell your excuse or technicality is to say that you are waiting for an AMTSO review board test while you know better. Once again you are hiding behind technicalities. The only thing you have to do is to come out of hiding and answer my question, please. No more technicalities.

Peace.

26

Why professional AV testers should be entitled to neglect an established practice any other researcher is willingly accustomed to in their area of expertise? ???

Submitting a test for peer review should be first and foremost its tester responsibility and there is no reason to believe that a professional cannot submit his/her tests to AMTSO review board before publication. :-La

In a nutshell, if there will be any trick or technicality that will allow any researcher to forsake such otherwise established practice for AV tests alone it would be up to AMTSO to approve in a forthcoming meeting regarding “compliance”.

Though it is interesting to notice that even before a related meeting will be held there are already rumors twisting something like a mandatory peer review to bring forth a mandatory “Default-Allow AV-tests Compliance” leveraging on a controversial anybody other than the tester “burden of proof” 88)

It could be that hobbyists who are not AMTSO members might publish their AV test neglecting AMTSO review board but why professional AV testers would be willing to claim “compliance” but would not be willing to submit their test for review before publication even now that there is a board of peers from the same organization (AMTSO) they are affiliated meant for that purpose? ???

Is “compliance” gong to be the belief of any individual tester according to his/her interpretation of 9 generic principles alone or something more transparent like an official review before publication according to forthcoming “compliance” criteria and procedures?

Would it be really possible to bend transparency to the point that only “reported” tests ought to be reviewed for a soon to be defined “compliance”?
Big answers would surely be the ones that are going to be provided by forthcoming AMTSO meeting/s…

27

This is getting funny…
If you claim…you prove…
you don’t expect others to prove your claim for you!

Jaki…you are asking us questions about amtso and nss… We are Comodo… pls go ask these questions about NSS tests which are claimed to be AMTSO compliant to both NSS and AMTSO… And stop asking us to prove a 3rd party test on their behalf even though there are channels for them to do that themselves…

melih

28

You, obviously misread or misunderstood what I said. I did not ask you pertinently about about the inner workings of the NSSLABS. I was asking about how the AMTSO review board works. I asked you a very simple true or false question and there are no three ways to answer it. I hate to repeat myself but here the question again. Can the AMTSO review board on its own review a purported dynamic test without that test being reported to it as not having entirely or at all any AMTSO dynamic principles merit?

Here is your trick again Melih: You keep saying "I’m waiting for an approved AMTSO review board test. But what does an approved AMTSO review board test mean? Well you never say and here is why: you know very well that the unwary forum user would never care to check your statements to see if they check in real-time beyond the smoke and mirrors. If the user does not know that the AMTSO review board does not directly review test; therefore that user would take your word for it.

Consequently, the user would say in fairness to Melih let us wait. Well then CIS v4, v5 or even v6 will come and go and you will still say I’m waiting for an AMTSO review board test. The first principle of the AMTSO is to protect the public and I think you agree since Comodo is a member of the AMTSO; thus you should take it upon yourself, in order to protect the public, to report the NSSLABS test to the AMTSO review board and walk the walk, more importantly desist with the smoke and mirrors.

Peace.

29

I agree with you totally jacki,
From my perspective i see above topic and these things come to my mind :=
why is comodo so reluctant to go for these tests ?
Other top rated companies are going into test? why it seems that we are going round & round in circles?
If its community driven decision more people willing for comodo to participate in test please see it again, if tests are not done properly i dont think many of the top rated antimalware companies would be willing to go for such test.

30

Jaki

with all due respect you are now crossing the line by your accusations which are all based on your misunderstanding of the AMTSO guidelines and the review board.

There are many tests people do from Matousec to the recent Russian AV test and we always come as one of the top players! If you want tests, there are MANY out there that tested CIS…We still believe in AMTSO compliant tests, period!

Melih

31

I’m not crossing the line just by asking a simple question. So to you when you are asked a question that’s crossing the line, how can that be? Moreover, if you think that I do not understand the AMTSO guidlines, please educate me first by answering my question, please.

Well, well I really do not believe that I would live to a day where you would give your tacit support to tests that are not approved by the AMTSO review board such as matousec and the Russian anti-malre.ru. Remember it is your own words and your own admission, because CIS came on top. Ladies and gentlemen Melih is on board and thus the motion has been carried unanimously.

Here is my logic now, since Melih approves of non AMTSO review board tests like matousec and anti-maware.ru and yes he took credit for them; consequently, he has finally relented to allow CIS to be tested by NSSLABS and av-comnparatives. Hipip Hooray… ;D

Peace.

32

Jaki

With all due respect, I don’t think you are understanding what I am writing or you purposely choose to misinterpret what i write.

1)you are using words like smoke and mirror, trick etc. This is utterly disrespectful
2)you are misunderstanding what AMTSO review board is (here is the document for you to read.
3)you are expecting Comodo to prove NSS tests are or are not compliant with AMTSO (it has nothing to do with us, take it up with either NSS or AMTSO).
4)you are misinterpreting, what I believe is a very clear statement that we support AMTSO compliant tests even though there are other public tests for Comodo.

AMTSO exists for a reason! All these companies have become a member of AMTSO for a reason! We believe in AMTSO compliant tests!

Melih

33

@Melih

first of all, you should not link to the PDF, bypassing the AMTSO license Agreement. Please remove it. The AMTSO license agreement has been introduced on purpose.

second, you, as a member of AMTSO, may consider to show up at some of the meetings, as it seems you are misinterpreting some of the work AMTSO has done so far and the involved procedures.

If you read the paper you linked to, you will see that by now, when the RAB reviews a test, all they do is checking if the nine fundamental principles have been accomplished (see example 8.d. in the PDF). If they are accomplished, the test is, as you say, “AMTSO compliant” (althought it seems you introduced this term by yourself). This can as well apply also to static tests. So, you should better say you want “a dynamic test which follows the fundamental principles”.

As it has been pointed out, AMTSO can not review all tests out there, especially not in advance. You may have read that some tests are already considered under review since some time. Whether the test in question in this thread is compliant or not, is up to the RAB to evaluate (and should - as Melih said - not be discussed here).

@all: for a better understanding of AMTSO, the review analysis, etc., please read the AMTSO website/documents, the ESET blog post of David or the post on the AVC forum.

34

If your statements are true please educate me about the AMTSO then; since Comodo is one of its members, at least you must know its fundamentals. Moreover, you still have not answered my simple question. My question is comprised of two possible answers true or false. Really, it is very simple.

Based upon your previous post I must conclude that you have indeed confirmed that you do condone tests that are NOT approved by the AMTSO review board, however selective. Sincerely, I do not want you to take offense when I’m saying this: Please Melih answer the question.

Peace.

35

ah ha, out of the blue mar56 appears, thanks. Anyway it seems to be that av-comparatives was right:

"
Quoted
AMTSO has a document with 9 “fundamental principles of testing”. You can find it on their website. Basically, a test needs to follow those fundamental principles to be “compliant” (though that term has no formal definition at the moment). This means that ANY kind of test, as long as it follows those 9 principles, can be considered compliant. You will find many testers that already follow those principles. And many tests which do not. Therefore, I advise readers to check out the principles and check if a test is in their own eyes compliant or not."

reference:

http://www.av-comparatives.org/forum/index.php?page=Thread&threadID=949

Peace.

36

Thanks for the heads up. I have removed the link. But fyi: i got the link from Google (3rd link from top with a pdf doc). Perhaps you should get in touch with Google and ask them to remove the link as this might be violating AMTSO licensing. Btw is Google violating the licensing? We did send a representative early on, but the team is doing great work we didn’t see that we could add any more to the good work being carried out.

Yes for clarification: We are looking for Dynamic Tests which are AMTSO Compliant. But as you will appreciate you can’t call a Static Test as AMTSO compliant if it takes an Anti Malware product and only tests one aspect of the anti-malware product without measuring the effectiveness and performance of the anti-malware product in a balanced way. So its very much dependent on what is being tested. You have to apply relevant tests to relevant products! So I would respectfully still say saying AMTSO Compliant test for CIS should be sufficient as doing Static tests on CIS would not be "measure the anti-malware product (CIS) in a balanced way. But you know what…I really don’t care :slight_smile: If you want to call it Dynamic Tests…happy to oblige… :slight_smile:

Melih

PS: mar56…why the secrecy? why not reveal who you are?

37

Hallo mar56, are you sure about that?

Can you comment on the 2nd (target) and 7th paragraph of procedures section of the AMTSO Analysis of Review Process document approved on May 7, 2009?

It states that the purpose is not limited to checking the 9 principles alone like you claimed but “AMTSO Guidelines and Principles as currently in effect”…

ref: AMTSO Guidelines and Principles currently in effect

The document also mention that the described process and procedures are a starting point and some changes might be expected.

Is the reduction of the criteria to be reviewed to the 9 principles alone like you claimed one of such expected improvements?
How much time will now take the RAB to check those reduced criteria?
Or was such improvement made to increase transparency to the public so they can easily and rapidly check compliance by themselves thus superseding the need for the RAB?

When a tester claims to be AMTSO compliant – and many have started to do that – or uses phrasing that implies compliance, such as "following AMTSO principles", what does that mean?

It looks David Harley focused his article on many testers who already claimed compliance or used phrasing that implied compliance such as “following AMTSO principles”.

How many compliance implying/claiming tests in total such testers made specifically available to the general public since May 7, 2009?
How many of of such tests were submitted to the RAB?
How many of such RAB requests filed for such tests were signed by their respective testers’?

What concerns me right now is that bitter experience suggests that if a tester makes a point of claiming that his methodology is conformant with the AMTSO [url=http://www.amtso.org/documents.html]guidelines[/url], quite a few people will accept that claim uncritically
David Harley made explicit mention about the AMTSO guidelines, did he mean the 9 principles alone or maybe the changes you mentioned post-date his article?

I guess nobody asked about all tests out there but only the ones explicitly claiming AMTSO compliance or using phrasing that imply such compliance.

What actually is the estimated monthly average number of such AMTSO implying/claiming tests specifically available to the general public?

Or was it meant that there are already so many publicly available AMTSO implying/claiming tests out there that the RAB cannot review them in a timely fashion even prioritizing the ones from AMTSO’s own members?

Besides it looks like some came to believe that a tester member of AMTSO cannot willingly submit his/her tests before publication, is there any AMTSO principle that prevent/forbid them to submit their own tests to the RAB before publication or eventually publish such test pending review taking care to publicly update the test to mention the RAB analysis results?

Thanks in advance for you replies.

38

If that is so Melih, would you submit CIS to NSSLABS for a “dynamic” test? I think it is quite clear that based upon my petition and Kyle’s most CIS users would like CIS to be tested. Would you defy the wishes of your own users or would you heed to our call?

Peace.

39

I wonder why the poll did mention NSSLABS and not simply AMTSO compliant tests?

Is the only know test implying compliance made by an AMTSO member since May 7, 2009 ?

AMTSO’s stance has been misinterpreted as meaning that dynamic testing is automatically compliant
Was compliance implied for such test because dynamic tests were considered automatically compliant or maybe it was implied so only according to the 9 principles alone ?

Besides were the criteria really reduced to 9 principles alone like some rumors have been claiming?

I wonder if the DIY compliance checks similar rumors have been spreading apply also to a recent dynamic test mentioned in these forums…

40

Hi Melih

You have the right to disagree with NSSLABS if you want to; that is certainly your own prerogative. However, if you doubt the NSSLABS AMTSO dynamic claims, then you also have the right to report it to the AMTSO review board. If you do not report it to the AMSTO that’s where I will have a problem with you as member of the AMTSO. You should not violate the AMTSO first principle and that’s what you will end-up doing if you do not convey your doubt(s) to the AMTSO review board with regard to the recent NSSLABS dynamic test.

Peace.