Poll

Should CIS be submitted to NSSLABS for an AMTSO compliant test?

Yes
No

Author Topic: Petition for CIS to be Tested by NSSLABS  (Read 14687 times)

Offline Jaki

  • Computer Security Testing Group
  • Comodo's Hero
  • *****
  • Posts: 550
Re: Petition for CIS to be Tested by NSSLABS
« Reply #30 on: November 11, 2009, 02:01:55 PM »
Jaki

with all due respect you are now crossing the line by your accusations which are all based on your misunderstanding of the AMTSO guidelines and the review board.

There are many tests people do from Matousec to the recent Russian AV test and we always come as one of the top players! If you want tests, there are MANY out there that tested CIS.....We still believe in AMTSO compliant tests, period!

Melih

I'm not crossing the line just by asking a simple question. So to you when you are asked a question that's crossing the line, how can that be? Moreover, if you think that I do not understand the AMTSO guidlines, please educate me first by answering my question, please.

Well, well I really do not believe that I would live to a day where you would give your tacit support to tests that are not approved by the AMTSO review board such as matousec and the Russian anti-malre.ru. Remember it is your own words and your own admission, because CIS came on top. Ladies and gentlemen Melih is on board and thus the motion has been carried unanimously.

Here is my logic now, since Melih approves of non AMTSO review board tests like matousec and anti-maware.ru and yes he took credit for them; consequently, he has finally relented to allow CIS to be tested by NSSLABS and av-comnparatives. Hipip Hooray..... ;D

Peace.
« Last Edit: November 11, 2009, 02:14:30 PM by Jaki »
"Anything you scan will be scanned against you; if you are smart, you will stop scanning." --Vundo
"Detecting and cleaning are futile, my growing family members will eventually hack you." --Virut

Offline Melih

  • CEO - Comodo
  • Administrator
  • Comodo's Hero
  • *****
  • Posts: 13450
    • Video Blog
Re: Petition for CIS to be Tested by NSSLABS
« Reply #31 on: November 11, 2009, 03:34:56 PM »
Jaki

With all due respect, I don't think you are understanding what I am writing or you purposely choose to misinterpret what i write.

1)you are using words like smoke and mirror, trick etc. This is utterly disrespectful
2)you are misunderstanding what AMTSO review board is (here is the document for you to read.
3)you are expecting Comodo to prove NSS tests are or are not compliant with AMTSO (it has nothing to do with us, take it up with either NSS or AMTSO).
4)you are misinterpreting, what I believe is a very clear statement that we support AMTSO compliant tests even though there are other public tests for Comodo.


AMTSO exists for a reason! All these companies have become a member of AMTSO for a reason! We believe in AMTSO compliant tests!

Melih
« Last Edit: November 11, 2009, 04:25:32 PM by Melih »

Offline mar56

  • Newbie
  • *
  • Posts: 1
Re: Petition for CIS to be Tested by NSSLABS
« Reply #32 on: November 11, 2009, 04:09:42 PM »
[at]Melih

first of all, you should not link to the PDF, bypassing the AMTSO license Agreement. Please remove it. The AMTSO license agreement has been introduced on purpose.

second, you, as a member of AMTSO, may consider to show up at some of the meetings, as it seems you are misinterpreting some of the work AMTSO has done so far and the involved procedures.

If you read the paper you linked to, you will see that by now, when the RAB reviews a test, all they do is checking if the nine fundamental principles have been accomplished (see example 8.d. in the PDF). If they are accomplished, the test is, as you say, "AMTSO compliant" (althought it seems you introduced this term by yourself). This can as well apply also to static tests. So, you should better say you want "a dynamic test which follows the fundamental principles".

As it has been pointed out, AMTSO can not review all tests out there, especially not in advance. You may have read that some tests are already considered under review since some time. Whether the test in question in this thread is compliant or not, is up to the RAB to evaluate (and should - as Melih said - not be discussed here).

[at]all: for a better understanding of AMTSO, the review analysis, etc., please read the AMTSO website/documents, the ESET blog post of David or the post on the AVC forum.

Offline Jaki

  • Computer Security Testing Group
  • Comodo's Hero
  • *****
  • Posts: 550
Re: Petition for CIS to be Tested by NSSLABS
« Reply #33 on: November 11, 2009, 04:12:09 PM »
Jaki

With all due respect, I don't think you are understanding what I am writing or you purposely choose to misinterpret what i write.

1)you are using words like smoke and mirror, trick etc. This is utterly disrespectful
2)you are misunderstanding what AMTSO review board is (here is the document for you to read amtso.org/uploads/amtso-analysis-of-reviews-process.pdf ).
3)you are expecting Comodo to prove NSS tests are or are not compliant with AMTSO (it has nothing to do with us, take it up with either NSS or AMTSO).
4)you are misinterpreting, what I believe is a very clear statement that we support AMTSO compliant tests even though there are other public tests for Comodo.


AMTSO exists for a reason! All these companies have become a member of AMTSO for a reason! We believe in AMTSO compliant tests!

Melih

If your statements are true please educate me about the AMTSO then; since Comodo is one of its members, at least you must know its fundamentals. Moreover, you still have not answered my simple question. My question is comprised of two possible answers true or false. Really, it is very simple.

Based upon your previous post I must conclude that you have indeed confirmed that you do condone tests that are NOT approved by the AMTSO review board, however selective. Sincerely, I do not want you to take offense when I'm saying this: Please Melih answer the question.


Peace.
"Anything you scan will be scanned against you; if you are smart, you will stop scanning." --Vundo
"Detecting and cleaning are futile, my growing family members will eventually hack you." --Virut

Offline Jaki

  • Computer Security Testing Group
  • Comodo's Hero
  • *****
  • Posts: 550
Re: Petition for CIS to be Tested by NSSLABS
« Reply #34 on: November 11, 2009, 04:16:02 PM »
[at]Melih

first of all, you should not link to the PDF, bypassing the AMTSO license Agreement. Please remove it. The AMTSO license agreement has been introduced on purpose.

second, you, as a member of AMTSO, may consider to show up at some of the meetings, as it seems you are misinterpreting some of the work AMTSO has done so far and the involved procedures.

If you read the paper you linked to, you will see that by now, when the RAB reviews a test, all they do is checking if the nine fundamental principles have been accomplished (see example 8.d. in the PDF). If they are accomplished, the test is, as you say, "AMTSO compliant" (althought it seems you introduced this term by yourself). This can as well apply also to static tests. So, you should better say you want "a dynamic test which follows the fundamental principles".

As it has been pointed out, AMTSO can not review all tests out there, especially not in advance. You may have read that some tests are already considered under review since some time. Whether the test in question in this thread is compliant or not, is up to the RAB to evaluate (and should - as Melih said - not be discussed here).

[at]all: for a better understanding of AMTSO, the review analysis, etc., please read the AMTSO website/documents, the ESET blog post of David or the post on the AVC forum.

ah ha, out of the blue mar56 appears, thanks. Anyway it seems to be that av-comparatives was right:

"
    Quoted
    AMTSO has a document with 9 “fundamental principles of testing”. You can find it on their website. Basically, a test needs to follow those fundamental principles to be "compliant" (though that term has no formal definition at the moment). This means that ANY kind of test, as long as it follows those 9 principles, can be considered compliant. You will find many testers that already follow those principles. And many tests which do not. Therefore, I advise readers to check out the principles and check if a test is in their own eyes compliant or not."


reference:

http://www.av-comparatives.org/forum/index.php?page=Thread&threadID=949


Peace.
« Last Edit: November 11, 2009, 05:10:59 PM by Jaki »
"Anything you scan will be scanned against you; if you are smart, you will stop scanning." --Vundo
"Detecting and cleaning are futile, my growing family members will eventually hack you." --Virut

Offline Melih

  • CEO - Comodo
  • Administrator
  • Comodo's Hero
  • *****
  • Posts: 13450
    • Video Blog
Re: Petition for CIS to be Tested by NSSLABS
« Reply #35 on: November 11, 2009, 04:38:38 PM »
[at]Melih

first of all, you should not link to the PDF, bypassing the AMTSO license Agreement. Please remove it. The AMTSO license agreement has been introduced on purpose.

second, you, as a member of AMTSO, may consider to show up at some of the meetings, as it seems you are misinterpreting some of the work AMTSO has done so far and the involved procedures.

If you read the paper you linked to, you will see that by now, when the RAB reviews a test, all they do is checking if the nine fundamental principles have been accomplished (see example 8.d. in the PDF). If they are accomplished, the test is, as you say, "AMTSO compliant" (althought it seems you introduced this term by yourself). This can as well apply also to static tests. So, you should better say you want "a dynamic test which follows the fundamental principles".

As it has been pointed out, AMTSO can not review all tests out there, especially not in advance. You may have read that some tests are already considered under review since some time. Whether the test in question in this thread is compliant or not, is up to the RAB to evaluate (and should - as Melih said - not be discussed here).

[at]all: for a better understanding of AMTSO, the review analysis, etc., please read the AMTSO website/documents, the ESET blog post of David or the post on the AVC forum.

Thanks for the heads up. I have removed the link. But fyi: i got the link from Google (3rd link from top with a pdf doc). Perhaps you should get in touch with Google and ask them to remove the link as this might be violating AMTSO licensing. Btw is Google violating the licensing? We did send a representative early on, but the team is doing great work we didn't see that we could add any more to the good work being carried out.

Yes for clarification: We are looking for Dynamic Tests which are AMTSO Compliant. But as you will appreciate you can't call a Static Test as AMTSO compliant if it takes an Anti Malware product  and only tests one aspect of the anti-malware product without measuring the effectiveness and performance of the anti-malware product in a balanced way. So its very much dependent on what is being tested. You have to apply relevant tests to relevant products! So I would respectfully still say saying AMTSO Compliant test for CIS should be sufficient as doing Static tests on CIS would not be "measure the anti-malware product (CIS) in a balanced way. But you know what..I really don't care :) If you want to call it Dynamic Tests...happy to oblige... :)

Melih

PS: mar56...why the secrecy? why not reveal who you are?
« Last Edit: November 11, 2009, 04:53:46 PM by Melih »

Offline Endymion

  • Comodo's Hero
  • *****
  • Posts: 1360
  • Reality is subordinate to perception.
    • Faces -The Madman (Kahlil Gibran, 1918)
Re: Petition for CIS to be Tested by NSSLABS
« Reply #36 on: November 11, 2009, 06:17:41 PM »
If you read the paper you linked to, you will see that by now, when the RAB reviews a test, all they do is checking if the nine fundamental principles have been accomplished (see example 8.d. in the PDF).

Hallo mar56, are you sure about that?

Can you comment on the  2nd (target) and 7th paragraph of procedures section of the AMTSO Analysis of Review Process  document approved on  May 7, 2009?

It states that the purpose is not limited to checking the 9 principles alone like you claimed but "AMTSO Guidelines and Principles as currently in effect"...

ref: AMTSO Guidelines and Principles currently in effect

The document also mention  that the described process  and procedures  are a  starting point and some changes might be expected.

Is the reduction of the criteria to be reviewed to the 9 principles alone like you claimed one of such expected improvements?
How much time will now take the RAB to check those reduced criteria?
Or was such improvement made to increase transparency to the public so they can easily and rapidly check compliance by themselves thus superseding the need for the RAB?

When a tester claims to be AMTSO compliant – and many have started to do that – or uses phrasing that implies compliance, such as "following AMTSO principles", what does that mean?

It looks David Harley focused his article on many testers who already claimed compliance or used phrasing that implied compliance such as "following AMTSO principles".

How many compliance implying/claiming tests in total such testers made specifically available to the general public  since  May 7, 2009?
How many of of such tests were submitted to the RAB?
How many of such RAB requests filed for such tests were signed by their respective testers'?


What concerns me right now is that bitter experience suggests that if a tester makes a point of claiming that his methodology is conformant with the AMTSO guidelines, quite a few people will accept that claim uncritically
David Harley made explicit mention about the AMTSO guidelines, did he mean the 9 principles alone or maybe the changes you mentioned post-date his article?


As it has been pointed out, AMTSO can not review all tests out there, especially not in advance.

I guess nobody asked about all tests out there but only the ones  explicitly claiming AMTSO compliance or using phrasing that imply such compliance.

What actually is the estimated monthly average number of such AMTSO implying/claiming tests specifically available to the general public?

Or was it meant that there are already so many publicly available AMTSO implying/claiming tests out there that the RAB cannot review them in a timely fashion even prioritizing the ones from AMTSO's own members?

Besides it looks like some came to believe that a tester member of AMTSO cannot willingly submit his/her tests before publication, is there any AMTSO principle that prevent/forbid them to submit their own tests to the RAB before publication or eventually publish such test pending review taking care to publicly update the test to mention the RAB analysis results?

Thanks in advance for you replies.

« Last Edit: November 11, 2009, 09:50:38 PM by Endymion »
I have learnt silence from the talkative, toleration from the intolerant, and kindness from the unkind; yet strange, I am ungrateful to these teachers.
Kahlil Gibran (1883 - 1931)

Offline Jaki

  • Computer Security Testing Group
  • Comodo's Hero
  • *****
  • Posts: 550
Re: Petition for CIS to be Tested by NSSLABS
« Reply #37 on: November 12, 2009, 09:05:16 AM »
So I would respectfully still say saying AMTSO Compliant test for CIS should be sufficient as doing Static tests on CIS would not be "measure the anti-malware product (CIS) in a balanced way. But you know what..I really don't care :) If you want to call it Dynamic Tests...happy to oblige... :)

Melih

PS: mar56...why the secrecy? why not reveal who you are?

If that is so Melih, would you submit CIS to NSSLABS for a "dynamic" test? I think it is quite clear that based upon my petition and Kyle's most CIS users would like CIS to be tested. Would you defy the wishes of your own users or would you heed to our call?

Peace.
« Last Edit: November 12, 2009, 09:07:35 AM by Jaki »
"Anything you scan will be scanned against you; if you are smart, you will stop scanning." --Vundo
"Detecting and cleaning are futile, my growing family members will eventually hack you." --Virut

Offline Endymion

  • Comodo's Hero
  • *****
  • Posts: 1360
  • Reality is subordinate to perception.
    • Faces -The Madman (Kahlil Gibran, 1918)
Re: Petition for CIS to be Tested by NSSLABS
« Reply #38 on: November 12, 2009, 09:54:50 AM »
I wonder why the poll did mention NSSLABS and not simply AMTSO compliant tests?

Is the only know test implying compliance made by an AMTSO member since May 7, 2009 ?


AMTSO’s stance has been misinterpreted as meaning that dynamic testing is automatically compliant
Was compliance implied for such test because dynamic tests were considered automatically compliant  or maybe it was implied so only according to the 9 principles alone ?

Besides were the criteria really reduced to 9 principles alone like some rumors have been claiming?

I wonder if the DIY compliance checks similar rumors have been spreading apply also to a recent dynamic test mentioned in these forums...

« Last Edit: November 12, 2009, 11:06:18 AM by Endymion »
I have learnt silence from the talkative, toleration from the intolerant, and kindness from the unkind; yet strange, I am ungrateful to these teachers.
Kahlil Gibran (1883 - 1931)

Offline Jaki

  • Computer Security Testing Group
  • Comodo's Hero
  • *****
  • Posts: 550
Re: Petition for CIS to be Tested by NSSLABS
« Reply #39 on: November 12, 2009, 12:39:58 PM »
Hi Melih

You have the right to disagree with NSSLABS if you want to; that is certainly your own prerogative. However, if you doubt the NSSLABS AMTSO dynamic claims, then you also have the right to report it to the AMTSO review board. If you do not report it to the AMSTO that's where I will have a problem with you as member of the AMTSO. You should not violate the AMTSO first principle and that's what you will end-up doing if you do not convey your doubt(s) to the AMTSO review board with regard to the recent NSSLABS dynamic test.


Peace.

"Anything you scan will be scanned against you; if you are smart, you will stop scanning." --Vundo
"Detecting and cleaning are futile, my growing family members will eventually hack you." --Virut

Offline Endymion

  • Comodo's Hero
  • *****
  • Posts: 1360
  • Reality is subordinate to perception.
    • Faces -The Madman (Kahlil Gibran, 1918)
Re: Petition for CIS to be Tested by NSSLABS
« Reply #40 on: November 12, 2009, 01:49:53 PM »
Even accounting for some rumors there is still no reason to believe that a tester cannot submit his/her own test to the RAB by themselves.

Besides with so many rumors and uncertainties about a yet to be defined "compliance" it looks that AMTSO members will have to find common grounds in forthcoming meetings as it appear that even the Review process is expected to change.

Anything else appear a moot point...
« Last Edit: November 12, 2009, 02:15:41 PM by Endymion »
I have learnt silence from the talkative, toleration from the intolerant, and kindness from the unkind; yet strange, I am ungrateful to these teachers.
Kahlil Gibran (1883 - 1931)

Offline bluebert

  • Newbie
  • *
  • Posts: 13
Re: Petition for CIS to be Tested by NSSLABS
« Reply #41 on: December 01, 2009, 05:25:12 AM »
I understand that the NSS Labs tests are currently being looked at formally by the AMTSO Review Analysis Board.

There isn't a formal process for any tester to demonstrate "compliance" voluntarily right now. But there isn't, as far as I can see, anything to stop a tester asking for a review analysis for a test that's taken place, either. But that would, at best, validate a specific test, not the testing organization.

The way I read David Harley's comments, AMTSO might well take several steps towards a tester being able to demonstrate intent to comply at its next meeting, but that's a lot different to some form of certification, and I can't see AMTSO being able to offer certification as things are: it doesn't have the resources.

The arguments here about static versus dynamic puzzle me. You can run a static test and still be compliant, and a dynamic test can be non-compliant. Obviously a test can also be dynamic yet not give a well-rounded view of a product's abilities: that's one of the complaints vendors have made about the NSS tests. But if dynamic tests are resource intensive, "full product" tests are even more so.

AMTSO seems interested in persuading the testers to use dynamic testing wherever practical, but that doesn't mean that all tests are now required to be dynamic. There's nothing in the principles to stop a tester focusing on one aspect of a product as long as he tests properly, his conclusions are consistent with the data, and so on.


Offline Melih

  • CEO - Comodo
  • Administrator
  • Comodo's Hero
  • *****
  • Posts: 13450
    • Video Blog
Re: Petition for CIS to be Tested by NSSLABS
« Reply #42 on: December 01, 2009, 05:30:01 AM »
I understand that the NSS Labs tests are currently being looked at formally by the AMTSO Review Analysis Board.


Thank you for this!
We wish NSS Labs all the best and hope that they can be the first one to get a green light from the review board. Comodo will sure to be knocking on NSS's door as soon as they get a green light.

thanks
Melih

 

Seo4Smf 2.0 © SmfMod.Com | Smf Destek