AV-Comparatives provides a methodology listing of its procedures on its website, specifically here: http://www.av-comparatives.org/seiten/ergebnisse/methodology.pdf
AV-C and AV-Test are the only tests whose results I personally consider to be reliable right now, other than my own, of course. Unfortunately, Andreas Marx of AV-Test doesn't seem to make his organization's test methodology readily available, but from what I've seen he does respond to email queries from the public.
I read that paper (august 2008 revision dated 15/09/2008) and found it difficult to understand.
Is detection rate used as a selection criteria to deem a vendor eligible to possibly receive samples?
If so can you explain what sampleset it is used in that case?
The full one?
The one who includes only malware not older than one year?
Does that paper explicitely state what is the minumum detection criteria?
I believe you're confused simply because you cannot seem to recognize the fact that testers not sharing samples doesn't necessarily mean they believe their samples are "private property" that belong to them.
I see. In that regard I'm surely confused. Even though it doesn't necessarily mean that, I feel concerned in that regard and I believe such aspects should be thoroughfully and publicly documented. Sure it will not be something diffiult to do.
I was a participant in an online conversion with IBK of AV-C some time ago, where we were told in response to a question that the criterias were set in place to prevent indiscriminate sharing of samples with non-trustworthy vendors. Even though the exact percentage to qualify to receive samples from AV-C doesn't seem to be publicly available anymore, I've seen no reason so far to doubt IBK's claim, as his methodology appears to be in line with his stated aims.
Then you surely asked what sampleset is used for that type of vetting procedure.
Still I would like to know what you consider a non discriminatory selection criteria leaving to the readers the effort to verify if any tester or vendor do use your suggested criteria.
Of course, I may be wrong, and you know perfectly well what you're talking about. But so far I've seen no evidence to back up your claims, either from you or anyone else. If you have proof that testers actually deliberately discriminate against vendors they don't like and deliberately withhold samples from them even though they meet the minimum criteria, please do share.
Do I have a possible way to verify that? Does this mean any AV tester who could possibly deny sample disclosure provide a way to verify eligibility criteria?
Eg in case a detection rate test is used would it be reasonable to assume that a list of CRC hashes of missed samples would be provided?
This would at least allow a rejected vendor to know if they possibly were able to gather an undetected sample at a later time.EDIT:
I just noticed that the above quote regarded deliberately discriminatory
criteria. This does not mean I endorse that description and I wish to make my excuses for carelessy replying to that without explicitly clarifying this point. I also feel the term "subjective", "biased" or "flawed" to be a fitting substitute for all the times I borrowed the term "discriminatory" in my replies.
I'm not a biologist, so I must admit I do not know enough to hold a debate in that area. Fortunately, however, this thread concerns computer viruses and not biological ones. To return to the topic at hand: in what way exactly do you believe that a tester not wanting to become a collector for vendors who don't have their own facilities is an unreasonable concern?
IMHO tester should be concerned how to carry her/his test correctly. The mere fact an AV tester can use a rare sample to test any AV is not relevant too provided they follow their publicly available methodology. I'm sure we will likely unable to agree on which can be or cannot be inferred by such tests in the context of the stated methodology.
This is not enough IMHO to neglect as irrelevant the difference with pharmaceutical companies. You can feel free to do so though.
Although I will wonder if you think that a pharmaceutical company could be legitimately prevented to gather a sample to develop a vaccine and if such case could be considered 'leeching'.
There is an advantage, yes. But last I checked testers do not demand perfect performance from vendors before sharing samples with them. There would be nothing to share anyway, if that were the case. There is simply a minimum baseline to be achieved.
I'm more likely to guess that vendors with many partnerships, well established marketshare and who possibly started to develop their AV more than five years ago maybe will have almost nothing to get from a whatsoever AV tester. I guess the a tester could possibly 'leech' some samples instead. I would be surely interested to read any documentation about these aspects.
I think that the ability to obtain samples within one week is a very reasonable minimum baseline indeed, given that the aim should be zero-day protection and one week leaves a VERY big margin of error. But it appears that you don't agree. How much of a margin do you think would be acceptable, then? One month? Six?
I used one week as example in first instance, though I wondered if it was reasonable. Thanks for providing such answer. This partly solve the nowhere-to-be found paradox you previously described and provide reader a more specific context to a previous reply of mine
I don't have any specific expectation and I would be more inclined to consider reasonable the average result of specific tests designed to measure that in a representative sample of AV brands.
Provided that the bias caused by cross-vendor sharing partnerships would be removed. Even though I'm interested to know if such tests are available.
What private agreements, exactly? You display a disturbing consistency of NOT providing any explanations at all - let alone evidence - behind your repeated insinuations.
Oh my! Is private agreement
an insinuation? The private nature of an agreement doesn't surely mean that such agreement will not publicly announced. Anyway IMHO I guess Public utility
may be a better alternative to the current AV ecosystem and I hope no one will consider that disturbing.
"Eg using a minimum detection rate as a selection criteria means that the sampleset composition only evaluate malware gathering abilities..."
Does that refresh your memory?
As I stated that having a sample it is only the first step to research and build an appropriate countermeasure being it an a AV signature that only work on a specific patented AV engine, a removal application, a patented heuristic detection engine or a patented HIPS technology in my first reply to your post
I still wonder how could you infer this
Again, this is to prove the point that the popular misconception that vendor has sample = nothing else matters is false.
using that refreshing reference posted way later than my first reply.
I realize you were looking for an explanation of why sample sharing should be regulated by private agreements, but a little reading of what I've said will reveal to you that my explanation was that there is no such thing "regulations" and "private agreements".
What does specific circumstances under tightly-controlled conditions with vetted individuals
Yes there is no regulation like for Public Utilities
Sharing is currently carried on a per case basis with individual agreements between private parties.
They have no way of regulating or restricting me as an amateur, and they have no way of doing the same to large international corporations in the industry. Samples can and will be obtained if one wants to do so, and there is nothing anyone can do about it short of banning the internet. Again, until you can stop using this fallacy as the crux of your arguments, we'll simply go around in circles.
Yes everyone can privately gather maleware samples. They can also privately choose to share them or not.
I guess everyone could read all your rearmks and then decide whenever my viewpoint was based on a fallacy or not. I did not assume you considered it otherwise and I thank you for your efforts to describe your viewpoint.