I have never seen malware drop out of the sandbox. Also it is a little fishy that Jiangmin did so good. The reason I say fishy is that I believe most of their malware samples are ones that are prevalent in the asian market, so that means AV vendors that are based in the asian market will see those samples first compared to companies located in other parts of the world. I would hope that they use malware that is a good representation of the whole world not just one part of it. But this is just my opinion of it based on personal experience.
also I think they are mistaking dropped file for a real system infection. Remember v4 had many more dropped files onto the hard drive compared to V5 and if you didn’t know how the comodo sandbox specifically works you would assume that the system was infected even though it is not.
well I don’t know how they counted the dynamic detection, did they just count the AV pop ups, the HIPS pop ups, sandbox? Their test does not say how they counted dynamic detection, we need more information on how they came up with their numbers.
As of now i have yet not discussed Dynamic Detection part with Jeffrey wrt to results, but in general he stated following about methodology:
For HIPS test, mostly we make the detection based on the notification of the HIPS module, e.g. if the HIPS notify the users there is a malicious behavior and users should make a deny decision, we then deny the execution. When there is no clear notification that show the behavior is malicious or dangerous, we make the allow decision. And the final decision is whether there is an infection to the PC(cause data loss, infection of system files, etc. ) We make this also in our Total protection test.
For sandbox, in dynamic block test, if it automatically executes in sandbox and no files leak out of the sandbox, then it is a successful block. For dynamic test, if it is automatically sandboxed, then it is regarded as a fp and if the window with three options: sandbox, allow, deny. Then we choose allow and then see if the client block the installation of normal software.
So far my discussion with Jeffrey has been limited to his definition of false-positive wrt Sandbox. So i have yet to gather information from Jeffrey if he has encountered any malware which by passed Sandbox and was able to infect system in the sense that it was able to persist.
In general as you know Sandbox doesn’t allow to drop files in privileged folders, like Windows or start up folders.