I think a technically true test of “leaking” for a firewall will not involve user interaction. Obviously if the user gets an alert that application xyz is trying to access the internet, the user has the option to deny or allow (and, the rogue app/leaktest is not very sneaky).
So a more accurate test would be if the “leak” can bypass the firewall rules without any user interaction required. Or, be stopped by the firewall rules w/o any user interaction.
IMO, the current leaktests commonly deployed to “test” firewalls seem to all require user interaction when they try to do their stuff. Taking a realistic approach, is malware really going to be that simple to block? I hardly think so.
If we’re looking at a firewall (and not a HIPS), there’s probably no way for a dedicated firewall to simply silently block these types of things because they’re going to be the result of a system hijack anyway, deeply rooted, and exploiting known system processes/vulnerabilities to escape to the outer world. The only thing it could be defaulted for is to build a safelist of all known processes/applications and all possible legit combinations of the same (to avoid the hijacking issue), all legit IP protocols (of all levels) for said legit apps & combinations of apps, along with a blacklist of known bad IPs (with the ability to identify other values for the IP, such as hex, etc). Then the FW (again, no HIPS), would allow the known safe, disallow (silently - no user interaction) the unknown, block bad IPs for outbound, and so on. Does that sound crazy? While the NIST lists could help to identify known applications, combinations of apps seems impossible. Not to mention, when you get some off-the-wall application/process that’s not known to the FW (but is to the user), it would be blocked silently (remember, no user interaction for a “true” leaktest), which is just insane.
So… user interaction is the only way to realistically run these tests, and that (as we all know) will always be flawed based on the dependence on users’ choices. No matter how informative the prompt, there are always going to be wrong choices. That’s just the way it is, IMO.
I still think the only accurate leaktest cannot depend on user interaction (ie, the human element), but that there is no way this can be legitimately accomplished (remember HAL?
).
LM