Leak Test Results

[Guest]

[Melih]

Updated: 27 November 2008

Hi Everyone

In order to make sure everything is easily found, lets put the results of our test results to this thread. So pls go ahead and report back your test results here and lets have at least 2 or more people confirm the results before we can update our results. Then we can put these results at testmypcsecurity.com for everyone's benefit.

Here are the product list i have got from testmypcsecurity. by all means if you are testing other products pls include it here.

These scores are out of 340. The highest that can be achieved is 340

        Product                                                   Result
Comodo Internet Security --------------->340/340
Kaspersky Internet Security ------------->270/340
Agnitum Outpost Firewall Pro ------------>250/340
Jetico Personal Firewall 
Sunbelt Personal Firewall  --------------->50/340
Avira Premium Security Suite ------------>90/340
Online Armor Personal Firewall ----------->290-340/340
Online Armor Free  ---------------------->290-340/340   
Norton Internet Security 2009  ---------->50/340 
BitDefender Internet Security ----------->20/340    
ZoneAlarm Pro Firewall ------------------>220/340
ZoneAlarm Free Firewall ----------------->40/340
Iolo Personal Firewall   
Panda Internet Security 2009 ------------>20/340
GoldTach Personal Firewall --------------->40/340
AVG AntiVirus Plus Firewall -------------->20/340
Spyware Terminator 2.5 ----------------->80/340
F-Secure Internet Security 2009 --------->20/340
DefenseWall ---------------------------->300/340

Thank you

Melih

PS: Mods: pls feel free to add the results to this post so that we have one place for all results.

[JamesFrance]

Here is mine running CIS with proactive security:

COMODO Leaktests v.1.1.0.1
Date   09:51:02 - 16/11/2008
OS   Windows XP SP3 build 2600
1. Hijacking: ActiveDesktop   Protected
2. Hijacking: AppinitDlls   Protected
3. Hijacking: ChangeDebuggerPath   Protected
4. Hijacking: StartupPrograms   Protected
5. Hijacking: SupersedeServiceDll   Protected
6. Hijacking: UIHost   Protected
7. Hijacking: Userinit   Protected
8. Hijacking: WinlogonNotify   Protected
9. Impersonation: BITS   Protected
10. Impersonation: Coat   Protected
11. Impersonation: DDE   Protected
12. Impersonation: ExplorerAsParent   Protected
13. Impersonation: OLE automation   Protected
14. InfoSend: DNS Test   Protected
15. InfoSend: ICMP Test   Protected
16. Injection: AdvancedProcessTermination   Protected
17. Injection: APC dll injection   Protected
18. Injection: CreateRemoteThread   Protected
19. Injection: DupHandles   Protected
20. Injection: KnownDlls   Protected
21. Injection: ProcessInject   Protected
22. Injection: Services   Protected
23. Injection: SetThreadContext   Protected
24. Injection: SetWindowsHookEx   Protected
25. Injection: SetWinEventHook   Protected
26. Invasion: DebugControl   Protected
27. Invasion: FileDrop   Protected
28. Invasion: PhysicalMemory   Protected
29. Invasion: RawDisk   Protected
30. Invasion: Runner   Protected
31. RootkitInstallation: ChangeDrvPath   Protected
32. RootkitInstallation: DriverSupersede   Protected
33. RootkitInstallation: LoadAndCallImage   Protected
34. RootkitInstallation: MissingDriverLoad   Protected
Score   340/340

[offchu]

COMODO Leaktests v.1.1.0.3
Date   21:31:38 - 15.11.2008

OS   Windows XP SP3 build 2600

1. RootkitInstallation: MissingDriverLoad   Protected
2. RootkitInstallation: LoadAndCallImage   Protected
3. RootkitInstallation: DriverSupersede   Protected
4. RootkitInstallation: ChangeDrvPath   Protected
5. Invasion: Runner   Protected
6. Invasion: RawDisk   Protected
7. Invasion: PhysicalMemory   Protected
8. Invasion: FileDrop   Protected
9. Invasion: DebugControl   Protected
10. Injection: SetWinEventHook   Protected
11. Injection: SetWindowsHookEx   Protected
12. Injection: SetThreadContext   Protected
13. Injection: Services   Protected
14. Injection: ProcessInject   Protected
15. Injection: KnownDlls   Protected
16. Injection: DupHandles   Protected
17. Injection: CreateRemoteThread   Protected
18. Injection: APC dll injection   Protected
19. Injection: AdvancedProcessTermination   Protected
20. InfoSend: ICMP Test   Protected
21. InfoSend: DNS Test   Protected
22. Impersonation: OLE automation   Protected
23. Impersonation: ExplorerAsParent   Protected
24. Impersonation: DDE   Protected
25. Impersonation: Coat   Protected
26. Impersonation: BITS   Protected
27. Hijacking: WinlogonNotify   Protected
28. Hijacking: Userinit   Protected
29. Hijacking: UIHost   Protected
30. Hijacking: SupersedeServiceDll   Protected
31. Hijacking: StartupPrograms   Protected
32. Hijacking: ChangeDebuggerPath   Protected
33. Hijacking: AppinitDlls   Protected
34. Hijacking: ActiveDesktop   Protected
Score   340/340

[subset]

Hi,

without testing guidelines all results a pretty meaningless.
Admin? LUA? Default program settings? Highest security settings?
You should really try to bring this to a more mature level, or it ends in another farce.

Cheers

[3xist]

COMODO Leaktests v.1.1.0.3
Date   10:41:23 AM - 11/16/2008
OS   Windows XP SP3 build 2600
1. RootkitInstallation: MissingDriverLoad   Protected
2. RootkitInstallation: LoadAndCallImage   Protected
3. RootkitInstallation: DriverSupersede   Protected
4. RootkitInstallation: ChangeDrvPath   Protected
5. Invasion: Runner   Protected
6. Invasion: RawDisk   Protected
7. Invasion: PhysicalMemory   Protected
8. Invasion: FileDrop   Protected
9. Invasion: DebugControl   Protected
10. Injection: SetWinEventHook   Protected
11. Injection: SetWindowsHookEx   Protected
12. Injection: SetThreadContext   Protected
13. Injection: Services   Protected
14. Injection: ProcessInject   Protected
15. Injection: KnownDlls   Protected
16. Injection: DupHandles   Protected
17. Injection: CreateRemoteThread   Protected
18. Injection: APC dll injection   Protected
19. Injection: AdvancedProcessTermination   Protected
20. InfoSend: ICMP Test   Protected
21. InfoSend: DNS Test   Protected
22. Impersonation: OLE automation   Protected
23. Impersonation: ExplorerAsParent   Protected
24. Impersonation: DDE   Protected
25. Impersonation: Coat   Protected
26. Impersonation: BITS   Protected
27. Hijacking: WinlogonNotify   Protected
28. Hijacking: Userinit   Protected
29. Hijacking: UIHost   Protected
30. Hijacking: SupersedeServiceDll   Protected
31. Hijacking: StartupPrograms   Protected
32. Hijacking: ChangeDebuggerPath   Protected
33. Hijacking: AppinitDlls   Protected
34. Hijacking: ActiveDesktop   Protected
Score   340/340

Josh

[cvsa]

Win xp sp3 - CIS (firewall only with Proactive security) and Kaspersky AV 2009 - note: (Kaspersky blocks CLT dowloading and launch ) -

COMODO Leaktests v.1.1.0.3
Date   10:45:20 - 11/16/2008
OS   Windows XP SP3 build 2600
1. RootkitInstallation: MissingDriverLoad   Protected
2. RootkitInstallation: LoadAndCallImage   Protected
3. RootkitInstallation: DriverSupersede   Protected
4. RootkitInstallation: ChangeDrvPath   Protected
5. Invasion: Runner   Protected
6. Invasion: RawDisk   Protected
7. Invasion: PhysicalMemory   Protected
8. Invasion: FileDrop   Protected
9. Invasion: DebugControl   Protected
10. Injection: SetWinEventHook   Protected
11. Injection: SetWindowsHookEx   Protected
12. Injection: SetThreadContext   Protected
13. Injection: Services   Protected
14. Injection: ProcessInject   Protected
15. Injection: KnownDlls   Protected
16. Injection: DupHandles   Protected
17. Injection: CreateRemoteThread   Protected
18. Injection: APC dll injection   Protected
19. Injection: AdvancedProcessTermination   Protected
20. InfoSend: ICMP Test   Protected
21. InfoSend: DNS Test   Protected
22. Impersonation: OLE automation   Protected
23. Impersonation: ExplorerAsParent   Protected
24. Impersonation: DDE   Protected
25. Impersonation: Coat   Protected
26. Impersonation: BITS   Protected
27. Hijacking: WinlogonNotify   Protected
28. Hijacking: Userinit   Protected
29. Hijacking: UIHost   Protected
30. Hijacking: SupersedeServiceDll   Protected
31. Hijacking: StartupPrograms   Protected
32. Hijacking: ChangeDebuggerPath   Protected
33. Hijacking: AppinitDlls   Protected
34. Hijacking: ActiveDesktop   Protected
Score   340/340
(C) COMODO 2008

[Melih]

Hi,

without testing guidelines all results a pretty meaningless.
Admin? LUA? Default program settings? Highest security settings?
You should really try to bring this to a more mature level, or it ends in another farce.

Cheers

I think the issue will come when people have differnt scores for the same product, in which case I am sure the discussions between them will resolve the differences.

Its a good experiement about reallife scenerios and how products do in real life scenerios.

Melih

[DarkButterfly]

Agnitum Outpost Firewall Pro 2009 (latest version) with HIPS component set for maximum protection and no automatic rules created.

COMODO Leaktests v.1.1.0.1
Date   23:15:55 - 16-11-2008

OS   Windows Vista SP1 build 6001

1. Hijacking: ActiveDesktop   Vulnerable
2. Hijacking: AppinitDlls   Protected
3. Hijacking: ChangeDebuggerPath   Protected
4. Hijacking: StartupPrograms   Vulnerable
5. Hijacking: SupersedeServiceDll   Vulnerable
6. Hijacking: UIHost   Protected
7. Hijacking: Userinit   Protected
8. Hijacking: WinlogonNotify   Protected
9. Impersonation: BITS   Protected
10. Impersonation: Coat   Protected
11. Impersonation: DDE   Protected
12. Impersonation: ExplorerAsParent   Protected
13. Impersonation: OLE automation   Protected
14. InfoSend: DNS Test   Protected
15. InfoSend: ICMP Test   Protected
16. Injection: AdvancedProcessTermination   Vulnerable
17. Injection: APC dll injection   Protected
18. Injection: CreateRemoteThread   Protected
19. Injection: DupHandles   Vulnerable
20. Injection: KnownDlls   Vulnerable
21. Injection: ProcessInject   Protected
22. Injection: Services   Protected
23. Injection: SetThreadContext   Protected
24. Injection: SetWindowsHookEx   Protected
25. Injection: SetWinEventHook   Protected
26. Invasion: DebugControl   Protected
27. Invasion: FileDrop   Vulnerable
28. Invasion: PhysicalMemory   Protected
29. Invasion: RawDisk   Vulnerable
30. Invasion: Runner   Protected
31. RootkitInstallation: ChangeDrvPath   Vulnerable
32. RootkitInstallation: DriverSupersede   Protected
33. RootkitInstallation: LoadAndCallImage   Protected
34. RootkitInstallation: MissingDriverLoad   Protected
Score   250/340


(C) COMODO 2008

dang! I do love Comodo Firewall Pro  (L)

[panic]

Hi,

without testing guidelines all results a pretty meaningless.
Admin? LUA? Default program settings? Highest security settings?
You should really try to bring this to a more mature level, or it ends in another farce.

Cheers

+1

Can you please provide a list of what should be included in the guidelines?

Ewen :-)

[Stargazer]

CIS, using all Defense+ monitor settings on:

COMODO Leaktests v.1.1.0.1
Date   下午 06:17:27 - 2008/11/17
OS   Windows XP SP3 build 2600
1. Hijacking: ActiveDesktop   Protected
2. Hijacking: AppinitDlls   Protected
3. Hijacking: ChangeDebuggerPath   Protected
4. Hijacking: StartupPrograms   Protected
5. Hijacking: SupersedeServiceDll   Protected
6. Hijacking: UIHost   Protected
7. Hijacking: Userinit   Protected
8. Hijacking: WinlogonNotify   Protected
9. Impersonation: BITS   Protected
10. Impersonation: Coat   Protected
11. Impersonation: DDE   Protected
12. Impersonation: ExplorerAsParent   Protected
13. Impersonation: OLE automation   Protected
14. InfoSend: DNS Test   Protected
15. InfoSend: ICMP Test   Protected
16. Injection: AdvancedProcessTermination   Protected
17. Injection: APC dll injection   Protected
18. Injection: CreateRemoteThread   Protected
19. Injection: DupHandles   Protected
20. Injection: KnownDlls   Protected
21. Injection: ProcessInject   Protected
22. Injection: Services   Protected
23. Injection: SetThreadContext   Protected
24. Injection: SetWindowsHookEx   Protected
25. Injection: SetWinEventHook   Protected
26. Invasion: DebugControl   Protected
27. Invasion: FileDrop   Protected
28. Invasion: PhysicalMemory   Protected
29. Invasion: RawDisk   Protected
30. Invasion: Runner   Protected
31. RootkitInstallation: ChangeDrvPath   Protected
32. RootkitInstallation: DriverSupersede   Protected
33. RootkitInstallation: LoadAndCallImage   Protected
34. RootkitInstallation: MissingDriverLoad   Protected
Score   340/340

 

When using "COMODO - Internet Security" preset, the "Invasion: RawDisk" test will fail. Using "COMODO - Proactive Security" preset will be just fine. (Well, I use my custom rules anyway)

[don67]

COMODO Leaktests v.1.1.0.1

Date   4:33:59 PM - 11/17/2008

OS   Windows XP SP3 build 2600

1. Hijacking: ActiveDesktop   Protected
2. Hijacking: AppinitDlls   Protected
3. Hijacking: ChangeDebuggerPath   Protected
4. Hijacking: StartupPrograms   Protected
5. Hijacking: SupersedeServiceDll   Protected
6. Hijacking: UIHost   Protected
7. Hijacking: Userinit   Protected
8. Hijacking: WinlogonNotify   Protected
9. Impersonation: BITS   Protected
10. Impersonation: Coat   Protected
11. Impersonation: DDE   Vulnerable
12. Impersonation: ExplorerAsParent   Vulnerable
13. Impersonation: OLE automation   Protected
14. InfoSend: DNS Test   Protected
15. InfoSend: ICMP Test   Protected
16. Injection: AdvancedProcessTermination   Protected
17. Injection: APC dll injection   Protected
18. Injection: CreateRemoteThread   Protected
19. Injection: DupHandles   Protected
20. Injection: KnownDlls   Protected
21. Injection: ProcessInject   Protected
22. Injection: Services   Protected
23. Injection: SetThreadContext   Protected
24. Injection: SetWindowsHookEx   Protected
25. Injection: SetWinEventHook   Protected
26. Invasion: DebugControl   Protected
27. Invasion: FileDrop   Protected
28. Invasion: PhysicalMemory   Protected
29. Invasion: RawDisk   Protected
30. Invasion: Runner   Protected
31. RootkitInstallation: ChangeDrvPath   Protected
32. RootkitInstallation: DriverSupersede   Protected
33. RootkitInstallation: LoadAndCallImage   Protected
34. RootkitInstallation: MissingDriverLoad   Protected
Score   320/340

why i have two fails?
11. Impersonation: DDE                   Vulnerable
12. Impersonation: ExplorerAsParent   Vulnerable

my firewall setting is default i'm using CIS latest version

[Flate]

I did the test using Trend Micro internet Security (what we have on my parents PC). The test did not go very well.. (see screen shot for results)

Edit: I noticed I may have set the firewall down a bit (a higher levels it blocked without asking and did not remember anything) so I attach two new images: one with firewall on max, and one with firewall + defense against internet theft on max.

Edit: I used vista x32 on a admin account with Windows firewall deactivated and Windows Defender deactivated and UAC on.

[subset]

Can you please provide a list of what should be included in the guidelines?

This is not a list, just some thoughts about application and OS settings.

Relatet to applications, all should be tested with default settings and highest security settings.
Because there is a big difference if you test
- KIS 2009 with default settings (Automatic Mode) or with Interactive Mode.
- NIS 2009 with default settings or with Advanced Event Monitoring enabled.
- Outpost with default settings (Optimal Host Protection) or with Advanced Host Protection.
etc. etc.

Testers should be able to select the best settings for the programs they test and afterwards readers of the results would recognize, that some of their programs are very, very weak with default settings (KIS, NIS, ...) and only offer a medium or high protection if they use custom settings.

Related to OS, at least all with the same settings, like for example with
- XP: Admin, Windows Firewall deactivated.
- Vista: Admin, Windows Firewall, Windows Defender and UAC deactivated.

With Vista 32bit it makes a real big difference, if you test with Admin account and Windows Firewall & Windows Defender deactivated (110/340) or with LUA and Windows Firewall & Windows Defender activated (250/340).



Just as a note. These OS results without any 3rd party security software seem to be pretty weird and I hardly can image, that they are intended by the developers of this test program.

Cheers

[Kyle]

I think what Melih wants is the results from the "Real World"
Perhaps not have a guide line to follow, Just list the environment and settings these tests were conducted.

[Melih]

I think what Melih wants is the results from the "Real World"
Perhaps not have a guide line to follow, Just list the environment and settings these tests were conducted.

Exactly Kyle.... afterall our users are not in a test environment! So how does security work for them out of box in the realworld, that is the question. If everyone is making the same mistake in the realworld and misconfiguring something which is then causing their security to be vulnerable, then these tests will show that, or it will also show providers who can't provide high security out of box because they haven't found a way of making it practical for usability.

Melih

[Tags:]