The issue is triggered by Sun Java activex control which was built with old Microsoft Compiler that used techniques that are now prevented by
DEP.
Older versions of ATL, and by older I mean pre-Visual C++ 2005, used dynamically generated code in small isolated cases. Obviously, without the appropriate APIs this is going to cause problems on a DEP-enabled computer, because you can't execute data. This code is referred to as a "thunk" and versions of ATL in VC++ 2005 and later work correctly with DEP.
The comment above is correct except for the suggestion that we could fix the Java Plug-In by adding one line of code. Our discussions with Microsoft have indicated that only they can fix the problem with IE 7. Making the API call in our ActiveX control would have no effect from what they have told us. They have also said that IE 8 will contain this API call, meaning that ActiveX controls like ours compiled with older compilers will work again.
ActiveX plugins which trigger this issue have to be ported to more recent Miscrosoft Compilers
Developer Call to Action
If you build Internet Explorer add-ons, you can help ensure users enjoy a smooth upgrade to IE8 by taking the following steps today:
If your code depends on older versions of ATL, please rebuild it with ATL v7.1 SP1 or later (Visual Studio 2005 includes ATL 8.0)
Set the /NXCompat linker option to indicate that your extension is compatible with DEP/NX
Test your code with DEP/NX enabled using IE8 Beta 1 on Windows Vista SP1. (Alternatively, test with IE7 on Windows Vista after enabling the DEP/NX option. To enable DEP/NX for IE7: Run IE as an administrator, then set the appropriate checkbox in the Tools > Internet Options > Advanced tab)
Opt your code into other available defenses like stack defense (/GS), safe exception handling (/SafeSEH), and ASLR (/DynamicBase)
It is likely that Comodo BO Prevention engine featured by Safesurf or Comodo Memory Firewall was designed to minimize the chances users will not able to run legitimate application or plugins thus continuing to offer protection in scenarios like this where the users are supposed to disable DEP for Internet Explorer in order to use Java plugin.
(R)