After successfully installing the free rules thanks to this https://forums.comodo.com/free-modsecurity-rules-comodo-web-application-firewall/how-to-install-the-free-rules-t107242.0.html extremely helpful post, my server is performing GREAT ! However, for the life of me I cannot stop two Chinese crawlers, EasouSpider and Baiduspider, both sending 100s of hits to my hosted sites.
In the logs they show as:
HTTP/1.1" 301 - “-” “Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
HTTP/1.1” 403 17914 “-” "Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
I have added any possible combination of “EasouSpider” and “Baiduspider” to my userdata_bl_agents file (like Baidu, ^Baidu, Baiduspider, “Baiduspider/2.0” etc. etc. ) and they still get through!
Anyone with the same issue can give me a clue??
Notice you already have HTTP/1.1" 403 (forbidden) answer for this kind of requests.
I am confused by the fact that the entries are in the logs. I mean, I added ‘Yandex’ spider, and there’s no more trace of ‘Yandex’ in my logs (301 or whatever), whereas for example WordFence (a security WordPress plugin with live view) shows constant accesses from both the Chinese. Even if they get a 301 they still keep hitting and using bandwidth.
CWAF rules didn’t produce 301 response code at this time. It is possible that you have some module on web server which redirects some requests before they reach mod_security module. I guess this could be mod_rewrite. You can test switching mod_rewrite off and put test results here.