Aboundex bot

Free Modsecurity rules - Comodo Web Application Fi
1

Anybody else have issues with this bot? I think technically Aboundex is a “search engine,” which is probably why it hasn’t landed on any “bad bot” lists, but their bot is really obtrusive. It hammers sites and scrapes all of the links very quickly, which results in a large CPU usage spike. They also seem to totally ignore robots.txt files.

I have several thousand hits within a 3 minute period with the following data in my logs:

IP: 173.192.34.95
User Agent: Aboundex/0.3 (http://www.aboundex.com/crawler/)

I’ve got a dozen or so VPS servers, and this is a repeat offender. I would really like to see a rule blocking this bot, and I don’t think there would be any SEO repercussions

edit: I should probably also mention that aboundex.com has been down for a very long time. I don’t think it’s even a legit business.

2

You always can blacklist custom User-Agents using userdata_bl_agents file. Just put each User-Agent you want to blacklist starting from new line.
Also Aboundex say that you can block crawling:

To block Aboundexbot from your entire web site you add this to your robots.txt file:

User-agent: Aboundexbot
Disallow: /

3

Thanks. If Comodo opts not to block it, then that’s definitely a valid option, though I’ll have to go through a lot of servers and make the change to each. I wanted to make a post because I suspect I am not the only one with this annoyance. Doing it on a per-site basis isn’t an option with 500+ urls.

4

Aboundex will be added to bl_scanners within next update, because it ignores robots.txt policy.