Recently we’ve seen servers being overloaded by the 80legs.com web crawler. Here’s a quick rewrite rule that can be placed in .htaccess to block 80legs:
|
1 2 3 4 |
RewriteCond %{HTTP_USER_AGENT} 80legs RewriteRule .* - [F,L] |
Here’s a rule to block by user agent with Nginx:
|
1 2 3 |
if ($http_user_agent ~ "80legs") { return 403; } |
2 Responses
that broke access to the website….
RewriteCond %{HTTP_USER_AGENT} 80legs
RewriteRule .* – [F,L]
if ($http_user_agent ~ “80legs”) { return 403; }
gt, the “if ($http_user …” line is for Nginx servers only and should not be used on your Apache server.
Lee