Recently we’ve seen servers being overloaded by the 80legs.com web crawler. Here’s a quick rewrite rule that can be placed in .htaccess to block 80legs:
|
1 2 |
RewriteCond %{HTTP_USER_AGENT} 80legs RewriteRule .* - [F,L] |
Here’s a rule to block by user agent with Nginx:
|
1 |
if ($http_user_agent ~ "80legs") { return 403; } |