Using .htaccess to block bad bots

[Andy]

Adding the following directives to your .htaccess files allows you to control which bots are denied and allowed to access your forum.

    #   Deny and Allow bots by User-Agent
    SetEnvIfNoCase User-Agent "bot|crawler|fetcher|headlesschrome|inspect|spider" bad_bot
    SetEnvIfNoCase User-Agent "bingbot|duckduckgo|googlebot|yahoo" good_bot
    Deny from env=bad_bot
    Allow from env=good_bot

Enter the above directives into your .htaccess file as shown here:

 

[Andy]

If you would like to add additional bad bots, you add them on this line. Keep in mind that by having "bot" already entered, that will cover any bot with the work "bot" in the user agent.

SetEnvIfNoCase User-Agent "bot|crawler|fetcher|headlesschrome|inspect" bad_bot

Just add the | symbol followed by the name of the bad bot.

If you would like to add good bots, you add them on this line.

SetEnvIfNoCase User-Agent "bingbot|duckduckgo|googlebot|yahoo" good_bot

Just add the | symbol followed by the name of the good bot. Note that I prefer to remove the bingbot as I consider it a bad bot.

 

[Andy]

I suggest installing the following two add-ons. With them you can easily identify bad bots which might not be identified with the directives above. Once identified you can update the .htaccess bad bots directives.

https://www.xf2addons.com/resources/access-log.690/

https://www.xf2addons.com/resources/broken-links.468/

 

[Andy]

Is "bingbot" a bad bot? I think it is because it does not update its search criteria. Bingbot searches for links long gone and is essentially wasting lots of bandwidth.