Using .htaccess to block bad bots

Andy

Administrator
Staff member
Adding the following directives to your .htaccess files allows you to control which bots are denied and allowed to access your forum.

Code:
    #   Deny and Allow bots by User-Agent
    SetEnvIfNoCase User-Agent "bot|crawler|fetcher|headlesschrome|inspect|spider" bad_bot
    SetEnvIfNoCase User-Agent "bingbot|duckduckgo|googlebot|yahoo" good_bot
    Deny from env=bad_bot
    Allow from env=good_bot

Enter the above directives into your .htaccess file as shown here:

1655743209284.png
 

Andy

Administrator
Staff member
If you would like to add additional bad bots, you add them on this line. Keep in mind that by having "bot" already entered, that will cover any bot with the work "bot" in the user agent.

SetEnvIfNoCase User-Agent "bot|crawler|fetcher|headlesschrome|inspect" bad_bot

Just add the | symbol followed by the name of the bad bot.

If you would like to add good bots, you add them on this line.

SetEnvIfNoCase User-Agent "bingbot|duckduckgo|googlebot|yahoo" good_bot

Just add the | symbol followed by the name of the good bot. Note that I prefer to remove the bingbot as I consider it a bad bot.
 

Andy

Administrator
Staff member
Is "bingbot" a bad bot? I think it is because it does not update its search criteria. Bingbot searches for links long gone and is essentially wasting lots of bandwidth.
 
Top