Tab – Bot Management

Tab - Bot Management

Fields/buttons

Description

Log only

Do not enforce blocking based on bot detection.

Enforce client cookie support

Only clients implementing a cookie store can access the application through this mapping if enabled. In contrast to regular browsers, most bots do not implement a cookie store and will, therefore, be blocked if this setting is enabled.

Source IP address whitelist

Requests from a client (e.g., a bot) with an IP address in the selected list are forwarded regardless of whether the client supports a cookie store. To define a source IP list, see article Submenu – IP Address Lists.

Well-Known Bot Exceptions

Check User-Agent

Check the User-Agent to determine if a bot is well-known and do not block such bots.

Clients sending one of the following User-Agent header values are treated as well-known bots:

  • AhrefsBot
  • Amazonbot
  • AppleBot
  • archive.org_bot
  • Baiduspider
  • bingbot
  • BingPreview
  • DataForSeoBot
  • DMBrowser
  • DuckDuckBot
  • Embedly
  • facebookexternalhit
  • Googlebot
  • InfoTigerBot
  • LinkedInBot
  • localsearch-web
  • MSNBot
  • Qwantify
  • Semrush
  • SiteAuditBot
  • Slurp
  • Yahoo Link Preview
  • YandexBot

Enforce source-domain

If enabled, a reverse IP lookup for well-known bots is performed to verify that the client's IP address belongs to the operator of a well-known bot. This prevents bots from pretending to be a well-known bot by sending a fake User-Agent header.

We strongly recommend enabling the Enforce source-domain option to block fake User-Agent headers.

The following domains are considered as domains of operators operating well-known bots:

  • ahrefs.com
  • applebot.apple.com
  • archive.org
  • baidu.com
  • baidu.jp
  • bl.semrush.com
  • bot.semrush.com
  • crawl.amazonbot.amazon
  • crawl.yahoo.net
  • dataforseo.com
  • dotcom-monitor.com
  • duckduckgo.com
  • embed.ly
  • fbsv.net
  • fwd.linkedin.com
  • googlebot.com
  • google.com
  • infotiger.com
  • qwant.com
  • search.ch
  • search.msn.com
  • yandex.com
  • yandex.net
  • yandex.ru

Custom Bot Exceptions

Check User-Agent

Check User-Agent – if enabled, custom bots are not blocked. Custom bots are identified by providing a User-Agent and Source-Domain pattern.

If Check User-Agent and Enforce source-domain are enabled, both patterns User-Agent pattern and Source-domain pattern have to match such that a request is not blocked.

User-Agent pattern

Do not block bots whose User-Agent header matches this pattern.

Enforce source-domain

Do not block bots whose source-domain matches the Source-domain pattern.

Source-domain pattern

The field for the source-domain pattern.

Ensure that the pattern ends with a $ to prevent wrong matches. For example, the pattern \.example\.com$ should be used instead of example.com. The pattern example.com would also match:

  • myexample.com
  • example.com.otherdomain.org
  • examplexcom.otherdomain.org
  • etc.