F5 Logo MyF5
Search
  1. MyF5 Home
  2. BIG-IP Application Security Manager: Implementations
  3. Configuring Bot Defense
Manual Chapter : Configuring Bot Defense

Applies To:

Show Versions Show Versions

BIG-IP ASM

  • 14.1.3, 14.1.2, 14.1.0
Manual Chapter

Configuring Bot Defense

About bot defense

Application Security Manager™ (ASM) can proactively defend your applications against automated attacks by web robots, called
bots
 for short. This defense method, called
bot defense
, can prevent layer 7 DoS attacks, web scraping, and brute force attacks from starting. By preventing bots from accessing the web site, proactive bot defense protects against these attacks as well.
Bot defense helps identify and mitigate attacks before they cause damage to the site. This feature inspects most traffic, but requires fewer resources than traditional web scraping and brute force protections. You can use bot defense in addition to the brute force protections that are available in ASM security policies. Bot defense is enforced through a bot defense profile, and does not require a security policy.
When clients access a protected web site for the first time, the system sends a JavaScript challenge to the browser. Therefore, if you plan to use this feature, it is important that clients use browsers that allow JavaScript.
If the client successfully evaluates the challenge and resends the request with a valid cookie, the system allows the client to reach the server. Requests that do not answer the challenge remain unanswered and are not sent to the server. Requests sent to non-HTML URLs without the cookie are dropped and considered to be bots.
You can configure whitelists of sources and URLs to consider safe so that the system does not need to validate them. This speeds up access time to the web site. If your application accesses many cross-domain resources and you have a list of those domains, you may want to enable validation of cross-domain requests to those domains.

About bot defense profile templates

Bot defense profile templates specify
Mitigation and Verification Settings
 default values. When selecting a bot defense profile template, security and business requirements should be considered.
Relaxed
Balanced
Strict
Browser
Challenge-free verification
Verify after access (blocking)
Verify before access
Device ID Mode
None
Generate after access
Generate after access
Trusted Bot
Alarm
Alarm
Alarm
Untrusted Bot
Alarm
Alarm
Block
Suspicious Browser
Alarm
CAPTCHA
Block
Malicious Bot
Block
Block
Block
Unknown
None
Rate limit
Block
DoS Attack Mitigation Mode
Disabled
Enabled
Enabled
API Access for Browsers and Mobile Applications
Disabled
Enabled
Enabled

Bot defense relaxed template

A relaxed bot defense profile defines a permissive security policy that performs basic non-intrusive verification of browsers; strong verification of mobile apps using Anti-Bot Mobile Security SDK; blocks malicious bots and allows all other clients. Malicious bots are detected mostly by using bot signatures. This template provides basic protection with very low risk of false positives.

Bot defense balanced template

A bot defense balanced template defines a moderate security policy that performs advanced verification of browsers; strong verification of mobile apps using Anti-bot Mobile Security SDK; blocks malicious bots; initiates a CAPTCHA challenge for suspicious browsers; limits the total request rate producted by unknown bots and allows trusted and untrusted bots. Malicious bots and suspicious browsers are identified by using both anomaly detection algorithms and bot signatures. This template provides an advanced protection level with reduced latency impact because browser verification is performed by injecting the challenge in the HTTP response.

Bot defense strict template

A strict bot defense profile defines a strict security policy that performs advanced verification of browsers; strong verification of mobile apps using Anti-Bot Mobile Security SDK; and blocks all bots except trusted bots. This template provides the most advanced and strict protection level using all capabilities of bot defense. Browser clients are not allowed access unless they pass proactive verification. Mobile client security access requires the use of the Anti-Bot Mobile ASK.

Creating a bot defense profile

Because this defense mechanism uses reverse lookup, you need to configure a DNS Server (
System
Configuration
Device
DNS
) and a DNS Resolver (
Network
DNS Resolver
DNS Resolver List
) for it to work. The DNS Resolver must use the default route domain in its Route Domain Name field. If you are not sure of the default route domain, you can check it under
Network
Route Domains
. The Partition Default field is defined as Yes for the default route domain.
You can configure Application Security Manager (ASM) to protect your web site against attacks by bots before the attacks occur. Bot defense checks all traffic (except whitelisted URLs) coming to the web site, not simply suspicious traffic. Bot defense uses a set of JavaScript evaluations and bot signatures to make sure that browsers visiting your web site are legitimate.
This task described how to create a bot defense profile using the bot defense system default configurations. The enforcement mode is Transparent, meaning that violations will be logged but not mitigated and the profile template is Balanced, meaning that browser verification is after access and device IDs are generated after access.
  1. On the Main tab, click
    Security
    Bot Defense
    Bot Defense Profiles
    .
  2. Click
    Create
    .
    The
    Bot Profile Configuration
     screen opens on the
    General Settings
     tab.
  3. Enter the
    Profile Name
     and click
    Create
    .
You have now configured a bot defense profile.
After you have configured a bot defense profile, you must assign it to a virtual server. Only then will bot defense protection begin on network traffic.

Assigning a bot defense profile to a virtual server

Before beginning to configure bot defense logging, ensure that you have configured a remote publisher. The logging format is Splunk (comma-separated key value pairs).
  1. On the Main tab, click
    Security
    Event Logs
    Logging Profiles
    Create New Logging Profile
    .
  2. Enter a
    Profile Name
     and enable
    Bot Defense
    .
  3. In the
    Bot Defense
     tab, select the desired Remote Publisher.
    The recommended configuration is:
    • Log Requests by Classification
      : Unknown enabled
    • Log Requests by Mitigation Action
      : all enabled except None.
  4. Click
    Create
     to save the configuration.
  5. On the Main tab, click
    Local Traffic
    Virtual Servers
    Virtual Server List
     and select the virtual server to associate the bot defense logging to.
  6. Click
    Security
    Policies
    .
  7. Under Policy Settings, for Bot Defense Profile, select
    Enabled
     and select the bot defense profile from the menu.
  8. In the
    Log Profile
     section, select local-bot-defense and the remote bot defense logging you created from the
    Available
     list and move it to the
    Selected
     list.
  9. Click
    Update
     to save the Policy Settings.
You can view the bot defense traffic by navigating to
Security
Event Logs
Bot Defense
Bot Traffic
.

Configuring bot defense logging

Before beginning to configure bot defense logging, ensure that you have configured remote logging to Splunk for your system. Both the F5 DevCentral and Splunk websites have information on how to configure BIG-IP to send logs to a Splunk platform. Local logging is not recommended.
  1. On the
    Main
     tab, click
    Security
    Event Logs
    Logging Profiles
    .
    The Logging Profile screen opens.
  2. Click the name of an existing logging profile (or create a new one, then open it).
    The Logging Profile Properties screen opens.
  3. Enable
    Bot Defense
    .
    The Bot Defense tab opens.
  4. From the
    Bot Defense
     tab, select your preconfigured
    Remote Publisher
     from the drop down list.
  5. Enable the log details you want to capture.
  6. Click
    Update
     to save the logging profile.
  7. On the Main tab, click
    Local Traffic
    Virtual Servers
    Virtual Server List
    .
  8. Select the virtual server to associate the bot defense logging profile to.
    The Properties tab opens.
  9. Click the
    Security
    Policies
     tab.
  10. Enable
    DoS Protection Profile
    .
  11. In the
    Log Profile
     section, select the bot defense profile from the
    Available
     list and move it to the
    Selected
     list.
  12. Click
    Update
     to save the Policy Settings.
You can view the bot defense logs by navigating to
Security
Event Logs
Bot Defense
Requests
.

Viewing bot defense traffic

Once you have a configured bot defense profile and network traffic, you can review your bot defense traffic statistics.
Use the bot defense traffic statistics to show if and where you need to refine your bot defense profile for maximum protection and minimum false positives.
  1. On the Main tab, click
    Security
    Event Logs
    Bot Defense
    Bot Traffic
    .
    The Bot Traffic screen opens with all configured virtual servers listed and all bot traffic summarized.
  2. Select the virtual server whose traffic you want to review and then select the desired traffic time period.
    From here, you can drill down into detected bots.
Use the bot statistics to tighten or loosen the virtual server's bot defense profile configuration.

Creating a bot defense whitelist

You can specify sources, URLs or a combination of source and URL on a whitelist that the system does not check for bot attacks. Sources on the whitelist are trusted and never blocked. Whitelist items are applied to traffic in the order in which they appear in the whitelist. Drag and drop the items to reorder them as needed.
This task describes how to create a whitelist item for a specific bot defense profile.
The system includes 2 predefined whitelists to avoid false negative detection: /favicon.ico and /apple-touch-icon*.png. These whitelists should be aligned according to their location of a protected application, if applicable.
  1. On the Main tab, click
    Security
    Bot Defense
    Bot Defense Profiles
    .
  2. Click the name of an existing bot defense profile (or create a new one, then open it) and then click the
    Whitelist
     tab.
  3. Click
    Create
    .
  4. Enter the whitelist item properties and click
    Add
    .
  5. If needed, drag and drop the whitelist item to reorder the list.
The whitelist you created is specific to the bot defense profile you created it in.

Using bot defense microservices

A bot defense microservice allows you to use specific verification and mitigation actions based on a request's URL and hostname. Microservices provide protection from OWASP Automated Threats by hardening mitigation and verification settings for a set of URLs and hostname. Each microservice type complements Brute Force and Credential Stuffing features by filtering out detected bots that attempt to log in. Use a microservice when you want to block a specific URL.
Microservices created for a security policy do not apply to a bot profile. You must create bot profile-specific microservices for a specific bot defense profile.
  1. On the Main tab, click
    Security
    Bot Defense
    Bot Defense Profiles
    .
  2. Click the name of an existing bot defense profile (or create a new one, then open it) and then click the
    Microservice Protection
     tab.
  3. Click
    Create
    .
  4. Enter your microservice configuration, including the service type of the microservice ( e.g. Login Protection or Intellectual Property Harvesting) and click
    Add
    .
In
Security
Event Logs
Bot Defense
Bot Traffic
 you can filter traffic by microservice(s).

Using DoS Attack Mitigation Mode

This mode requires DoS protection enabled on the same virtual server as the Bot Defense profile.
Bot attacks are typically performed by non-browser clients. DoS Attack Mitigation Mode blocks all bots and performs verification before access for browser clients. This will block DoS attackers and allow legitimate users through. When
DoS Attack Mitigation Mode
 is enabled, the following
Mitigation and Verification Settings
 will be in effect during a DoS attack. This mode is enabled by default in Balanced and Strict profile templates.
Mitigation
Setting
Browser
Verify before access
Trusted Bots
Alarm
Untrusted Bots
Block
Suspicious Browsers
Block
Malicious Bots
Block
Unknown
Block
  1. On the Main tab, click
    Security
    Bot Defense
    Bot Defense Profiles
    .
  2. Click the name of an existing bot defense profile (or create a new one, then open it) and then click the
    Bot Mitigation Settings
     tab.
  3. Under
    Strict Mitigation Enforcement Cases
    , ensure that
    DoS Attack Mitigation Mode
     is enabled.
In
Security
Event Logs
Bot Defense
Bot Requests
, the Bot Defense log event includes the Enforced By field and shows that the event was handled by this mode. There is no specific log event to show that this mode is activated or inactivated.

Using API access for browsers and mobile applications

Use this when you want to ensure that browser AJAX requests to REST API are verified and allowed while bots trying to access the REST API are blocked. The system will require a valid cookie that was allocated to browser clients during Verify Before or Verify After Access actions. When
API Access for Browsers and Mobile Applications
 is enabled, the following
Mitigation and Verification Settings
 will be in effect during access to API URLs. This feature is disabled by default in a relaxed profile.
Mitigation
Setting
Browser
Verify before access
Trusted Bots
Block
Untrusted Bots
Block
Suspicious Browsers
Block
Malicious Bots
Block
Unknown
Block
A URL is considered an API URL in the following cases:
  • Content-type request header matches
    json
     or
    xml
    .
  • Content-type response header with values
    json
     or
    xml
    .
  • Request has X-Security-Request header and the Single Page Application option is enabled.
  • Request is already AJAX-qualified.
  1. On the Main tab, click
    Security
    Bot Defense
    Bot Defense Profiles
    .
  2. Click the name of an existing bot defense profile (or create a new one, then open it) and then click the
    Bot Mitigation Settings
     tab.
  3. Under
    Strict Mitigation Enforcement Cases
    , ensure that
    API Access for Browsers and Mobile Applications
     is enabled.
In
Security
Event Logs
Bot Defense
Bot Requests
, the Bot Defense log event includes the Enforced By field and shows that the event was handled by this mode. There is no specific log event to show that this mode is activated or inactivated.

Enforcing staged bot signatures

Signatures that are updated by Live Update are moved to staging. Requests that match signatures in staging are logged but not mitigated. You need to periodically review
Signature Enforcement
and choose which signatures to enforce to maintain optimum bot defense.
  1. On the Main tab, click
    Security
    Bot Defense
    Bot Defense Profiles
    .
  2. Click the name of the profile with
    Signature Staging upon Update
     enabled and then click the
    Signature Enforcement
     tab.
  3. Review the number of signatures ready to be enforced; select those you want to enforce and click
    Enforce
    .

About cross domain requests

Bot defense allows you to specify which cross domain requests are legal.
Cross domain requests
 are HTTP requests for resources from a different domain than the domain of the resource making the request.
If your application accesses many cross domain resources and you have a list of those domains, you can validate cross domain requests to those domains.
For example, your web site uses two domains,
site1.com
 (the main site) and
site2.com
 (where resources are stored). You can configure this in the Bot Defense profile by choosing when to validate cross domain requests (rather than the default to allow all cross domain requestions without validation) and specifying both of the web sites in the list of related site domains. When the browser makes a request to
site1.com
, it gets cookies for both
site1.com
 and
site2.com
 independently and simultaneously, and cross domain requests from
site1.com
 to
site2.com
 are allowed.
If only
site1.com
 is configured as a related site domain, when the browser makes a request to
site1.com
, it gets a cookie for
site1.com
 only. If the browser makes a cross-domain request to get an image from
site2.com
, it gets a cookie and is allowed only if it already has a valid
site1.com
 cookie.

Bot defense and CORS

Cross-Origin Resource Sharing (CORS)
 is a way that web sites can allow resources from another origin access to your site (that is, domain + protocol + port) such as when using AJAX, @font-face, and a few other cases. Bot Defense offers three different ways for handle cross domain requests:
  • Allow all requests
  • Allow configured domains; validate in bulk
  • Allow configured domains; validate upon request
A common type of cross-domain request is when an HTML page references resources from other domains, such as embedded images, style sheets (CSS), and JavaScript. Bot Defense supports this type of cross-domain request, and you can configure specific domains from which to allow resources in the
Cross Domain Requests
 setting.
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information