Configuring Bot Defense

About bot defense

Application Security Manager™ (ASM) can proactively defend your applications against automated attacks by web robots, called bots for short. This defense method, called bot defense, can prevent layer 7 DoS attacks, web scraping, and brute force attacks from starting. By preventing bots from accessing the web site, proactive bot defense protects against these attacks as well.

Bot defense helps identify and mitigate attacks before they cause damage to the site. This feature inspects most traffic, but requires fewer resources than traditional web scraping and brute force protections. You can use bot defense in addition to the brute force protections that are available in ASM security policies. Bot defense is enforced through a bot defense profile, and does not require a security policy.

When clients access a protected web site for the first time, the system sends a JavaScript challenge to the browser. Therefore, if you plan to use this feature, it is important that clients use browsers that allow JavaScript.

If the client successfully evaluates the challenge and resends the request with a valid cookie, the system allows the client to reach the server. Requests that do not answer the challenge remain unanswered and are not sent to the server. Requests sent to non-HTML URLs without the cookie are dropped and considered to be bots.

You can configure whitelists of sources and URLs to consider safe so that the system does not need to validate them. This speeds up access time to the web site. If your application accesses many cross-domain resources and you have a list of those domains, you may want to enable validation of cross-domain requests to those domains.

About bot defense profile templates

Bot defense profile templates specify Mitigation and Verification Settings default values. When selecting a bot defense profile template, security and business requirements should be considered.

Bot defense relaxed template

A relaxed bot defense profile defines a permissive security policy that performs basic non-intrusive verification of browsers; strong verification of mobile apps using Anti-Bot Mobile Security SDK; blocks malicious bots and allows all other clients. Malicious bots are detected mostly by using bot signatures. This template provides basic protection with very low risk of false positives.

Bot defense balanced template

A bot defense balanced template defines a moderate security policy that performs advanced verification of browsers; strong verification of mobile apps using Anti-bot Mobile Security SDK; blocks malicious bots; initiates a CAPTCHA challenge for suspicious browsers; limits the total request rate produced by unknown bots and allows trusted and untrusted bots. Malicious bots and suspicious browsers are identified by using both anomaly detection algorithms and bot signatures. This template provides an advanced protection level with reduced latency impact because browser verification is performed by injecting the challenge in the HTTP response.

Bot defense strict template

A strict bot defense profile defines a strict security policy that performs advanced verification of browsers; strong verification of mobile apps using Anti-Bot Mobile Security SDK; and blocks all bots except trusted bots. This template provides the most advanced and strict protection level using all capabilities of bot defense. Browser clients are not allowed access unless they pass proactive verification. Mobile client security access requires the use of the Anti-Bot Mobile SDK.

Creating a bot defense profile

Because this defense mechanism uses reverse lookup, you need to configure a DNS Server (System > Configuration > Device > DNS) and a DNS Resolver (Network > DNS Resolver > DNS Resolver List) for it to work. The DNS Resolver must use the default route domain in its Route Domain Name field. If you are not sure of the default route domain, you can check it under Network > Route Domains. The Partition Default field is defined as Yes for the default route domain.

You can configure Application Security Manager (ASM) to protect your web site against attacks by bots before the attacks occur. Bot defense checks all traffic (except whitelisted URLs) coming to the web site, not simply suspicious traffic. Bot defense uses a set of JavaScript evaluations and bot signatures to make sure that browsers visiting your web site are legitimate.

This task described how to create a bot defense profile using the bot defense system default configurations. The enforcement mode is Transparent, meaning that violations will be logged but not mitigated and the profile template is Balanced, meaning that browser verification is after access and device IDs are generated after access.

1. On the Main tab, click Security > Bot Defense > Bot Defense Profiles.

2. Click Create.

   The Bot Profile Configuration screen opens on the General Settings tab.

3. Enter the Profile Name and click Create.

You have now configured a bot defense profile.

After you have configured a bot defense profile, you must assign it to a virtual server. Only then will bot defense protection begin on network traffic.

Assigning a bot defense profile to a virtual server

Before beginning to configure bot defense logging, ensure that you have configured a remote publisher. The logging format is Splunk (comma-separated key value pairs).

1. On the Main tab, click Security > Event Logs > Logging Profiles > Create New Logging Profile.

2. Enter a Profile Name and enable Bot Defense.

3. In the Bot Defense tab, select the desired Remote Publisher.

   The recommended configuration is:

   • Log Requests by Classification: Unknown enabled
   • Log Requests by Mitigation Action: all enabled except None.

4. Click Create to save the configuration.

5. On the Main tab, click Local Traffic > Virtual Servers > Virtual Server List and select the virtual server to associate the bot defense logging to.

6. Click Security > Policies.

7. Under Policy Settings, for Bot Defense Profile, select Enabled and select the bot defense profile from the menu.

8. In the Log Profile section, select local-bot-defense and the remote bot defense logging you created from the Available list and move it to the Selected list.

9. Click Update to save the Policy Settings.

You can view the bot defense traffic by navigating to Security > Event Logs > Bot Defense > Bot Traffic.

Configuring bot defense logging

Before beginning to configure bot defense logging, ensure that you have configured remote logging to Splunk for your system. Both the F5 DevCentral and Splunk websites have information on how to configure BIG-IP to send logs to a Splunk platform. Local logging is not recommended.

1. On the Main tab, click Security > Event Logs > Logging Profiles.

   The Logging Profile screen opens.

2. Click the name of an existing logging profile (or create a new one, then open it).

   The Logging Profile Properties screen opens.

3. Enable Bot Defense.

   The Bot Defense tab opens.

4. From the Bot Defense tab, select your preconfigured Remote Publisher from the drop down list.

5. Enable the log details you want to capture.

6. Click Update to save the logging profile.

7. On the Main tab, click Local Traffic > Virtual Servers > Virtual Server List.

8. Select the virtual server to associate the bot defense logging profile to.

   The Properties tab opens.

9. Click the Security > Policies tab.

10. Enable DoS Protection Profile.

11. In the Log Profile section, select the bot defense profile from the Available list and move it to the Selected list.

12. Click Update to save the Policy Settings.

You can view the bot defense logs by navigating to Security > Event Logs > Bot Defense > Requests.

Viewing bot defense traffic

Once you have a configured bot defense profile and network traffic, you can review your bot defense traffic statistics.

Use the bot defense traffic statistics to show if and where you need to refine your bot defense profile for maximum protection and minimum false positives.

1. On the Main tab, click Security > Event Logs > Bot Defense > Bot Traffic.

   The Bot Traffic screen opens with all configured virtual servers listed and all bot traffic summarized.

2. Select the virtual server whose traffic you want to review and then select the desired traffic time period.

   From here, you can drill down into detected bots.

Use the bot statistics to tighten or loosen the virtual server’s bot defense profile configuration.

Creating a bot defense whitelist

You can specify sources, URLs or a combination of source and URL on a whitelist that the system does not check for bot attacks. Sources on the whitelist are trusted and never blocked. Whitelist items are applied to traffic in the order in which they appear in the whitelist. Drag and drop the items to reorder them as needed.

This task describes how to create a whitelist item for a specific bot defense profile.

Note: The system includes 2 predefined whitelists to avoid false negative detection: /favicon.ico and /apple-touch-icon*.png. These whitelists should be aligned according to their location of a protected application, if applicable.

1. On the Main tab, click Security > Bot Defense > Bot Defense Profiles.

2. Click the name of an existing bot defense profile (or create a new one, then open it) and then click the Whitelist tab.

3. Click Create.

4. Enter the whitelist item properties and click Add.

5. If needed, drag and drop the whitelist item to reorder the list.

The whitelist you created is specific to the bot defense profile you created it in.

Using bot defense microservices

A bot defense microservice allows you to use specific verification and mitigation actions based on a request’s URL and hostname. Microservices provide protection from OWASP Automated Threats by hardening mitigation and verification settings for a set of URLs and hostname. Each microservice type complements Brute Force and Credential Stuffing features by filtering out detected bots that attempt to log in. Use a microservice when you want to block a specific URL.

Note: Microservices created for a security policy do not apply to a bot profile. You must create bot profile-specific microservices for a specific bot defense profile.

1. On the Main tab, click Security > Bot Defense > Bot Defense Profiles.

2. Click the name of an existing bot defense profile (or create a new one, then open it) and then click the Microservice Protection tab.

3. Click Create.

4. Enter your microservice configuration, including the service type of the microservice ( e.g. Login Protection or Intellectual Property Harvesting) and click Add.

In Security > Event Logs > Bot Defense > Bot Traffic you can filter traffic by microservice(s).

Using DoS Attack Mitigation Mode

This mode requires DoS protection enabled on the same virtual server as the Bot Defense profile.

Bot attacks are typically performed by non-browser clients. DoS Attack Mitigation Mode blocks all bots and performs verification before access for browser clients. This will block DoS attackers and allow legitimate users through. When DoS Attack Mitigation Mode is enabled, the following Mitigation and Verification Settings will be in effect during a DoS attack. This mode is enabled by default in Balanced and Strict profile templates.

1. On the Main tab, click Security > Bot Defense > Bot Defense Profiles.

2. Click the name of an existing bot defense profile (or create a new one, then open it) and then click the Bot Mitigation Settings tab.

3. Under Strict Mitigation Enforcement Cases, ensure that DoS Attack Mitigation Mode is enabled.

In Security > Event Logs > Bot Defense > Bot Requests, the Bot Defense log event includes the Enforced By field and shows that the event was handled by this mode. There is no specific log event to show that this mode is activated or inactivated.

Using API access for browsers and mobile applications

Use this when you want to ensure that browser AJAX requests to REST API are verified and allowed while bots trying to access the REST API are blocked. The system will require a valid cookie that was allocated to browser clients during Verify Before or Verify After Access actions. When API Access for Browsers and Mobile Applications is enabled, the following Mitigation and Verification Settings will be in effect during access to API URLs. This feature is disabled by default in a relaxed profile.

A URL is considered an API URL in the following cases:

• Content-type request header matches json or xml.
• Content-type response header with values json or xml.
• Request has X-Security-Request header and the Single Page Application option is enabled.
• Request is already AJAX-qualified.

1. On the Main tab, click Security > Bot Defense > Bot Defense Profiles.

2. Click the name of an existing bot defense profile (or create a new one, then open it) and then click the Bot Mitigation Settings tab.

3. Under Strict Mitigation Enforcement Cases, ensure that API Access for Browsers and Mobile Applications is enabled.

In Security > Event Logs > Bot Defense > Bot Requests, the Bot Defense log event includes the Enforced By field and shows that the event was handled by this mode. There is no specific log event to show that this mode is activated or inactivated.

Enforcing staged bot signatures

Signatures that are updated by Live Update are moved to staging. Requests that match signatures in staging are logged but not mitigated. You need to periodically review Signature Enforcement and choose which signatures to enforce to maintain optimum bot defense.

1. On the Main tab, click Security > Bot Defense > Bot Defense Profiles.

2. Click the name of the profile with Signature Staging upon Update enabled and then click the Signature Enforcement tab.

3. Review the number of signatures ready to be enforced; select those you want to enforce and click Enforce.

About cross domain requests

Bot defense allows you to specify which cross domain requests are legal. Cross domain requests are HTTP requests for resources from a different domain than the domain of the resource making the request.

If your application accesses many cross domain resources and you have a list of those domains, you can validate cross domain requests to those domains.

For example, your web site uses two domains, site1.com (the main site) and site2.com (where resources are stored). You can configure this in the Bot Defense profile by choosing when to validate cross domain requests (rather than the default to allow all cross domain requestions without validation) and specifying both of the web sites in the list of related site domains. When the browser makes a request to site1.com, it gets cookies for both site1.com and site2.com independently and simultaneously, and cross domain requests from site1.com to site2.com are allowed.

If only site1.com is configured as a related site domain, when the browser makes a request to site1.com, it gets a cookie for site1.com only. If the browser makes a cross-domain request to get an image from site2.com, it gets a cookie and is allowed only if it already has a valid site1.com cookie.

Bot defense and CORS

Cross-Origin Resource Sharing (CORS) is a way that web sites can allow resources from another origin access to your site (that is, domain + protocol + port) such as when using AJAX, @font-face, and a few other cases. Bot Defense offers three different ways for handle cross domain requests:

• Allow all requests
• Allow configured domains; validate in bulk
• Allow configured domains; validate upon request

A common type of cross-domain request is when an HTML page references resources from other domains, such as embedded images, style sheets (CSS), and JavaScript. Bot Defense supports this type of cross-domain request, and you can configure specific domains from which to allow resources in the Cross Domain Requests setting.