F5 Logo MyF5
Search
  1. MyF5 Home
  2. Knowledge Centers
  3. BIG-IP ASM
  4. BIG-IP Application Security Manager: Attack and Bot Signatures
  5. Assigning Bot Signatures to Security Policies
Manual Chapter : Assigning Bot Signatures to Security Policies

Applies To:

Show Versions Show Versions

BIG-IP ASM

  • 12.1.6, 12.1.5, 12.1.4, 12.1.3, 12.1.2, 12.1.1, 12.1.0
Manual Chapter
Table of Contents | << Previous Chapter | Next Chapter >>

Assigning Bot Signatures to Security Policies

About bot signatures

Bot signatures identify web robots by looking for specific patterns in the headers of incoming HTTP requests. DoS Layer 7 bot detection includes many signatures that identify bots, and you can also write your own for customized bot defense.

Bot signatures carefully identify bots and have a low rate of producing false positive results. The signatures identify the type of bot for classification and investigative purposes, and can distinguish between benign and malicious bots.

Benign bots can be useful for providing Internet services such as search engine bots, index crawlers, site monitors, and those used to establish availability and response time. Some environments may not want to block benign bot traffic. But attackers use malicious bots for more harmful purposes such as harvesting email addresses, producing spam, and developing exploitation tools. You may want to block malicious bots because they can orchestrate DoS attacks, waste internet resources, and search for vulnerabilities to exploit in your application.

Being able to classify bots allows you to treat them differently. You can report, block, or do nothing when a signature matches a malicious or benign bot. Further, malicious and benign bots fall into more specific bot signature categories that can be handled as needed. You can create new categories if needed for custom bot signatures.

Using proactive bot defense

For you to use proactive bot defense, client browsers accessing your web site must be able to accept JavaScript. Because this defense mechanism uses reverse lookup, you need to configure a DNS Server ( System > Configuration > Device > DNS ) and a DNS Resolver ( Network > DNS Resolvers > DNS Resolver List ) for it to work.
You can configure Application Security Manager™ (ASM) to protect your web site against attacks by web robots (called bots, for short) before the attacks occur. Proactive bot defense checks all traffic (except whitelisted URLs) coming to the web site, not simply suspicious traffic. This DoS protection uses a set of JavaScript evaluations and bot signatures to make sure that browsers visiting your web site are legitimate.
Important: Proactive bot defense has limitations if your web site uses Cross-Origin Resource Sharing (CORS), for example, with AJAX requests.
  1. On the Main tab, click Security > DoS Protection > DoS Profiles .
    The DoS Profiles list screen opens.
  2. Click the name of an existing DoS profile (or create a new one).
    The DoS Profile Properties screen for that profile opens.
  3. On the left, under Application Security, click General Settings, and ensure that Application Security is enabled.
    If Application Security is disabled, click Enabled.
    The screen displays additional settings.
  4. On the left, click Proactive Bot Defense.
  5. To set the Operation Mode to enable proactive bot defense, click Edit and specify when to implement proactive bot defense.
    Option Description
    During Attacks Checks all traffic during a DoS attack, and prevents detected attacks from escalating.
    Always Checks all traffic at all times, and prevents DoS attacks from starting.
    Important: If you enable Proactive Bot Defense and your web site uses CORS (Cross-Origin Resource Sharing), we recommend that you add the CORS URLs to the proactive bot URL whitelist.
    The system enables Bot Signatures to enforce Proactive Bot Defense. By default, the system blocks requests from highly suspicious browsers and displays a default CAPTCHA (or visual character recognition) challenge to browsers that are suspicious.
  6. By default, the Block requests from suspicious browsers setting is enabled. You can clear either Block Suspicious Browsers or CAPTCHA Challenge if you do not want to block suspicious browsers or send a CAPTCHA challenge.
    You can also change the CAPTCHA response by clicking CAPTCHA Settings. (Another task explains how to configure CAPCHA when setting up DoS protection.)
  7. In the Grace Period field, type the number of seconds to wait before the system blocks suspected bots.
    The default value is 300 seconds.
    The grace period allows web pages (including complex pages such as those which include images, JS, and CSS) the time to be recognized as non-bots, receive validation, and completely load without unnecessarily dropping requests.
    The grace period begins after the client is validated, a configuration change occurs, or when proactive bot defense starts as a result of a detected DoS attack or high latency.
  8. Using the Cross-Domain Requests setting, specify how the system validates cross-domain requests (such as requests for non-HTML resources like embedded images, CSS style sheets, XML, JavaScript, or Flash).
    Cross-domain requests are requests with different domains in the Host and Referrer headers.
    Option Description
    Allow all requests

    Allows requests arriving to a non-HTML URL referred by a different domain and without a valid cookie if they pass a simple challenge. The system sends a challenge that tests basic browser capabilities, such as HTTP redirects and cookies.
    Allow configured domains; validate in bulk Allows requests to other related internal or external domains that are configured in this section, and validates the related domains in advance. The requests to related site domains must include a valid cookie from one of the site domains; the external domains are allowed if they pass a simple challenge. Choose this option if your web site does not use many domains, and then include them all in the lists below.

    Also, if your website uses CORs, select this option and then specify the WebSocket domain in the Related Site Domains list.
    Allow configured domains; validate upon request Allows requests to other related internal or external domains that are configured in this section. The requests to related site domains must include a valid cookie from the main domain; the external domains are allowed if they pass a simple challenge. Choose this option if your web site uses many domains, and include the main domain in the list below.
  9. If you selected one of the Allow configured domains options in the last step, you need to add Related Site Domains that are part of your web site, and Related External Domains that are allowed to link to resources in your web site.
  10. In the URL Whitelist setting, add the non-HTML resource URLs for which the web site expects to receive requests and that you want the system to consider safe.
    Type URLs in the form /index.html, then click Add.. Wildcards are supported.
    Tip: If your web site uses CORS, add the CORS URLs to the whitelist, otherwise, they will be blocked.
    The system does not perform proactive bot defense on requests to the URLs in this list.
  11. Click Update to save the DoS profile.
You have now configured proactive bot defense which protects against DDoS, web scraping, and brute force attacks (on the virtual servers that use this DoS profile).

The system sends a JavaScript challenge to traffic accessing the site for the first time. Legitimate traffic answers the challenge correctly, and resends the request with a valid cookie; then it is allowed to access the server. The system drops requests sent by browsers that do not answer the system’s initial JavaScript challenge (considering those requests to be bots). The system also automatically enables bot signatures and blocks bots known to be malicious.

If proactive bot detection is always running, ASM™ filters out bots before they manage to build up an attack on the system and cause damage. If using proactive bot defense only during attacks, once ASM detects a DoS attack, the system uses proactive bot defense for the duration of the attack.

Proactive bot defense is used together with the active mitigation methods specified in TPS- and stress-based detection. Any request that is not blocked by the active mitigation method still has to pass the proactive bot defense mechanism to be able to reach the server (unless it is on the URL whitelist). Proactive bot defense blocks requests to CORS (Cross-Origin Resource Sharing) URLs not on the URL whitelist.

Configuring bot signature checking

If you need to create custom bot signatures and categories for your application, you should do this before configuring bot signature checking. Navigate to Security > Options > DoS Protection > Bot Signatures . Otherwise, you can use the system-supplied bot signatures and categories.

Because this defense mechanism uses reverse lookup, you need to configure a DNS Server ( System > Configuration > Device > DNS ) and a DNS Resolver ( Network > DNS Resolvers > DNS Resolver List ) for it to work.

Bot signature checking is typically used with proactive bot defense (and is enabled by default when you use proactive bot defense). The system performs bot signature checking, which identifies known bots as legitimate or malicious based on their HTTP characteristics. You can specify whether to ignore, report, or block certain categories of malicious or benign bots. You can also disable specific bot signatures, if needed.
  1. On the Main tab, click Security > DoS Protection > DoS Profiles .
    The DoS Profiles list screen opens.
  2. Click the name of an existing DoS profile (or create a new one).
    The DoS Profile Properties screen for that profile opens.
  3. On the left, under Application Security, click General Settings, and ensure that Application Security is enabled.
    If Application Security is disabled, click Enabled.
    The screen displays additional settings.
  4. On the left, click Bot Signatures to display the settings.
  5. For the Bot Signature Check setting, click Edit and select Enabled if it is not already selected.
  6. In the Bot Signature Categories field, for each category of bots, both malicious and benign, select the action to take when a request matches a signature in that category.
    Option Action
    None Ignore requests in this category.
    Report Log requests in this category.
    Block Block and report requests in this category.
    You can select one action for all malicious or all benign categories, or have different actions for separate categories.
    Note:

    These settings override the Proactive Bot Defense settings. For example, requests from bots in any category, if set to Block, are always blocked.

  7. If certain signatures need to be disabled, in the Bot Signatures List options, move the signatures to the Disabled Signatures list.
  8. Click Update to save the DoS profile.
You have specified how to perform bot signature checking on your system. By comparing the bot signatures with requests, the system can identify those made by different categories of bots and will ignore, report, or block requests from bots it discovers.
If using bot signature checking, you will want to keep the signatures up to date. You can configure bot signatures (and all other signatures) to be updated automatically or update them manually using the Security Updates feature. A security update downloads the latest new and updated bot signatures and attack signatures.
Table of Contents | << Previous Chapter | Next Chapter >>
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5 Networks, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information