Manual Chapter :
Monitoring Bot Defense Activity
Applies To:
Show Versions![Show Versions](/etc/designs/pcx/techdocs/images/expandversions.gif)
BIG-IQ Centralized Management
- 8.3.0, 8.2.0, 8.1.0, 8.0.0
Monitoring Bot Defense Activity
Isolating bot traffic
Your bot defense profile defines the security policy applied to bot traffic,
once a bot request is detected by the virtual server. Once it detects bot requests, your system
will mitigate the traffic according to the policy configured to the corresponding
profile. The volume of denied traffic, as opposed to accepted traffic, can indicate a need to
adjust your traffic mitigation protocols within your bot defense profile.
For managed BIG-IP devices running version 14.1.x, or later, you can centrally
monitor the data for all the bot traffic detected by your bot defense profiles. Use
this data to evaluate whether you need to fine-tune your current bot defense
configuration on the host BIG-IP device.
Evaluating bot defense traffic
You can evaluate all bot traffic detected by your bot defense profiles using the Bot Traffic Dashboard (
). This screen provides a summary of how much bot traffic was detected, the most common types of bot requests to your application, and how these requests were managed. You can use the data on this screen to drill down into more granular information about the virtual servers that reported data. ![](/content/dam/projects/techdocs/bigiq-8-0-0/central-mgmt/images/bot-summary-dashboard.png/_jcr_content/renditions/original)
Time Resolution
The time setting is a filter where you can select the period of time that is displayed over the entire dashboard. The charts display data reported by virtual servers with a bot defense profile over the selected period of time.
Summary
The summary row of charts organizes bot request data by status, class, and mitigation. You can click the title of any of these charts to view a list of the virtual servers reporting bot traffic data.
- TRAFFIC BY STATUSdisplays the distribution of request outcomes compared to the total number of requests. Traffic by status indicates the volume of accepted versus denied traffic.
- TRAFFIC BY CLASSdisplays the distribution of the type of bot requests to your application. Traffic by class can indicate which types of bot traffic are trying to access your application, and whether your bot defense profile requires specific mitigation settings.
- TRAFFIC BY MITIGATIONdisplays the distribution of actions taken in response to the bot requests. Traffic by mitigation can indicate which bot defense mitigation settings are most commonly used against detected bot traffic. This can also indicate whether your current enforcement mode is allowing bot traffic to your application, or blocking valid traffic.
Status Trends
The status trends charts display the average trends of bot request outcomes over the selected period of time. The ALL TRAFFIC chart displays the outcome of all requests, while the TRAFFIC BY CLASS charts display the most common bot classes detected, and their request outcomes.
Mitigation
The mitigation charts display the most common mitigation actions for accepted and denied bot requests, over the selected period of time.
As shown in the image, if there are fewer than three aspects reported, the charts display NO DATA.
Review bot traffic by virtual server
Before you can view bot traffic isolated by virtual server, you must have a bot defense profile running on a BIG-IP device version 14.1 or later. You must also have statistics enabled on the BIG-IP device.
You can view virtual servers reporting unusual or high bot activity to ensure that their bot defense profile is properly configured to manage and mitigate bot traffic.
- Go to.The screen displays current summary information about all traffic processed by your bot defense profiles. You can change the time period using the control at the top left of the screen.
- Using the charts at the top of the screen, evaluate the total number of requests and the overall distribution of traffic by status, class, or mitigation over the selected period of time.
- To find a specific time in which the total bot traffic volume either increased or decreased, view the ALL TRAFFIC chart in the center row of the screen.
- To view the most common types of bot traffic and status over time, use the charts under TRAFFIC BY CLASS.These charts can indicate when your virtual servers detected a specific type of bot requests.
- To identify the most common mitigation actions over time, view the MITIGATION charts at the bottom of the screen.These charts indicate the most common mitigation actions for accepted bot requests and denied bot requests. Accepted mitigation actions, such as Alarm, , indicate an increase in potentially harmful bot traffic that was accepted due to a Transparent enforcement mode on your bot defense profile. An increase in denied mitigation actions can indicate false positives, and might require additional evaluation of traffic class by virtual server.
- To display a list of virtual servers and their distribution of data by status, class, or mitigation, click theTRAFFIC BY STATUS,TRAFFIC BY CLASS, orTRAFFIC BY MITIGATIONheaders.The screen displays a list of virtual servers that detected bot traffic using a bot defense profile. The summary bar at the top of the screen displays data from all bot requests.
- To isolate bot request data from a specific virtual server, click the virtual server's row.Data for the selected virtual server is displayed at the bottom of the screen.
Once you have isolated a virtual server, click
VIEW REQUEST LOG
to review additional details about the bot requests. If you have finished troubleshooting an issue with your virtual server's bot request management, edit the bot defense profile on the virtual server's host BIG-IP system.View bot request logs
To view bot request logs, you must have configured bot logging. For more information, see . You also must have a bot defense profile enabled to a BIG-IP device running version 14.1 or later. Ensure that Web Application Security is activated for your DCD services.
Configure bot defense logging
You can view a complete list of the bot requests detected by bot defense. This allows you to better identify details of specific requests, or request types, to your applications.
- Go to.The screen displays a list of all bot requests. Each request in the list displays request parameters detected by your bot defense.
- Click a row to display additional details of the selected bot request.
- To filter the request log by a specific virtual server, bot class, Bot Defense action (Request Status) you can use the filter bar in the upper right side of the list. To sort requests, click the column header.
- Use the Request Details area to identify the logged request's general information, including the URL, source and destination IP, Host BIG-IP device, and the bot defense profile.
- Use the Request Details area to identify general information about detected about the request.General information includes contents of the request header, the request targets, current status, and mitigation action.
- To view the entire request header, see the Request area at the bottom of the screen.
- Use the Bot Details area to see why the system identified the request as a bot request.SeeBot classesandBot Categoriesfor more information about these bot details.
- Use the Verification Action and Challenge Status area to view details about why a request received a request status and mitigation action.
Create a new log filter
You can create new filters to better manage the
events in your logs. The filters are based on a fixed set of query parameters, with an
option to manually enter all available parameters into a query expression. For more
details about the required syntax, see
Query expression
syntax for log
filters.
- From the log screen, click the filter icon at the top right of the screen (
).
- ClickCreate.The New Filter configuration popup screen opens.
- Type a uniqueFilter Name.
- In the Query Parameters area, add the query information.Adding information to these fields automatically populates theQuery Expressionbox. Refer to the Query expression syntax for log filters to view all query options.
- Once you have the custom filter the way you want it, clickSave & Apply.
The new filter is added to the filter list. You can select this filter later to query the list according to the set parameters.
Query expression syntax for log filters
On the New Filter configuration popup screen, the Query Expression area for creating a new log filter
requires specific syntax. To manually run query parameters, use the syntax requirements
listed here.
General Syntax
- Express elements of the filter query as key value pairs, separated by a colon, such asprofile_name:"MyCurrentProfile".
- Use the following operators within a filter query.OperatorUsage ExampleANDThis:p1 AND bar:(A AND B AND "another value")AND NOTAND NOT qux:errorORname:"this is a name" OR bar:(A OR B OR C)OR NOTOR NOT qux:error*support_id:*123*. This operator can only be used for text fields.
- Enclose values that have spaces within quotation marks, such askey:"two words".
- Query any field for more than one value by enclosing the values with parentheses, such askey:(a b "two words"). In this case, the default operator is OR.
- Only pre-defined values are allowed for fields with a type of multi-value. These values are listed in the Query Parameters area, next to the relevant field.
- In a policy name, you must include the full path to the policy, such as/Common/MyPolicy.
Dates
- Values with a type of date can accept valid date formats, such as'Oct 30, 2017 00:00:00'.
- Values of the date range type can accept input in the format of[min_date...max_date], such as'[Oct 30, 2017 00:00:00...Oct 30, 2017 06:00:00]'. The date range might also contain only minimum without maximum, and the reverse, such as'[Oct 30, 2017 00:00:00...]'or'[...Oct 30, 2017 00:00:00]'.
Numeric Values
- Values of the numeric range type can accept input in the format of[min...max], such as'[1...100]'. The numeric range might also contain only minimum without maximum, and the reverse, such as'[1...]'or'[...100]'.
Bot classes
The bot defense process classifies client traffic to verify the threat level,
and then applies the configured mitigation action. The bot profile classifies clients using
browser and mobile app verification tests, bot signatures, and anomaly detection algorithms.
Bot signatures and anomalies are grouped into these classes.
Bot Class | Description |
---|---|
Browser | Browser clients that successfully passed
browser verification tests |
Mobile Application | Mobile app clients with Anti-Bot Mobile SDK, and predefined mobile apps
that successfully passed verification. |
Trusted Bot | Clients that are detected with search engine signatures. |
Untrusted Bot | Clients that are detected with signatures for non-malicious tools
and bots, such as crawlers, site monitors, and HTTP libraries. |
Suspicious Browser | Browser clients that failed specific browser verification tests.
|
Malicious Bot | Malicious clients that are detected using bot signatures, browser
verification tests, and anomaly detection heuristics. These bots can
include DoS tools, exploit tools, and vulnerability scanners. |
Unknown | Clients that were not classified by any other class. Typically, these
are non-browser clients that cannot be identified using known bot
signatures. |
Bot categories
This list shows default bot signatures and anomalies that are used to identify bots. They use specific
patterns in the headers of the incoming HTTP request. These signatures are categorized in order to classify the bot's threat against your system, and to perform the most
effective mitigation action. The table provides a description of these bot signature
categories and their associated bot class.
Bot Category | Associated Bot Class | Type |
---|---|---|
DoS Tool | Malicious Bot | Signature |
E-mail Collector | Malicious Bot | Signature |
Exploit Tool | Malicious Bot | Signature |
Network Scanner | Malicious Bot | Signature |
Spyware | Malicious Bot | Signature |
Vulnerability Scanner | Malicious Bot | Signature |
Web Spider | Malicious Bot | Signature |
Webserver Stress Tool | Malicious Bot | Signature |
Mobile App without SDK | Mobile Application | Signature |
Search Engine | Trusted Bot | Signature |
Crawler | Untrusted Bot | Signature |
Headless Browser | Untrusted Bot | Signature |
HTTP Library | Untrusted Bot | Signature |
RSS Reader | Untrusted Bot | Signature |
Search Bot | Untrusted Bot | Signature |
Service Agent | Untrusted Bot | Signature |
Site Monitor | Untrusted Bot | Signature |
Social Media Agent | Untrusted Bot | Signature |
Spam Bot | Untrusted Bot | Signature |
Web Downloader | Untrusted Bot | Signature |
Browser Automation | Malicious Bot | Anomaly |
Browser Masquerading | Malicious Bot | Anomaly |
Classification Evasion | Malicious Bot | Anomaly |
Headless Browser Anomalies | Malicious Bot | Anomaly |
Illegal Mobile App | Malicious Bot | Anomaly |
Malicious Browser Extensions | Malicious Bot | Anomaly |
Mobile App Automation | Malicious Bot | Anomaly |
Mobile App Masquerading | Malicious Bot | Anomaly |
OWASP Automated Threat | Malicious Bot | Anomaly |
Search Engine Masquerading | Malicious Bot | Anomaly |
Suspicious Browser Extensions | Suspicious Browser | Anomaly |
Suspicious Browser Types | Suspicious Browser | Anomaly |
Bot mitigation and browser verification
Requests that match bot criteria undergo evaluation and
processing by the bot profile criteria. The request status indicates the outcome of the bot
defense evaluation process. This table describes the mitigation actions. Additional request
results that appear in the Bot Traffic Dashboard are also listed in the table below (To view
bot dashboards see
).This information is based on the configuration of bot defense for the
managed BIG-IP device. Bot defense profiles are supported by BIG-IP devices running
version 14.1, or later. To view your bot profile, go to
Mitigation Action | Request Status | Description |
---|---|---|
None | Accepted | Request was passed by the server with no
mitigation action. |
Alarmed | Accepted | Request was passed to the server, but triggered an alarm. This could
indicate a possible bot threat detected under a transparent bot profile.
|
CAPTCHA | Varies depending on challenge outcome | A CAPTCHA challenge was sent to the client. The request is
either accepted or denied, based on the challenge result. |
Block | Denied | The request was not passed to the server and a blocking notification
was sent. |
Honeypot Page |
Redirect |
The request is redirected to a honeypot
response page. The following mitigation is
supported by BIG-IP version 15.0, or later |
Redirect to Pool |
Redirect |
The request was passed to an alternate server
pool. The following mitigation is supported by
BIG-IP version 15.0, or later |
TCP Reset |
Denied |
The request was not passed to the server, and
the connection was terminated. This occurs when the configured rate
limit for an unknown bot class is exceeded, or if this option is
configured for the bot class. |
Rate Limit (Unknown bot only) |
Denied |
The system processes the configured number of allowed transactions
per second, and terminates connections with unknown bot types, once the
system reaches threshold. |
Verification Action |
Request Status |
Description |
---|---|---|
Device ID Challenge |
Denied |
Device ID challenge was sent to the client,
and the client failed to solve the challenge. The request was not passed
to the server. |
Browser Verification Challenge |
Denied |
Browser verification challenge was sent to the
client, and the client failed to solve the challenge. The request was
not passed to the server. |
Allowlisted |
Accepted |
Request contained a allowlisted signature and was passed by the
server with no mitigation action. |
Aggregated (Data Result) |
Denied |
Requests with various mitigation actions.
This mitigation is rare and occurs when
the system groups multiple denied requests with more than one
mitigation action. |
Value |
Request Status |
Description |
---|---|---|
Allowlisted |
Accepted |
Request contained a allowlisted signature and was passed by the
server with no mitigation action. |
Aggregated (Data Result) |
Denied |
Requests with various mitigation actions.
This mitigation is rare and occurs when
the system groups multiple denied requests with more than one
mitigation action. |