F5 Logo MyF5
Search
  1. MyF5 Home
  2. BIG-IQ: Monitoring and Reports
  3. Configuring Statistics Collection
Manual Chapter : Configuring Statistics Collection

Applies To:

Show Versions Show Versions
Manual Chapter

Configuring Statistics Collection

BIG-IP device configuration requirements for viewing statistics in BIG-IQ

Before you can enable statistics collection for centralized management, you must ensure that the BIG-IP device has the proper configuration. The proper configuration varies depending on the version of the BIG-IP device. The minimum supported BIG-IP device is version 12.1.0. BIG-IQ has limited visibility for BIG-IP devices prior to 13.1.0.5.
For details about how to configure statistics visibility, based on the BIG-IP version, see
Enabling statistics collection during device discovery
.
For details on how to access statistic information, based on the BIG-IP version and service, refer to
Statistics compatibility and visibility
.

Monitoring BIG-IP statistics in BIG-IQ

Visibility of statistics in BIG-IQ depends on the version of your managed BIG-IP devices. Devices running versions 13.1.X, or earlier, have limited statistics visibility support within BIG-IQ. Below outlines the compatibility and what to expect when accessing Analytics (AVR) data within BIG-IQ. For more information, see the supporting documentation found in the
BIG-IQ Centralized Management: Monitoring and Reports
 guide.

Statistics visibility of managed BIG-IP devices

The format in which statistics are presented in the BIG-IQ environment, depends on the managed version of BIG-IP and the service presented. Refer to the table to access statistics visibility, based on the managed device version. Ensure that the managed device configuration meets the requirements outlined below.

Minimum configuration requirements:

BIG-IP Version 13.1.x or earlier
  • Ports 22 and 443 on each BIG-IP device must be open for the BIG-IQ DCD to retrieve data.
  • There must be a Data Collection Device (DCD) configured to your BIG-IQ.
BIG-IP Version 13.1.0.5 or later
  • You must have AVR provisioned for each BIG-IP device.
  • BIG-IQ needs to provide access on Port 443 to receive BIG-IP AVR data.
  • There must be a Data Collection Device (DCD) configured to your BIG-IQ.
    To view statistics, ensure that the licenses for your managed BIG-IP devices include root access. A BIG-IP license running in Appliance Mode, will not allow for statistics visibility in the BIG-IQ environment.

Where to view statistics

Location of service statistics per managed BIG-IP version
BIG-IP v12.1
BIG-IP v13.0
BIG-IP v13.1
BIG-IP v13.1.0.5
BIG-IP v14.0
BIG-IP v14.1
BIG-IP v15.0 or later
Device Traffic
Monitoring
DASHBOARDS
Device
Local Traffic (General)
Monitoring
DASHBOARDS
Local Traffic
Local Traffic (HTTP)
Not available to this version
Monitoring
DASHBOARDS
Local Traffic
HTTP
DNS (General)*
Monitoring
DASHBOARDS
DNS
Network Firewall (General)
Monitoring
REPORTS
Security
Network Firewall
Reporting
Network Firewall information is provided by ACL, IP Reputation, and IPS.
Network Firewall (ACL)
Not applicable to this version
Monitoring
DASHBOARDS
AFM
Network Security (IP Reputation)
Not applicable to this version
Monitoring
DASHBOARDS
AFM
Network Firewall (IPS)
Not applicable to this version
Monitoring
DASHBOARDS
IPS
Web Application Security (General)
Monitoring
REPORTS
Security
Web Application Security
Reporting
Monitoring
DASHBOARDS
Web Application Security
Web Application Security (Bot)
Not available to this version
Monitoring
DASHBOARDS
Bot Traffic
DDoS (Shared Security)
Not available to this version
Monitoring
DASHBOARDS
DDoS
Application Summary
Applications
APPLICATIONS
(limited statistics visibility)
Applications
APPLICATIONS
Secure Web Gateway
Not available to this version
Monitoring
DASHBOARDS
SWG
SSLO**
Not available to this version
Monitoring
DASHBOARDS
SSLO
Access
Monitoring
DASHBOARDS
Access
Not available to this version
*Top Charts are only available to BIG-IP version 13.1.0.5 or later.
**SSLO support is available to versions 5.4 to 5.9.

How do I start viewing BIG-IP device statistics from BIG-IQ?

To start viewing statistics for a BIG-IP device, you must have enabled statistics collection for that device. You can do that either during or after adding the device to the BIG-IP Devices inventory list on the BIG-IQ system. You also need to install, configure, and add a data collection device before you can view statistics for your managed BIG-IP devices.

Enabling statistics collection during device discovery

Before you can enable statistics for BIG-IP devices:
  • There must be a BIG-IQ data collection device (DCD) configured for the BIG-IQ device.
  • The BIG-IP device must be located in your network and running a compatible software version. Refer to K34133507#cm6.0.1 for more information.
    • For BIG-IP devices running version 13.1.0.5 or later, you must have AVR provisioned.
  • For BIG-IP devices running versions prior to 13.1.0.5, configure Ports 22 and 443 must be open to traffic from the BIG-IQ DCD to the managed BIG-IP devices.
  • For BIG-IP devices running version 13.1.0.5 or later, BIG-IQ needs to provide access on Port 443 so that the BIG-IP AVR module can send statistics to the BIG-IQ DCD
One way to enable statistics collection for BIG-IP devices is to do it when you add those devices to the BIG-IQ system inventory. Adding devices to the inventory is referred to as
device discovery
. If the devices you want to enable have already been discovered, refer to
Enabling collection after device discovery
.
The ADC component is automatically included (first) any time you discover or import services for a device.
You do not need to discover and import a device’s configuration to collect and view statistics for it. You just need to establish trust between your BIG-IQ and the device. If you do not discover and import the device configuration, the virtual servers, pool, pool members, and iRules will be visible in the statistics dimension panes, but these objects will not appear in the configuration page for those objects. Also, you will not be able to manage these objects in BIG-IQ. If you decide you want to manage these objects, you can discover and import the BIG-IP device’s configuration later without interrupting statistics collection.
  1. At the top of the screen, click
    Devices
    .
  2. Click the
    Add Device(s)
     button.
  3. For
    IP Address
    , type the IPv4 or IPv6 address of the device.
  4. Type the
    User Name
     and
    Password
     for the device.
  5. If this device is part of a DSC group, for the
    Cluster Display Name
     setting, specify how to handle it:
    • For an existing DSC group, select
      Use Existing
       from the list, and then select the name of the DSC group from the next list.
    • To create a new DSC group, select
      Create New
       from the list, and type a name in the field.
    For BIG-IQ to properly associate the devices in the same DSC group, the
    Cluster Display Name
     must be the same for all members in a group.
    There can be up to eight members in a DSC group.
  6. If this device is configured in a DSC group or you are creating a new DSC group, for the
    Cluster Properties
    , specify how to handle it:
    • Initiate BIG-IP DSC sync when deploying configuration changes (Recommended)
      : Select this option if you want this device to automatically synchronize configuration changes with other members in the DSC.
    • Allow deployment when DSC configured devices have changes pending ( Not Recommended)
      : Select this option if you want to deploy changes to this device even if there are changes pending for devices in the DSC group.
      This option is not recommended, because it can lead to unpredictable results.
    • Ignore BIG-IP DSC sync when deploying configuration changes
      : Select this option if you want to manually synchronize configurations changes between members in the DSC group.
  7. Click the
    Add
     button at the bottom of the screen.
    The BIG-IQ system opens communication to the BIG-IP device, and checks the BIG-IP device framework.
    The BIG-IQ system can properly manage a BIG-IP device only if the BIG-IP device is running a compatible version of the REST framework.
  8. If a framework upgrade is required, in the popup window, in the
    Root User Name
     and
    Root Password
     fields, type the root user name and password for the BIG-IP device, and click
    Continue
    .
  9. If in addition to basic management tasks (like software upgrades, license management, and UCS backups) you also want to centrally manage this device's configurations for licensed services, select the check box next to each service you want to discover.
    You can also select these service configuration after you add the BIG-IP device to the inventory.
  10. To enable statistics collection for this BIG-IP device, under Statistics monitoring, select the check box next to each service you want to collect statistics for, and then click
    Continue
    .
    If you want to enable statistics collection without managing any services, just clear the check boxes for all services.
  11. Click the
    Add
     button at the bottom of the screen.

Enable statistics collection for devices

Before you can enable statistics collection for a BIG-IP device using this method:
  • The device must already be in the BIG-IQ system inventory.
  • There must be a BIG-IQ data collection device configured for the BIG-IQ device.
  • For BIG-IP device version 13.1.0.5 or later, AVR must be provisioned.
Generally, if you want to collect statistics for a BIG-IP device, you enable statistics collection when you discover it. But you can enable or disable statistics collection for a device any time it is convenient for you.
  1. At the top of the screen, click
    Devices
    .
  2. Click the name of the device you want to enable statistics collection for.
  3. On the left, click
    Statistics Collection
    .
  4. To begin statistics collection, for
    Collect Statistics Data
    , select
    Enabled
    .
  5. For
    Modules/Services
    , click the check box for the types of statistics you want to collect.
  6. For
    Frequency
    , next to
    Collect every
    , select the interval at which you want to collect statistics from this device.
After you enable statistics collection for a device, data for that device begins aggregating along with any other devices for which you are collecting data. Two buttons (
View Health Statistics
, and
View Traffic Statistics
) are added to the properties page for enabled devices. Clicking either of these takes you directly to the overview page for the statistics type you clicked.

Statistics retention policy overview

When you choose how much raw data to retain, you need to consider how much disk space you have available. The controls on this screen are simple to set up, but understanding how they work takes a bit of explanation.
The fields on the Statistics Retention Policy screen all work in similar fashion. One way to understand how these fields work is to think of your data storage space as a set of containers. The values you specify on this screen determine how much storage space each container consumes. Because data is saved for the time periods you specify, the longer the time period that you specify, the more space you consume. The disk storage that is consumed depends on several factors.
  • The number of BIG-IP devices you manage
  • The number of objects on the BIG-IP devices you manage (for example, virtual servers, pools, pool members, and iRules)
  • The frequency of statistics collection
  • The data retention policy
  • The data replication policy
There are three key concepts to understand about how the retention policy works.
How long is data in each container retained?
Data is retained in each container for the time period you specify. When the specified level is reached, the oldest chunk of data is deleted. For example, if you specify a raw data value of 48 hours, then when 48 hours of raw data accumulate, the next hour of incoming raw data causes the oldest hour to be deleted.
When does data from one container pass on to the next?
Data passes from one container to the next in increments that are the size of the next (larger) container. That is, every 60 minutes, the last 60 minutes of raw data is aggregated into a data set and passed to the
Hour(s)
container. Every 24 hours, the last 24 hours of hourly data is aggregated into a data set and passed to the
Day(s)
 container, and so on for the
Month(s)
 container.
What about limits?
Limit Max Storage to
 specifies the percentage of total disk space that you want data to consume on the data collection devices in your cluster.
If more disk space is consumed than the percentage you specified, BIG-IQ takes two actions:
  1. New statistical data is not accepted until the available disk space complies with the
    Limit max storage to
     setting.
  2. Statistical data not required to calculate the next higher time layer is removed (for example, you need 60 minutes of raw data to aggregate to the Hours level). Data is removed starting with the raw data container, then the hourly data container, then the daily time container. This process stops when storage consumption is below the
    Limit max storage to
     setting.
The BIG-IQ takes this action to prevent data corruption when storage is completely exhausted.

The aggregation policy for your statistics data

There is a default statistics aggregation policy for the data added to your data collection device. The aggregation policy impacts the quality of the entity data, per dimension, over time. This optimizes the disk usage, and allows for high quality data for short-term analysis and troubleshooting for raw, hour, or even day time layers of data. Long term data storage provides insights into global statistics over time, but are not recommended for troubleshooting.

Manage the retention policy for your statistics data

Before you can set the statistics retention policy, you must have added a data collection device.
You can manage the settings that determine how your statistics data is retained. The highest quality data is the raw data, (data that has not been averaged), but that consumes a lot of disk space, so you need to consider your needs in choosing your data retention settings.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    Statistics Data Collection
    .
    The Statistics Collection Status screen opens.
  3. On the left, click
    Statistics Data Collection
    .
    The Statistics Collection Status screen displays the percentage of available disk space currently consumed by statistics data for each container.
  4. To change the retention settings for your statistics data, click
    Configure
    .
    The Statistics Retention Policy screen opens.
  5. In the
    Keep real-time (raw) data up to
     field, type the number of hours of raw data to retain.
    You must specify a minimum of 1 hour, so that there is sufficient data to average and create a data point for the
    Keep hourly data up to
     container.
  6. In the
    Keep hourly data up to
     field, type the number of hourly data points to retain.
    You must specify a minimum of 24 hours, so that there is sufficient data to average and create a data point for the
    Keep daily data up to
     container.
  7. In the
    Keep daily data up to
     field, type the number of daily data points to retain.
    You must specify a minimum of 31 days, so that there is sufficient data to average and create a data point for the
    Keep monthly data up to
     container.
  8. In the
    Keep monthly data up to
     field, type the number of monthly data points to retain.
    Once the specified number of months passes, the oldest monthly data set is deleted.
  9. In the
    Limit max storage to
     field, type the percentage of disk space that you want collected data to consume before the oldest monthly data set is deleted.
  10. Expand Advanced Settings, and then select the
    Enable Replicas
     check box.
    Replicas
     are copies of a data set that are available to the DCD cluster when one or more devices within that cluster become unavailable. By default, data replication for statistics is not enabled. Disabling replication reduces the amount of disk space required for data retention. However, this provides no protection from data corruption that can occur when you remove a data collection device. You should enable replicas to provide this protection.
  11. When you are satisfied with the values specified for data retention, click
    Save & Close
    .

Reviewing captured traffic details

Traffic capturing prompts the system to log traffic request and response headers and payload data, based on specific collection requirements. You enable traffic capturing in your Analytics profile to monitor a known application issue, such as trouble with throughput or latency, or a known factor that can impact application performance, such as HTTP method, or client IP address. You can specify these traffic aspects to later examine application statistics, and troubleshoot captured transactions.
Once enabled, you can examine the captured traffic to explore details, such as the payload of captured transactions, requested URLs and response size. When traffic capturing is enabled, you can view data about captured traffic within the charts for HTTP traffic statistics.

Configure traffic capturing for troubleshooting

Before you begin, you need to ensure that AVR is provisioned on your managed BIG-IP devices, and that Statistics Collection is enabled on your BIG-IQ per device (
Devices
BIG-IP DEVICES
<DEVICE NAME>
STATISTICS COLLECTION
).
You can configure your HTTP analytics profile to capture traffic headers and additional transaction details.Once configured, you can review captured traffic, based upon specific transaction parameters and performance thresholds.
  1. Go to
    Configuration
    LOCAL TRAFFIC
    Profiles
    .
    This screen lists the profiles that are configured for the managed BIG-IP devices in your network.
  2. Select the HTTP Analytics profile you wish to edit.
    The
    analytics
     profile is a default profile for all HTTP Analytics management. If you are creating a new HTTP Analytics profile, make sure to select the
    Override All
     check box to change the settings inherited by the parent profile.
  3. Select
    Captured Traffic External Logging
     to ensure that captured traffic data is logged on BIG-IQ.
    To store traffic on the BIG-IP traffic management device, you can select
    Captured Traffic Internal Logging
    . This stores the data on the BIG-IP device's local logs.
    Once you select a traffic capturing logging location, the Capture Filter area becomes available. This allows you to further configure which traffic you would like to capture.
  4. In the Capture Filter area, enter a unique name for the capturing profile in the
    Capturing Profile Name
     field.
    This step is mandatory to save traffic capture logging settings to the profile.
  5. From the
    Capture Request Details
     and
    Capture Response Details
     lists, select the options that indicate the part of the traffic to capture.
    Option
    Description
    None
    Specifies that the system does not capture request (or response) data.
    Headers
    Specifies that the system captures request (or response) header data only.
    Body
    Specifies that the system captures the body of requests (or responses) only.
    All
    Specifies that the system captures all request (or response) data, including header and body.
  6. For
    DoS Activity
    , select the option that indicates which DoS traffic is captured.
    Option
    Description
    Any
    Specifies that the system captures any traffic regardless of DoS activity.
    Mitigated by Application DoS
    Specifies that the system only captures DoS traffic if it was mitigated.
  7. For
    Protocols
    , specify whether the system only captures traffic with
    HTTP
    , or
    HTTPS
     protocols.
  8. For
    Qualified for JavaScript Injection
    , select
    Qualified only
     to specify that the system only captures traffic that qualifies for JavaScript injection, which includes the following conditions:
    • The HTTP content is not compressed
    • The HTTP content-type is
      text/html
      .
    • The HTTP content contains an HTML
      <head>
       tag
  9. Customize the dimension filters, according to your application needs, to capture the portion of traffic to that you need for troubleshooting.
    Dimension filters capture traffic according to defined aspects of the transaction's configuration, or header/payload contents. By focusing in on the data and limiting the type of information that is captured, you can troubleshoot particular areas of an application more efficiently. For example, capture only requests or responses, specific status codes or methods, or headers containing a specific string.
    Virtual Servers
    Select
    All
     to capture traffic for all Virtual servers.
    Select
    Only
     to capture traffic from specific virtual servers. To specify, add virtual servers to the
    Selected Virtual Servers
     list from the
    Available Virtual Servers
     list.
    Nodes
    Select
    All
    to capture traffic from all nodes.
    Select
    Only
     to capture traffic from specific nodes. To specify, add nodes to the
    Selected Nodes
     list from the
    Available Nodes
     list.
    Response Status Codes
    Select
    All
    to capture traffic, regardless of the HTTP status response code.
    Select
    Only
     to capture traffic with specific response status codes. To specify, add response status codes to the
    Selected Status Codes
     list from the
    Available Status Codes
     list.
    HTTP Methods
    Select
    All
    to capture traffic, regardless of the HTTP request method.
    Select
    Only
     to capture traffic with requests that contain a specific HTTP method. To specify, add methods to the
    Selected Methods
     list from the
    Available Methods
     list.
    URL
    Select
    All
     to capture traffic with requests for any URL.
    Select
    Starts With
     to only capture traffic with requests for URLs that start with a specific string.
    If you select this option, and leave the list blank, the system will not capture any traffic.
    Select
    Does not start with
     to capture traffic with requests for URLs except for those that start with a specific string.
    You can add up to 10 different strings to the list. If the list is blank, the system will capture traffic with requests for any URL.
    User Agent
    Select
    All
     to capture traffic sent from any browser.
    Select
    Contains
     to only capture traffic sent from a browser that contains a specific string.
    You can add up to 10 different strings to the list. If the list is blank, the system will capture traffic sent from any browser.
    Client IP Address
    Select
    All
     to capture traffic sent to, or from, any client IP address.
    Select
    Only
     to only capture traffic sent to or from a specific client IP address.
    You can add up to 10 different IP addresses to the list. If the list is blank, the system will capture traffic sent to, or from, any IP address.
    Request Containing String
    Select
    All
     to capture all traffic.
    Select
    Search in
     filter captured traffic that includes a specific string contained in the request.
    Response Containing String
    Select
    All
     to capture all traffic.
    Select
    Search in
     filter captured traffic that includes a specific string contained in the response.
  10. Click
    Save & Close
    .

Review captured traffic

To display captured traffic, your virtual server must be assigned an HTTP analytics profile that has captured traffic enabled, with external logging.
You can troubleshoot details of captured HTTP traffic to your applications and virtual servers. This information can provide details of request/response headers and payload sent to your managed application. Captured traffic information is found within the following dashboards that provide HTTP traffic visibility:
  • Device Traffic:
    Monitoring
    DASHBOARDS
    Device
    Traffic
    .
  • DDoS HTTP Analysis:
    Monitoring
    DASHBOARDS
    DDoS
    HTTP Analysis
    .
  • Local Traffic:
    Monitoring
    DASHBOARDS
    Local Traffic
    HTTP
    .
  1. Navigate to one of the monitoring dashboards that display HTTP traffic data.
  2. Select the
    Traffic Capturing
     button above the charts.
    Selecting this option overlays captured traffic data over the charts, and adds a traffic capturing filter in the Dimensions pane.
  3. To filter captured traffic based on a specific host object, such as a BIG-IP system (
    BIG-IP Host Names
    ), application (
    Applications Services
    ), or virtual server (
    Virtual Servers
    ), expand the dimension widgets in the Dimensions pane to the right of the charts.
    You can select multiple dimension objects from multiple dimensions. With each selection, the charts and dimensions filter displayed data according to your selections.
  4. To filter captured traffic based on server latency and payload volume metrics, expand the
    Traffic Capturing Filters
     found in the dimensions pane.
    For latency metrics, you can enter a range, or set a greater or less than filter value.
  5. To view traffic details, select a traffic capturing icon from within the chart to display an information table.
    You can click the rows within the displayed table to view additional request/response header and payload information.
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information