Configuring Statistics Collection

BIG-IP device configuration requirements for viewing statistics in BIG-IQ

Before you can enable statistics collection for centralized management, you must ensure that the BIG-IP device has the proper configuration. The proper configuration varies depending on the version of the BIG-IP device. The minimum supported BIG-IP device is version 12.1.0. BIG-IQ has limited visibility for BIG-IP devices prior to 13.1.0.5.

For details about how to configure statistics visibility, based on the BIG-IP version, see Enabling statistics collection during device discovery.

For details on how to access statistic information, based on the BIG-IP version and service, refer to Statistics compatibility and visibility.

Monitoring BIG-IP statistics in BIG-IQ

Visibility of statistics in BIG-IQ depends on the version of your managed BIG-IP devices. Devices running versions 13.1.X, or earlier, have limited statistics visibility support within BIG-IQ. Below outlines the compatibility and what to expect when accessing Analytics (AVR) data within BIG-IQ. For more information, see the supporting documentation found in the BIG-IQ Centralized Management: Monitoring and Reports guide.

Statistics visibility of managed BIG-IP devices

The format in which statistics are presented in the BIG-IQ environment, depends on the managed version of BIG-IP and the service presented. Refer to the table to access statistics visibility, based on the managed device version. Ensure that the managed device configuration meets the requirements outlined below.

Minimum configuration requirements:

BIG-IP Version 13.1.x or earlier

• Ports 22 and 443 on each BIG-IP device must be open for the BIG-IQ DCD to retrieve data.

• There must be a Data Collection Device (DCD) configured to your BIG-IQ.

BIG-IP Version 13.1.0.5 or later

• You must have AVR provisioned for each BIG-IP device.

• BIG-IQ needs to provide access on Port 443 to receive BIG-IP AVR data.

• There must be a Data Collection Device (DCD) configured to your BIG-IQ.

  Note: To view statistics, ensure that the licenses for your managed BIG-IP devices include root access. A BIG-IP license running in Appliance Mode, will not allow for statistics visibility in the BIG-IQ environment.

Where to view statistics

*Top Charts are only available to BIG-IP version 13.1.0.5 or later.

**SSLO support is available to versions 5.4 to 5.9.

How do I start viewing BIG-IP device statistics from BIG-IQ?

To start viewing statistics for a BIG-IP device, you must have enabled statistics collection for that device. You can do that either during or after adding the device to the BIG-IP Devices inventory list on the BIG-IQ system. You also need to install, configure, and add a data collection device before you can view statistics for your managed BIG-IP devices.

Enabling statistics collection during device discovery

Before you can enable statistics for BIG-IP devices:

• There must be a BIG-IQ data collection device (DCD) configured for the BIG-IQ device.
• The BIG-IP device must be located in your network and running a compatible software version. Refer to https://support.f5.com/csp/article/K34133507#cm6.0.1 for more information.
  • For BIG-IP devices running version 13.1.0.5 or later, you must have AVR provisioned.
• For BIG-IP devices running versions prior to 13.1.0.5, configure Ports 22 and 443 must be open to traffic from the BIG-IQ DCD to the managed BIG-IP devices.
• For BIG-IP devices running version 13.1.0.5 or later, BIG-IQ needs to provide access on Port 443 so that the BIG-IP AVR module can send statistics to the BIG-IQ DCD

One way to enable statistics collection for BIG-IP devices is to do it when you add those devices to the BIG-IQ system inventory. Adding devices to the inventory is referred to as device discovery. If the devices you want to enable have already been discovered, refer to Enabling collection after device discovery.

Note: The ADC component is automatically included (first) any time you discover or import services for a device.

Note: You do not need to discover and import a device’s configuration to collect and view statistics for it. You just need to establish trust between your BIG-IQ and the device. If you do not discover and import the device configuration, the virtual servers, pool, pool members, and iRules will be visible in the statistics dimension panes, but these objects will not appear in the configuration page for those objects. Also, you will not be able to manage these objects in BIG-IQ. If you decide you want to manage these objects, you can discover and import the BIG-IP device’s configuration later without interrupting statistics collection.

1. At the top of the screen, click Devices.

2. Click the Add Device(s) button.

3. For IP Address, type the IPv4 or IPv6 address of the device.

4. Type the User Name and Password for the device.

5. If this device is part of a Device Service Cluster (DSC) group, for the Cluster Display Name setting, specify how to handle it:

   • For an existing DSC group, select Use Existing from the list, and then select the name of the DSC group from the next list.
   • To create a new DSC group, select Create New from the list, and type a name in the field. For BIG-IQ to properly associate the devices in the same DSC group, the Cluster Display Name must be the same for all members in a group.

   There can be up to 8 members in a DSC group.

   Important: For BIG-IP devices with ASM services, you can only add five devices at a time. If the BIG-IP device(s) provisioned with ASM is part of a DSC cluster, that device must also be a member of a sync-only device group, and ASM synchronization must be enabled on one of the devices in the DSC group. Without these DSC group settings, deploying changes to the ASM device can cause the cluster to become out of sync. For details on configuring these groups, refer to https://support.f5.com/csp/article/K12200102 or the ASM implementations chapter Automatically Synchronizing Application Security Configurations on support.f5.com.

6. If this device is configured in a Device Service Clustering (DSC) group, for the Deployment Settings, specify how to handle it:

   Note: For more information about DSC groups, see What is a BIG-IP Device Service Clustering (DSC) group and how do I start managing it from BIG-IQ?.

   • Initiate BIG-IP DSC sync when deploying configuration changes (Recommended): Select this option if this device is part of a DSC group and you want BIG-IQ to automatically synchronize configuration changes with other members in the DSC group from BIG-IQ.

     • Allow deployment when DSC configured devices have changes pending (Not Recommended) this option pushes changes to all the devices in the cluster independently, ignoring the logical device cluster configuration. This option is intended for temporary use when a BIG-IP cluster node becomes unavailable, you would like to push changes to individual devices, until the node comes back online.

       Note: If multiple nodes are online, the entire BIG-IP DSC group will go into a Change Pending state. This can only be resolved by the system administrator.

   • Ignore BIG-IP DSC sync when deploying configuration changes: Select this option if you want to manually synchronize configurations changes between members in the DSC group.

7. Click the Add button at the bottom of the screen.

   The BIG-IQ system opens communication to the BIG-IP device, and checks the BIG-IP device framework.

   Note: The BIG-IQ system can properly manage a BIG-IP device only if the BIG-IP device is running a compatible version of the REST framework.

8. If a framework upgrade is required, in the popup window, in the Root User Name and Root Password fields, type the root user name and password for the BIG-IP device, and click Continue.

9. If in addition to basic management tasks (like software upgrades, license management, and UCS backups) you also want to centrally manage this device’s configurations for licensed services, select the check box next to each service you want to discover.

   You can also select these service configuration after you add the BIG-IP device to the inventory.

10. To enable statistics collection for this BIG-IP device, under Statistics monitoring, select the check box next to each service you want to collect statistics for, and then click Continue.

    Note: If you want to enable statistics collection without managing any services, just clear the check boxes for all services.

11. Click the Add button at the bottom of the screen.

Enable statistics collection for devices

Before you can enable statistics collection for a BIG-IP device using this method:

• The device must already be in the BIG-IQ system inventory.
• There must be a BIG-IQ data collection device configured for the BIG-IQ device.
• For BIG-IP device version 13.1.0.5 or later, AVR must be provisioned.

Generally, if you want to collect statistics for a BIG-IP device, you enable statistics collection when you discover it. But you can enable or disable statistics collection for a device any time it is convenient for you.

1. At the top of the screen, click Devices.

2. Click the name of the device you want to enable statistics collection for.

3. On the left, click Statistics Collection.

4. To begin statistics collection, for Collect Statistics Data, select Enabled.

5. For Modules/Services, click the check box for the types of statistics you want to collect.

6. For Frequency, next to Collect every, select the interval at which you want to collect statistics from this device.

After you enable statistics collection for a device, data for that device begins aggregating along with any other devices for which you are collecting data. Two buttons (View Health Statistics, and View Traffic Statistics) are added to the properties page for enabled devices. Clicking either of these takes you directly to the overview page for the statistics type you clicked.

Statistics retention policy overview

When you choose how much raw data to retain, you need to consider how much disk space you have available. The controls on this screen are simple to set up, but understanding how they work takes a bit of explanation.

The fields on the Statistics Retention Policy screen all work in similar fashion. One way to understand how these fields work is to think of your data storage space as a set of containers. The values you specify on this screen determine how much storage space each container consumes. Because data is saved for the time periods you specify, the longer the time period that you specify, the more space you consume. The disk storage that is consumed depends on several factors.

• The number of BIG-IP devices you manage
• The number of objects on the BIG-IP devices you manage (for example, virtual servers, pools, pool members, and iRules)
• The frequency of statistics collection
• The data retention policy
• The data replication policy

There are three key concepts to understand about how the retention policy works.

The aggregation policy for your statistics data

There is a default statistics aggregation policy for the data added to your data collection device. The aggregation policy impacts the quality of the entity data, per dimension, over time. This optimizes the disk usage, and allows for high quality data for short-term analysis and troubleshooting for raw, hour, or even day time layers of data. Long term data storage provides insights into global statistics over time, but are not recommended for troubleshooting.

Manage the retention policy for your statistics data

Before you can set the statistics retention policy, you must have added a data collection device.

You can manage the settings that determine how your statistics data is retained. The highest quality data is the raw data, (data that has not been averaged), but that consumes a lot of disk space, so you need to consider your needs in choosing your data retention settings.

1. At the top of the screen, click System, then, on the left, click BIG-IQ DATA COLLECTION and then select BIG-IQ Data Collection Cluster.

   The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.

   • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
   • Under Configuration, you can access the screens that control DCD cluster performance.

2. Under the screen name, click Configuration > Statistics Data Collection.

   The Statistics Collection Status screen opens.

3. On the left, click Statistics Data Collection.

   The Statistics Collection Status screen displays the percentage of available disk space currently consumed by statistics data for each container.

4. To change the retention settings for your statistics data, click Configure.

   The Statistics Retention Policy screen opens.

5. In the Keep real-time (raw) data up to field, type the number of hours of raw data to retain.

   You must specify a minimum of 1 hour, so that there is sufficient data to average and create a data point for the Keep hourly data up to container.

6. In the Keep hourly data up to field, type the number of hourly data points to retain.

   You must specify a minimum of 24 hours, so that there is sufficient data to average and create a data point for the Keep daily data up to container.

7. In the Keep daily data up to field, type the number of daily data points to retain.

   You must specify a minimum of 31 days, so that there is sufficient data to average and create a data point for the Keep monthly data up to container.

8. In the Keep monthly data up to field, type the number of monthly data points to retain.

   Once the specified number of months passes, the oldest monthly data set is deleted.

9. In the Limit max storage to field, type the percentage of disk space that you want collected data to consume before the oldest monthly data set is deleted.

10. Expand Advanced Settings, and then select the Enable Replicas check box.

    Replicas are copies of a data set that are available to the DCD cluster when one or more devices within that cluster become unavailable. By default, data replication for statistics is not enabled. Disabling replication reduces the amount of disk space required for data retention. However, this provides no protection from data corruption that can occur when you remove a data collection device. You should enable replicas to provide this protection.

11. When you are satisfied with the values specified for data retention, click Save & Close.

Troubleshoot HTTP Traffic by Reviewing Captured Traffic

Reviewing captured traffic details

Traffic capturing prompts the system to log traffic request and response headers and payload data, based on specific collection requirements. You enable traffic capturing in your Analytics profile to monitor a known application issue, such as trouble with throughput or latency, or a known factor that can impact application performance, such as HTTP method, or client IP address. You can specify these traffic aspects to later examine application statistics, and troubleshoot captured transactions.

Once enabled, you can examine the captured traffic to explore details, such as the payload of captured transactions, requested URLs and response size. When traffic capturing is enabled, you can view data about captured traffic within the charts for HTTP traffic statistics.

Configure traffic capturing for troubleshooting

Before you begin, you need to ensure that AVR is provisioned on your managed BIG-IP devices, and that Statistics Collection is enabled on your BIG-IQ per device (Devices > BIG-IP DEVICES > <DEVICE NAME> > STATISTICS COLLECTION).

You can configure your HTTP analytics profile to capture traffic headers and additional transaction details.Once configured, you can review captured traffic, based upon specific transaction parameters and performance thresholds.

1. Go to Configuration > LOCAL TRAFFIC > Profiles.

   This screen lists the profiles that are configured for the managed BIG-IP devices in your network.

2. Select the HTTP Analytics profile you wish to edit.

   The analytics profile is a default profile for all HTTP Analytics management. If you are creating a new HTTP Analytics profile, make sure to select the Override All check box to change the settings inherited by the parent profile.

3. Select Captured Traffic External Logging to ensure that captured traffic data is logged on BIG-IQ.

   To store traffic on the BIG-IP traffic management device, you can select Captured Traffic Internal Logging. This stores the data on the BIG-IP device’s local logs.

   Once you select a traffic capturing logging location, the Capture Filter area becomes available. This allows you to further configure which traffic you would like to capture.

4. In the Capture Filter area, enter a unique name for the capturing profile in the Capturing Profile Name field.

   This step is mandatory to save traffic capture logging settings to the profile.

5. From the Capture Request Details and Capture Response Details lists, select the options that indicate the part of the traffic to capture.

   Option	Description
   None	Specifies that the system does not capture request (or response) data.
   Headers	Specifies that the system captures request (or response) header data only.
   Body	Specifies that the system captures the body of requests (or responses) only.
   All	Specifies that the system captures all request (or response) data, including header and body.

6. For DoS Activity, select the option that indicates which DoS traffic is captured.

   Option	Description
   Any	Specifies that the system captures any traffic regardless of DoS activity.
   Mitigated by Application DoS	Specifies that the system only captures DoS traffic if it was mitigated.

7. For Protocols, specify whether the system only captures traffic with HTTP, or HTTPS protocols.

8. For Qualified for JavaScript Injection, select Qualified only to specify that the system only captures traffic that qualifies for JavaScript injection, which includes the following conditions:

   • The HTTP content is not compressed
   • The HTTP content-type is text/html.
   • The HTTP content contains an HTML <head> tag

9. Customize the dimension filters, according to your application needs, to capture the portion of traffic to that you need for troubleshooting.

   Tip: Dimension filters capture traffic according to defined aspects of the transaction’s configuration, or header/payload contents. By focusing in on the data and limiting the type of information that is captured, you can troubleshoot particular areas of an application more efficiently. For example, capture only requests or responses, specific status codes or methods, or headers containing a specific string.

10. Click Save & Close.

Review captured traffic

To display captured traffic, your virtual server must be assigned an HTTP analytics profile that has captured traffic enabled, with external logging.

You can troubleshoot details of captured HTTP traffic to your applications and virtual servers. This information can provide details of request/response headers and payload sent to your managed application. Captured traffic information is found within the following dashboards that provide HTTP traffic visibility:

• Device Traffic: Monitoring > DASHBOARDS > Device > Traffic.
• DDoS HTTP Analysis: Monitoring > DASHBOARDS > DDoS > HTTP Analysis.
• Local Traffic: Monitoring > DASHBOARDS > Local Traffic > HTTP.

1. Navigate to one of the monitoring dashboards that display HTTP traffic data.

2. Select the Traffic Capturing button above the charts.

   Selecting this option overlays captured traffic data over the charts, and adds a traffic capturing filter in the Dimensions pane.

3. To filter captured traffic based on a specific host object, such as a BIG-IP system (BIG-IP Host Names), application (Applications Services), or virtual server (Virtual Servers), expand the dimension widgets in the Dimensions pane to the right of the charts.

   You can select multiple dimension objects from multiple dimensions. With each selection, the charts and dimensions filter displayed data according to your selections.

4. To filter captured traffic based on server latency and payload volume metrics, expand the Traffic Capturing Filters found in the dimensions pane.

   For latency metrics, you can enter a range, or set a greater or less than filter value.

5. To view traffic details, select a traffic capturing icon from within the chart to display an information table.

   You can click the rows within the displayed table to view additional request/response header and payload information.