Manual Chapter :
Configuring Statistics Collection
Applies To:
Show Versions![Show Versions](/etc/designs/pcx/techdocs/images/expandversions.gif)
Configuring Statistics Collection
BIG-IP device configuration requirements for viewing statistics in BIG-IQ
Before you can enable statistics collection for centralized management, you must ensure that the BIG-IP device has the proper configuration. The proper configuration varies depending on the version of the BIG-IP device. The minimum supported BIG-IP device is version 12.1.0. BIG-IQ has limited visibility for BIG-IP devices prior to 13.1.0.5.
For details about how to configure statistics visibility, based on the BIG-IP version, see
Enabling statistics collection during device discovery
.For details on how to access statistic information, based on the BIG-IP version and service, refer to
Statistics compatibility and visibility
.Monitoring BIG-IP statistics in BIG-IQ
Visibility of statistics in BIG-IQ depends on the version of
your managed BIG-IP devices. Devices running versions 13.1.X, or earlier, have limited
statistics visibility support within BIG-IQ. Below outlines the compatibility and what to
expect when accessing Analytics (AVR) data within BIG-IQ. For more information, see the
supporting documentation found in the
BIG-IQ Centralized
Management: Monitoring and Reports
guide.Statistics visibility of managed BIG-IP devices
The format in which statistics are presented in the BIG-IQ environment, depends on the managed version of BIG-IP and the service presented. Refer to the table to access statistics visibility, based on the managed device version. Ensure that the managed device configuration meets the requirements outlined below.
Minimum configuration requirements:
- BIG-IP Version 13.1.x or earlier
- Ports 22 and 443 on each BIG-IP device must be open for the BIG-IQ DCD to retrieve data.
- There must be a Data Collection Device (DCD) configured to your BIG-IQ.
- BIG-IP Version 13.1.0.5 or later
- You must have AVR provisioned for each BIG-IP device.
- BIG-IQ needs to provide access on Port 443 to receive BIG-IP AVR data.
- There must be a Data Collection Device (DCD) configured to your BIG-IQ.To view statistics, ensure that the licenses for your managed BIG-IP devices include root access. A BIG-IP license running in Appliance Mode, will not allow for statistics visibility in the BIG-IQ environment.
Where to view statistics
BIG-IP v12.1 | BIG-IP v13.0 | BIG-IP v13.1 | BIG-IP v13.1.0.5 | BIG-IP v14.0 | BIG-IP v14.1 | BIG-IP v15.0 or later | |
---|---|---|---|---|---|---|---|
Device Traffic | |||||||
Local Traffic (General) | |||||||
Local Traffic (HTTP) | Not available to this version | ||||||
DNS (General)* | |||||||
Network Firewall (General) | Network Firewall information is
provided by ACL, IP Reputation, and IPS. | ||||||
Network Firewall (ACL) | Not applicable to
this version | ||||||
Network Security (IP Reputation) | Not applicable to this version | ||||||
Network Firewall (IPS) | Not applicable to
this version | ||||||
Web Application Security (General) | |||||||
Web Application Security (Bot) | Not
available to this version | ||||||
DDoS (Shared Security) | Not available to
this version | ||||||
Application Summary | (limited statistics visibility) | ||||||
Secure Web Gateway | Not available to this version | ||||||
SSLO** | Not available to
this version | ||||||
Access | Not available to this
version |
*Top Charts are only available to BIG-IP version 13.1.0.5 or later.
**SSLO support is available to versions 5.4 to 5.9.
How do I start viewing BIG-IP device
statistics from BIG-IQ?
To start viewing statistics for a BIG-IP device, you must have enabled statistics collection for that device.
You can do that either during or after adding the device to the BIG-IP Devices inventory
list on the BIG-IQ system. You also need to
install, configure, and add a data collection device before you can view statistics for
your managed BIG-IP devices.
Enabling statistics collection during
device discovery
Before you can enable statistics for BIG-IP devices:
- There must be a BIG-IQ data collection device (DCD) configured for the BIG-IQ device.
- The BIG-IP device must be located in your network and running a compatible software version. Refer to K34133507#cm6.0.1 for more information.
- For BIG-IP devices running version 13.1.0.5 or later, you must have AVR provisioned.
- For BIG-IP devices running versions prior to 13.1.0.5, configure Ports 22 and 443 must be open to traffic from the BIG-IQ DCD to the managed BIG-IP devices.
- For BIG-IP devices running version 13.1.0.5 or later, BIG-IQ needs to provide access on Port 443 so that the BIG-IP AVR module can send statistics to the BIG-IQ DCD
One way to enable statistics
collection for BIG-IP devices is to do it when you add those devices to the BIG-IQ
system inventory. Adding devices to the inventory is referred to as
device
discovery
. If the devices you want to enable have already been discovered,
refer to Enabling collection after device discovery
. The ADC component is
automatically included (first) any time you discover or import services for a
device.
You do not need
to discover and import a device’s configuration to collect and view statistics for
it. You just need to establish trust between your BIG-IQ and the device. If you do
not discover and import the device configuration, the virtual servers, pool, pool
members, and iRules will be visible in the statistics dimension panes, but these
objects will not appear in the configuration page for those objects. Also, you will
not be able to manage these objects in BIG-IQ. If you decide you want to manage
these objects, you can discover and import the BIG-IP device’s configuration later
without interrupting statistics collection.
- At the top of the screen, clickDevices.
- Click theAdd Device(s)button.
- ForIP Address, type the IPv4 or IPv6 address of the device.
- Type theUser NameandPasswordfor the device.
- If this device is part of a DSC group, for theCluster Display Namesetting, specify how to handle it:
- For an existing DSC group, selectUse Existingfrom the list, and then select the name of the DSC group from the next list.
- To create a new DSC group, selectCreate Newfrom the list, and type a name in the field.
For BIG-IQ to properly associate the devices in the same DSC group, theCluster Display Namemust be the same for all members in a group.There can be up to eight members in a DSC group. - If this device is configured in a DSC group or you are creating a new DSC group, for theCluster Properties, specify how to handle it:
- Initiate BIG-IP DSC sync when deploying configuration changes (Recommended): Select this option if you want this device to automatically synchronize configuration changes with other members in the DSC.
- Allow deployment when DSC configured devices have changes pending ( Not Recommended): Select this option if you want to deploy changes to this device even if there are changes pending for devices in the DSC group.This option is not recommended, because it can lead to unpredictable results.
- Ignore BIG-IP DSC sync when deploying configuration changes: Select this option if you want to manually synchronize configurations changes between members in the DSC group.
- Click theAddbutton at the bottom of the screen.The BIG-IQ system opens communication to the BIG-IP device, and checks the BIG-IP device framework.The BIG-IQ system can properly manage a BIG-IP device only if the BIG-IP device is running a compatible version of the REST framework.
- If a framework upgrade is required, in the popup window, in theRoot User NameandRoot Passwordfields, type the root user name and password for the BIG-IP device, and clickContinue.
- If in addition to basic management tasks (like software upgrades, license management, and UCS backups) you also want to centrally manage this device's configurations for licensed services, select the check box next to each service you want to discover.You can also select these service configuration after you add the BIG-IP device to the inventory.
- To enable statistics collection for this BIG-IP device, under Statistics monitoring, select the check box next to each service you want to collect statistics for, and then clickContinue.If you want to enable statistics collection without managing any services, just clear the check boxes for all services.
- Click theAddbutton at the bottom of the screen.
Enable statistics collection for
devices
Before you can enable statistics
collection for a BIG-IP device using this method:
- The device must already be in the BIG-IQ system inventory.
- There must be a BIG-IQ data collection device configured for the BIG-IQ device.
- For BIG-IP device version 13.1.0.5 or later, AVR must be provisioned.
Generally, if you want to collect
statistics for a BIG-IP device, you enable statistics collection when you discover it.
But you can enable or disable statistics collection for a device any time it is
convenient for you.
- At the top of the screen, clickDevices.
- Click the name of the device you want to enable statistics collection for.
- On the left, clickStatistics Collection.
- To begin statistics collection, forCollect Statistics Data, selectEnabled.
- ForModules/Services, click the check box for the types of statistics you want to collect.
- ForFrequency, next toCollect every, select the interval at which you want to collect statistics from this device.
After you enable statistics
collection for a device, data for that device begins aggregating along with any other
devices for which you are collecting data. Two buttons (
View Health
Statistics
, and View Traffic Statistics
) are
added to the properties page for enabled devices. Clicking either of these takes you
directly to the overview page for the statistics type you clicked.Statistics retention policy overview
When you choose how much raw data to retain, you need to consider how
much disk space you have available. The controls on this screen are simple to set up,
but understanding how they work takes a bit of explanation.
The fields on the Statistics Retention Policy screen all work in similar
fashion. One way to understand how these fields work is to think of your data storage
space as a set of containers. The values you specify on this screen determine how much
storage space each container consumes. Because data is saved for the time periods you
specify, the longer the time period that you specify, the more space you consume. The
disk storage that is consumed depends on several factors.
- The number of BIG-IP devices you manage
- The number of objects on the BIG-IP devices you manage (for example, virtual servers, pools, pool members, and iRules)
- The frequency of statistics collection
- The data retention policy
- The data replication policy
How long is data in each container
retained? | Data is retained in each container for the
time period you specify. When the specified level is reached, the oldest
chunk of data is deleted. For example, if you specify a raw data value
of 48 hours, then when 48 hours of raw data accumulate, the next hour of
incoming raw data causes the oldest hour to be deleted. |
When does data from one container pass on to
the next? | Data passes from one container to the next in
increments that are the size of the next (larger) container. That is,
every 60 minutes, the last 60 minutes of raw data is aggregated into a
data set and passed to the Hour(s) container. Every 24 hours, the last 24 hours
of hourly data is aggregated into a data set and passed to the
Day(s)
container, and so on for the Month(s)
container. |
What about limits? | Limit Max Storage to specifies the percentage of total
disk space that you want data to consume on the data collection devices
in your cluster. If more disk space is consumed
than the percentage you specified, BIG-IQ takes two actions:
|
The aggregation policy for your statistics data
There is a default statistics aggregation policy for the data added to
your data collection device. The aggregation policy impacts the quality of the entity data,
per dimension, over time. This optimizes the disk usage, and allows for high quality data
for short-term analysis and troubleshooting for raw, hour, or even day time layers of data.
Long term data storage provides insights into global statistics over time, but are not
recommended for troubleshooting.
Manage the
retention policy for your statistics data
Before you can set the statistics retention policy, you must have added a data
collection device.
You can manage the settings that determine how
your statistics data is retained. The highest quality data is the raw data, (data that
has not been averaged), but that consumes a lot of disk space, so you need to consider
your needs in choosing your data retention settings.
- At the top of the screen, clickSystem, then, on the left, clickBIG-IQ DATA COLLECTIONand then selectBIG-IQ Data Collection Cluster.The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
- Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
- Under Configuration, you can access the screens that control DCD cluster performance.
- Under the screen name, click.The Statistics Collection Status screen opens.
- On the left, clickStatistics Data Collection.The Statistics Collection Status screen displays the percentage of available disk space currently consumed by statistics data for each container.
- To change the retention settings for your statistics data, clickConfigure.The Statistics Retention Policy screen opens.
- In theKeep real-time (raw) data up tofield, type the number of hours of raw data to retain.You must specify a minimum of 1 hour, so that there is sufficient data to average and create a data point for theKeep hourly data up tocontainer.
- In theKeep hourly data up tofield, type the number of hourly data points to retain.You must specify a minimum of 24 hours, so that there is sufficient data to average and create a data point for theKeep daily data up tocontainer.
- In theKeep daily data up tofield, type the number of daily data points to retain.You must specify a minimum of 31 days, so that there is sufficient data to average and create a data point for theKeep monthly data up tocontainer.
- In theKeep monthly data up tofield, type the number of monthly data points to retain.Once the specified number of months passes, the oldest monthly data set is deleted.
- In theLimit max storage tofield, type the percentage of disk space that you want collected data to consume before the oldest monthly data set is deleted.
- Expand Advanced Settings, and then select theEnable Replicascheck box.Replicasare copies of a data set that are available to the DCD cluster when one or more devices within that cluster become unavailable. By default, data replication for statistics is not enabled. Disabling replication reduces the amount of disk space required for data retention. However, this provides no protection from data corruption that can occur when you remove a data collection device. You should enable replicas to provide this protection.
- When you are satisfied with the values specified for data retention, clickSave & Close.
Reviewing captured traffic details
Traffic capturing prompts the system to log traffic request
and response headers and payload data, based on specific collection
requirements. You enable traffic capturing in your Analytics profile to
monitor a known application issue, such as trouble with throughput or
latency, or a known factor that can impact application performance, such
as HTTP method, or client IP address. You can specify these traffic
aspects to later examine application statistics, and troubleshoot
captured transactions.
Once enabled, you can examine the captured traffic to explore details, such as
the payload of captured transactions, requested URLs and response size.
When traffic capturing is enabled, you can view data about captured
traffic within the charts for HTTP traffic statistics.
Configure traffic capturing for troubleshooting
Before
you begin, you need to ensure that AVR is provisioned on your managed BIG-IP devices,
and that Statistics Collection is enabled on your BIG-IQ per device (
). You can configure your HTTP analytics profile to
capture traffic headers and additional transaction details.Once configured, you can
review captured traffic, based upon specific transaction parameters and performance
thresholds.
- Go to.This screen lists the profiles that are configured for the managed BIG-IP devices in your network.
- Select the HTTP Analytics profile you wish to edit.Theanalyticsprofile is a default profile for all HTTP Analytics management. If you are creating a new HTTP Analytics profile, make sure to select theOverride Allcheck box to change the settings inherited by the parent profile.
- SelectCaptured Traffic External Loggingto ensure that captured traffic data is logged on BIG-IQ.To store traffic on the BIG-IP traffic management device, you can selectCaptured Traffic Internal Logging. This stores the data on the BIG-IP device's local logs.Once you select a traffic capturing logging location, the Capture Filter area becomes available. This allows you to further configure which traffic you would like to capture.
- In the Capture Filter area, enter a unique name for the capturing profile in theCapturing Profile Namefield.This step is mandatory to save traffic capture logging settings to the profile.
- From theCapture Request DetailsandCapture Response Detailslists, select the options that indicate the part of the traffic to capture.OptionDescriptionNoneSpecifies that the system does not capture request (or response) data.HeadersSpecifies that the system captures request (or response) header data only.BodySpecifies that the system captures the body of requests (or responses) only.AllSpecifies that the system captures all request (or response) data, including header and body.
- ForDoS Activity, select the option that indicates which DoS traffic is captured.OptionDescriptionAnySpecifies that the system captures any traffic regardless of DoS activity.Mitigated by Application DoSSpecifies that the system only captures DoS traffic if it was mitigated.
- ForProtocols, specify whether the system only captures traffic withHTTP, orHTTPSprotocols.
- ForQualified for JavaScript Injection, selectQualified onlyto specify that the system only captures traffic that qualifies for JavaScript injection, which includes the following conditions:
- The HTTP content is not compressed
- The HTTP content-type istext/html.
- The HTTP content contains an HTML<head>tag
- Customize the dimension filters, according to your application needs, to capture the portion of traffic to that you need for troubleshooting.Dimension filters capture traffic according to defined aspects of the transaction's configuration, or header/payload contents. By focusing in on the data and limiting the type of information that is captured, you can troubleshoot particular areas of an application more efficiently. For example, capture only requests or responses, specific status codes or methods, or headers containing a specific string.Virtual ServersSelectAllto capture traffic for all Virtual servers.SelectOnlyto capture traffic from specific virtual servers. To specify, add virtual servers to theSelected Virtual Serverslist from theAvailable Virtual Serverslist.NodesSelectAllto capture traffic from all nodes.SelectOnlyto capture traffic from specific nodes. To specify, add nodes to theSelected Nodeslist from theAvailable Nodeslist.Response Status CodesSelectAllto capture traffic, regardless of the HTTP status response code.SelectOnlyto capture traffic with specific response status codes. To specify, add response status codes to theSelected Status Codeslist from theAvailable Status Codeslist.HTTP MethodsSelectAllto capture traffic, regardless of the HTTP request method.SelectOnlyto capture traffic with requests that contain a specific HTTP method. To specify, add methods to theSelected Methodslist from theAvailable Methodslist.URLSelectAllto capture traffic with requests for any URL.SelectStarts Withto only capture traffic with requests for URLs that start with a specific string.If you select this option, and leave the list blank, the system will not capture any traffic.SelectDoes not start withto capture traffic with requests for URLs except for those that start with a specific string.You can add up to 10 different strings to the list. If the list is blank, the system will capture traffic with requests for any URL.User AgentSelectAllto capture traffic sent from any browser.SelectContainsto only capture traffic sent from a browser that contains a specific string.You can add up to 10 different strings to the list. If the list is blank, the system will capture traffic sent from any browser.Client IP AddressSelectAllto capture traffic sent to, or from, any client IP address.SelectOnlyto only capture traffic sent to or from a specific client IP address.You can add up to 10 different IP addresses to the list. If the list is blank, the system will capture traffic sent to, or from, any IP address.Request Containing StringSelectAllto capture all traffic.SelectSearch infilter captured traffic that includes a specific string contained in the request.Response Containing StringSelectAllto capture all traffic.SelectSearch infilter captured traffic that includes a specific string contained in the response.
- ClickSave & Close.
Review captured traffic
To display captured traffic, your virtual server must
be assigned an HTTP analytics profile that has captured traffic enabled, with external
logging.
You can troubleshoot details of captured HTTP
traffic to your applications and virtual servers. This information can provide details
of request/response headers and payload sent to your managed application. Captured
traffic information is found within the following dashboards that provide HTTP traffic
visibility:
- Device Traffic:.
- DDoS HTTP Analysis:.
- Local Traffic:.
- Navigate to one of the monitoring dashboards that display HTTP traffic data.
- Select theTraffic Capturingbutton above the charts.Selecting this option overlays captured traffic data over the charts, and adds a traffic capturing filter in the Dimensions pane.
- To filter captured traffic based on a specific host object, such as a BIG-IP system (BIG-IP Host Names), application (Applications Services), or virtual server (Virtual Servers), expand the dimension widgets in the Dimensions pane to the right of the charts.You can select multiple dimension objects from multiple dimensions. With each selection, the charts and dimensions filter displayed data according to your selections.
- To filter captured traffic based on server latency and payload volume metrics, expand theTraffic Capturing Filtersfound in the dimensions pane.For latency metrics, you can enter a range, or set a greater or less than filter value.
- To view traffic details, select a traffic capturing icon from within the chart to display an information table.You can click the rows within the displayed table to view additional request/response header and payload information.