Manual Chapter :
Reporting Usage Data to an External Analytics Server
Applies To:
Show VersionsBIG-IP LTM
- 17.1.2, 17.1.1, 17.1.0, 17.0.0, 16.1.5, 16.1.4, 16.1.3, 16.1.2, 16.1.1
BIG-IP PEM
- 17.1.2, 17.1.1, 17.1.0, 17.0.0, 16.1.5, 16.1.4, 16.1.3, 16.1.2, 16.1.1, 16.1.0
Reporting Usage Data to an External Analytics Server
Overview: Reporting usage data to an external analytics server
In Policy Enforcement Manager™, you can create a rule within an enforcement
policy that instructs the system to send usage data in high-speed logging (HSL) format to an
external analytics server. The rule specifies what type of reporting data you are interested in;
one of the actions it can take with the traffic is to send the information collected about it for
processing to a centralized analytics server.
The system sends the information as a set of comma-separated values by means of SYSLOG
transport. You can choose to use the session-based, flow-based or transactional reporting format,
depending on the level of granularity you need.
For example, a rule might collect session-based information about all audio and video traffic.
You can specify how often to log the data and set the destination as an HSL server or pool.
Transactional Policy Enforcement
, provides the ability to report each of the HTTP
transaction and sends the report to a HSL publisher. Each transaction report information is
specific to that transaction only. The transactional reports are used for analytics and high
level granularity for application and subscriber visibility. Task summary
Creating a log
publisher
Create a log publisher to specify where the BIG-IP
system sends alert messages.
If you
want alerts sent to a remote syslog server, you need to create two log publishers,
one for the local syslog server and one for the remote syslog
server.
- On the Main tab, click.The Log Publishers screen opens.
- ClickCreate.
- In theNamefield, type a unique, identifiable name for this publisher.
- For theDestinationssetting, selectlocal-syslogfrom theAvailablelist, and click<<to move the destination to theSelectedlist.
- ClickFinished.The list of Log Publishers appears, showing the Log Publisher you just created.
- If you want to have alerts sent to a remote syslog server, repeat steps 2-5, and at step 4 select the log destination that you created previously from theAvailablelist.
Creating a rule for high-speed logging for session reporting
Before you can create a high-speed logging (HSL) rule, you need to create a
publisher that defines the destination server or pool where the HSL logs are
sent.
In an enforcement policy, a rule can specify that session statistics about the
traffic affected by the rule are sent to an external high-speed logging
server.
- On the Main tab, click.The Policies screen opens.
- Click the name of the enforcement policy you want to add rules to.The properties screen for the policy opens.
- In the Policy Rules area, clickAdd.The New Rule screen opens.
- In theNamefield, type a name for the rule.
- In thePrecedencefield, type an integer that indicates the precedence for the rule in relation to the other rules. Number 1 has the highest precedence. Rules with higher precedence are evaluated before other rules with lower precedence.All rules in a policy are run concurrently. Precedence takes effect when there are conflicting rules. The conflict occurs when the traffic matches two rules and the policy actions from these rules differ. For example, if you have rule 1 with precedence 10 andGate Statusdisabled for a search engine, and you have rule 2 with precedence 11 andGate Statusenabled, then rule 1 is processed first because it has higher precedence. Rules conflict if they have identical or overlapping classification criteria (for the traffic that matches more than one rule). In some cases, different policy actions are not conflicting, and hence, applied in parallel.
- Use the Classification, URL, Flow, and Custom Criteria tabs to identify the traffic that you want to be affected by this rule.
- From theUsage Reportinglist, selectEnabled.
- From theUsage Report Granularitylist, selectSessionto log details about subscribers and application sessions.
- In theUsage Volume Thresholdsetting, specify in octets, the threshold to send HSL reporting records. You can send reporting data from uplink traffic, to downlink traffic and the total traffic volume before logging the information.
- In theUsage Destinationsetting, specify where to send the usage monitoring data:
- In theGxfield selectEnabledfor the BIG-IP system to send usage monitoring data over a Gx interface. You can then type a string for theGx Monitoring Keythat is used for usage monitoring.When you selectSessionin theReport Granularityfield, theGxfield appears.
- From theHSLlist, select the name of the publisher that specifies the server or pool of remote HSL servers to send the logs and select the format script of the report from theFormat Scriptlist.
- Select theRADIUS Accountingoption from the destination. From theRADIUS AAA Virtuallist, select the RADIUS AAA virtual that you have created before.
If you are using a formatted destination, select the publisher that matches your log servers, such as Remote Syslog, Splunk, or ArcSight.There are noFormat Scriptsfor transactional reporting. - In theUsage Intervalfield, type an integer that specifies how frequently HSL reporting data is sent.
- For theSession Reporting Fieldsetting, move the fields that you want to see in the logs from theAvailablelist to theSelectedlist.
- ClickFinished.
You have created a rule that sends data about the traffic to external high-speed
logging servers. The CSV reporting format differs depending on whether the report
granularity is flow-based or session-based.
Creating a rule for high-speed logging for flow reporting
Before you can create a high-speed logging (HSL) rule, you need to create a
publisher that defines the destination server or pool where the HSL logs are
sent.
In an enforcement policy, a rule can specify that flow statistics about the traffic
affected by the rule are sent to an external high-speed logging server.
- On the Main tab, click.The Policies screen opens.
- Click the name of the enforcement policy you want to add rules to.The properties screen for the policy opens.
- In the Policy Rules area, clickAdd.The New Rule screen opens.
- In theNamefield, type a name for the rule.
- In thePrecedencefield, type an integer that indicates the precedence for the rule in relation to the other rules. Number 1 has the highest precedence. Rules with higher precedence are evaluated before other rules with lower precedence.All rules in a policy are run concurrently. Precedence takes effect when there are conflicting rules. The conflict occurs when the traffic matches two rules and the policy actions from these rules differ. For example, if you have rule 1 with precedence 10 andGate Statusdisabled for a search engine, and you have rule 2 with precedence 11 andGate Statusenabled, then rule 1 is processed first because it has higher precedence. Rules conflict if they have identical or overlapping classification criteria (for the traffic that matches more than one rule). In some cases, different policy actions are not conflicting, and hence, applied in parallel.
- Use the Classification, URL, Flow, and Custom Criteria tabs to identify the traffic that you want to be affected by this rule.
- From theUsage Reportinglist, selectEnabled.
- From theReport Granularitylist, selectFlow, for more granular reporting of every TCP connection.
- In theVolume Thresholdsetting, specify in octets, the threshold to send HSL reporting records. You can send reporting data from uplink traffic, to downlink traffic and the total traffic volume before logging the information.
- In theIntervalfield, type an integer that specifies how frequently HSL reporting data is sent.
- In theDestinationsetting, specify where to send the usage monitoring data:
- From theHSLlist, select the name of the publisher that specifies the server or pool of remote HSL servers to send the logs.
- Select theFormat Scriptlistand select the format script of the report from theFormat Scriptlist.
- Select theRADIUS Accountingoption from the destination. From theRADIUS AAA Virtuallist, select the RADIUS AAA virtual that you have created before.
If you are using a formatted destination, select the publisher that matches your log servers, such as Remote Syslog, Splunk, or ArcSight. - For theFlow Reporting Fieldsetting, move the fields that you want to see in the logs from theAvailablelist to theSelectedlist.
- ClickFinished.
You have created a rule that sends data about the traffic to external high-speed
logging servers. The CSV reporting format differs depending on whether the report
granularity is flow-based or session-based.
Creating a high-speed logging rule for transactional reporting
Before you can create a high-speed logging (HSL) rule, you need to create a
publisher that defines the destination server or pool where the HSL logs are
sent.
In an enforcement policy, a rule can specify that transactional statistics about
traffic affected by the rule are sent to an external high-speed logging
server.
- On the Main tab, click.The Policies screen opens.
- From the Transactional list, selectEnabledif you want the BIG-IP system to allow policy enforcement on each HTTP transaction.
- Click the name of the enforcement policy you want to add rules to.The properties screen for the policy opens.
- In the Policy Rules area, clickAdd.The New Rule screen opens.
- In theNamefield, type a name for the rule.
- In thePrecedencefield, type an integer that indicates the precedence for the rule in relation to the other rules. Number 1 has the highest precedence. Rules with higher precedence are evaluated before other rules with lower precedence.All rules in a policy are run concurrently. Precedence takes effect when there are conflicting rules. The conflict occurs when the traffic matches two rules and the policy actions from these rules differ. For example, if you have rule 1 with precedence 10 andGate Statusdisabled for a search engine, and you have rule 2 with precedence 11 andGate Statusenabled, then rule 1 is processed first because it has higher precedence. Rules conflict if they have identical or overlapping classification criteria (for the traffic that matches more than one rule). In some cases, different policy actions are not conflicting, and hence, applied in parallel.
- Use the Classification, URL, Flow, and Custom Criteria tabs to identify the traffic that you want to be affected by this rule.
- From theUsage Reportinglist, selectEnabled.
- From theReport Granularitylist, selectTransaction, for more granular reporting of every HTTP transaction.
- In theAdditional HTTP Informationsetting, specify in bytes, the HTTPHostname, the HTTPUser Agent, and the HTTPURI.
- In theDestinationsetting, specify where to send the usage monitoring data:
- From theHSLlist, select the name of the publisher that specifies the server or pool of remote HSL servers to send the logs.
- Select theRADIUS Accountingoption from the destination. From theRADIUS AAA Virtuallist, select the RADIUS AAA virtual that you created earlier.
If you are using a formatted destination, select the publisher that matches your log servers, such as Remote Syslog, Splunk, or ArcSight.There are noFormat Scriptsfor transactional reporting. - For theTransaction Reporting Fieldsetting, move the fields that you want to see in the logs from theAvailablelist to theSelectedlist.
- ClickFinished.
You have created a rule that sends transactional data about the traffic to external
high-speed logging servers. You can now assign the policy to an active
subscriber.
Creating a high-speed logging rule for device detection and tethering
You can specify a reporting destination where reports are sent out whenever the
subscribers go from a non-tethering state to a tethering state, or vice-versa. Before
you can create a high-speed logging (HSL) rule, you need to create a publisher that
defines the destination server or pool where the HSL logs are sent. In an enforcement
policy, a rule can specify that tethering details are sent to an external high-speed
logging server.
- On the Main tab, click.The Policies screen opens.
- Click the name of the enforcement policy you want to add rules to.The properties screen for the policy opens.
- In the Policy Rules area, clickAdd.The New Rule screen opens.
- In theNamefield, type a name for the rule.
- In thePrecedencefield, type an integer that indicates the high precedence for the rule in relation to the other rules. Number 1 has the highest precedence. Rules with higher precedence are evaluated before other rules with lower precedence.
- In theReportingsetting, specify where to send the tethering detection data:
- From theHSLlist, select the name of the publisher that specifies the server or pool of remote HSL servers to send the logs.
- From theFormat Scriptlist, select the format script of the report from theFormat Scriptlist.
The format script is previously configured inpage. - ClickFinished.
You have created a rule that sends device detection and tethering data about the traffic to external high-speed logging servers.
Session-based reporting format
In an enforcement policy, a rule can send session-based information
about traffic that matches certain criteria to an external high-speed
logging (HSL) server. The logs include the following comma-separated values
in the order listed.
Field |
Description |
---|---|
PEM id |
Identifies the reporting module (PEM) and the field value is
23003143. |
Version |
Indicates the version of the format for backward
compatibility. |
Timestamp seconds |
The time the information was logged (along with the timestamp
in milliseconds), specifies seconds using UNIX time
format. |
Timestamp msec |
The time the information was logged (along with the timestamp
in seconds), specifies milliseconds using UNIX time
format. |
Report type |
The type of report. Always set to 3 for session-based
reporting. |
Subscriber ID |
A unique identifier (up to 64 characters) for the subscriber
initiating the session, such as a phone number. The subscriber ID
type determines the format. |
Subscriber ID type |
The format of the subscriber ID. It can be E.164, IMSI, NAI,
or Private. |
3GPP parameters |
The list of 3GPP parameters, which can be imsi, imeisv,
tower_id, or username. |
Policy ID |
The Identification of the policy. |
Rule ID |
The Identification of the policy rule. |
Application ID |
A unique number that represents a particular application, and
is used for classifying traffic. |
Last Sent |
The time, in seconds, since the last log entry was
sent. |
Bytes in |
The number of bytes received during this session. |
Bytes out |
The number of bytes sent during this session. |
Concurrent flows |
Always 0 (unsupported). |
Opened flows |
Always 0 (unsupported). |
Terminated flows |
Always 0 (unsupported). |
Total transactions |
Always 0 (unsupported). |
Successful transactions |
Always 0 (unsupported). |
Aggregated category duration |
Summary of the duration of all flows for the session. |
Reason |
The reason for sending the record. It can be 0 - reserved, 1
- volume threshold reached, 2- interval time, 3 - subscriber
logout, or 4 - inactivity. |
Example session-based reporting format
Oct 10 17:19:45 172.31.63.64 23003143,1349914925,546879,3,404234567123456,IMSI,linux,f501, 404234567123456,35827001,16394,1349914913,5469633,308908379, 0,0,0,0,0,5052,1 Oct 10 17:19:57 172.31.63.64 23003143,1349914937,546661,3,404234567123456,IMSI,linux,f501, 404234567123456,35827001,16394,1349914925,5550857,313317479, 0,0,0,0,0,5063,1 Oct 10 17:20:09 172.31.63.64 23003143,1349914949,546676,3,404234567123456,IMSI,linux,f501, 404234567123456,35827001,16394,1349914937,5636605,318053179, 0,0,0,0,0,5074,1
Flow-based reporting format
In an enforcement policy, a rule can send flow-based information about traffic that matches certain criteria to an external high-speed logging (HSL) server. The logs include the following comma-separated values in the order listed.
Field | Description |
---|---|
PEM id | Identifies the reporting module (PEM) and the field value is
23003143. |
Version | Indicates the version of the format for backward compatibility. |
Timestamp seconds | The time the information was logged in UNIX time format. |
Timestamp msec | The msecs time value of the timestamp in UNIX time format. |
Report type | The type of report; 0 – flow start, 1 – flow interim, 2 – flow end. |
Subscriber ID | A unique identifier (up to 64 characters) for the subscriber initiating the session, such as a phone number. The subscriber ID type determines the format. |
Subscriber ID type | The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private. |
Source IP | The IPv4 source address in the IP packet header. |
Source port | The source port the subscriber. |
Destination IP | The IPv4 destination address in the IP packet header. |
Destination IP address | The destination IP of the traffic. |
Destination port | The destination port for the traffic. |
Protocol | The protocol of the traffic for this flow, TCP or UDP. |
Route Domain | The route domain this flow belongs to. |
VLAN | The VLAN this flow belongs to. |
Application ID | A unique number that represents a particular application in this flow; it is used for classifying traffic. |
Urlcat ID | The URL category id that the flow belongs to. |
Flow start time seconds | The time, in seconds, the flow started in UNIX time format. |
Flow start time msecs | The time in milliseconds of the flow start time. |
Flow end time seconds | The time the flow ended in UNIX time format. |
Flow end time msecs | The time in milliseconds of the flow end time. |
Transactions count | The count of full transactions seen in the flow. |
Bytes in | The number of bytes received during this flow. |
Bytes out | The number of bytes sent during this flow. |
SNI | The Server Name Indication value in SSL traffic. |
Video Content Provider | The name of the Video Content Provider. |
Video Resolution | The number of pixels(vertical) of the video resolution. |
Video Bitrate | The number of bits processed per unit time. |
Handshake RTT | The Round Trip time of the handshake mechanism. |
Example flow-based reporting format
Sep 13 13:48:58 172.31.63.60 23003143,1347546777,654398,0,4086007577,E164,2001::10,52784,2001::2,80,6, 67,1347546774,628630,4278124286,4278124286,331,156, www.youtube.com, youtube, 720p, 11234567, 123 Sep 13 13:48:58 172.31.63.60 23003143,1347546777,654398,2,4086007577,E164,2001::10,52784,2001::2,80,6, 67,1347546774,628630,1347546775,382473,547,864, www.youtube.com, youtube, 720p, 11234567, 123
Transaction-based reporting format
In an enforcement policy, a rule can send transaction-based information about traffic
that matches certain criteria to an external high-speed logging (HSL) server. The logs include
the following comma-separated values in the order listed.
Field |
Description |
---|---|
PEM id |
Identifies the reporting module (PEM) and the field value is 23003143. |
Version |
Indicates the version of the format for backward compatibility. |
Record type |
The type of report; 10 – transactional. |
Transaction Number |
The sequential number of transaction in this flow (starting from 1). |
Subscriber ID |
A unique identifier (up to 64 characters) for the subscriber initiating the
session, such as a phone number. The subscriber ID type determines the format. |
Subscriber ID type |
The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private. |
Source IP |
The IPv4 source address in the IP packet header. |
Source port |
The source port the subscriber. |
Destination IP |
The IPv4 destination address in the IP packet header. |
Destination port |
The destination port for the traffic. |
Protocol, TCP/UDP |
The protocol of the traffic for this flow, TCP or UDP. |
Route Domain ID |
The route domain ID of the traffic. |
VLAN ID |
The VLAN ID of the traffic. |
Application/Category ID |
A unique number that represents the most relevant application or category that is
classified for the transaction. |
URL Category ID |
A unique number that represents the first (most relevant) URL category that is
classified for the transaction. |
Transaction Classification result |
Reports all classification tokens from the classification engine. The traffic classification result is stored using multiple tokens (8
application/category token identifiers and 4 URL token identifiers) and reported
using a CSV format. |
Transaction Start, seconds |
The transaction timestamp (seconds) in UNIX time format, when an HTTP request is
received. |
Transaction Start, msecs |
The transaction timestamp (msecs) in UNIX time format when an HTTP request is
received. |
Transaction Stop, seconds |
The transaction timestamp (seconds) in UNIX time format when the corresponding
HTTP response is received. |
Transaction Stop, msecs |
The transaction timestamp (msecs) in UNIX time format when the corresponding HTTP
response is received. |
Transaction Upstream Volume, bytes |
The number of HTTP request bytes for this transaction. |
Transaction Downstream Volume, bytes |
The number of HTTP response bytes for this transaction. |
Skipped Transactions of this kind |
The number of transactional reports skipped within the flow since the last
successfully transmission in the flow. |
HTTP information: |
The HTTP request/response information presented in a CSV format containing the
following fields:
|
Example transaction-based reporting format
Jan 15 11:36:27 localhost info tmm[29503]: 23003143,10,1.0.0,1,12341234,IMSI,10.10.10.212,32965,10.10.10.217,80,6,0,311,67,0, 67,16394,0,0,0,0,0,0,0,0,0,0,1389123382,694,1389123382,697,127,80799103,0,200, 0,10.10.10.217,0,Wget/1.13.4 (linux-gnu),0,/index_long.html Jan 15 11:36:28 localhost info tmm[29503]: 23003143,10,1.0.0,2,12341234,IMSI,10.10.10.212,32965,10.10.10.217,80,6,0,311,67,0, 67,16394,0,0,0,0,0,0,0,0,0,0,1389123384,264,1389123384,267,127,80799103,0,200, 0,10.10.10.217,0,Wget/1.13.4 (linux-gnu),0,/index_long.html Jan 15 11:36:33 localhost info tmm[29503]: 23003143,10,1.0.0,3,12341234,IMSI,10.10.10.212,32965,10.10.10.217,80,6,0,311,67,0, 67,16394,0,0,0,0,0,0,0,0,0,0,1389123385,572,1389123385,574,127,80799103,0,200, 0,10.10.10.217,0,Wget/1.13.4 (linux-gnu),0,/index_long.html Jan 15 11:36:33 localhost info tmm[29503]: 23003143,10,1.0.0,4,12341234,IMSI,10.10.10.212,32965,10.10.10.217,80,6,0,311,67,0, 67,16394,0,0,0,0,0,0,0,0,0,0,1389123387,968,1389123387,970,127,80799103,0,200, 0,10.10.10.217,0,Wget/1.13.4 (linux-gnu),0,/index_long.html Jan 15 11:36:45 localhost info tmm[29503]: 23003143,10,1.0.0,5,12341234,IMSI,10.10.10.212,32965,10.10.10.217,80,6,0,311,67,0, 67,16394,0,0,0,0,0,0,0,0,0,0,1389123399,196,1389123399,201,127,80799103,0,200, 0,10.10.10.217,0,Wget/1.13.4 (linux-gnu),0,/index_long.html
Device Type and OS-based reporting format
In an enforcement policy, a rule can send Device Type and OS (DTOS)-based information
about traffic that matches certain criteria to an external high-speed logging (HSL) server. The
logs include the following comma-separated values in the order listed.
Field |
Description |
---|---|
Report id |
Identifies the reporting module (PEM) and the field value is
23003143 . |
Report type |
The type of report. Always set to 5 for tethering-based reporting. |
Report version |
Indicates the version of the format for backward compatibility. |
Report timestamp seconds |
The time the information was logged (along with the timestamp in milliseconds),
specifies seconds using UNIX time format. |
Report timestamp millisecs |
The time the information was logged (along with the timestamp in seconds),
specifies milliseconds using UNIX time format. |
Subscriber ID |
A unique identifier (up to 64 characters) for the subscriber initiating the
session, such as a phone number. The subscriber ID type determines the format. |
Subscriber type |
The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private. |
Subscriber IMEISV |
The IMEISV value for the subscriber. |
Device name |
The name of the device obtained from the TacDB. |
Device OS |
The device OS obtained form the TacDB. |
UA-based OS |
The OS determined from the user-agent for the sampled flow. |
TCP fingerprint-based OS |
The OS determined from the TCP fingerprints of the sampled flow. |
TCP window size |
The Window size from the sampled TCP flow. |
Source port |
The source port of the sampled flow. |
TTL |
The time to live (TTL) value of the sampled flow. |
TCP header length |
The header length of the sampled TCP flow. |
TCP window scaling factor |
The window scaling factor of the sampled TCP flow. |
Record Reason |
The reason for sending the record. Set to 14 when tethering is detected, or set
to 15 when tethering is not detected. |
Operating Systems detected |
Not Applicable |
Example Device-based reporting format
Mar 6 12:07:26 172.31.63.95 23003143,5,1.0.0,1425672465,542,310150123456654,IMSI,012430000098765,Apple-iPhone 4 model MC603KS,iOS,,, 0,0,0,0,0,14,Linux;Windows; Mar 6 12:07:56 172.31.63.95 23003143,5,1.0.0,1425672495,542,310150123456654,IMSI,012430000098765,Apple-iPhone 4 model MC603KS,iOS,,, 0,0,0,0,0,15,
Tethering-based reporting format
In an enforcement policy, a rule can send tethering-based information about traffic
that matches certain criteria to an external high-speed logging (HSL) server. The logs include
the following comma-separated values in the order listed.
Field |
Description |
---|---|
Report id |
Identifies the reporting module (PEM) and the field value is
23003143 . |
Report type |
The type of report. Always set to 5 for tethering-based reporting. |
Report version |
Indicates the version of the format for backward compatibility. |
Report timestamp seconds |
The time the information was logged (along with the timestamp in milliseconds),
specifies seconds using UNIX time format. |
Report timestamp millisecs |
The time the information was logged (along with the timestamp in seconds),
specifies milliseconds using UNIX time format. |
Subscriber ID |
A unique identifier (up to 64 characters) for the subscriber initiating the
session, such as a phone number. The subscriber ID type determines the format. |
Subscriber type |
The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private. |
Subscriber IMEISV |
The IMEISV value for the subscriber. |
Device name |
The name of the device obtained from the TacDB. |
Device OS |
The device OS obtained form the TacDB. |
UA-based OS |
Not Applicable |
TCP fingerprint-based OS |
Not Applicable |
TCP window size |
Not Applicable |
Source port |
Not Applicable |
TTL |
Not Applicable |
TCP header length |
Not Applicable |
TCP window scaling factor |
Not Applicable |
Record Reason |
The reason for sending the record. Set to 14 when tethering is detected, or set
to 15 when tethering is not detected. |
Operating Systems detected |
Contains the list of operating systems that were detected during the tethering
detection sampling interval. |
Example: Tethering-based reporting format
Mar 6 12:07:26 172.31.63.95 23003143,5,1.0.0,1425672465,542,310150123456654,IMSI,012430000098765,Apple-iPhone 4 model MC603KS,iOS,,, 0,0,0,0,0,14,Linux;Windows; Mar 6 12:07:56 172.31.63.95 23003143,5,1.0.0,1425672495,542,310150123456654,IMSI,012430000098765,Apple-iPhone 4 model MC603KS,iOS,,, 0,0,0,0,0,15,