F5 Logo MyF5
Search
  1. MyF5 Home
  2. Knowledge Centers
  3. BIG-IP PEM
  4. BIG-IP Policy Enforcement Manager: Implementations
  5. Reporting Usage Data to an External Analytics Server
Manual Chapter : Reporting Usage Data to an External Analytics Server

Applies To:

Show Versions Show Versions

BIG-IP PEM

  • 11.6.5, 11.6.4, 11.6.3, 11.6.2, 11.6.1
Manual Chapter
Table of Contents | << Previous Chapter | Next Chapter >>

Reporting Usage Data to an External Analytics Server

Overview: Reporting usage data to an external analytics server

In Policy Enforcement Manager™, you can create a rule within an enforcement policy that instructs the system to send usage data in high-speed logging (HSL) format to an external analytics server. The rule specifies what type of reporting data you are interested in; one of the actions it can take with the traffic is to send the information collected about it for processing to a centralized analytics server.

The system sends the information as a set of comma-separated values by means of SYSLOG transport. You can choose to use the session-based, flow-based or transactional reporting format, depending on the level of granularity you need.

For example, a rule might collect session-based information about all audio and video traffic. You can specify how often to log the data and set the destination as an HSL server or pool.

Transactional Policy Enforcement, provides the ability to report each of the HTTP transaction and sends the report to a HSL publisher. Each transaction report information is specific to that transaction only. The transactional reports are used for analytics and high level granularity for application and subscriber visibility.

Task summary

Creating a publisher

Before you create a publisher, you have to create a HSL pool that needs to be associated to a destination. Ensure that at least one destination associated with a pool of remote log servers exists on the BIG-IP® system.
Create a publisher to specify where the BIG-IP system sends log messages for specific resources.
  1. On the Main tab, click System > Logs > Configuration > Log Publishers .
    The Log Publishers screen opens.
  2. Click Create.
  3. In the Name field, type a unique, identifiable name for this publisher.
  4. For the Destinations setting, select a destination from the Available list, and click << to move the destination to the Selected list.
    Note: If you are using a formatted destination, select the destination that matches your log servers, such as Remote Syslog, Splunk, or ArcSight.
  5. Click Finished.

Creating a rule for high-speed logging for session reporting

Before you can create a high-speed logging (HSL) rule, you need to create a publisher that defines the destination server or pool where the HSL logs are sent.
In an enforcement policy, a rule can specify that session statistics about the traffic affected by the rule are sent to an external high-speed logging server.
  1. On the Main tab, click Policy Enforcement > Policies .
    The Policies screen opens.
  2. Click the name of the enforcement policy you want to add rules to.
    The properties screen for the policy opens.
  3. In the Policy Rules area, click Add.
    The New Rule screen opens.
  4. In the Name field, type a name for the rule.
  5. In the Precedence field, type an integer that indicates the precedence for the rule in relation to the other rules. Number 1 has the highest precedence. Rules with higher precedence are evaluated before other rules with lower precedence.
    Tip: All rules in a policy are run concurrently. Precedence takes effect when there are conflicting rules. The conflict occurs when the traffic matches two rules and the policy actions from these rules differ. For example, if you have a rule 1 with precedence 10 with Gate Status disabled for a search engine and you have rule 2 with precedence 11 with Gate Status enabled, then rule 1 is processed first because it has higher precedence. Rules conflict if they have identical or overlapping classification criteria (for the traffic that matches more than one rule). In some cases, different policy actions are not conflicting and hence applied in parallel.
  6. Use the Classification, URL, Flow, and Custom Criteria tabs to identify the traffic that you want to be affected by this rule.
  7. From the Reporting list, select Enabled.
  8. From the Report Granularity list, select Session to log details about subscribers and application sessions.
  9. In the Volume Threshold setting, specify in octets, the threshold to send HSL reporting records. You can send reporting data from uplink traffic, to downlink traffic and the total traffic volume before logging the information.
  10. In the Destination setting, specify where to send the usage monitoring data:
    • In the Gx field select Enabled for the BIG-IP system to send usage monitoring data over a Gx interface. You can then type a string for the Gx Monitoring Key that is used for usage monitoring.
      Note: When you select Session in the Report Granularity field, the Gx field appears.
    • From the HSL list, select the name of the publisher that specifies the server or pool of remote HSL servers to send the logs and select the format script of the report from the Format Script list.
    • Select the RADIUS Accounting option from the destination. From the RADIUS AAA Virtual list, select the RADIUS AAA virtual that you have created before.
    Note: If you are using a formatted destination, select the publisher that matches your log servers, such as Remote Syslog, Splunk, or ArcSight.
    Note: There are no Format Scripts for transactional reporting.
  11. In the Interval field, type an integer that specifies how frequently HSL reporting data is sent.
  12. For the Session Reporting Field setting, move the fields that you want to see in the logs from the Available list to the Selected list.
  13. Click Finished.
You have created a rule that sends data about the traffic to external high-speed logging servers. The CSV reporting format differs depending on whether the report granularity is flow-based or session-based.

Creating a rule for high-speed logging for flow reporting

Before you can create a high-speed logging (HSL) rule, you need to create a publisher that defines the destination server or pool where the HSL logs are sent.
In an enforcement policy, a rule can specify that flow statistics about the traffic affected by the rule are sent to an external high-speed logging server.
  1. On the Main tab, click Policy Enforcement > Policies .
    The Policies screen opens.
  2. Click the name of the enforcement policy you want to add rules to.
    The properties screen for the policy opens.
  3. In the Policy Rules area, click Add.
    The New Rule screen opens.
  4. In the Name field, type a name for the rule.
  5. In the Precedence field, type an integer that indicates the precedence for the rule in relation to the other rules. Number 1 has the highest precedence. Rules with higher precedence are evaluated before other rules with lower precedence.
    Tip: All rules in a policy are run concurrently. Precedence takes effect when there are conflicting rules. The conflict occurs when the traffic matches two rules and the policy actions from these rules differ. For example, if you have a rule 1 with precedence 10 with Gate Status disabled for a search engine and you have rule 2 with precedence 11 with Gate Status enabled, then rule 1 is processed first because it has higher precedence. Rules conflict if they have identical or overlapping classification criteria (for the traffic that matches more than one rule). In some cases, different policy actions are not conflicting and hence applied in parallel.
  6. Use the Classification, URL, Flow, and Custom Criteria tabs to identify the traffic that you want to be affected by this rule.
  7. From the Reporting list, select Enabled.
  8. From the Report Granularity list, select Flow, for more granular reporting of every TCP connection.
  9. In the Volume Threshold setting, specify in octets, the threshold to send HSL reporting records. You can send reporting data from uplink traffic, to downlink traffic and the total traffic volume before logging the information.
  10. In the Interval field, type an integer that specifies how frequently HSL reporting data is sent.
  11. In the Destination setting, specify where to send the usage monitoring data:
    • From the HSL list, select the name of the publisher that specifies the server or pool of remote HSL servers to send the logs.
    • Select the Format Script listand select the format script of the report from the Format Script list.
    • Select the RADIUS Accounting option from the destination. From the RADIUS AAA Virtual list, select the RADIUS AAA virtual that you have created before.
    Note: If you are using a formatted destination, select the publisher that matches your log servers, such as Remote Syslog, Splunk, or ArcSight.
  12. For the Flow Reporting Field setting, move the fields that you want to see in the logs from the Available list to the Selected list.
  13. Click Finished.
You have created a rule that sends data about the traffic to external high-speed logging servers. The CSV reporting format differs depending on whether the report granularity is flow-based or session-based.

Creating a high-speed logging rule for transactional reporting

Before you can create a high-speed logging (HSL) rule, you need to create a publisher that defines the destination server or pool where the HSL logs are sent.
In an enforcement policy, a rule can specify that transactional statistics about traffic affected by the rule are sent to an external high-speed logging server.
  1. On the Main tab, click Policy Enforcement > Policies .
    The Policies screen opens.
  2. From the Transactional list, select Enabled if you want the BIG-IP system to allow policy enforcement on each HTTP transaction.
  3. Click the name of the enforcement policy you want to add rules to.
    The properties screen for the policy opens.
  4. In the Policy Rules area, click Add.
    The New Rule screen opens.
  5. In the Name field, type a name for the rule.
  6. In the Precedence field, type an integer that indicates the precedence for the rule in relation to the other rules. Number 1 has the highest precedence. Rules with higher precedence are evaluated before other rules with lower precedence.
    Tip: All rules in a policy are run concurrently. Precedence takes effect when there are conflicting rules. The conflict occurs when the traffic matches two rules and the policy actions from these rules differ. For example, if you have a rule 1 with precedence 10 with Gate Status disabled for a search engine and you have rule 2 with precedence 11 with Gate Status enabled, then rule 1 is processed first because it has higher precedence. Rules conflict if they have identical or overlapping classification criteria (for the traffic that matches more than one rule). In some cases, different policy actions are not conflicting and hence applied in parallel.
  7. Use the Classification, URL, Flow, and Custom Criteria tabs to identify the traffic that you want to be affected by this rule.
  8. From the Reporting list, select Enabled.
  9. From the Report Granularity list, select Transaction, for more granular reporting of every HTTP transaction.
  10. In the Additional HTTP Information setting, specify in bytes, the HTTP Hostname, the HTTP User Agent, and the HTTP URI.
  11. In the Destination setting, specify where to send the usage monitoring data:
    • From the HSL list, select the name of the publisher that specifies the server or pool of remote HSL servers to send the logs.
    • Select the RADIUS Accounting option from the destination. From the RADIUS AAA Virtual list, select the RADIUS AAA virtual that you created earlier.
    Note: If you are using a formatted destination, select the publisher that matches your log servers, such as Remote Syslog, Splunk, or ArcSight.
    Note: There are no Format Scripts for transactional reporting.
  12. For the Transaction Reporting Field setting, move the fields that you want to see in the logs from the Available list to the Selected list.
  13. Click Finished.
You have created a rule that sends transactional data about the traffic to external high-speed logging servers. You can now assign the policy to an active subscriber.

Session-based reporting format

In an enforcement policy, a rule can send session-based information about traffic that matches certain criteria to an external high-speed logging (HSL) server. The logs include the following comma-separated values in the order listed.

Field Description
PEM id Identifies the reporting module (PEM) and the field value is 23003143.
Version Indicates the version of the format for backward compatibility.
Timestamp seconds The time the information was logged (along with the timestamp in milliseconds), specifies seconds using UNIX time format.
Timestamp msec The time the information was logged (along with the timestamp in seconds), specifies milliseconds using UNIX time format.
Report type The type of report. Always set to 3 for session-based reporting.
Subscriber ID A unique identifier (up to 64 characters) for the subscriber initiating the session, such as a phone number. The subscriber ID type determines the format.
Subscriber ID type The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private.
3GPP parameters The list of 3GPP parameters, which can be imsi, imeisv, tower_id, or username.
Policy ID The Identification of the policy.
Rule ID The Identification of the policy rule.
Application ID A unique number that represents a particular application, and is used for classifying traffic.
Last Sent The time, in seconds, since the last log entry was sent.
Bytes in The number of bytes received during this session.
Bytes out The number of bytes sent during this session.
Concurrent flows Always 0 (unsupported).
Opened flows Always 0 (unsupported).
Terminated flows Always 0 (unsupported).
Total transactions Always 0 (unsupported).
Successful transactions Always 0 (unsupported).
Aggregated category duration Summary of the duration of all flows for the session.
Reason The reason for sending the record. It can be 0 - reserved, 1 - volume threshold reached, 2- interval time, 3 - subscriber logout, or 4 - inactivity.

Example session-based reporting format

Oct 10 17:19:45 172.31.63.64
23003143,1349914925,546879,3,404234567123456,IMSI,linux,f501,
404234567123456,35827001,16394,1349914913,5469633,308908379,
0,0,0,0,0,5052,1
Oct 10 17:19:57 172.31.63.64
23003143,1349914937,546661,3,404234567123456,IMSI,linux,f501,
404234567123456,35827001,16394,1349914925,5550857,313317479,
0,0,0,0,0,5063,1
Oct 10 17:20:09 172.31.63.64
23003143,1349914949,546676,3,404234567123456,IMSI,linux,f501,
404234567123456,35827001,16394,1349914937,5636605,318053179,
0,0,0,0,0,5074,1

Flow-based reporting format

In an enforcement policy, a rule can send flow-based information about traffic that matches certain criteria to an external high-speed logging (HSL) server. The logs include the following comma-separated values in the order in which the attributes were added (available to selected list).

Field Description
PEM id Identifies the reporting module (PEM) and the field value is 2300314.
Version Indicates the version of the format for backward compatibility.
Timestamp seconds The time the information was logged in UNIX time format.
Timestamp msec The msecs time value of the timestamp (in decimal number).
Report type The type of report; 0 – flow start, 1 – flow interim, 2 – flow end.
Subscriber ID A unique identifier (up to 64 characters) for the subscriber initiating the session, such as a phone number. The subscriber ID type determines the format.
Subscriber ID type The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private.
Source IP The IPv4 source address in the IP packet header.
Source port The source port the subscriber.
Destination IP The IPv4 destination address in the IP packet header.
Destination port The destination port for the traffic.
Protocol The protocol of the traffic for this flow, TCP or UDP.
Route Domain The route domain this flow belongs to.
VLAN The VLAN this flow belongs to.
Application ID A unique number that represents a particular application in this flow; it is used for classifying traffic.
Urlcat ID The URL category id that the flow belongs to.
Flow start time seconds The time, in seconds, the flow started in UNIX time format.
Flow start time msecs The time in milliseconds of the flow start time.
Flow end time seconds The time the flow ended in UNIX time format.
Flow end time msecs The time in milliseconds of the flow end time.
Transactions count The count of full transactions seen in the flow.
Bytes in The number of bytes received during this flow.
Bytes out The number of bytes sent during this flow.

Example flow-based reporting format

Sep 13 13:48:58 172.31.63.60
23003143,1347546777,654398,0,4086007577,E164,2001::10,52784,2001::2,80,6,
67,1347546774,628630,4278124286,4278124286,331,156
Sep 13 13:48:58 172.31.63.60
23003143,1347546777,654398,2,4086007577,E164,2001::10,52784,2001::2,80,6,
67,1347546774,628630,1347546775,382473,547,864

Transaction-based reporting format

In an enforcement policy, a rule can send transaction-based information about traffic that matches certain criteria to an external high-speed logging (HSL) server. The logs include the following comma-separated values in the order listed.

Field Description
PEM id Identifies the reporting module (PEM) and the field value is 23003143.
Version Indicates the version of the format for backward compatibility.
Record type The type of report; 10 – transactional.
Transaction Number The sequential number of transaction in this flow (starting from 1).
Subscriber ID A unique identifier (up to 64 characters) for the subscriber initiating the session, such as a phone number. The subscriber ID type determines the format.
Subscriber ID type The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private.
Source IP The IPv4 source address in the IP packet header.
Source port The source port the subscriber.
Destination IP The IPv4 destination address in the IP packet header.
Destination port The destination port for the traffic.
Protocol, TCP/UDP The protocol of the traffic for this flow, TCP or UDP.
Route Domain ID The route domain ID of the traffic.
VLAN ID The VLAN ID of the traffic.
Application/Category ID A unique number that represents the most relevant application or category that is classified for the transaction.
URL Category ID A unique number that represents the first (most relevant) URL category that is classified for the transaction.
Transaction Classification result Reports all classification tokens from the classification engine.
Note: The traffic classification result is stored using multiple tokens (8 application/category token identifiers and 4 URL token identifiers) and reported using a CSV format.
Transaction Start, seconds The transaction timestamp (seconds) in UNIX time format, when an HTTP request is received.
Transaction Start, msecs The transaction timestamp (msecs) in UNIX time format when an HTTP request is received.
Transaction Stop, seconds The transaction timestamp (seconds) in UNIX time format when the corresponding HTTP response is received.
Transaction Stop, msecs The transaction timestamp (msecs) in UNIX time format when the corresponding HTTP response is received.
Transaction Upstream Volume, bytes The number of HTTP request bytes for this transaction.
Transaction Downstream Volume, bytes The number of HTTP response bytes for this transaction.
Skipped Transactions of this kind The number of transactional reports skipped within the flow since the last successfully transmission in the flow.
HTTP information: The HTTP request/response information presented in a CSV format containing the following fields:
  • HTTP Transaction Response Code
  • HTTP Hostname field truncated (indicates that the Hostname field is truncated due to excessive length)
  • HTTP Hostname
  • HTTP User Agent field truncated (indicates that the User Agent field is truncated due to excessive length)
  • HTTP User Agent
  • HTTP URI field truncated (indicates that the URI field is truncated due to excessive length)
  • HTTP URI

Example transaction-based reporting format

Jan 15 11:36:27 localhost info tmm[29503]:
23003143,10,1.0.0,1,12341234,IMSI,10.10.10.212,32965,10.10.10.217,80,6,0,311,67,0,
67,16394,0,0,0,0,0,0,0,0,0,0,1389123382,694,1389123382,697,127,80799103,0,200,
0,10.10.10.217,0,Wget/1.13.4 (linux-gnu),0,/index_long.html
Jan 15 11:36:28 localhost info tmm[29503]:
23003143,10,1.0.0,2,12341234,IMSI,10.10.10.212,32965,10.10.10.217,80,6,0,311,67,0,
67,16394,0,0,0,0,0,0,0,0,0,0,1389123384,264,1389123384,267,127,80799103,0,200,
0,10.10.10.217,0,Wget/1.13.4 (linux-gnu),0,/index_long.html
Jan 15 11:36:33 localhost info tmm[29503]:
23003143,10,1.0.0,3,12341234,IMSI,10.10.10.212,32965,10.10.10.217,80,6,0,311,67,0,
67,16394,0,0,0,0,0,0,0,0,0,0,1389123385,572,1389123385,574,127,80799103,0,200,
0,10.10.10.217,0,Wget/1.13.4 (linux-gnu),0,/index_long.html
Jan 15 11:36:33 localhost info tmm[29503]:
23003143,10,1.0.0,4,12341234,IMSI,10.10.10.212,32965,10.10.10.217,80,6,0,311,67,0,
67,16394,0,0,0,0,0,0,0,0,0,0,1389123387,968,1389123387,970,127,80799103,0,200,
0,10.10.10.217,0,Wget/1.13.4 (linux-gnu),0,/index_long.html
Jan 15 11:36:45 localhost info tmm[29503]:
23003143,10,1.0.0,5,12341234,IMSI,10.10.10.212,32965,10.10.10.217,80,6,0,311,67,0,
67,16394,0,0,0,0,0,0,0,0,0,0,1389123399,196,1389123399,201,127,80799103,0,200,
0,10.10.10.217,0,Wget/1.13.4 (linux-gnu),0,/index_long.html
Table of Contents | << Previous Chapter | Next Chapter >>
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5 Networks, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information