Applies To:
Show VersionsBIG-IP APM
- 12.1.6, 12.1.5, 12.1.4, 12.1.3, 12.1.2, 12.1.1, 12.1.0
Reports, Logs, and Statistics
About SWG data for threat monitoring
After Secure Web Gateway (SWG) starts proxying web access, it provides information that you can use to monitor threats and to fine-tune URL filters.
On a BIG-IP® system with Access Policy Manager®, SWG can provide logs and reports.
On a BIG-IP system with an SWG subscription, SWG can provide overview statistics in addition to logs and reports.
Overview: Monitoring Internet traffic for threats
You can view Secure Web Gateway (SWG) statistics on the BIG-IP® system and adjust URL filters to handle new threats based on the information that you gather from logs and reports.
Before you begin, event logging should be configured. SWG reports and charts depend on event logging for URL filters. For event logging to occur, log settings must be configured and then specified in the access profile, and a Category Lookup item must be run in the per-request policy.
Task summary
About the Secure Web Gateway Overview
The Secure Web Gateway (SWG) overview provides multiple reports and charts that summarize the top requests, such as top URLs, top categories by blocked request count, top users by permitted request count or by blocked request count, and so on. The overview can be customized to show the specific type of data that you are interested in.
In addition to the reports and charts on the overview, SWG provides the All Requests and Blocked Requests reports and charts. The reports can be filtered to show the information that you want to see.
Configuring statistics collection for SWG reports
Examining statistics on the SWG Overview
Focusing the Overview on security threats
Exporting or emailing SWG statistics
Creating an SMTP server configuration
Implementation result
Secure Web Gateway (SWG) is configured to produce reports and charts.
About the reporting interval for charts and reports
The system updates the statistics for charts and reports at five minute intervals: at five minutes after the hour, ten minutes after the hour, and so on.
Charts and data that you export from charts reflect the publishing interval of five minutes. For example, if you request data for the time period 12:40-13:40, the data in the chart or in the file that you export is for the time period 12:35-13:35. By default, the BIG-IP® system displays one hour of data.
About statistics aggregation for weekly and longer time ranges
Secure Web Gateway (SWG) reports and charts for weekly, monthly, and yearly time ranges include statistics up through the previously completed hour. The system performs hourly updates to the aggregated statistics.
About Secure Web Gateway statistics
Secure Web Gateway (SWG) reports display statistical information about web traffic on your system. These details are available:
- Actions
- Action (allowed, blocked, or confirmed) taken on the URL request.
- Client IP address
- IP address from which the request for the URL originated.
- Host Name
- When available, host name from which the request for the URL originated.
- Categories
- Name of the preconfigured or custom URL category into which a requested URL falls.
- URLs
- Requested URL.
- URL filters
- Name of the URL filter SWG applied to the request based on the schedule in the scheme.
- Security categories
- The security category of the URL if it was blocked, because it matched a security
category.Note: Security categories are available on a BIG-IP® system with an SWG subscription.
- Users
- Name of the user that made the request, if available. Note: Configuring your system to identify users is optional.
- SSL bypass
- Whether the request was bypassed (yes or no). Note: Configuring your system to omit certain SSL traffic from inspection is optional.
Overview: Configuring remote high-speed APM and SWG event logging
You can configure the BIG-IP® system to log information about Access Policy Manager® (APM® ) and Secure Web Gateway events and send the log messages to remote high-speed log servers.
When configuring remote high-speed logging of events, it is helpful to understand the objects you need to create and why, as described here:
Object | Reason |
---|---|
Pool of remote log servers | Create a pool of remote log servers to which the BIG-IP system can send log messages. |
Destination (unformatted) | Create a log destination of Remote High-Speed Log type that specifies a pool of remote log servers. |
Destination (formatted) | If your remote log servers are the ArcSight, Splunk, or Remote Syslog type, create an additional log destination to format the logs in the required format and forward the logs to a remote high-speed log destination. |
Publisher | Create a log publisher to send logs to a set of specified log destinations. |
Log Setting | Add event logging for the APM system and configure log levels for it or add logging for URL filter events, or both. Settings include the specification of up to two log publishers: one for access system logging and one for URL request logging. |
Access profile | Add log settings to the access profile. The log settings for the access profile control logging for the traffic that comes through the virtual server to which the access profile is assigned. |
Association of remote high-speed logging configuration objects
Task summary
Perform these tasks to configure remote high-speed APM and SWG event logging on the BIG-IP system.Task list
About the default-log-setting
Access Policy Manager® (APM®) provides a default-log-setting. When you create an access profile, the default-log-setting is automatically assigned to it. The default-log-setting can be retained, removed, or replaced for the access profile. The default-log-setting is applied to user sessions only when it is assigned to an access profile.
Regardless of whether it is assigned to an access profile, the default-log-setting applies to APM processes that run outside of a user session. Specifically, on a BIG-IP® system with an SWG subscription, the default-log-setting applies to URL database updates.
Creating a pool of remote logging servers
Creating a remote high-speed log destination
Create a log destination of the Remote High-Speed Log type to specify that log messages are sent to a pool of remote log servers.
Creating a formatted remote high-speed log destination
Create a formatted logging destination to specify that log messages are sent to a pool of remote log servers, such as Remote Syslog, Splunk, or ArcSight servers.
Creating a publisher
Configuring log settings for access system and URL request events
Disabling logging
About event log levels
Event log levels are incremental, ranging from most severe (Emergency) to least severe (Debug). Setting an event log level to Warning for example, causes logging to occur for warning events, in addition to events for more severe log levels. The possible log levels, in order from highest to lowest severity are:
- Emergency
- Alert
- Critical
- Error
- Warning
- Notice (the default log level)
- Informational
- Debug
APM log example
The table breaks a typical Access Policy Manager® (APM®) log entry into its component parts.
An example APM log entry
Feb 2 12:37:05 site1 notice tmm[26843]: 01490500:5: /Common/for_reports:Common: bab0ff52: New session from client IP 10.0.0.1 (ST=/CC=/C=) at VIP 20.0.0.1 Listener /Common/site1_http (Reputation=Unknown)
Information Type | Example Value | Description |
---|---|---|
Timestamp | Feb 2 12:37:05 | The time and date that the system logged the event message. |
Host name | site1 | The host name of the system that logged the event message. Because this is typically the host name of the local machine, the appearance of a remote host name could be of interest. |
Log level | notice |
The text value of the log level for the message. |
Service | tmm |
The process that generated the event. |
PID | [26843] | The process ID. |
Log ID | 01490500 | A code that signifies the product, a subset of the product, and a message number. |
Level | 5 | The numeric value of the log level for the message. |
Partition | /Common/for_reports:Common | The partition.to which configuration objects belong. |
Session ID | bab0ff52 | The ID associated with the user session. |
Log message | New session from client IP 10.0.0.1 (ST=/CC=/C=) at VIP 20.0.0.1 Listener /Common/site1_http (Reputation=Unknown) | The generated message text. |
About local log destinations and publishers
The BIG-IP® system provides two local logging destinations:
- local-db
- Causes the system to store log messages in the local MySQL database. Log messages published to this destination can be displayed in the BIG-IP Configuration utility.
- local-syslog
- Causes the system to store log messages in the local Syslog database. Log messages published to this destination are not available for display in the BIG-IP Configuration utility.
The BIG-IP system provides a default log publisher for local logging, sys-db-access-publisher; initially, it is configured to publish to the local-db destination and the local-syslog destination. Users can create other log publishers for local logging.
Configuring a log publisher to support local reports
Viewing an APM report
Viewing URL request logs
Configuring a log publisher to supply local syslogs
Preventing logging to the /var/log/apm file
About local log storage locations
The BIG-IP® system publishes logs for portal access traffic and for connections to virtual desktops (VDI) to the /var/log/rewrite* files. APM® cannot publish these logs to remote destinations.
APM can publish URL request logs to remote or local destinations. Logs published to the local-db destination are stored in the local database and are available for display from the Configuration utility. Logs published to the local-syslog destination are stored in the /var/log/urlfilter.log file.
APM can publish access system logs to remote or local destinations. Logs published to the local-db destination are stored in the local database. Logs in the local database are available for display in APM reports. Logs published to the local-syslog destination are stored in the /var/log/apm file.
Code expansion in Syslog log messages
The BIG-IP® system log messages contain codes that provide information about the system. You can run the Linux command cat log |bigcodes |less at the command prompt to expand the codes in log messages to provide more information. For example:
Jun 14 14:28:03 sccp bcm56xxd [ 226 ] : 012c0012 : (Product=BIGIP Subset=BCM565XXD) : 6: 4.1 rx [ OK 171009 Bad 0 ] tx [ OK 171014 Bad 0 ]
About configurations that produce duplicate log messages
Event log duplication
The figure illustrates a configuration that writes duplicate logs. Two log publishers specify the same log destination, local-db. Each log publisher is specified in one of the log settings that are assigned to an access profile. Logs are written to the local-db destination twice.
Methods to prevent or eliminate duplicate log messages
Duplicate log messages are written when the same log destination is specified by two or more log publishers and more than one of the log publishers is specified in the log settings that are assigned to an access profile.
One way to avoid or eliminate this problem is to specify only one log setting for each access profile. Another is to ensure that the log publishers you associate with log settings for an access profile do not contain duplication log destinations.
About log level configuration
Log levels can be configured in various ways that depend on the specific functionality. Log levels for access portal traffic and for connections to virtual desktops are configured in the System area of the product. The log level for the URL database download is configured in the default-log-setting in the Access Policy Event Logs area of the product. The log level for NTLM authentication of Microsoft Exchange clients is configured using the ECA option in any log setting. Other access policy (and Secure Web Gateway) log levels are configured in any log setting.
Updating the log level for NTLM for Exchange clients
Configuring logging for the URL database
Setting log levels for Portal Access and VDI events
Change the logging level for access policy events when you need to increase or decrease the minimum severity level at which Access Policy Manager® (APM®) logs that type of event. Follow these steps to change the log level for events that are related to portal access traffic or related to connections to virtual desktops (VDI).