Manual Chapter : Connect BIG-IP data logs to a Data Collection Device Cluster
Applies To:Show Versions
BIG-IQ Centralized Management
- 8.2.0, 8.1.0
Connect BIG-IP data logs to a Data Collection Device Cluster
Data logging over multiple DCDs
Data logging over multiple data collection devices (DCDs) streamlines the data collection process across your BIG-IQ system. This is done by load balancing log messages, sent from BIG-IP devices, among the multiple DCDs in your configuration. Completing this configuration ensures that your logging profiles are configured to have high availability, in the event that a DCD is unavailable, while preventing unnecessary duplication of data.
Load Balancing events over DCDs
To ensure high availability among your DCDs, for the purpose of logging events from your BIG-IP devices, you need to configure a pool of DCDs. For more information about configuring the DCD pool, see Create a remote logging pool of DCDs.
Each service module requires its own DCD pool. This ensures that the pool is configured with the proper port listeners for each module's log messages. For modules that share the same port number, such as ASM and Bot Defense, you can use the same DCD pool.
Connecting event logs to DCD pool
Once you have created your DCD pool(s), you can configure remote logging to your BIG-IQ DCDs. Depending on your service module, this process may vary. For more information about each modules' log configuration requirements, see Overview of configuring HA DCD logging per service module.
Overview of configuring HA DCD logging per service module
Using BIG-IQ, you can view and manage statistics and event data generated by the services on your BIG-IP devices. You collect BIG-IP logging data on a Data Collection Device (DCD) cluster. If your system configuration includes multiple DCDs it is recommended to create a pool that optimizes and streamlines data collection from your managed BIG-IP devices. Before you create and configure your logging profiles, you must complete this procedure. To create a pool of DCDs, see
Create a remote logging pool of DCDs.
If you only have one DCD in your configuration, you do not need to configure a remote logging pool, and you can connect your logging profile directly to your DCD. This option does not provide high availability and log messages will not be collected in the event that your DCD is not available.
The work flow to configure data to route from the BIG-IP devices to your data collection device (DCD) cluster depends on the type of data you want to collect and the services that run on your BIG-IP devices. The table below outlines the procedures required for each service module. For end to end configuration of high availability logging, including configuration of a logging profile using BIG-IQ, go to the documentation referred to in
Logging profile configuration.
If you have multiple modules, you need to configure one remote logging DCD pool per service port.
Logging profile configuration
Web Application Security (ASM or Adv. WAF)
Managing Web Application Security Loggingsection in
BIG-IQ Web Application Security
Logging Bot Defense Requestssection in
Managing Bot Defense Using BIG-IQ
Fraud Protection Services (FPS)
The following process is for BIG-IP devices running version 15.0 or higher. For information about setting up logging to BIG-IQ for versions 14.1 or earlier, see
Configure BIG-IP a remote logging profile for BIG-IP FPS.
This process can only be done using the BIG-IP interface (version 15.0 or higher). You cannot create an FPS logging profile using BIG-IQ.
Logging FPS over multiple DCDssection in
BIG-IQ: Fraud Protection Service.
Logging DoS Protection eventssection in
Managing DDoS attacks Using BIG-IQ
Network Security (AFM)
Logging Network Security eventssection in
Managing Network Security Using BIG-IQ
Access (APM and IP Security)
Logging Access eventssection in
Centrally Managing Access Groups Using BIG-IQ
Create a remote logging pool of DCDs
For this process you will need the following:
- Three or more data collection devices (recommended).This process only applies to BIG-IQ configurations with multiple DCD devices.
- A logging profile that has remote storage enabled, and is attached to a virtual server on a BIG-IP device that hosts the service module and its policy.
- [ASM/Web Application Security only] A load balancing BIG-IP device. This is a device that hosts the virtual server that load balances logging messages to the pool of DCDs, but does NOT host ASM service module policies or logging profiles.
To optimize data logging of messages from your BIG-IP devices to multiple DCDs, you can configure a BIG-IP system to load balance these messages among the DCDs in your BIG-IQ configuration. This process prevents duplication of information in the consolidated data repository, while also providing high availability for your log messages in the case that one or more DCDs become unavailable.
- At the top of the screen, clickConfiguration, then, on the left, click .
- Type a uniqueNamefor the pool.
- From theDevicelist, select a load balancing BIG-IP device that provides the load balancing service to the DCD pool.For ASM/ Web Application security logging information, be sure to select a BIG-IP device that is different from the device that hosts your virtual server with the service module policy.
- In theHealth Monitorsfield, select the/Common/httpoption.
- ForNode Type, selectNew Node.
- Add aNode Name(optional).
- Add the DCD IP address in theAddressfield and enter a service port for thePortfield.Service module port numbers:Service ModuleModule in BIG-IPPort NumberWeb Application SecurityASM and Adv. WAF8514Bot Defense8514Fraud Protection ServiceFPS8008DoS ProtectionDoS Protection8020Network SecurityAFM8018Access (APM), IP Security (IPSec)APM and IP Security9997If you have multiple service modules sending logging data, you will need to create a separate DCD pool for each module's port number.
- Ensure thatState (on BIG-IQ)isEnabled.
- ClickSave & Close
- Repeat steps 6-10 for all DCDs in your configuration.
- Create a virtual server to host your DCD pool
- Go to.
- From theNamefield add a name.
- From theDevicefield, select the host device from step 4.
- In theDestination Address/Maskfield add the IP address of the virtual server that hosts the logging profile.If your managed BIG-IP device uses high speed traffic logging (HSL) pools, you must apply the Self-IP address. You cannot apply the Management IP address. For more information about using HSL pools on BIG-IP devices, see K17398.For Web Application Security, you must configure the DCD pool on a separate BIG-IP device from the device used to host the logging profile.
- In theService Portfield enter the service port numbers that matches your logging profile's module (see table in step 8).
- In theSource Address Translationfield selectAuto Map.
- In the Resources area, clickDefault Pooland select the name of the DCD pool.
- ClickSave & Close.
The load balancing configuration for your DCD pool is complete, you now need to ensure that the log messages from the virtual server that hosts your service module policy is directed to your newly configured virtual server.
Configure your remote logging protocol to the newly created virtual server that hosts the DCD pool. This process generally includes creation of a log publisher, log destination, and a logging profile, but varies depending on the service module. Ensure that you follow the procedures that match the logging profile's module data. For more information about your module's work flow, and its configuration documentation see
Overview of configuring HA DCD logging per service module.