F5 Logo MyF5
Search
  1. MyF5 Home
  2. Setting up and Configuring a BIG-IQ Centralized Management Solution
  3. Connect BIG-IP data logs to a Data Collection Device Cluster
Manual Chapter : Connect BIG-IP data logs to a Data Collection Device Cluster

Applies To:

Show Versions Show Versions

BIG-IQ Centralized Management

  • 8.2.0, 8.1.0
Manual Chapter

Connect BIG-IP data logs to a Data Collection Device Cluster

Data logging over multiple DCDs

Data logging over multiple data collection devices (DCDs) streamlines the data collection process across your BIG-IQ system. This is done by load balancing log messages, sent from BIG-IP devices, among the multiple DCDs in your configuration. Completing this configuration ensures that your logging profiles are configured to have high availability, in the event that a DCD is unavailable, while preventing unnecessary duplication of data.

Load Balancing events over DCDs

To ensure high availability among your DCDs, for the purpose of logging events from your BIG-IP devices, you need to configure a pool of DCDs. For more information about configuring the DCD pool, see Create a remote logging pool of DCDs.
Each service module requires its own DCD pool. This ensures that the pool is configured with the proper port listeners for each module's log messages. For modules that share the same port number, such as ASM and Bot Defense, you can use the same DCD pool.

Connecting event logs to DCD pool

Once you have created your DCD pool(s), you can configure remote logging to your BIG-IQ DCDs. Depending on your service module, this process may vary. For more information about each modules' log configuration requirements, see Overview of configuring HA DCD logging per service module.

Overview of configuring HA DCD logging per service module

Using BIG-IQ, you can view and manage statistics and event data generated by the services on your BIG-IP devices. You collect BIG-IP logging data on a Data Collection Device (DCD) cluster. If your system configuration includes multiple DCDs it is recommended to create a pool that optimizes and streamlines data collection from your managed BIG-IP devices. Before you create and configure your logging profiles, you must complete this procedure. To create a pool of DCDs, see
Create a remote logging pool of DCDs
.
If you only have one DCD in your configuration, you do not need to configure a remote logging pool, and you can connect your logging profile directly to your DCD. This option does not provide high availability and log messages will not be collected in the event that your DCD is not available.
The work flow to configure data to route from the BIG-IP devices to your data collection device (DCD) cluster depends on the type of data you want to collect and the services that run on your BIG-IP devices. The table below outlines the procedures required for each service module. For end to end configuration of high availability logging, including configuration of a logging profile using BIG-IQ, go to the documentation referred to in
Logging profile configuration
.
If you have multiple modules, you need to configure one remote logging DCD pool per service port.
To configure
Procedure overview
Logging profile configuration
Web Application Security (ASM or Adv. WAF)
  1. Create a remote logging pool of DCDs for service port
    8514
    .
  2. Create a logging profile that is configured to the remote logging pool of DCDs.
    Web Application Security does not require a Log Publisher. As such, it requires two separate BIG-IP host devices: One to distribute log messages over the DCD pool, and another to host the logging profile.
Managing Web Application Security Logging
 section in
BIG-IQ Web Application Security
Bot Defense
  1. Create a remote logging pool of DCDs for service port
    8514
  2. Create Log Destination for your DCD pool.
  3. Create a high-speed remote Log Publisher.
  4. Create a logging profile for Bot Defense and associate it with the Log Publisher. See
    Configure logging for Bot Defense Requests
    .
Logging Bot Defense Requests
 section in
Managing Bot Defense Using BIG-IQ
Fraud Protection Services (FPS)
The following process is for BIG-IP devices running version 15.0 or higher. For information about setting up logging to BIG-IQ for versions 14.1 or earlier, see
Configure BIG-IP a remote logging profile for BIG-IP FPS
.
This process can only be done using the BIG-IP interface (version 15.0 or higher). You cannot create an FPS logging profile using BIG-IQ.
  1. Create a remote logging pool of DCDs for service port
    8008
  2. Create Log Destination for your DCD pool.
  3. Create a high-speed remote Log Publisher.
  4. (optional) Create an Anti-Fraud logging profile on your host BIG-IP device. See
    Configure BIG-IP a remote Log Profile for BIG-IP FPS
     for version support information.
  5. On your BIG-IP device, associate the remote DCD pool and Log Publisher with the Anti-Fraud profile alerts.
Logging FPS over multiple DCDs
 section in
BIG-IQ: Fraud Protection Service
.
Dos Protection
  1. Create a remote logging pool of DCDs for service port
    8020
  2. Create Log Destination for your DCD pool.
  3. Use the automated tool to create a logging profile directly on the virtual server. See
    Configure logging for DoS Protection and Network Security
    .
  4. Add the DCD pool Log Destination to the automatically generated remote Log Publisher.
Logging DoS Protection events
 section in
Managing DDoS attacks Using BIG-IQ
Network Security (AFM)
  1. Create a remote logging pool of DCDs for service port
    8018
  2. Create Log Destination for your DCD pool.
  3. Use the automated tool to create a logging profile directly on the virtual server.
  4. Add the DCD pool Log Destination to the automatically generated remote Log Publisher.
Logging Network Security events
 section in
Managing Network Security Using BIG-IQ
Access (APM and IP Security)
  1. Create a remote logging pool of DCDs for service port
    9997
  2. Create Log Destination for your DCD pool.
  3. Use the automated tool to create a logging profile. See
    Configure logging for Access Policy Manager
    .
  4. Add the DCD pool log destination to the automatically generated remote Log Publisher.
Logging Access events
 section in
Centrally Managing Access Groups Using BIG-IQ

Create a remote logging pool of DCDs

For this process you will need the following:
  • Three or more data collection devices (recommended).
    This process only applies to BIG-IQ configurations with multiple DCD devices.
  • A logging profile that has remote storage enabled, and is attached to a virtual server on a BIG-IP device that hosts the service module and its policy.
  • [ASM/Web Application Security only] A load balancing BIG-IP device. This is a device that hosts the virtual server that load balances logging messages to the pool of DCDs, but does NOT host ASM service module policies or logging profiles.
To optimize data logging of messages from your BIG-IP devices to multiple DCDs, you can configure a BIG-IP system to load balance these messages among the DCDs in your BIG-IQ configuration. This process prevents duplication of information in the consolidated data repository, while also providing high availability for your log messages in the case that one or more DCDs become unavailable.
  1. At the top of the screen, click
    Configuration
    , then, on the left, click
    LOCAL TRAFFIC
    Pools
    .
  2. Click
    Create
  3. Type a unique
    Name
     for the pool.
  4. From the
    Device
     list, select a load balancing BIG-IP device that provides the load balancing service to the DCD pool.
    For ASM/ Web Application security logging information, be sure to select a BIG-IP device that is different from the device that hosts your virtual server with the service module policy.
  5. In the
    Health Monitors
     field, select the
    /Common/http
     option.
  6. For
    Node Type
    , select
    New Node
    .
  7. Add a
    Node Name
     (optional).
  8. Add the DCD IP address in the
    Address
     field and enter a service port for the
    Port
     field.
    Service module port numbers:
    Service Module
    Module in BIG-IP
    Port Number
    Web Application Security
    ASM and Adv. WAF
    8514
    Bot Defense
    8514
    Fraud Protection Service
    FPS
    8008
    DoS Protection
    DoS Protection
    8020
    Network Security
    AFM
    8018
    Access (APM), IP Security (IPSec)
    APM and IP Security
    9997
    If you have multiple service modules sending logging data, you will need to create a separate DCD pool for each module's port number.
  9. Ensure that
    State (on BIG-IQ)
     is
    Enabled
    .
  10. Click
    Save & Close
  11. Repeat steps 6-10 for all DCDs in your configuration.
  12. Create a virtual server to host your DCD pool
    1. Go to
      Configuration
      LOCAL TRAFFIC
      Virtual Servers
      .
    2. Click
      Create
      .
    3. From the
      Name
       field add a name.
    4. From the
      Device
       field, select the host device from step 4.
    5. In the
      Destination Address/Mask
       field add the IP address of the virtual server that hosts the logging profile.
      If your managed BIG-IP device uses high speed traffic logging (HSL) pools, you must apply the Self-IP address. You cannot apply the Management IP address. For more information about using HSL pools on BIG-IP devices, see K17398.
      For Web Application Security, you must configure the DCD pool on a separate BIG-IP device from the device used to host the logging profile.
    6. In the
      Service Port
       field enter the service port numbers that matches your logging profile's module (see table in step 8).
    7. In the
      Source Address Translation
       field select
      Auto Map
      .
    8. In the Resources area, click
      Default Pool
       and select the name of the DCD pool.
    9. Click
      Save & Close
      .
The load balancing configuration for your DCD pool is complete, you now need to ensure that the log messages from the virtual server that hosts your service module policy is directed to your newly configured virtual server.
Configure your remote logging protocol to the newly created virtual server that hosts the DCD pool. This process generally includes creation of a log publisher, log destination, and a logging profile, but varies depending on the service module. Ensure that you follow the procedures that match the logging profile's module data. For more information about your module's work flow, and its configuration documentation see
Overview of configuring HA DCD logging per service module
.
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information