F5 Logo MyF5
Search
  1. MyF5 Home
  2. Planning and Implementing a BIG-IQ Deployment
  3. Deploying a Data Collection Device
Manual Chapter : Deploying a Data Collection Device

Applies To:

Show Versions Show Versions

BIG-IQ Centralized Management

  • 7.0.0
Manual Chapter

Deploying a Data Collection Device

How do I deploy a data collection device cluster?

To manage the data generated by BIG-IP devices on BIG-IQ Centralized Management, you deploy a network of devices called a
data collection device (DCD) cluster
, and then configure that cluster to meet your business needs.
To deploy a DCD cluster, you should:
  • Prepare your network environment and architecture (refer to
    Planning a BIG-IQ Centralized Management Deployment
     in
    Planning a BIG-IQ Centralized Management Deployment
     on
    support.f5.com
     for details).
  • Install and configure the platform you plan to use to run the BIG-IQ system. The platform can either be a physical device or a virtual device. To use a physical device, you need a BIG-IQ 7000 series device. To use a virtual device, the solution you choose depends on the environment you choose. Supported platforms for this release are listed below. Use the guide appropriate for the platform you use to complete the installation. All of these guides are posted on
    support.f5.com
    .
    If you choose this platform:
    Refer to this guide for installation details:
    BIG-IQ 7000 Series
    Platform Guide: BIG-IQ 7000 Series
    Amazon Web Services
    F5 BIG-IQ Centralized Management 6.0.0 and Amazon Web Services: Setup
    Citrix XenServer:
    F5 BIG-IQ Centralized Management 6.0.0 and Citrix XenServer: Setup
    KVM
    F5 BIG-IQ Centralized Management 6.0.0 and Linux KVM: Setup
    Microsoft Azure
    F5 BIG-IQ Centralized Management 6.0.0 and Microsoft Azure: Setup
    Microsoft Hyper-V
    F5 BIG-IQ Centralized Management 6.0.0 and Microsoft Hyper-V: Setup
    VMware NSX-V
    F5 BIG-IQ Centralized Management 6.0.0 and VMware ESXi: Setup
    Xen Project
    F5 BIG-IQ Centralized Management 6.0.0 and Linux Xen Project: Setup
  • Install, configure, discover and activate the DCDs that manage your BIG-IP device data.
  • Define an external location to store snapshots.
  • Enable data collection for the DCD cluster.
  • Configure a BIG-IP system to send alerts or events to the cluster (if needed).
  • If you want an HA configuration:
    • Install and configure a BIG-IQ peer.
    • If you want your HA configuration to failover automatically, install and configure a quorum device.

Licensing and setting up a data collection device

The BIG-IQ data collection device runs as a virtual machine in supported hypervisors, or on the BIG-IQ 7000 series platform. You license the data collection device (DCD) using the base registration key. The
base registration key
 is a character string that the F5 license server uses to provide access to data collection device features.
You must use the correct license type when you license a DCD. You must use a license that uses the SKU: F5-BIQ-VE-LOG-NODE.
You license data collection device in one of the following ways:
  • If the system has access to the internet, you can have the data collection device contact the F5 license server and automatically activate the license.
  • If the system is not connected to the internet, you can manually retrieve the activation key from a system that is connected to the internet, and transfer it to the data collection device.
  • If your data collection device is in a closed-circuit network (CCN) that does not allow you to export any encrypted information, you must open a case with F5 support.
When you license the data collection device, you:
  • Specify a host name for the system.
  • Assign a management port IP address.
  • Specify the IP address of your DNS server and the name of the DNS search domain.
  • Specify the IP address of your Network Time Protocol (NTP) servers and select a time zone.
  • Change the administrator’s default admin and root passwords.

Automatic license and initial setup for a DCD

You must have a base registration key before you can license the BIG-IQ system. If you do not have a base registration key, contact the F5 Networks sales group (
f5.com
). After you set up your BIG-IQ VE or set up your BIG-IQ 7000 Series, you can install the BIG-IQ software license.
If the data collection device system is connected to the public internet, you can follow these steps to automatically perform the license activation and perform the initial setup.
  1. Use a browser to log in to BIG-IQ by typing
    https://
    <management_IP_address>
    , where
    <management_IP_address>
     is the address you specified for device management.
  2. In
    Base Registration Key
    , type or paste the BIG-IQ registration key.
    If you are setting up a data collection device, you have to use a registration key that supports a data collection device license.
  3. In
    Add-On Keys
    , paste any additional license key you have.
  4. For
    Activation Method
    , select
    Automatic
    , click the
    Activate
    button, and then click the
    Next
     button.
    If you are setting up this device for the first time, the Accept User Legal Agreement screen opens.
  5. To accept the license agreement, click the
    Agree
     button, and then click the
    Next
     button.
  6. Type a
    Passphrase
     that satisfies the requirements specified on screen, and then type the same phrase for
    Confirm Passphrase
    .
    The DCD uses the pass phrase to generate a Master Key. This pass phrase must be the same on all of the devices in the DCD cluster. Make sure you keep track of the pass phrase, because it cannot be recovered if you lose it.
    • Make sure you keep track of the pass phrase, because it cannot be recovered if you lose it. To protect the security of this device, you must have the passphrase used to generate the master key before you can change the master key.
    • If this BIG-IQ is not part of an HA or DCD configuration, you can change the Master Key any time from the
      System
      THIS DEVICE
      General Properties
       screen.
    • To add a BIG-IQ to an HA or DCD configuration, its master key must match the key for the other devices in the HA or DCD configuration. So if the passphrase is different and you do not know what it is, the only way to add that BIG-IQ to a cluster is to reset it to its factory defaults; However, that reset destroys any data on that BIG-IQ.
    • Finally, when you backup and restore a BIG-IQ, the master key is backed up with the rest of the data, and you cannot restore that data onto a BIG-IQ that has a different master key, so without that key you will be unable to have this BIG-IQ and it's data in an HA or DCD configuration.
    If you are setting up a Microsoft Azure VE, and you type an entry in any of the fields, you will not be able to continue successfully. The only way to proceed is to leave all of the fields empty and click the
    Next
     button at the bottom of the screen. This allows the system to use the first-time access credentials you specified previously.
  7. In the
    Old Password
     fields, type the default admin and root passwords, and then type a new password in the
    Password
     field and click the
    Next
     button at the bottom of the screen.
    If your license supports both BIG-IQ Data Collection Device and BIG-IQ Central Management Console, the System Personality screen displays. Otherwise the Management Address screen opens.
  8. If you are prompted with the System Personality screen, select the option you're licensed for, and then click OK. If you are not prompted, proceed to the next step.
    You cannot undo this choice. Once you license a device as a BIG-IQ Management Console, you can't change your mind and license it as a Data Collection Device.
    The Management Address screen opens.
  9. In
    Hostname
    , type a fully-qualified domain name (FQDN) for the system.
    The FQDN can consist of letters and numbers, as well as the characters underscore ( _ ), dash ( - ), or period ( . ).
  10. Type the
    Management Port IP Address
     and
    Management Port Route
    .
    The management port IP address must be in Classless Inter-Domain Routing (CIDR) format. For example:
    10.10.10.10/24
    .
  11. Specify what you want the DCD to use for the
    Discovery Address
    .
    The DCD advertises this address to other devices that want to communicate with it. DCD nodes communicate using their respective discovery addresses.
    • To use the management IP address, select
      Use Management Address
      .
    • To use the internal self IP address, select
      Self IP Address
      , and type the IP address.
      F5 strongly recommends using the internal self IP address as the Discovery Address for a DCD.
      The self IP address must be in Classless Inter-Domain Routing (CIDR) format. For example:
      10.10.10.10/24
      .
  12. Click the
    Next
     button at the bottom of the screen.
  13. In the
    DNS Lookup Servers
     field, type the IP address of your DNS server.
    You can click the
    Test Connection
     button to verify that BIG-IQ can reach that IP address.
  14. In the
    DNS Search Domains
     field, type the name of your search domain.
    The DNS search domain list allows the BIG-IQ system to search for local domain lookups to resolve local host names.
  15. In the
    Time Servers
     field, type the IP addresses of your Network Time Protocol (NTP) server.
    You can click the
    Test Connection
     button to verify that BIG-IQ can reach the IP address.
  16. From the
    Time Zone
     list, select your local time zone.
  17. Click the
    Next
     button at the bottom of the screen.
  18. If the details are as you intended, click
    Launch
     to continue; if you want to make corrections, use the
    Previous
     button to navigate back to the screen you want to change.

Manual license and initial setup for a DCD

You must have a base registration key before you can license the BIG-IQ system. If you do not have a base registration key, contact the F5 Networks sales group (
f5.com
). After you set up your BIG-IQ VE or set up your BIG-IQ 7000 Series, you can install the BIG-IQ software license.
If the BIG-IQ system is not connected to the public internet, you can follow these steps to contact the F5 license web portal then perform the initial setup.
  1. Use a browser to log in to BIG-IQ by typing
    https://
    <management_IP_address>
    , where
    <management_IP_address>
     is the address you specified for device management.
  2. In
    Base Registration Key
    , type or paste the BIG-IQ registration key.
    If you are setting up a data collection device, you have to use a registration key that supports a data collection device license.
  3. In
    Add-On Keys
    , paste any additional license key you have.
  4. For
    Activation Method
    , select
    Manual
     and click the
    Get Dossier
     button.
    The BIG-IQ system refreshes and displays the dossier in the
    Device Dossier
     field.
  5. Select and copy the text displayed in
    Device Dossier
    .
  6. Click the
    Access F5 manual activation web portal
     link.
    The Activate F5 Product site opens.
  7. Into the
    Enter your dossier
     field, paste the dossier.
    Alternatively, if you saved the file, click the
    Choose File
     button and navigate to it.
  8. Click
    Next
    .
    • If you are setting up this device for the first time, the Accept User Legal Agreement screen opens. To accept the license agreement, select
      I have read and agree to the terms of this license
      , and click
      Next
      . The licensing server creates the license key text.
    • If you have set up this device before, the licensing server goes right to generating the license text.
  9. Copy the license key.
  10. In the
    License Text
     field on BIG-IQ, paste the license text.
  11. Click the
    Activate
     button.
  12. Click the
    Next
     button at the bottom of the screen.
  13. Type a
    Passphrase
     that satisfies the requirements specified on screen, and then type the same phrase for
    Confirm Passphrase
    .
    The DCD uses the pass phrase to generate a Master Key. This pass phrase must be the same on all of the devices in the DCD cluster. Make sure you keep track of the pass phrase, because it cannot be recovered if you lose it.
    • Make sure you keep track of the pass phrase, because it cannot be recovered if you lose it. To protect the security of this device, you must have the passphrase used to generate the master key before you can change the master key.
    • If this BIG-IQ is not part of an HA or DCD configuration, you can change the Master Key any time from the
      System
      THIS DEVICE
      General Properties
       screen.
    • To add a BIG-IQ to an HA or DCD configuration, its master key must match the key for the other devices in the HA or DCD configuration. So if the passphrase is different and you do not know what it is, the only way to add that BIG-IQ to a cluster is to reset it to its factory defaults; However, that reset destroys any data on that BIG-IQ.
    • Finally, when you backup and restore a BIG-IQ, the master key is backed up with the rest of the data, and you cannot restore that data onto a BIG-IQ that has a different master key, so without that key you will be unable to have this BIG-IQ and it's data in an HA or DCD configuration.
    If you are setting up a Microsoft Azure VE, and you type an entry in any of the fields, you will not be able to continue successfully. The only way to proceed is to leave all of the fields empty and click the
    Next
     button at the bottom of the screen. This allows the system to use the first-time access credentials you specified previously.
  14. In the
    Old Password
     fields, type the default admin and root passwords, and then type a new password in the
    Password
     field and click the
    Next
     button at the bottom of the screen.
    If your license supports both BIG-IQ Data Collection Device and BIG-IQ Central Management Console, the System Personality screen displays. Otherwise the Management Address screen opens.
  15. If you are prompted with the System Personality screen, select the option you're licensed for, and then click OK. If you are not prompted, proceed to the next step.
    You cannot undo this choice. Once you license a device as a BIG-IQ Management Console, you can't change your mind and license it as a Data Collection Device.
    The Management Address screen opens.
  16. Select the System Personality option you're licensed for, and then click the
    Next
     button.
  17. In
    Hostname
    , type a fully-qualified domain name (FQDN) for the system.
    The FQDN can consist of letters and numbers, as well as the characters underscore ( _ ), dash ( - ), or period ( . ).
  18. Type the
    Management Port IP Address
     and
    Management Port Route
    .
    The management port IP address must be in Classless Inter-Domain Routing (CIDR) format. For example:
    10.10.10.10/24
    .
  19. Specify what you want the DCD to use for the
    Discovery Address
    .
    The DCD advertises this address to other devices that want to communicate with it. DCD nodes communicate using their respective discovery addresses.
    • To use the management IP address, select
      Use Management Address
      .
    • To use the internal self IP address, select
      Self IP Address
      , and type the IP address.
      F5 strongly recommends using the internal self IP address as the Discovery Address for a DCD.
      The self IP address must be in Classless Inter-Domain Routing (CIDR) format. For example:
      10.10.10.10/24
      .
  20. Click the
    Next
     button at the bottom of the screen.
  21. In the
    DNS Lookup Servers
     field, type the IP address of your DNS server.
    You can click the
    Test Connection
     button to verify that BIG-IQ can reach that IP address.
  22. In the
    DNS Search Domains
     field, type the name of your search domain.
    The DNS search domain list allows the BIG-IQ system to search for local domain lookups to resolve local host names.
  23. In the
    Time Servers
     field, type the IP addresses of your Network Time Protocol (NTP) server.
    You can click the
    Test Connection
     button to verify that BIG-IQ can reach the IP address.
  24. From the
    Time Zone
     list, select your local time zone.
  25. Click the
    Next
     button at the bottom of the screen.
  26. If the details are as you intended, click
    Launch
     to continue; if you want to make corrections, use the
    Previous
     button to navigate back to the screen you want to change.

Add BIG-IQ SSL certificates to the data collection device cluster

Before you add a data collection device (DCD) for a BIG-IQ you've enabled SSL certificate verification for, you need to add the SSL certificate for the DCD to the BIG-IQ so you can validate the end-user host. This is required for each DCD in the cluster. BIG-IQ validates the SSL certificate presented by the communicating host either against a list of certificates you provide (for example, self-signed certificates), or internal or public certificate authority certificates.
SSL certificate verification is disabled by default. If you haven’t enabled SSL verification, you do not need to complete this task for your DCD cluster.
  1. Save the BIG-IQ SSL public key certificates on your local system.
  2. At the top of the screen, click
    System
    .
  3. On the left, click
    SSL CERTIFICATION VERIFICATION
    .
  4. Click
    Import
    .
  5. From the
    Import Type
     list, select
    Certificate
    .
  6. Type a
    Name
     for this BIG-IQ certificate.
    BIG-IQ stores and identifies this certificate by the name you specify here. Therefore, if the certificate you are importing is currently named
    mycertificate.crt
    , but you when you import it you name it
    f5.crt
    , BIG-IQ renames the certificate as you specified, to
    f5.crt
    .
  7. Click
    Upload File
     and navigate to the certificate.
You can now discover the DCD to add it to the BIG-IQ DCD cluster.

Discover and activate a data collection device

If you configured SSL certificate verification for the BIG-IQ by enabling the
Verify Hosts
 setting from the
System
SSL CERTIFICATE VERIFICATION
 screen, you must add the SSL certificates to any data collection device (DCD) that you want to discover.
Using BIG-IQ Centralized Management, you can discover a DCD and add it to the BIG-IQ Data Collection Cluster, so that the BIG-IQ system can access its data. You can then use that DCD to receive and process event logs, alerts, and statistics data from multiple BIG-IP systems. This unified view makes browsing easier, and provides a complete view of application alert or event activity and statistics data.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
    BIG-IQ Data Collection Devices
    .
    The BIG-IQ Data Collection Devices screen opens listing the data collection devices in the cluster. The Services column lists the BIG-IP services monitored by each DCD. If no services are enabled for a DCD, this column displays
    Add Services
     instead.
  2. Click
    Add
    .
  3. On the New BIG-IQ Data Collection Device screen, specify the details for this DCD:
    1. In
      Discovery/Listener Address
      , type one of the self IP addresses for this DCD.
      The BIG-IQ system uses this address to discover the DCD. The DCD uses this address to listen for alerts from your managed devices.
    2. In
      Username
      , type the user name for an administrator on the data collection device (for example,
      admin
      ).
    3. In
      Password
      , type the password for an administrator on the data collection device (for example,
      admin
      ).
    4. In
      Data Collection IP Address
      , type one of the self IP addresses for this DCD.
      The DCD uses this address to exchange data and replicas with other DCDs in the cluster.
      The DCD and BIG-IQ should both use the same VLAN.
    5. Note the
      Data Collection Port
       value (
      9300
      ). This field displays the number of the port that DCDs in your cluster use for internal polling and communication with each other.
      You cannot change the port, but knowing the port number may be useful in resolving DCD communications issues.
    6. For
      Zone
      , either select the disaster recovery zone in which you want this DCD to reside, or use the default setting.
      • If your organization does not use disaster recovery zones, use
        default
        .
      • If disaster recovery zones have been created, select the zone for this device and click
        Update
        .
      • If you want to create a disaster recovery zone:
        • Select
          Create New
          . A new text box opens.
        • Type the name for the new zone in text box, and click
          Update
          .
      You set up the zones so that the BIG-IQ devices and DCDs in your cluster are distributed equitably for disaster recovery purposes.
      When you change the setting for the
      Zone
      , the DCD cluster restarts. Data collection is interrupted until the service resumes.
    7. Click the
      Add
       button at the bottom of the screen to add the data collection device to the system.
      This operation might take a minute or two.
  4. Repeat the preceding steps for each data collection device you want to configure.
  5. To activate the services you want to monitor on each DCD, on the BIG-IQ Data Collection Devices screen, in the Services column, click
    Add Services
    .
    The Services screen for the data collection device opens.
  6. For the service you want to add, confirm that the
    Listener Address
     specifies the correct self IP address on the data collection device, and then click
    Activate
    .
    For Web Application Security, you can resolve insecure connection issues between devices and the Centralized Policy Builder. To establish a secure connection, click
    Enable
     under the Secure Policy Builder field.
    When the service is successfully added, the
    Service Status
     changes to
    Active
    .
  7. Click
    Save & Close
    .
After it has been discovered and activated, this data collection device collects the data generated by the configured BIG-IP systems. Thus, BIG-IQ provides a single view of all alert or event entries and statistics data.
The
Total Document Count
 is not a report of the number of alerts or events sent to the data collection device. Instead, it is a sum of various document types sent to the data collection device. Events and alerts are included in this list, but this total includes other document types as well.

Decide whether to configure log indices

The Indices settings specify the physical characteristics of how the data collection device manages your data. The DCD stores data coming in fromBIG-IP devices in a data index. As data is received, it accumulates in the current index. When the accumulated data reaches the rotation threshold that you set, four things happen.
  • A new current index is created.
  • BIG-IP data begins accumulating in the new index.
  • The former current index becomes one of the retained indices.
  • If the total number of indexes is now larger than the retained index count, the oldest one is dropped.
When you set up index rotation, you determine what triggers the rotation threshold.
The ideal configuration for log indices depends on the flow of data your devices send to the DCD. The default settings are designed to satisfy most user scenarios, but you might want to explore the settings for the data types that you plan to send to the DCD, to make sure that those settings meet your needs.

Modify alert log indices for Access

Before you can configure the indices for a data collection device, you must activate services for the components that you want to collect data for.
The
Indices
 settings specify the physical characteristics of how the data collection device manages your data. The DCD stores data coming in from BIG-IP devices in a data index. As data is received, it accumulates in the current index. When the accumulated data reaches the rotation threshold you set, four things happen.
  • A new current index is created.
  • BIG-IP data begins accumulating in the new index.
  • The former current index becomes one of the retained indices.
  • If the total number of indexes is now larger than the retained index count, the oldest one is dropped.
When you set up index rotation, you determine what triggers the rotation threshold.
The ideal log indices configuration depends on the flow of data your devices send to the DCD. Use the rotation type that best suits your business needs.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    Logging Data Collection
    .
    The Logging Data Collection Settings screen opens.
  3. For Access Policy (APM), click the
    Configure
     button.
    The Access Indices screen opens.
  4. Perform the next two steps for each section on this screen.
    To avoid a mismatch in the reports generated from your logging data, use the same indices values for the
    access-event-logs
     and
    access stats
    .
  5. Specify the
    Rotation Type
    .
    • To chunk your data based on the amount of data:
      1. Select
        Size Based
      2. For the
        Max Index Size
        , type the size of the indexes you want to create.
      For example, if you type
      1000
      , when the index size reaches 1 GB, it becomes a retained index and new data from your BIG-IP begins accumulating in a new current index. If your
      Retained Index Count
       is set to 10, then the maximum disk space used by these indexes will be approximately 10 GB.
    • To chunk your data based on the increments of time:
      1. Select Time Based
      2. For the
        Rotation Period
        , specify a time unit, and type how many of those units you want to comprise indexes you want to create.
      For example, if you type
      .5
       and select
      Hours
      , a new index is created every half hour. If your
      Retained Index Count
       is set to 10, then each retained index will contain approximately 5 hours of data.
  6. For the
    Retained Index Count
    , type the total number of indices you want to store on the DCD.
    This setting determines the maximum amount of data stored on the DCD. When this limit is reached, the oldest data is truncated or discarded. For example, if you set the number of indices to 10 and each index is 1 GB, then you must have 10 GB of storage available on your DCD.
  7. Click
    Save & Close
     to save the indices configuration settings.

Modifying event log indices for FPS

Before you can configure the indices for a data collection device, you must activate services for the components that you want to collect data for.
The
Indices
 settings specify the physical characteristics of how the data collection device manages your data. The DCD stores data coming in from BIG-IP devices in a data index. As data is received, it accumulates in the current index. When the accumulated data reaches the rotation threshold you set, four things happen.
  • A new current index is created.
  • BIG-IP data begins accumulating in the new index.
  • The former current index becomes one of the retained indices.
  • If the total number of indexes is now larger than the retained index count, the oldest one is dropped.
When you set up index rotation, you determine what triggers the rotation threshold.
The ideal log indices configuration depends on the flow of data your devices send to the DCD. Use the rotation type that best suits your business needs.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    Logging Data Collection
    .
    The Logging Data Collection Settings screen opens.
  3. For Fraud Protection (FPS), click the
    Configure
     button.
    The FPS Indices screen opens.
  4. Specify the
    Rotation Type
    .
    • To chunk your data based on the amount of data:
      1. Select
        Size Based
      2. For the
        Max Index Size
        , type the size of the indexes you want to create.
      For example, if you type
      1000
      , when the index size reaches 1 GB, it becomes a retained index and new data from your BIG-IP begins accumulating in a new current index. If your
      Retained Index Count
       is set to 10, then the maximum disk space used by these indexes will be approximately 10 GB.
    • To chunk your data based on the increments of time:
      1. Select Time Based
      2. For the
        Rotation Period
        , specify a time unit, and type how many of those units you want to comprise indexes you want to create.
      For example, if you type
      .5
       and select
      Hours
      , a new index is created every half hour. If your
      Retained Index Count
       is set to 10, then each retained index will contain approximately 5 hours of data.
  5. For the
    Retained Index Count
    , type the total number of indices you want to store on the DCD.
    This setting determines the maximum amount of data stored on the DCD. When this limit is reached, the oldest data is truncated or discarded. For example, if you set the number of indices to 10 and each index is 1 GB, then you must have 10 GB of storage available on your DCD.
  6. Click
    Save & Close
     to save the indices configuration settings.

Modify alert log indices for Web Application Security

Before you can configure the indices for a data collection device, you must activate the services for the components that you want to collect data for.
The
Indices
 settings specify the physical characteristics of how the data collection device manages your data. The DCD stores data coming in from BIG-IP devices in a data index. As data is received, it accumulates in the current index. When the accumulated data reaches the rotation threshold you set, four things happen.
  • A new current index is created.
  • BIG-IP data begins accumulating in the new index.
  • The former current index becomes one of the retained indices.
  • If the total number of indexes is now larger than the retained index count, the oldest one is dropped.
When you set up index rotation, you determine what triggers the rotation threshold.
The ideal log indices configuration depends on the flow of data your devices send to the DCD. Use the rotation type that best suits your business needs.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    Logging Data Collection
    .
    The Logging Data Collection Settings screen opens.
  3. For Web Application Security (ASM), click the
    Configure
     button.
    The ASM Indices screen opens.
  4. Specify the
    Rotation Type
    .
    • To chunk your data based on the amount of data:
      1. Select
        Size Based
      2. For the
        Max Index Size
        , type the size of the indexes you want to create.
      For example, if you type
      1000
      , when the index size reaches 1 GB, it becomes a retained index and new data from your BIG-IP begins accumulating in a new current index. If your
      Retained Index Count
       is set to 10, then the maximum disk space used by these indexes will be approximately 10 GB.
    • To chunk your data based on the increments of time:
      1. Select Time Based
      2. For the
        Rotation Period
        , specify a time unit, and type how many of those units you want to comprise indexes you want to create.
      For example, if you type
      .5
       and select
      Hours
      , a new index is created every half hour. If your
      Retained Index Count
       is set to 10, then each retained index will contain approximately 5 hours of data.
  5. For the
    Retained Index Count
    , type the total number of indices you want to store on the DCD.
    This setting determines the maximum amount of data stored on the DCD. When this limit is reached, the oldest data is truncated or discarded. For example, if you set the number of indices to 10 and each index is 1 GB, then you must have 10 GB of storage available on your DCD.
  6. Click
    Save & Close
     to save the indices configuration settings.

Modifying alert log indices for IPsec

Before you can configure the indices for a data collection device, you must activate services for the components that you want to collect data for.
The
Indices
 settings specify the physical characteristics of how the data collection device manages your data. The DCD stores data coming in from BIG-IP devices in a data index. As data is received, it accumulates in the current index. When the accumulated data reaches the rotation threshold you set, four things happen.
  • A new current index is created.
  • BIG-IP data begins accumulating in the new index.
  • The former current index becomes one of the retained indices.
  • If the total number of indexes is now larger than the retained index count, the oldest one is dropped.
When you set up index rotation, you determine what triggers the rotation threshold.
The ideal log indices configuration depends on the flow of data your devices send to the DCD. Use the rotation type that best suits your business needs.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    Logging Data Collection
    .
    The Logging Data Collection Settings screen opens.
  3. For Fraud Protection (FPS), click the
    Configure
     button.
    The FPS Indices screen opens.
  4. Specify the
    Rotation Type
    .
    • To chunk your data based on the amount of data:
      1. Select
        Size Based
      2. For the
        Max Index Size
        , type the size of the indexes you want to create.
      For example, if you type
      1000
      , when the index size reaches 1 GB, it becomes a retained index and new data from your BIG-IP begins accumulating in a new current index. If your
      Retained Index Count
       is set to 10, then the maximum disk space used by these indexes will be approximately 10 GB.
    • To chunk your data based on the increments of time:
      1. Select Time Based
      2. For the
        Rotation Period
        , specify a time unit, and type how many of those units you want to comprise indexes you want to create.
      For example, if you type
      .5
       and select
      Hours
      , a new index is created every half hour. If your
      Retained Index Count
       is set to 10, then each retained index will contain approximately 5 hours of data.
  5. For the
    Retained Index Count
    , type the total number of indices you want to store on the DCD.
    This setting determines the maximum amount of data stored on the DCD. When this limit is reached, the oldest data is truncated or discarded. For example, if you set the number of indices to 10 and each index is 1 GB, then you must have 10 GB of storage available on your DCD.
  6. Click
    Save & Close
     to save the indices configuration settings.

Statistics retention policy overview

When you choose how much raw data to retain, you need to consider how much disk space you have available. The controls on this screen are simple to set up, but understanding how they work takes a bit of explanation.
The fields on the Statistics Retention Policy screen all work in similar fashion. One way to understand how these fields work is to think of your data storage space as a set of containers. The values you specify on this screen determine how much storage space each container consumes. Because data is saved for the time periods you specify, the longer the time period that you specify, the more space you consume. The disk storage that is consumed depends on several factors.
  • The number of BIG-IP devices you manage
  • The number of objects on the BIG-IP devices you manage (for example, virtual servers, pools, pool members, and iRules)
  • The frequency of statistics collection
  • The data retention policy
  • The data replication policy
There are three key concepts to understand about how the retention policy works.
How long is data in each container retained?
Data is retained in each container for the time period you specify. When the specified level is reached, the oldest chunk of data is deleted. For example, if you specify a raw data value of 48 hours, then when 48 hours of raw data accumulate, the next hour of incoming raw data causes the oldest hour to be deleted.
When does data from one container pass on to the next?
Data passes from one container to the next in increments that are the size of the next (larger) container. That is, every 60 minutes, the last 60 minutes of raw data is aggregated into a data set and passed to the
Hour(s)
container. Every 24 hours, the last 24 hours of hourly data is aggregated into a data set and passed to the
Day(s)
 container, and so on for the
Month(s)
 container.
What about limits?
Limit Max Storage to
 specifies the percentage of total disk space that you want data to consume on the data collection devices in your cluster.
If more disk space is consumed than the percentage you specified, BIG-IQ takes two actions:
  1. New statistical data is not accepted until the available disk space complies with the
    Limit max storage to
     setting.
  2. Statistical data not required to calculate the next higher time layer is removed (for example, you need 60 minutes of raw data to aggregate to the Hours level). Data is removed starting with the raw data container, then the hourly data container, then the daily time container. This process stops when storage consumption is below the
    Limit max storage to
     setting.
The BIG-IQ takes this action to prevent data corruption when storage is completely exhausted.

Manage the retention policy for your statistics data

Before you can set the statistics retention policy, you must have added a data collection device.
You can manage the settings that determine how your statistics data is retained. The highest quality data is the raw data, (data that has not been averaged), but that consumes a lot of disk space, so you need to consider your needs in choosing your data retention settings.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    Statistics Data Collection
    .
    The Statistics Collection Status screen opens.
  3. On the left, click
    Statistics Data Collection
    .
    The Statistics Collection Status screen displays the percentage of available disk space currently consumed by statistics data for each container.
  4. To change the retention settings for your statistics data, click
    Configure
    .
    The Statistics Retention Policy screen opens.
  5. In the
    Keep real-time (raw) data up to
     field, type the number of hours of raw data to retain.
    You must specify a minimum of 1 hour, so that there is sufficient data to average and create a data point for the
    Keep hourly data up to
     container.
  6. In the
    Keep hourly data up to
     field, type the number of hourly data points to retain.
    You must specify a minimum of 24 hours, so that there is sufficient data to average and create a data point for the
    Keep daily data up to
     container.
  7. In the
    Keep daily data up to
     field, type the number of daily data points to retain.
    You must specify a minimum of 31 days, so that there is sufficient data to average and create a data point for the
    Keep monthly data up to
     container.
  8. In the
    Keep monthly data up to
     field, type the number of monthly data points to retain.
    Once the specified number of months passes, the oldest monthly data set is deleted.
  9. In the
    Limit max storage to
     field, type the percentage of disk space that you want collected data to consume before the oldest monthly data set is deleted.
  10. Expand Advanced Settings, and then select the
    Enable Replicas
     check box.
    Replicas
     are copies of a data set that are available to the DCD cluster when one or more devices within that cluster become unavailable. By default, data replication for statistics is not enabled. Disabling replication reduces the amount of disk space required for data retention. However, this provides no protection from data corruption that can occur when you remove a data collection device. You should enable replicas to provide this protection.
  11. When you are satisfied with the values specified for data retention, click
    Save & Close
    .

Configure secure communications for data collection device

You need a signed SSL certificate before you can configure HTTPS communications to a data collection device.
If you want to secure the communications between the BIG-IP devices and your data collection device cluster using SSL encryption, you must provide a signed SSL certificate to the BIG-IP devices and F5 BIG-IQ Centralized Management systems. You do this by configuring both the BIG-IP device and the data collection device.
The BIG-IP device that generates Fraud Protection Service alerts must be configured to send its alerts to the data collection device (DCD). This process is documented in a separate guide. The guide
F5 Fraud Protection Service: Configuration, Version 13.0
 provides complete setup instructions for using FPS on a BIG-IP system. Complete the standard setup as documented in the guide, except when you configure the alert server pool, add your DCDs to an alerts pool using their internal self IP addresses.
  1. Use SSH to log in to the data collection device.
  2. Replace the content of the
    /etc/httpd/conf/ssl.crt/
     directory on the data collection device with your signed SSL certificate.
  3. Replace the content of the
    /etc/httpd/conf/ssl.key/
     directory on the data collection device with your signed SSL key.
  4. To apply these changes to the data collection device, type:
    bigstart restart webd
     and then press Enter.
  5. Log out of the data collection device.

Add a proxy for secure communication

Before you can perform this task, you must be logged in as Admin, and you must have configured a proxy server that your data collection device (DCD) cluster can access.
As a security precaution, you may want to configure a proxy to route communications. For example you might use it to route your forwarded alerts or download alert rules from the security operations center. Or you might want to use a proxy to avoid exposing the DCD when you download ASM signature files.
To use a proxy for Fraud Protection Service, you must configure a proxy on each device (every DCD and both the primary and the secondary BIG-IQ devices) in the cluster. The proxy names you specify for each node in the cluster must match exactly, but the IP address and port number for the proxy can be different from device to device.
  1. At the top of the screen, click
    System
    .
  2. On the left, click
    PROXIES
    .
  3. On the Proxies screen, click
    Add
    .
  4. For
    Name
    , type a name for the proxy you want to use.
    The proxy name must match across all devices in the cluster. The proxy addresses and port can vary.
  5. For
    Address
    , type the IP address of the proxy server.
  6. For
    Port
    , type the port that you want the proxy server to use.
  7. If the proxy server requires authentication, type the
    User Name
     and
    Password
     for the proxy.
  8. To add another proxy, click the plus sign in the upper right hand corner, and then repeat the preceding 4 steps.
  9. Click
    Save & Close
You need to add a proxy for each data collection device in the cluster.
Remember, the proxy name must match across all devices in the cluster. The proxy addresses and port can vary.

DCD snapshot overview

With snapshots of the data sent to your data collection devices (DCDs) you can preserve the logging and analytics information stored on your DCD cluster at a particular moment in time. Snapshots are created based on the snapshot schedules that you define. The primary use is to restore older versions of your data when you upgrade your BIG-IQ cluster. Although each snapshot only contains changes since the last snapshot was created, if your BIG-IP devices are adding data at a high rate, the snapshots can get quite large. Despite their incremental nature, there are no dependencies or relationships between snapshots that are taken at specific times. Unlike typical incremental restorations, there is no need to restore multiple snapshots to recover data.
Because the underlying Elastic Search (ES) cluster already replicates both event logs and analytics data across the entire DCD cluster, you don't need snapshots to prevent loss of data, except in the case of catastrophic failure. If a single DCD fails, the data is still available due to the replication that occurs across all DCDs in the cluster. If you had a major failure (multiple DCDs), you can use a snapshot to restore the entire cluster to a point in time.
You control the snapshot frequency using the Backup Schedule screen. Each organization determines their own recovery point objective (RPO) as part of their disaster recovery plan. Because the snapshot represents a point-in-time copy of the data in the ES cluster, you should align your snapshot schedule with your RPO. Take into account the amount of time or tolerance that your organization has for data loss. For example, if your RPO is 12 hours, then snapshots would be scheduled to occur every 12 hours. If all of the DCDs in your cluster failed 11 hours after the last successful snapshot, then you would lose any data collected after that snapshot. If this is not acceptable, you can schedule more frequent snapshots, but those extra backups require more storage. You can also mitigate that cost to some extent by carefully configuring your data retention settings.
On the External Storage & Snapshots screen, you can restore from the last known good snapshot. However, you cannot restore from previous snapshots. Use the External Storage & Snapshots screen to perform all DCD snapshot management. This ensures that your snapshot data is kept consistent, and that records are updated accordingly.
Before you proceed, keep in mind that there are several different ways to make backups of your data. Each of these methods backs up different things and is documented separately.
  • DCD snapshots (discussed here) back up the alert, event, and analytics data collected by your DCDs.
  • Configuration snapshots back up the settings for configuration objects that reside on your managed BIG-IP devices. Refer to
    Managing Configuration Snapshots
     in the
    BIG-IQ Centralized Management: Device
     article on
    support.f5.com
     for details.
  • To back up the configuration of a BIG-IQ system, you create a compressed user configuration set (UCS). Refer to
    BIG-IQ System File Management
     in the
    Planning and Implementing a BIG-IQ Centralized Management Deployment
     article on
    support.f5.com
     for details.
  • You also use a UCS file to back up the configuration of a managed BIG-IP device. Refer to
    Backup File Management
     in the
    BIG-IQ Centralized Management: Device
     article on
    support.f5.com
     for details.

Define external storage snapshots location

Before you configure the external snapshot storage location, collect the following information for the machine that will store your data collection device (DCD) snapshots:
  • IP address for the storage machine
  • Storage file path
  • User name, password, and (optionally) domain for the user account configured on the external storage device
  • Read/Write permissions for the storage file path
You need snapshots to perform software upgrades and to restore your old data.
Creating external storage so you can create snapshots is an optional task. However, F5 strongly recommends that you create snapshots to safeguard your data.
If you set up external storage for this logging node cluster in 5.1.and plan to retain that setup after you upgrade, continue setting up the external storage location. When you create DCD snapshots, they need to be stored on a machine other than the DCD. You define the location for the snapshot using the BIG-IQ Centralized Management device.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    External Storage & Snapshots
    .
    The External Storage & Snapshots screen opens.
  3. For
    External Storage
    , click
    Configure
    .
    The External Storage popup screen opens.
  4. In the
    User name
     and
    Password
     fields, type the user name and password for the user account configured on the external storage device.
  5. For the
    Domain
    , you can type the domain name for the user account configured on the external storage device.
  6. For the
    Storage Path
    , type the path to the external storage location.
    You can specify the device using the IP address or the host name. Additionally, you need to specify the path to the folder on the external storage device. For example:
    //<storage machine ip-address>/<storage-file-path>
    Remember, the folder you specify must have full read, write, and execute permissions.
  7. To test the settings just specified, click
    Test
    .
    A message displays to tell you whether the test completes successfully. If it does not, correct the settings and permissions.
  8. When the external storage is specified successfully, click
    Save
    .
The storage location is accessible to the all of the devices in the DCD cluster.

Define snapshot schedules

Before you define snapshot schedules, you must have defined the snapshot storage location.
You create snapshots of the data sent to your data collection devices (DCDs) to preserve the logging and analytics information stored on your DCD cluster at a particular moment in time. You control how frequently the BIG-IQ creates snapshots, based on your organization's unique business requirements. For a more detailed discussion of factors to consider when determining your schedule, refer to
DCD snapshot overview
 in the
Planning and Implementing a BIG-IQ Centralized Management Deployment
 article on
support.f5.com
.
You perform this task on the BIG-IQ Centralized Management device; not on the data collection device (DCD).
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    External Storage & Snapshots
    .
    The External Storage & Snapshots screen opens.
  3. To view the list of snapshot schedules for this device, in the External Storage & Snapshots area, for
    Snapshot Schedules
     click the
    View Schedules
     button.
    The BIG-IQ Data Collection Snapshot Schedule screen opens.
  4. To define a new snapshot schedule for this device, click
    Create
    .
    The New BIG-IQ Data Collection Snapshot Schedule screen opens.
  5. For the
    Snapshot Name Prefix
    , type the string that you want to use to identify the snapshots created by this schedule.
    For example,
    snapshot_
    .
  6. In
    Snapshots to Keep
    , specify the number of snapshots that you want to accumulate before they are deleted for space constraints.
    For example, if you specify
    25
    , then the system will retain a maximum of 25 snapshots before it starts to delete older snapshots as new snapshots are created. You can save up to 100.
  7. Define how you want the snapshots scheduled.
    Schedule the interval at which you want to create snapshots:
    You schedule the system to take snapshots indefinitely, at the frequency you specify.
    1. For the Schedule Type setting,select
      Repeat Interval
      .
    2. Specify the
      Snapshot Frequency
      .
    3. Select a time increment.
    For example, if you set the frequency to
    6
     and
    Hours
    , the first DCD snapshot is taken immediately (on
    Save & Close
    ). Subsequent snapshots are taken every 6 hours.
    Schedule specific days on which you want to create snapshots:
    You schedule the system to take snapshots on specific days.
    1. For the Schedule Type setting, select
      Days of the Week
      .
    2. For the
      Days of the Week
       setting, select the days on which you want backups to occur.
    3. For the
      Start Date
      , select the time (date, hour, minute, and AM or PM) on which you want backups to start.
  8. Click
    Save & Close
     to save the new schedule.

Overview of configuring the data collection device to BIG-IP device connection

The workflow to configure data to route from the BIG-IP® devices to your data collection device (DCD) cluster depends on the type of data you want to collect.
  • To collect statistics data, refer to
    Discover and activate a data collection device
    .
  • To collect Access Policy Manager® data, refer to
    Configuring remote logging for Access Policy Manager
    .
  • To collect Fraud Protection Services data, refer to
    Configuring BIG-IP FPS devices to route alerts to a data collection device
    .
  • To collect Web Application Security data, refer to:
    • Configuring the BIG-IP logging profile
    • Virtual servers that remote logging uses to route event logs
    • Assigning the logging profile to a virtual server

Configure remote logging for Access Policy Manager

BIG-IP devices that you configure for remote logging send Access reporting and SWG log report data to the BIG-IQ data collection device for storage and management.
  1. At the top left of the screen, click
    Monitoring
    DASHBOARDS
    Access
    .
  2. Click
    Remote Logging Configuration
    .
    The Remote Logging Configuration screen opens to display all of the discovered BIG-IP devices that are provisioned with the Access service.
  3. Select the BIG-IP devices for which you want to enable remote logging, and then click
    Configure
    .
    The hostname of the primary data collection device is displayed, and the status changes to let you know whether the enable request was successful.

Configuring BIG-IP FPS devices to route alerts to a data collection device

The BIG-IP device that generates Fraud Protection Service alerts must be configured to send its alerts to the data collection device (DCD). This process is documented in a separate guide. The guide
F5 Fraud Protection Service: Configuration, Version 13.0
 provides complete setup instructions for using FPS on a BIG-IP system. Complete the standard setup as documented in the guide, except when you configure the alert server pool, add your DCDs to an alerts pool using their internal self IP addresses.
Although DCDs use their own version of load balancing to level the data stored on each node, it is best practice to configure the BIG-IP pool members with a load balancing method that ensures smooth traffic flow to the DCDs. The load balancing method you configure should:
  • Distribute traffic between the nodes.
  • Ensure that, if a DCD goes offline, the BIG-IP device must still be able send traffic to the available DCDs without dropping alerts.
The default port to specify is
8008
, but you can use a different port if your DCD is configured for it. To ensure that alerts are received even if one DCD goes down, specify at least one alternative DCD.

Configure the BIG-IP logging profile

You configure the BIG-IP system by creating a logging profile and assigning the logging profile to a virtual server, and then deploying it to the BIG-IP system. The
logging profile
 defines the content of the events, and identifies the data collection device to which the events are sent.
For Web Application Security users, or users that want to ensure that a DCD is always available to receive log messages, you can create a load balancing DCD pool. This provides high availability in the case that a DCD becomes inactive. To complete this process, refer to:
Configuring high availability logging for multiple DCDs
.
  1. At the top of the screen, click
    Configuration
    .
  2. On the left, click
    SECURITY
    Shared Security
    Logging Profiles
    .
    The Logging Profiles screen opens to display the logging profiles that have been configured on this device.
  3. On the Logging Profiles screen, click
    Create
    .
    The New Logging Profile screen opens, showing the Properties information.
  4. On the Properties screen, edit as appropriate:
    1. In the
      Name
       field, type a unique name for this new profile. This field is required.
    2. For the
      Description
      , you can specify an optional description for the logging profile.
    3. For the
      Partition
      , you can specify the partition to which the logging profile belongs. Only users with access to a partition can view the objects (such as the logging profile) that it contains. If the logging profile resides in the
      Common
       partition, all users can access it. Although this field is pre-populated with
      Common
       by default, you can set the partition when creating logging profiles by typing a unique name for the partition.
      The partition with the name you specify must already exist on the BIG-IP device. No whitespace is allowed in the partition name.
    4. To specify the devices to which you want to deploy this logging profile, select the devices in the
      Available
       list, and click the right arrow to add them to the
      Selected
       list.
  5. On the left, click
    Application Security
    , and then select the
    Enabled
     check box.
    The screen displays the Application Security settings.
    1. Select the
      Remote Storage
      Enabled
       check box.
      The screen displays additional settings, and the
      Local Storage
       option becomes active.
    2. Clear the
      Local Storage
       check box.
    3. Specify the appropriate
      Logging Format
      .
      • If the BIG-IP device runs version 12.0 or later, select
        BIG-IQ
        .
      • If the BIG-IP device runs a version earlier than 12.0, select
        Comma-Separated Values
        . Several new settings appear.
        • For
          Storage Format
          , select
          User Defined
          .
        • In the
          Selected
           field, paste the following text: 
          
unit_hostname="%unit_hostname%",management_ip_address="%management_ip_address%",
http_class_name="%http_class_name%",web_application_name="%http_class_name%",policy_name="%policy_name%",
policy_apply_date="%policy_apply_date%",violations="%violations%",support_id="%support_id%",
request_status="%request_status%",response_code="%response_code%",ip_client="%ip_client%",
route_domain="%route_domain%",method="%method%",protocol="%protocol%",query_string="%query_string%",
x_forwarded_for_header_value="%x_forwarded_for_header_value%",sig_ids="%sig_ids%",sig_names="%sig_names%",
date_time="%date_time%",severity="%severity%",attack_type="%attack_type%",geo_location="%geo_location%",
ip_address_intelligence="%ip_address_intelligence%",username="%username%",session_id="%session_id%",
src_port="%src_port%",dest_port="%dest_port%",dest_ip="%dest_ip%",sub_violations="%sub_violations%",
virus_name="%virus_name%",uri="%uri%",request="%request%",violation_details="%violation_details%",
header="%headers%",response="%response%
          The line breaks in the example above were necessary due to screen width; remove all of them after you paste this data. It must be a single string with no white space.
    4. For
      Protocol
      , select
      TCP
      .
    5. For the
      Server Addresses
       settings, specify the address you want to use:
      1. In the
        IP Address
         field, type a data collection node's management IP address.
      If you have only one data collection node, or you do not require high availability, add a single IP Address to prevent duplication of data. If you would like to include high availability, you will need to load balance the log messages to your data collection nodes. For more information, see
      Configuring Web Application Security logging on multiple DCDs
      .
      1. Specify the port to use for your data.
        • If you are setting up a logging profile for Web Application Security, type
          8514
           in the
          Port
           field.
        • If you are setting up a logging profile for Fraud Protection Service, type
          8008
           in the
          Port
           field.
      2. Click the
        Add
         button to add the address and port to the list of servers.
    6. For the
      Maximum Entry Length
      , select
      64k
      .
    7. In the Storage Filter area, from the
      Request Type
       list, select
      All requests
      .
  6. If you want to specify Protocol Security options, on the left click
    Protocol Security
    , then select the
    Enabled
     check box: the Protocol Security settings display. Edit as appropriate.
  7. If you want to specify Network Firewall options, on the left click
    Network Firewall
    , then select the
    Enabled
     check box: the Network Firewall settings display. Edit as appropriate.
  8. If you want to specify Network Address Translation options, on the left click
    Network Address Translation
    , then select the
    Enabled
     check box: the Network Address Translation settings display. Edit as appropriate.
  9. If you want to specify DoS Protection options, on the left click
    DoS Protection
    , then select the
    Enabled
     check box: the DoS Protection settings display. Edit as appropriate.
  10. Click
    Save & Close
     to save the new profile.
The new logging profile is added to the list of profiles defined on this device.
Before you can begin using this profile, you must assign it to a virtual server and then deploy the virtual server to the BIG-IP device.

Configuring high availability logging for multiple DCDs

For this process you will need the following:
  • Three or more data collection devices (recommended). If you only have one DCD, you can configure data logging directly to the DCD's IP address during the BIG-IQ configuration process.
  • A BIG-IP device that hosts a virtual server that load balances logging messages to the pool of DCDs.
  • A separate BIG-IP device that hosts a virtual server with an ASM policy and an enabled HTTP logging profile.
To optimize data logging of Web Application Firewall messages from your BIG-IP devices to multiple DCDs, you can configure a BIG-IP system to load balance these messages among the DCDs. This process prevents duplication of information in the consolidated data repository, while also providing high availability for your log messages in the case that one or more DCDs become unavailable.
The following configuration process is conducted within your BIQ-IQ environment. Before you begin, ensure that you have two separate BIG-IP devices, as described in the pre-requisites.
  1. Create a pool of data collection devices (DCDs):
    1. Go to
      Configuration
      LOCAL TRAFFIC
      Pools
      .
    2. Click
      Create
    3. From the
      Name
       field add a name.
    4. From the
      Device
       list, select a host BIG-IP device that provides the load balancing service to the DCD pool.
      Be sure to select a device that is different from the device that hosts your virtual server with an ASM policy.
    5. In the
      Health Monitors
       field, select the
      /Common/http
       option.
    6. Click
      New Member
       to add New Nodes to the list, using the appropriate DCD IP addresses.
    7. From the
      Node Type
       field, select
      New Node
      .
    8. Add the DCD IP address in the
      Address
       field and select a service port for the
      Port
       field.
    9. Ensure that the State (on BIG-IQ)
       field is
      Enabled
      .
    10. Click
      Save & Close
    11. Repeat steps 2f-2j for all DCDs.
    12. From the New Pool Member screen, click
      Save & Close
      .
  2. Create a load balancing virtual server to the new pool by going to
    Configuration
    LOCAL TRAFFIC
    Virtual Servers
    .
    1. Click
      Create
    2. From the
      Name
       field add a name.
    3. From the
      Device
       field, select the device selected in step 2d.
    4. In the
      Destination Address/Mask
       field add the IP address of the virtual server that hosts ASM protection for your application (found on the other BIG-IP device).
    5. In the
      Service Port
       field enter
      8514
      , the designated service port for ASM.
    6. In the
      Source Address Translation
       field select
      Auto Map
      .
    7. In the Resources area, click
      Default Pool
       and select the pool created in step 2.
    8. Click
      Save & Close
      .
    The load balancing configuration for your DCD pool is complete, you now need to ensure that the log messages from the virtual server that hosts your ASM policy is directed to your newly configured virtual server.
  3. Create a new logging profile, by going to
    Configuration
    SECURITY
    Shared Security
    Logging Profiles
    :
    1. Click
      Create
      .
    2. In the
      Name
       field, add a unique name for the profile.
    3. Click
      APPLICATION SECURITY
       from the left menu, select
      Enabled
      .
    4. From the
      Remote Storage
       field, select
      Enabled
      .
    5. From the
      Server Addresses
       field, enter the
      IP Address
       and
      Port
       values of the virtual server created in step 2.
    6. Click the
      Add
       button next to the port value.
    7. In the Storage Filter area, from the
      Request Type
       field, select
      All Requests
      .
    8. Click
      Save & Close
      .
  4. Add the logging profile to the virtual server with the ASM security policy by going to
    Configuration
    SECURITY
    Shared Security
    Virtual Servers
    .
    1. Click the name of the virtual server with ASM security policy.
    2. From the
      Logging Profiles
       field, select the name of the logging profile created in step 3.
    3. Click
      Save & Close
      .
Your BIG-IQ Centralized Management now has high availability of logging data collection for its Web Application Security event logs. This prevents loss of messages in the case that one or more DCDs become unavailable.

Virtual servers that remote logging uses to route alert or event logs

You can either create a new virtual server on the BIG-IP device that creates the alert or event, or you can use a virtual server that already exists on that device.

Creating a virtual server for remote logging

If the device for which you are configuring remote logging does not have a virtual server, you need to create one.
  1. At the top of the screen, click
    Configuration
    .
  2. On the left, expand
    LOCAL TRAFFIC
    .
  3. Under
    LOCAL TRAFFIC
    , select
    Virtual Servers
    .
    The screen displays a list of virtual servers defined on this device.
  4. Click
    Create
    .
    The Virtual Servers - New Item screen opens.
  5. In the
    Name
     field, type in a name for the virtual server you are creating.
  6. From the
    Device
     list, select the device on which to create the virtual server.
  7. In the
    Description
     field, type in a brief description for the virtual server you are creating.
  8. For the
    Destination Address
    , type the IP address of the destination you want to add to the Destination list.
    The format for an IPv4 address is
    I<a>.I<b>.I<c>.I<d>
    . For example,
    172.16.254.1
    .
    The format for an IPv6 address is
    I<a>:I<b>:I<c>:I<d>:I<e>:I<f>:I<g>:I<h>.
    .
    For example,
    2001:db8:85a3:8d3:1319:8a2e:370:7348
    .
  9. In the
    Service Port
     field, type a service port number, or select a type from the list.
    When you select a type from the list, the value in the
    Service Port
     field changes to reflect the associated default, which you can change.
  10. Click
    Save
    .
    The system creates the new virtual server with the settings you specified.
  11. Click
    Save
     to save the assignment. Or, click
    Save & Close
    to save the assignment and return to the Virtual Servers screen.
A virtual server that can be used to route alert or event data to the logging node is created for the BIG-IP device.
Before the BIG-IP device can actually use this new virtual server, you must deploy it to the device.

Assign the logging profile to a virtual server

After configuring a logging profile on the BIG-IQ system, you must assign it to a virtual server and deploy it to the BIG-IP device from which you want to collect event logs.
  1. At the top of the screen, click
    Configuration
    .
  2. On the left, click
    SECURITY
    Shared Security
    Virtual Servers
    .
    The screen displays a list of virtual servers that are configured with devices that have been provisioned and discovered.
  3. On the Virtual Servers screen, click the name of the virtual server you want to use.
    The Virtual Servers - Properties screen opens.
  4. From the
    Log Profiles
     list, under
    Available
    , click a logging profile and move it to the
    Selected
     list.
  5. Click
    Save & Close
    to save the assignment and return to the Virtual Servers screen.
The virtual server is now associated with the logging profile.
Before the BIG-IP system(s) can start sending alert or event logs to the data collection device, you must deploy the changes you just made to the BIG-IP device.
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information