F5 Logo MyF5
Search
  1. MyF5 Home
  2. Planning and Implementing a BIG-IQ Deployment
  3. Managing a Data Collection Device Cluster
Manual Chapter : Managing a Data Collection Device Cluster

Applies To:

Show Versions Show Versions

BIG-IQ Centralized Management

  • 7.0.0
Manual Chapter

Managing a Data Collection Device Cluster

Data collection device best practices

There are a number of useful concepts to consider when you manage data collection devices for off-box log storage. This reference material might prove helpful in setting up and maintaining your data collection device (DCD) configuration.
In addition to the collection of best practices discussed here, you may want to consider how to take best advantage of the disaster recovery capabilities that the DCD can provide by deploying DCD clusters in multiple zones. For a basic discussion of these concepts, refer to
Managing Disaster Recovery Scenarios
 in the
Planning and Implementing a BIG-IQ Deployment
guide on
support.f5.com
As part of maintaining a DCD cluster, you might need to remove one or more devices from your DCD cluster. When you remove a DCD from the cluster, the data must be redistributed to the remaining devices in the cluster. Consequently, there are a couple things to bear in mind.
  • Whenever you move data, losing part or all of that data is a risk. Therefore, before you remove a DCD from the cluster, F5 recommends creating a snapshot to back up your data.
  • While the data is being redistributed, it might take a little longer to deliver the statistics data that drives your application service analytics.

Restore data collection device snapshots

Before initiating a snapshot restore, make sure that sufficient disk space is allocated to the
/var
 folder on the device to which you are restoring the snapshot.
You can use the BIG-IQ user interface to restore data collection device (DCD) snapshots.
  • The restore operation requires a down time during which no BIG-IQ or DCD work is performed.
  • During the restore operation, no data sent to the DCD is retained.
  • The restore operation restores only the data from the time before the chosen snapshot was created. Data from the time that the chosen snapshot was created to the current time is not restored.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    External Storage & Snapshots
    .
    The External Storage & Snapshots screen opens.
  3. You have two options for choosing a snapshot and starting the restore, using the settings in the External Storage & Snapshot area near the bottom of the screen.
    To restore from the most recent snapshot:
    Next to
    Last Snapshot/Time
    , click
    Restore Latest
    .
    To select the snapshot that you want to restore:
    1. Click the
      View History
       button.
    2. Choose the snapshot you want to restore, and click
      Restore
      .

Delete a data collection device snapshot

If you determine that there are issues with a specific snapshot, you can delete it so that you cannot accidentally restore to it in the future.
You perform this task on the BIG-IQ Centralized Management device; not on the data collection device (DCD).
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    External Storage & Snapshots
    .
    The External Storage & Snapshots screen opens.
  3. Near the bottom of the screen, click the
    View History
     button.
    The BIG-IQ Data Collection Snapshots screen opens.
  4. Browse through the list to find the snapshot you want to delete.
  5. Select the check box for the snapshot you want to delete, and click
    Delete
    .

Check the health of devices in a DCD cluster

You can view the health and status of the devices in your data collection device (DCD) cluster using the BIG-IQ Data Collection Device Settings screen. Use the information here, both before and after an upgrade to verify that the devices in your DCD cluster are configured as you expect it to be.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Click the
    Settings
     button.
    The Settings screen opens to display the current state of the DCD cluster defined for this BIG-IQ device.
  3. Inspect the DCD cluster details listed in the Summary area.
    Sub-screen
    What details are provided here?
    Status
    Look here for information about the current status of the cluster.
    Nodes
    Look here for information about the current state of the cluster nodes.
    Indexes
    Look here for information about the current state of the cluster indexes.
    Shards
    Look here for information about the current state of the cluster shards.
    Cluster Settings
    Look here for information about the DCD cluster configured for this device.
    External Storage & Snapshots
    Look here for summary information about the external storage location used to keep the backup snapshots you create for the DCD cluster configured for this device.
    Logging Data Collection
    Look here for summary information about the event and alert log indices that have been configured for this DCD.
    Statistics Data Collection
    Look here for details about the statistics data stored on this DCD.
    This information provides a fairly detailed overview that describes the DCD cluster you have created to store data. After you complete an upgrade, you can check the health to verify that the cluster restored successfully.

Index rotation policy

The optimum settings used to configure your data collection device (DCD) indices depend on a number of key factors.
  • The system provides the ability to dynamically create new indices based on either a specified interval or a specified size. The primary goal to consider when you make these decisions is how to maintain a maximum disk allocation for the DCD data, while maintaining capacity for new data that flows in.
  • Secondary considerations include search optimization, and the ability to optimize old indices to reduce their size.
  • Generally, the best policy is one that does not create unnecessary indices. The more indices, the lower the overall performance, because your searches have to deal with more shards. For example, if a module knows that it has a low indexing volume (thousands/day) then it makes the most sense to have a large aggregation per rotation (5 days or 30 days). For components like Web Application Security that probably have high indexing volumes, it makes more sense to rotate every 8 hours (which reduces the number of retained indices).
  • Index rotation also allows changing sharding and replica counts by changing the template on a given index type. New indices created from that template will contain the new shard and replica count properties.
This table shows the default configuration values for each index running on BIG-IQ Centralized Management. These values are based on anticipated data ingestion rates and typical usage patterns.
Component
Index Name
Minimum Number of DCDs
Rotation Policy
Retained Index Count
Approximate time window
Size of /var file system
Access
access-event-logs
2
Time/5 days
19
95 days
500 GB
Access
access-stats
2
Time/5 days
19
95 days
500 GB
Web Application Security
asmindex
2
Size/100000 MB
5
N/A
500 GB
FPS
websafe
2
Time/30 days
100
8 years
10 GB
If multiple modules are running on a given DCD or if you have higher inbound data rates, you might have to adjust these values to keep the
/var
 file system from filling up. (There is a default alert to warn of this when the file system becomes 80% full.)
The simplest resolution is to revise the retained index count; lowering this value reduces the disk space requirements, but it will also reduce the amount of data available for queries. For details on changing this setting, refer to the modifying indices topic for the component you are configuring.

Change the minimum number of master eligible devices

You can manage the minimum number of devices that must be available for the cluster to be considered operational. If the number of available devices is less than the value specified for the Minimum Master Eligible Devices, the cluster is deemed unhealthy.
  1. At the top of the screen, click
    System
    , then, on the left, click
    BIG-IQ DATA COLLECTION
     and then select
    BIG-IQ Data Collection Cluster
    .
    The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
    • Under Summary, you can view information detailing how much data is stored, as well as how the data is stored.
    • Under Configuration, you can access the screens that control DCD cluster performance.
  2. Under the screen name, click
    Configuration
    Cluster Settings
    .
    The Cluster Settings screen opens.
  3. To change this setting, click
    Override
    .
    The button text changes to
    Update
    .
  4. In the
    Minimum Master Eligible Devices
    field, type or select the new minimum number of healthy devices for this DCD cluster, and click
    Update
    .
    The system updates the setting.
  5. When you are satisfied with the minimum number of devices setting, click
    Cancel
     to close the screen.
Contact Support Contact Support

Have a Question?

Support and Sales >

About F5

Education

F5 Sites

Support Tasks




©2023 F5, Inc. All rights reserved.


Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information