Manual Chapter :
Managing a Data
Collection Device Cluster
Applies To:
Show VersionsBIG-IQ Centralized Management
- 6.0.0
Managing a Data
Collection Device Cluster
Data collection device
best practices
There are a number of useful concepts to consider when you manage data
collection devices for off-box log storage. This reference material might prove helpful in
setting up and maintaining your data collection device (DCD) configuration.
In addition to the collection of best practices discussed here, you may want
to consider how to take best advantage of the disaster recovery capabilities that the DCD can
provide by deploying DCD clusters in multiple zones. For a basic discussion of these concepts,
refer to
Managing Disaster Recovery Scenarios
in the Planning and Implementing a BIG-IQ Deployment
guide on support.f5.com
As
part of maintaining a DCD cluster, you might need to remove one or more devices from your DCD
cluster. When you remove a DCD from the cluster,
BIG-IQ® Centralized Management moves the data to another
device in the cluster. Whenever you move data, losing part or all of that data is a risk.
Therefore, before you remove a DCD from the cluster, F5 recommends creating a snapshot to back up
your logging data.
Restore data collection device
snapshots
Before initiating a snapshot restore, make sure that sufficient
disk space is allocated to the
/var
folder on the device to which you are restoring the
snapshot.You can use the BIG-IQ user interface to restore data
collection device (DCD) snapshots.
- The restore operation requires a down time during which no BIG-IQ or DCD work is performed.
- During the restore operation, no data sent to the DCD is retained.
- The restore operation restores only the data from the time before the chosen snapshot was created. Data from the time that the chosen snapshot was created to the current time is not restored.
- At the top of the screen, clickSystem, then, on the left, clickBIG-IQ DATA COLLECTIONand then selectBIG-IQ Data Collection Cluster.The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
- Under SUMMARY, you can view information detailing how much data is stored, as well as how the data is stored.
- Under CONFIGURATION, you can access the screens that control DCD cluster performance.
- Under the screen name, click.The External Storage & Snapshots screen opens so you can specify how you want to schedule snapshots.
- You have two options for choosing a snapshot and starting the restore, using the settings in the External Storage & Snapshot area near the bottom of the screen.To restore from the most recent snapshot:Next toLast Snapshot/Time, clickRestore Latest.To select the snapshot that you want to restore:
- Click theView Historybutton.
- Choose the snapshot you want to restore, and clickRestore.
Delete a data collection device
snapshot
If you determine that there are issues with a specific snapshot, you can delete it so
that you cannot accidentally restore to it in the future.
You perform this task
on the BIG-IQ Centralized Management device; not on the data
collection device (DCD).
- At the top of the screen, clickSystem, then, on the left, clickBIG-IQ DATA COLLECTIONand then selectBIG-IQ Data Collection Cluster.The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
- Under SUMMARY, you can view information detailing how much data is stored, as well as how the data is stored.
- Under CONFIGURATION, you can access the screens that control DCD cluster performance.
- Under the screen name, click.The External Storage & Snapshots screen opens so you can specify how you want to schedule snapshots.
- Near the bottom of the screen, click theView Historybutton.The BIG-IQ Data Collection Snapshots screen opens.
- Browse through the list to find the snapshot you want to delete.
- Select the check box for the snapshot you want to delete, and clickDelete.
Check data collection device
health
You
can use the BIG-IQ Data Collection Device Settings screen to
review the overall health and status of the data collection devices you've configured.
You can use the data displayed on this screen both before and after an upgrade to verify
that your
data collection device (DCD)
cluster configuration is as you expect it to be.
- At the top of the screen, clickSystem, then, on the left, clickBIG-IQ DATA COLLECTIONand then selectBIG-IQ Data Collection Cluster.The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
- Under SUMMARY, you can view information detailing how much data is stored, as well as how the data is stored.
- Under CONFIGURATION, you can access the screens that control DCD cluster performance.
- Click theSettingsbutton.The Settings screen opens to display the current state of the DCD cluster defined for this BIG-IQ device.
- Inspect the DCD cluster details listed in the Summary area.Sub-screenWhat details are provided here?StatusLook here for information about the current state of the cluster.NodesLook here for information about the current state of the cluster nodes.IndexesLook here for information about the current state of the cluster indexes.ShardsLook here for information about the current state of the cluster shards.Cluster SettingsDisplays information for the DCD cluster configured for this device.External Storage & SnapshotsDisplays summary information about the external storage location used to keep the backup snapshots you create for the DCD cluster configured for this device.Logging Data CollectionDisplays summary information for the event and alert log indices that have been configured for this DCD.Statistics Data CollectionDisplays details about the statistics data stored on this DCD.This information provides a fairly detailed overview that describes the DCD cluster you have created to store data. After you complete an upgrade, you can check the health to verify that the cluster restored successfully.
Index rotation
policy
The optimum settings used to configure your data collection device (DCD)
indices depend on a number of key factors.
- The system provides the ability to dynamically create new indices based on either a specified interval or a specified size. The primary goal to consider when you make these decisions is how to maintain a maximum disk allocation for the DCD data, while maintaining capacity for new data that flows in.
- Secondary considerations include search optimization, and the ability to optimize old indices to reduce their size.
- Generally, the best policy is one that does not create unnecessary indices. The more indices, the lower the overall performance, because your searches have to deal with more shards. For example, if a module knows that it has a low indexing volume (thousands/day) then it makes the most sense to have a large aggregation per rotation (5 days or 30 days). For components like Web Application Security that probably have high indexing volumes, it makes more sense to rotate every 8 hours (which reduces the number of retained indices).
- Index rotation also allows changing sharding and replica counts by changing the template on a given index type. New indices created from that template will contain the new shard and replica count properties.
This table shows the default configuration values for each index running
on BIG-IQ® Centralized Management. These values are
based on anticipated data ingestion rates and typical usage patterns.
Component | Index Name | Minimum Number of DCDs | Rotation Policy | Retained Index Count | Approximate time window | Size of /var file system |
---|---|---|---|---|---|---|
Access | access-event-logs | 2 | Time/5 days | 19 | 95 days | 500 GB |
Access | access-stats | 2 | Time/5 days | 19 | 95 days | 500 GB |
Web Application Security | asmindex | 2 | Size/100000 MB | 5 | N/A | 500 GB |
FPS | websafe | 2 | Time/30 days | 100 | 8 years | 10 GB |
If multiple modules are running on a given DCD or if you have higher
inbound data rates, you might have to adjust these values to keep the
/var
file system from filling up. (There is a
default alert to warn of this when the file system becomes 80% full.)The simplest resolution is to revise the retained index count; lowering
this value reduces the disk space requirements, but it will also reduce the amount of data
available for queries. For details on changing this setting, refer to the modifying indices
topic for the component you are configuring.
Changing the minimum number of master
eligible devices
You can manage the minimum number of
devices that must be available for the cluster to be considered operational. If the
number of available devices is less than the value specified for the Minimum Master
Eligible Devices, the cluster is deemed unhealthy.
- At the top of the screen, clickSystem, then, on the left, clickBIG-IQ DATA COLLECTIONand then selectBIG-IQ Data Collection Cluster.The BIG-IQ Data Collection Cluster screen opens. On this screen, you can either view summary status for the data collection device cluster or access the screens that you can use to configure the DCD cluster.
- Under SUMMARY, you can view information detailing how much data is stored, as well as how the data is stored.
- Under CONFIGURATION, you can access the screens that control DCD cluster performance.
- Under the screen name, click.The Cluster Settings screen opens.
- To change this setting, clickOverride.The button text changes toUpdate.
- In theMinimum Master Eligible Devicesfield, type or select the new minimum number of healthy devices for this DCD cluster, and clickUpdate.The system updates the setting.
- When you are satisfied with the minimum number of devices setting, clickCancelto close the screen.