Applies To:
Show VersionsBIG-IQ Centralized Management
- 5.3.0
Data collection device best practices
There are a number of useful concepts to consider when you manage data collection devices for off-box log storage. This reference material might prove helpful in setting up and maintaining your data collection device (DCD) configuration.
Restore data collection device snapshots
You can use the BIG-IQ® user interface to restore data collection device (DCD) snapshots.
Please note:
- The restore operation requires a down time during which no BIG-IQ or DCD work is performed.
- During the restore operation, no data sent to the DCD is retained.
- The restore operation restores only the data from the time before the chosen snapshot was created. Data from the time that the chosen snapshot was created to the current time is not restored.
- Before initiating a snapshot restore, make sure that sufficient disk space is allocated to the /var folder on the device to which you are restoring the snapshot.
Delete a data collection device snapshot
If you determine that there are issues with a specific snapshot, you can delete it so that you cannot accidentally restore to it in the future.
Check data collection device health
Index rotation policy
The optimum settings used to configure your data collection device (DCD) indices depend on a number of key factors.
- The system provides the ability to dynamically create new indices based on either a specified interval or a specified size. The primary goal to consider when you make these decisions is how to maintain a maximum disk allocation for the DCD data, while maintaining capacity for new data that flows in.
- Secondary considerations include search optimization, and the ability to optimize old indices to reduce their size.
- Generally, the best policy is one that does not create unnecessary indices. The more indices, the lower the overall performance, because your searches have to deal with more shards. For example, if a module knows that it has a low indexing volume (thousands/day) then it makes the most sense to have a large aggregation per rotation (5 days or 30 days). For components like Web Application Security that probably have high indexing volumes, it makes more sense to rotate every 8 hours (which reduces the number of retained indices).
- Index rotation also allows changing sharding and replica counts by changing the template on a given index type. New indices created from that template will contain the new shard and replica count properties.
This table shows the default configuration values for each index running on BIG-IQ® Centralized Management. These values are based on anticipated data ingestion rates and typical usage patterns.
Component | Index Name | Minimum Number of DCDs | Rotation Policy | Retained Index Count | Approximate time window | Size of /var file system |
---|---|---|---|---|---|---|
Access | access-event-logs | 2 | Time/5 days | 19 | 95 days | 500 GB |
Access | access-stats | 2 | Time/5 days | 19 | 95 days | 500 GB |
Web Application Security | asmindex | 2 | Size/100000 MB | 5 | N/A | 500 GB |
FPS | websafe | 2 | Time/30 days | 100 | 8 years | 10 GB |
If multiple modules are running on a given DCD or if you have higher inbound data rates, you might have to adjust these values to keep the /var file system from filling up. (There is a default alert to warn of this when the file system becomes 80% full.)
The simplest resolution is to revise the retained index count; lowering this value reduces the disk space requirements, but it will also reduce the amount of data available for queries. For details on changing this setting, refer to the modifying indices topic for the component you are configuring.