Manual Chapter : Troubleshooting BIG-IQ High Availability Auto Failover Issues

Applies To:

Show Versions Show Versions

BIG-IQ Centralized Management

  • 7.1.0, 7.0.0
Manual Chapter

Troubleshooting BIG-IQ High Availability Auto Failover Issues

BIG-IQ in an auto failover HA configuration with a floating IP address

A BIG-IQ HA configuration with automatic fail over consists of three components.
BIG-IQ HA Configuration with Auto Fail Over
Component
Description
Active BIG-IQ
The BIG-IQ currently managing all BIG-IP devices.
Standby BIG-IQ
The standby BIG-IQ that has the same configuration as the active BIG-IQ. The standby BIG-IQ automatically becomes active if the active BIG-IQ fails over.
Quorum DCD
A quorum data collection device (DCD) makes the deciding vote for which BIG-IQ will become active if communication is disrupted between the components of the BIG-IQ high availability configuration. If the quorum DCD is able to communicate with one of the BIG-IQ in the pair during the disruption, that BIG-IQ becomes active. The quorum DCD is not a peer BIG-IQ in an HA configuration.
A healthy BIG-IQ auto failover high availability configuration with a floating IP address looks like this.
The three members of the high availability configuration form a voting cluster and continuously monitor each others health. If the standby and quorum DCD agree that the active BIG-IQ is no longer able to perform its function, failover is triggered. The standby BIG-IQ is promoted to an active standalone BIG-IQ using the floating IP address, fencing off the formerly active BIG-IQ. After you resolve the issue that caused the failover, you can reestablish the BIG-IQ auto failover HA configuration.
The floating IP address must be on the same network (this configuration uses Gratuitous ARP packets) as the active and standby BIG-IQ systems’ local management address (interface eth0) and not any of the discovery self IP addresses. This does not restrict HA traffic; HA traffic can be on any of the available interfaces.
Floating IP addresses are not supported if your active and standby BIG-IQ systems are in a public cloud environment, such as AWS, Azure, or VMware.

Reestablishing a BIG-IQ auto failover HA configuration after failover

You can select the
HA error
alarm from the
Monitoring
ALERTS & NOTIFICATIONS
General Settings
screen to notify you if a BIG-IQ in your HA auto failover configuration becomes unresponsive. If an issue occurs, an HA Alert appears on the Alert Properties screen as
HA status has changed from OPERATIONAL to FAILED.
When an active BIG-IQ automatically fails over, the standby BIG-IQ becomes standalone and continues to manage your BIG-IP devices. After failover, you'll need to reestablish your BIG-IQ auto HA configuration.
  1. Log into the command line of the BIG-IQ that failed and type the following command:
    ha_reset -f <discovery IP>
    .
  2. From the currently-active standalone BIG-IQ, at the top of the screen, click
    System
    .
  3. On the left, click
    BIG-IQ HA
    .
  4. Click the
    BIG-IQ HA Settings
    button.
  5. Click the
    Remove HA Config
    button.
  6. Click the
    Save & Close
    button.
  7. Click the
    Add Standby
    button.
  8. In the
    IP Address
    field, type the discovery address you want to set up as the standby BIG-IQ.
    This is the same IP address the peers in a high availability configuration use to communicate.
    IPv6 short form addresses are not supported.
  9. Type the local administrative
    Username
    and
    Password
    for the system.
  10. Type the
    Root Password
    for the system.
  11. For the
    Failover setting
    , select
    Auto Failover
    .
    For auto failover to work, you must have the following ports open on the active and standby BIG-IQ as well as the quorum DCD.
    • TCP port 2224
    • UDP port 5404
    • UDP port 5405
  12. Click the
    Set Up Quorum Device
    button, select the quorum DCD from the list, and type its
    Root Password
    .
  13. If you want BIG-IQ to use a floating IP address when automatically failing over to the standby BIG-IQ, select the
    Enable Floating IP
    Enable
    check box and type the address.
    The floating IP address must be on the same network (this configuration uses Gratuitous ARP packets) as the active and standby BIG-IQ systems’ local management address (interface eth0) and not any of the discovery self IP addresses. This does not restrict HA traffic; HA traffic can be on any of the available interfaces. Floating IP addresses are not supported if your active and standby BIG-IQ systems are in a public cloud environment, such as AWS, Azure, or VMware.
    If you choose not to use a floating IP address and the active BIG-IQ fails over, you'll have to provide all users access to the newly active BIG-IQ by providing the IP address.
  14. Click the
    Add
    button to add this device to this high availability configuration.
The active BIG-IQ discovers the standby BIG-IQ and displays its status.

Refresh the database after a BIG-IQ in an auto failover HA configuration comes back online

You can select the HA error alarm from the
Monitoring
ALERTS & NOTIFICATIONS
Monitoring ALERTS & NOTIFICATIONS
General Settings
screen to notify you if a BIG-IQ in your HA auto failover configuration becomes unresponsive. If an issue occurs, an HA Alert appears on the Alert Properties screen as
HA status has changed from OPERATIONAL to FAILED
.
If the one of the BIG-IQ systems becomes inactive, the active BIG-IQ continues to manage traffic. If the inactive BIG-IQ becomes healthy again in less than 3 hours, it automatically rejoins the BIG-IQ auto failover HA configuration with no required intervention from you. But if the BIG-IQ has been down for over 3 hours, you'll need to re-sync the database between the active BIG-IQ and the newly-activated standby BIG-IQ.
If the BIG-IQ auto failover HA configuration has been inoperable for over 3 hours, you must recreate the configuration by adding a standby BIG-IQ and a BIG-IQ DCD quorum back to the standalone, active BIG-IQ.
  1. From the active BIG-IQ, at the top of the screen, click
    System
    .
  2. On the left, click
    BIG-IQ HA
    .
  3. Click the
    BIG-IQ HA Settings
    button.
  4. Click the
    Repair Database
    button to resynchronize the database between the active and standby BIG-IQ systems.
    This button displays only if the database needs to resynchronized.
  5. Click the
    Save & Close
    button.

BIG-IQ quorum DCD in an auto failover HA configuration fails

You can select the HA error alarm from the
Monitoring
ALERTS & NOTIFICATIONS
Monitoring ALERTS & NOTIFICATIONS
General Settings
screen to notify you the BIG-IQ quorum DCD in your HA auto failover configuration becomes unresponsive. If an issue occurs, an HA Alert appears on the Alert Properties screen as
HA status has changed from OPERATIONAL to FAILED
.
A quorum data collection device (DCD) makes the deciding vote for which BIG-IQ will become active if communication is disrupted between the components of the BIG-IQ high availability configuration. If the quorum DCD comes back up (even up to 10 hours later), it will automatically rejoin the HA auto failover configuration. But when the quorum DCD is down, automatic failover cannot occur. It's important to get the quorum DCD back online as quickly as possible. If the quorom DCD is unrepairable, perform these steps.
  1. From the active BIG-IQ device, at the top of the screen, click
    System
    .
  2. On the left, click
    BIG-IQ HA
    .
  3. Click the
    Remove Standby
    button.
  4. Click the
    Save & Close
    button.
  5. Click the
    Add Standby
    button.
  6. In the
    IP Address
    field, type the discovery address you want to set up as the standby BIG-IQ.
    This is the same IP address the peers in a high availability configuration use to communicate.
    IPv6 short form addresses are not supported.
  7. Type the local administrative
    Username
    and
    Password
    for the system.
  8. Type the
    Root Password
    for the system.
  9. For the Failover setting, select
    Auto Failover
    .
  10. Select the quorum DCD.
  11. Click the
    Add
    button at the bottom of the screen.