998877-3 : Purging User Script execution logs

Component: BIG-IQ Device Management

Symptoms:
BIG-IQ purges User Script execution logs after 14 days.

Conditions:
After 14 days, BIG-IQ purges User Script execution logs.

Impact:
You cannot change the 14-day purge setting from the user interface.

Workaround:
See Fix Text

Fix:
You can now configure the User Script execution log purge interval from the command line by setting a value for: platform.miscellaneous.userScriptTaskPurgeDays in /var/config/rest/config/restjavad.properties.json

998805-3 : Duplicated entries display when viewing Related Items for a particular object

Component: BIG-IQ Configuration - Security - Network Security

Symptoms:
If there are more than three items of a particular type(for example, Address List, Port List, Rules, Rule List) are attached to a Firewall policy on a managed BIG-IP device, BIG-IQ might display duplicate entries in a list when you click Show Related Items.

Conditions:
-- Attach more than 3 items of a particular kind to a Firewall Policy.
-- Select the Firewall Policy from the list and click Show Related Items.

Impact:
BIG-IQ displays duplicate objects.

Workaround:
This issue has no impact on BIG-IQ management functionality.

Fix:
This issue is fixed and you now see the correct number of objects when you click on Show Related Items for a policy.

996829-2 : User configured statistics retention might not work correctly for BIG-IQ v7.1.0 or v8.0.0

Component: AppIQ

Symptoms:
User configured analytics statistics retention (real-time, hourly, daily, monthly) might not work as expected in BIG-IQ v7.1.0 after upgrading from 7.0.X, or for BIG-IQ v8.0.0 after upgrading from BIG-IQ 7.1.0. This is setting is configured from BIG-IQ by navigating to System -> BIG-IQ DATA COLLECTION -> BIG-IQ Data Collection Cluster -> Configuration -> Statistics Data Collection -> Configure Retention.

Conditions:
For the issue to occur, the following conditions must be met:
1. Upgrade BIG-IQ v7.0.x to v7.1.0 or upgrade BIG-IQ v7.0.X to v7.1.0 and then to v8.0.0. NOTE: If there was an incremental upgrade from v7.0.x to v7.1.0.1, 7.1.0.2 or 7.1.0.3 this issue will not occur.
2. After upgrading, change the retention under System -> BIG-IQ DATA COLLECTION -> BIG-IQ Data Collection Cluster -> Configuration -> Statistics Data Collection -> Configure Retention. For example, adjust "real-time" to another value.
3. During the upgrade process, restart appiqconfig from BIG-IQ using the command: "bigstart restart appiqconfig"
4.View data in System -> BIG-IQ DATA COLLECTION -> BIG-IQ Data Collection Cluster -> Configuration -> Statistics Data Collection -> Configure Retention and see that the value is not what was previously configured.

Impact:
If default retention values were changed, and the config server was restarted, BIG-IQ reloads the previous retention value or might remove the user-configured retention data.
For example if the retention for raw-data was 10 hours and then was adjusted to 13 hours, then after the appiqconfig restart, the retention will be 10 hours again and 3 oldest indexes of raw data are deleted.

Workaround:
If this issue is recognized according to conditions above then the workaround is as follows:

For BIG-IQ v7.1.0:

1. From the BIG-IQ command line, run the following command:

curl -XPOST -s localhost:9200/metadata_indices_management/meta-data/_delete_by_query -H 'Content-Type: application/json' -d '
{"query":{"bool":{"must":{"match":{"indexFamily":"statistics"}},"must_not":{"term":{"_id":"statistics"}}}}}'

2. Restart the appiqconfig by running the following command:

bigstart restart appiqconfig

For BIG-IQ v8.0.0:

1. From the BIG-IQ command line, run the following command:

curl -XPOST -ks https://localhost:9200/metadata_indices_management/meta-data/_delete_by_query -H 'Content-Type: application/json' -d '
{"query":{"bool":{"must":{"match":{"indexFamily":"statistics"}},"must_not":{"term":{"_id":"statistics"}}}}}'

2. Restart the appiqconfig by running the following command:

bigstart restart appiqconfig

Fix:
Statistics data retained as expected following an upgrade.

996601-5 : Virtual Server, Pool, Pool Member and Node status might not display properly for BIG-IP versions 15.x or 16.x

Component: AppIQ

Symptoms:
BIG-IQ might not properly display the status of Virtual Servers, Pools, Pools Members and Nodes configured on managed BIG-IP devices running versions 15.x and 16.x that have DNS configured.

When this happens, BIG-IQ logs an the following in the /var/log/appiq/agentmanager.log:
java.lang.ArrayIndexOutOfBoundsException: 1

Conditions:
Managing BIG-IP devices running v15.X and 16.x that has one or more Wide-IP configured to the DNS(GTM) configuration with LTM status updates from DCDs enabled on BIG-IQ.

restcurl -X PUT /cm/adc-core/current-config/stats-refresh -d '
{ "isMonitorRunning": true,
"useAppIqDcd": true,
"pollingIntervalSeconds": 300 }'

Impact:
BIG-IQ doesn't properly display the status of LTM objects
on the application dashboards for DNS application services on the Applications > APPLICATIONS screen.

Workaround:
Use one of the following workarounds to fix this issue.

1. Disable the LTM status collection from the DCD and allow status to be collected directly by BIG-IQ. This option might increase the CPU usage, see article K91114310 titled: Reducing the performance impact of BIG-IQ statistics gathering on managed BIG-IP systems at support.f5.com/csp/article/K91114310.

2. Update the analytics iApp RPM. For more information, see article K53001642 titled: Updating the BIG-IQ iApp to prevent BIG-IP performance issues on managed BIG-IP systems and improve DNS statistics collection at https://support.f5.com/csp/article/K53001642

Fix:
The iApp script installed by the BIG-IQ on managed BIG-IP devices now allows the system to display objects as expected.

995837-2 : BIG-IQ displays the message "Waiting for BIG-IQ services to become available..." after upgrading to version 8.0.0★

Solution Article: K07869171

Component: REST Framework and TMOS Platform

Symptoms:
If you attempt to upgrade BIG-IQ to version 8.0.0 and there are more than 100,000 records for a single task in the BIG-IQ database, the upgrade does not complete successfully.

Conditions:
Upgrading BIG-IQ to version 8.0.0 when its database has more than 100,000 records.

Impact:
BIG-IQ displays a spinner with a message: Waiting for BIG-IQ services to become available...

Workaround:
To work around this issue, see: https://support.f5.com/csp/article/K07869171

Fix:
N/A

989929-4 : Sorting lists by description field

Component: REST Framework and TMOS Platform

Symptoms:
Due to limitations, we disabled the option to sort lists on the description field in BIG-IQ version 6.0.0.

Conditions:
Attempting to sort on description field for a list.

Impact:
Users can't sort lists on the description field.

Workaround:
There is no workaround.

Fix:
Sorting lists by description field is re-enabled.

Behavior Change:
You can now sort on the description field for lists.

977109-4 : BIG-IQ might report errors when deploying changes to BIG-IP devices

Component: BIG-IQ Local Traffic & Management

Symptoms:
BIG-IQ can sometimes display errors similar to the following when deploying changes to BIG-IP devices: 'Exception in verifyConfig: java.lang.NullPointerException'

Conditions:
This happens when BIG-IQ is managing BIG-IP devices with varying versions. It can also occur when a BIG-IP device has been upgraded multiple times.

Impact:
You cannot manage the BIG-IP device from BIG-IQ when that error occurs.

Workaround:
If you manage BIG-IP devices running the same version and you have no changes pending on BIG-IQ, run the following command from BIG-IQ to clear the error status: clear-rest-storage

If you manage several BIG-IP devices of varying versions then you will be unable to work around this issue.

Fix:
This issue is fixed. BIG-IQ no longer displays error messages when deploying BIG-IP changes.

909205-1 : BIG-IQ statistics reports are missing the latest event data

Component: AppIQ

Symptoms:
BIG-IQ statistics reports are missing the latest event data.

Conditions:
This occurs when the default system-generated Elasticsearch indices are manually deleted by administrators.

This can also happen when the disk becomes full and BIG-IQ runs out of disk space due to accumulating events/logs.

Impact:
This issue results in a loss of statistics data.

Workaround:
The workaround involves repairing the elasticsearch indices storing statistics data.

To identify the statistics that are impacted
    - list all index names & index aliases

     # curl -s localhost:9200/_cat/indices?v
     # curl -s localhost:9200/_cat/aliases?v

     - run below command (or create a periodic task) to detect affected/corrupted elasticsearch cluster indices

      # curl -s localhost:9200/_cat/indices?h=index | grep _writer

     - every index output in above command requires a repair procedure outlined in the remainder of article as the name of index is not expected to have '_writer' suffix (only names of index aliases will have the '_writer' suffix)
  
      lets say it reports youraffectedindex1_writer/youraffectedindex2_writer

    - for every index reported above you can see the size of statistics data that have been accumulated under it (this it the amount of data not getting reported in BIG-IQ GUI)

      # curl -s localhost:9200/_cat/indices | youraffectedindex1_writer
      # curl -s localhost:9200/_cat/indices | youraffectedindex2_writer

If statistics data for impacted/corrupt indices is non-critical (and permanent deletion is acceptable), then please follow procedure A. If you need to preserve the corrupt data proceed to procedure B. However, both procedures repair the impacted elasticsearch indices.

------------
Procedure A
------------

Use this procedure to permanently delete the corrupt data and repair the impacted indices

1. Deactivate the impacted service.

   - on BIG-IQ CM, navigate to System -> BIG-IQ DATA COLLECTION -> BIG-IQ Data Collection Devices

   - deactivate the impacted service (based on the reported indices earlier) for all DCDs

     Ex: Access / DOS Protection / Fraud Protection Service / IPSec / Network Security / Web Application Security

   - wait for 5 minutes to allow existing connections to close

   - on BIG-IQ DCD verify all external connections to special ports are closed:

       # netstat -an | grep -E "9997|8018|8514|8020|8008"

2. Delete the impacted elasticsearch index.

   - on CM (or DCD) remove the incorrect index from elasticsearch, ex:
      # curl -sX DELETE localhost:9200/youraffectedindex1_writer

       Note: above command should return {"acknowledged":true}

    - confirm the index is no longer reported by below command

       # curl -s localhost:9200/_cat/indices | grep _writer

3. Reactivate the service.

   - on BIG-IQ CM, navigate to System -> BIG-IQ DATA COLLECTION -> BIG-IQ Data Collection Devices and activate the impacted services again

------------
Procedure B
------------

Use this procedure to recover the corrupt data and repair the impacted indices

1. Deactivate the impacted service.
   
   see step #1 from Procedure A

2. Create a temporary elasticsearch index.

   - identify naming convention (including a date suffix) by observing output of below command

     # curl -s localhost:9200/_cat/indices?h=index

     ex: youraffectedindex1_2020-xx-xxtxx-xx-xx-xxxx


   - create a temporary index using name identified in above step

     # curl localhost:9200/youraffectedindex1_2020-xx-xxtxx-xx-xx-xxxx -X PUT -d {}

     ex: # curl localhost:9200/afmlogindex_2020-02-01t10-10-10-0100 -X PUT -d {}

3. Reindex data from corrupt index to the temporary elasticsearch index.

   - this may take time depending on size of data in affected index

   - begin the indexing process

      # curl -s localhost:9200/_reindex?wait_for_completion=false -d '{"source":{"index":"youraffectedindex1_writer"},"dest":{"index":"youraffectedindex1_2020-xx-xxtxx-xx-xx-xxxx"}}' | jq .

      ex: # curl -s localhost:9200/_reindex?wait_for_completion=false -d '{"source":{"index":"afmlogindex_writer"},"dest":{"index":"afmlogindex_2020-02-01t10-10-10-0100"}}' | jq .

  - above command will print a task-identifier

    ex: "task": "rur5BcBNTGqdtydEDjHMAA:22687961"

  - use above reported unique task-identifier and repeatedly query the progress of the task until it completes

    # curl -s localhost:9200/_tasks/rur5BcBNTGqdtydEDjHMAA:22687961 | jq .completed
true

   - verify the index status

     # curl -s localhost:9200/_cat/indices?v
     # curl -s localhost:9200/_cat/aliases?v

4. Remove the corrupted elasticsearch index.
   
   - see step #2 in Procedure A.

5. Create a new *_writer alias for the temporary elasticsearch index.

   - run below command to create an alias (for your newly created index) named as the index you just deleted

     # curl localhost:9200/_aliases -X POST -d '{"actions" : [ {"add" : { "index" : "youraffectedindex1_2020-xx-xxtxx-xx-xx-xxxx" , "alias" : "youraffectedindex1_writer" }} ] }'

     ex: # curl localhost:9200/_aliases -X POST -d '{"actions" : [ {"add" : { "index" : "afmlogindex_2020-02-01t10-10-10-0100" , "alias" : "afmlogindex_writer" }} ] }'

   - verify the index status

     # curl -s localhost:9200/_cat/indices?v
     # curl -s localhost:9200/_cat/aliases?v

6. Reactivate the impacted service.
 
   see step #3 in Procedure A.

7. Confirm newly initialized elasticsearch indices.

   - verify the index status

     # curl -s localhost:9200/_cat/indices?v
     # curl -s localhost:9200/_cat/aliases?v

   - confirm that you see a new index named yourimpactedindex1_YYYY_MM_DD and that yourimpactedindex1_writer points to this latest index instance
   - this happens because eventually index are rotated and removed (this also depends on the retention policy)

Fix:
Statistics and event data displays as expected.

823565-1 : BIG-IQ REST framework JVM process fails intermittently

Component: REST Framework and TMOS Platform

Symptoms:
The BIG-IQ REST framework JVM process (restjavad) fails intermittently when RAM memory is low or during highly intensive memory operations.

Conditions:
This happens when there is not enough RAM available for the normal operation of the BIG-IQ REST framework, especially in environments with a relatively large number of managed BIG-IP devices and related configuration objects, and a high volume of collected statistical data.

Impact:
BIG-IQ management interface and device management workflows are temporarily unavailable while restjavad restarts.

Workaround:
Increase the physical memory (RAM) that is available for BIG-IQ (hardware or virtual).

Fix:
This issue is fixed and restjavad no longer fails under these conditions.

1011941-4 : Admin users can run arbitrary commands on the BIG-IQ configuration utility

Component: BIG-IQ DNS Management

Symptoms:
Admin users can run arbitrary commands in the BIG-IQ configuration utility using unspecified methods.

Conditions:
Admin users can run arbitrary commands in the BIG-IQ configuration utility using unspecified methods.

Impact:
Admin users can run arbitrary commands in the BIG-IQ configuration utility using unspecified methods.

Workaround:
None

Fix:
Admin users are now limited with what commands they are allowed to use.

1011877 : Resizing partitions after upgrading

Component: REST Framework and TMOS Platform

Symptoms:
Unable to increase the /var partitions after upgrading to BIG-IQ version 8.0.0.

Conditions:
1) Create a virtual server of BIG-IQ 7.1.x
2) Increase the /var partition size
3) Upgrade BigIQ to 8.0.0
2) Try running the resizevol script as resizevol /var 272218920

Impact:
Unable to resize the /var partition with resizevol tool.

Workaround:
If you have already upgraded, you can resize the /var partition after the upgrade completes and BIG-IQ reboots using the command similar to: lvresize -rL 10G /dev/vg-db-sda/set.2._var

Fix:
The resizevol tool script is now included starting in BIG-IQ version 8.0.0.1.

Behavior Change:
You can now resize the /var partitions with the resizevol tool
using a command similar to lvresize -rL 10G /dev/vg-db-sda/set.2._var. When you run this command, BIG-IQ resizes the partition as specified.

1005049-1 : DCD cluster stops working after adding a DCD or a standby BIG-IQ to a high availability configuration

Component: AppIQ

Symptoms:
After adding a standby BIG-IQ to a high availability (HA) configuration or while (less likely) adding a DCD, the DCD cluster no longer functions and the restjavad.0.logs shows a message similar to the following:

[HTTP/1.1 503 Service Unavailable]
Open Distro not initialized

Conditions:
Adding a standby BIG-IQ in a high availability (HA) configuration, or (less likely) when adding a DCD.

Impact:
BIG-IQ is unable to receive new analytics statistics and events and cannot display existing events.

Workaround:
After adding the standby BIG-IQ for a high availability (HA) configuration, or a DCD, run the following command on the active BIG-IQ:

export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64 ; /bin/bash -c "/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -h `netstat -nap | grep ':9300 ' | grep LISTEN | awk '{print $4}' | rev | cut -c 6- | rev` --accept-red-cluster -cd /var/config/appiq/elasticsearch/utils/ -icl -nhnv -cacert /var/config/rest/elasticsearch/config/es_root-ca.pem -cert /var/config/rest/elasticsearch/config/es_admin.pem -key /var/config/rest/elasticsearch/config/es_admin-key.pem"

Fix:
This issue is fixed and DCD clusters work as expected after adding a standby BIG-IQ or a DCD.

1004421-2 : ASM policy is removed after LTM virtual server deployment

Component: BIG-IQ Web Application Security (ASM)

Symptoms:
BIG-IQ removes any configured ASM policies related to a BIG-IP virtual server when you deploy a full LTM configuration or deploy a change for LTM on that virtual server with the ASM policies.

Conditions:
LTM and ASM services are provisioned and discovered by BIG-IQ while adding a BIG-IP device.

Impact:
Web application security is disabled on the affected virtual servers.

Workaround:
Redeploy BIG-IP virtual servers licensed for ASM and re-import LTM with the auto-generated ASM policy for the virtual server.

Fix:
BIG-IQ no longer removes ASM policies after you deploy an LTM virtual server.

1000061-1 : Elasticsearch process in DCDs restarts every few hours

Component: AppIQ

Symptoms:
Attempting to aggregate statistics data from BIG-IQ can cause the Elasticsearch (ES) process in the data collection devices (DCDs) to restart every few hours. When this happens, the following exception appears in the elasticsearch/esnode.log file for affected DCDs:

java.lang.NoClassDefFoundError: org/elasticsearch/client/RestClient

Conditions:
Many statistic dimension combinations are defined for a specific module, which might create more unique ES documents than the allowed document limit per BIG-IP (50000 / DCDs num).

For example, a managed BIG-IP device configured with an HTTP module might have traffic from multiple client IPs, with each client IP having multiple destination URLs.

Impact:
When BIG-IQ attempts to aggregate the statistics data under these conditions, the ES process fails and then restarts, resulting in the loss of that aggregated statistics data and pending module events.

Workaround:
Upgrade F5 Elasticsearch plugins.

Fix:
This issue is fixed and the Elasticsearch process no longer restarts every few hours.

• The F5 Networks Technical Support web site: http://www.f5.com/support/
• The AskF5 web site: https://support.f5.com/csp/#/home
• The F5 DevCentral web site: http://devcentral.f5.com/

Generated: Tue Apr 27 13:35:22 2021 PDT

Copyright F5 Networks (2021) - All Rights Reserved