998877 : Purging User Script execution logs

Component: BIG-IQ Device Management

Symptoms:
BIG-IQ purges User Script execution logs after 14 days.

Conditions:
After 14 days, BIG-IQ purges User Script execution logs.

Impact:
You cannot change the 14-day purge setting from the user interface.

Workaround:
See Fix Text

Fix:
You can now configure the User Script execution log purge interval from the command line by setting a value for: platform.miscellaneous.userScriptTaskPurgeDays in /var/config/rest/config/restjavad.properties.json

998805-4 : Duplicated entries display when viewing Related Items for a particular object

Component: BIG-IQ Configuration - Security - Network Security

Symptoms:
If there are more than three items of a particular type(for example, Address List, Port List, Rules, Rule List) are attached to a Firewall policy on a managed BIG-IP device, BIG-IQ might display duplicate entries in a list when you click Show Related Items.

Conditions:
-- Attach more than 3 items of a particular kind to a Firewall Policy.
-- Select the Firewall Policy from the list and click Show Related Items.

Impact:
BIG-IQ displays duplicate objects.

Workaround:
This issue has no impact on BIG-IQ management functionality.

Fix:
This issue is fixed and you now see the correct number of objects when you click on Show Related Items for a policy.

998377-3 : Discovering BIG-IP devices that contain an interface name with forward slash / in it

Component: BIG-IQ Local Traffic & Management

Symptoms:
BIG-IQ fails to discover BIG-IP devices that contain an interface name with a forward slash in it.

Conditions:
Attempting to discover BIG-IP devices that contain an interface object with a forward slash.

Impact:
BIG-IQ fails to discover the BIG-IP device.

Workaround:
Do not use a forward slash in a BIG-IP device's interface name. If that's not an option, upgrade BIG-IQ version 8.0.0.1.

Fix:
BIG-IQ can now discover BIG-IP devices that contain a forward slash in an interface object's name.

996829-3 : User configured statistics retention might not work correctly for BIG-IQ v7.1.0 or v8.0.0

Component: AppIQ

Symptoms:
User configured analytics statistics retention (real-time, hourly, daily, monthly) might not work as expected in BIG-IQ v7.1.0 after upgrading from 7.0.X, or for BIG-IQ v8.0.0 after upgrading from BIG-IQ 7.1.0. This is setting is configured from BIG-IQ by navigating to System -> BIG-IQ DATA COLLECTION -> BIG-IQ Data Collection Cluster -> Configuration -> Statistics Data Collection -> Configure Retention.

Conditions:
For the issue to occur, the following conditions must be met:
1. Upgrade BIG-IQ v7.0.x to v7.1.0 or upgrade BIG-IQ v7.0.X to v7.1.0 and then to v8.0.0. NOTE: If there was an incremental upgrade from v7.0.x to v7.1.0.1, 7.1.0.2 or 7.1.0.3 this issue will not occur.
2. After upgrading, change the retention under System -> BIG-IQ DATA COLLECTION -> BIG-IQ Data Collection Cluster -> Configuration -> Statistics Data Collection -> Configure Retention. For example, adjust "real-time" to another value.
3. During the upgrade process, restart appiqconfig from BIG-IQ using the command: "bigstart restart appiqconfig"
4.View data in System -> BIG-IQ DATA COLLECTION -> BIG-IQ Data Collection Cluster -> Configuration -> Statistics Data Collection -> Configure Retention and see that the value is not what was previously configured.

Impact:
If default retention values were changed, and the config server was restarted, BIG-IQ reloads the previous retention value or might remove the user-configured retention data.
For example if the retention for raw-data was 10 hours and then was adjusted to 13 hours, then after the appiqconfig restart, the retention will be 10 hours again and 3 oldest indexes of raw data are deleted.

Workaround:
If this issue is recognized according to conditions above then the workaround is as follows:

For BIG-IQ v7.1.0:

1. From the BIG-IQ command line, run the following command:

curl -XPOST -s localhost:9200/metadata_indices_management/meta-data/_delete_by_query -H 'Content-Type: application/json' -d '
{"query":{"bool":{"must":{"match":{"indexFamily":"statistics"}},"must_not":{"term":{"_id":"statistics"}}}}}'

2. Restart the appiqconfig by running the following command:

bigstart restart appiqconfig

For BIG-IQ v8.0.0:

1. From the BIG-IQ command line, run the following command:

curl -XPOST -ks https://localhost:9200/metadata_indices_management/meta-data/_delete_by_query -H 'Content-Type: application/json' -d '
{"query":{"bool":{"must":{"match":{"indexFamily":"statistics"}},"must_not":{"term":{"_id":"statistics"}}}}}'

2. Restart the appiqconfig by running the following command:

bigstart restart appiqconfig

Fix:
Statistics data retained as expected following an upgrade.

996601-7 : Virtual Server, Pool, Pool Member and Node status might not display properly for BIG-IP versions 15.x or 16.x

Component: AppIQ

Symptoms:
BIG-IQ might not properly display the status of Virtual Servers, Pools, Pools Members and Nodes configured on managed BIG-IP devices running versions 15.x and 16.x that have DNS configured.

When this happens, BIG-IQ logs an the following in the /var/log/appiq/agentmanager.log:
java.lang.ArrayIndexOutOfBoundsException: 1

Conditions:
Managing BIG-IP devices running v15.X and 16.x that has one or more Wide-IP configured to the DNS(GTM) configuration with LTM status updates from DCDs enabled on BIG-IQ.

restcurl -X PUT /cm/adc-core/current-config/stats-refresh -d '
{ "isMonitorRunning": true,
"useAppIqDcd": true,
"pollingIntervalSeconds": 300 }'

Impact:
BIG-IQ doesn't properly display the status of LTM objects
on the application dashboards for DNS application services on the Applications > APPLICATIONS screen.

Workaround:
Use one of the following workarounds to fix this issue.

1. Disable the LTM status collection from the DCD and allow status to be collected directly by BIG-IQ. This option might increase the CPU usage, see article K91114310 titled: Reducing the performance impact of BIG-IQ statistics gathering on managed BIG-IP systems at support.f5.com/csp/article/K91114310.

2. Update the analytics iApp RPM. For more information, see article K53001642 titled: Updating the BIG-IQ iApp to prevent BIG-IP performance issues on managed BIG-IP systems and improve DNS statistics collection at https://support.f5.com/csp/article/K53001642

Fix:
The iApp script installed by the BIG-IQ on managed BIG-IP devices now allows the system to display objects as expected.

995837 : BIG-IQ displays the message "Waiting for BIG-IQ services to become available..." after upgrading to version 8.0.0★

Solution Article: K07869171

Component: REST Framework and TMOS Platform

Symptoms:
If you attempt to upgrade BIG-IQ to version 8.0.0 and there are more than 100,000 records for a single task in the BIG-IQ database, the upgrade does not complete successfully.

Conditions:
Upgrading BIG-IQ to version 8.0.0 when its database has more than 100,000 records.

Impact:
BIG-IQ displays a spinner with a message: Waiting for BIG-IQ services to become available...

Workaround:
To work around this issue, see: https://support.f5.com/csp/article/K07869171

Fix:
N/A

991181 : BIG IQ: GUI Search filter for scripts is not always correct

Component: REST Framework and TMOS Platform

Symptoms:
In BIG-IQ, the device script management search bar does not filter scripts for 'All' and 'Criteria Contains' as expected.

Conditions:
Go to the Scripts screen (Devices> SCRIPT MANAGEMENT> Scripts) and perform a text search on device scripts with the 'All' or 'Criteria Contains' filter settings.

Impact:
Search is not displaying results that correspond with the filter setting.

Fix:
Search displays results relevant to the search text and corresponding filter.

989929-5 : Sorting lists by description field

Component: REST Framework and TMOS Platform

Symptoms:
Due to limitations, we disabled the option to sort lists on the description field in BIG-IQ version 6.0.0.

Conditions:
Attempting to sort on description field for a list.

Impact:
Users can't sort lists on the description field.

Workaround:
There is no workaround.

Fix:
Sorting lists by description field is re-enabled.

Behavior Change:
You can now sort on the description field for lists.

985301 : Policy analyzer not authorized for a user with fine grained RBAC on policies

Component: BIG-IQ Web Application Security (ASM)

Symptoms:
Users with fine grained authorization on specific policies will not be able to access the Policy Analyzer feature.

Conditions:
1. create a resource group with 2 policies
2. create role of "web application security manager" + the resource group
3. create a user with this role
4. login with the user and go to Configuration > Security > Web Application Security > Policies.
5. Select a policy and then More. Select the Policy Analyzer option.

Impact:
The policy analyzer screen does not successfully load.

Fix:
The Policy Analyzer screen works as expected for users with fine grained policy authorization.

984869 : User with RBAC permissions to a virtual server cannot see its TCP statistics

Component: AppIQ

Symptoms:
A user with a strict custom role with LTM permissions for a specific virtual server cannot see the virtual server TCP statistics under Monitoring->DASHBOARDS->Local Traffic->TCP.

Conditions:
A user assigned a strict custom role with permissions to view a specific virtual server tries to view the virtual servers' TCP statistics under Monitoring->DASHBOARDS->Local Traffic->TCP.

Impact:
The user won't be able to view the TCP statistics for the virtual servers he is permitted to view.

Workaround:
View the virtual server TCP statistics from an user that is associated with the admin user role.

Fix:
A custom user with LTM permissions can now view TCP statistics.

978241 : Renewing certificates without extensions from BIG-IQ

Component: BIG-IQ Local Traffic & Management

Symptoms:
BIG-IQ cannot renew a certificate imported from a BIG-IP device is it does not have a file name extension (.crt)

Conditions:
-- BIG-IP certificate and key files do not have a file extension
-- Attempt to manage the certificate from BIG-IQ

Impact:
Certificate renewal fails.

Workaround:
Add extensions to certificate and key files and upload them to BIG-IQ.

Fix:
You can now renew certificates that do not contain extensions in the filename.

977109 : BIG-IQ might report errors when deploying changes to BIG-IP devices

Component: BIG-IQ Local Traffic & Management

Symptoms:
BIG-IQ can sometimes display errors similar to the following when deploying changes to BIG-IP devices: 'Exception in verifyConfig: java.lang.NullPointerException'

Conditions:
This happens when BIG-IQ is managing BIG-IP devices with varying versions. It can also occur when a BIG-IP device has been upgraded multiple times.

Impact:
You cannot manage the BIG-IP device from BIG-IQ when that error occurs.

Workaround:
If you manage BIG-IP devices running the same version and you have no changes pending on BIG-IQ, run the following command from BIG-IQ to clear the error status: clear-rest-storage

If you manage several BIG-IP devices of varying versions then you will be unable to work around this issue.

Fix:
This issue is fixed. BIG-IQ no longer displays error messages when deploying BIG-IP changes.

953921 : After upgrading BIG-IQ with DCDs the software versions do not get updated in the DCDs' user interface★

Component: BIG-IQ System User Interface

Symptoms:
After upgrading BIG-IQ with data collection devices (DCDs), the updated version do not display properly.

Conditions:
After upgrading BIG-IQ, from the DCD navigate to -> System -> SOFTWARE MANAGEMENT:

DCDs display the older version of BIG-IQ CM
DCDs display the correct version for themselves

Impact:
DCDs show the old version of the CM

Workaround:
NA

Fix:
This issue is fixed and the proper version now displays for BIG-IQ and DCDs after upgrading.

945389-3 : Audit Log Syslog Servers do not create successfully

Component: BIG-IQ Monitoring - Logs

Symptoms:
Audit Log Syslog Server entries do not create successfully and consequently do not send syslog events to remote syslog server, despite the entry appearing to have been created without issue.

/var/log/restjavad.X.log will log Null Pointer Exceptions whenever an Audit Log Syslog Server entry is created like:

[ERROR][06 May 2021 10:59:23 PDT][ RestServer] java.lang.NullPointerException
        at com.f5.rest.workers.configmgmtbase.auditLogger.syslogServer.SyslogClient.tryAndEstablishConnection(SyslogClient.java:263)
        at com.f5.rest.workers.configmgmtbase.auditLogger.syslogServer.SyslogClient.access$100(SyslogClient.java:39)
        at com.f5.rest.workers.configmgmtbase.auditLogger.syslogServer.SyslogClient$4.run(SyslogClient.java:283)
        at com.f5.rest.common.ScheduleTaskManager$2$1.run(ScheduleTaskManager.java:116)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Conditions:
Audit Log Syslog Server entries are created via GUI.

Impact:
Audit Log Syslog Server objects appear to be created successfully however they will not transmit syslog entries to remote server due to Null Point Exception during their creation.

Workaround:
None.

Fix:
Audit Log Syslog Server objects can now be successfully created without raising a Null Pointer Exception.

A subsequent issue now occurs which is documented in ID1017125 with a workaround of issuing "bigstart restart restjavad" once after the issue occurs.

927621-1 : Rolling upgrade blocked during Data Collection Device cluster upgrade★

Component: BIG-IQ Device Management

Symptoms:
An error occurs during a rolling upgrade of Data Collection Device (DCD) cluster"

"Detected multi-zone environment with insufficient number of DCDs in a single zone"

Conditions:
-- 3 or more DCDs across all zones
-- 2 or more DCDs in a single zone
-- None of the DCDs are in the same zone as the Configuration Management (CM) device

Impact:
You are unable to perform a rolling upgrade

Workaround:
Change the zone of the BIG-IQ CM device from 'default' to another zone by navigating to System-> This Device -> General Properties -> Edit

Impact of workaround: This will restart Elasticsearch and potentially initiate a rediscovery of all shards if a new master is elected.

909205-2 : BIG-IQ statistics reports are missing the latest event data

Component: AppIQ

Symptoms:
BIG-IQ statistics reports are missing the latest event data.

Conditions:
This occurs when the default system-generated Elasticsearch indices are manually deleted by administrators.

This can also happen when the disk becomes full and BIG-IQ runs out of disk space due to accumulating events/logs.

Impact:
This issue results in a loss of statistics data.

Workaround:
The workaround involves repairing the elasticsearch indices storing statistics data.

To identify the statistics that are impacted
    - list all index names & index aliases

     # curl -s localhost:9200/_cat/indices?v
     # curl -s localhost:9200/_cat/aliases?v

     - run below command (or create a periodic task) to detect affected/corrupted elasticsearch cluster indices

      # curl -s localhost:9200/_cat/indices?h=index | grep _writer

     - every index output in above command requires a repair procedure outlined in the remainder of article as the name of index is not expected to have '_writer' suffix (only names of index aliases will have the '_writer' suffix)
  
      lets say it reports youraffectedindex1_writer/youraffectedindex2_writer

    - for every index reported above you can see the size of statistics data that have been accumulated under it (this it the amount of data not getting reported in BIG-IQ GUI)

      # curl -s localhost:9200/_cat/indices | youraffectedindex1_writer
      # curl -s localhost:9200/_cat/indices | youraffectedindex2_writer

If statistics data for impacted/corrupt indices is non-critical (and permanent deletion is acceptable), then please follow procedure A. If you need to preserve the corrupt data proceed to procedure B. However, both procedures repair the impacted elasticsearch indices.

------------
Procedure A
------------

Use this procedure to permanently delete the corrupt data and repair the impacted indices

1. Deactivate the impacted service.

   - on BIG-IQ CM, navigate to System -> BIG-IQ DATA COLLECTION -> BIG-IQ Data Collection Devices

   - deactivate the impacted service (based on the reported indices earlier) for all DCDs

     Ex: Access / DOS Protection / Fraud Protection Service / IPSec / Network Security / Web Application Security

   - wait for 5 minutes to allow existing connections to close

   - on BIG-IQ DCD verify all external connections to special ports are closed:

       # netstat -an | grep -E "9997|8018|8514|8020|8008"

2. Delete the impacted elasticsearch index.

   - on CM (or DCD) remove the incorrect index from elasticsearch, ex:
      # curl -sX DELETE localhost:9200/youraffectedindex1_writer

       Note: above command should return {"acknowledged":true}

    - confirm the index is no longer reported by below command

       # curl -s localhost:9200/_cat/indices | grep _writer

3. Reactivate the service.

   - on BIG-IQ CM, navigate to System -> BIG-IQ DATA COLLECTION -> BIG-IQ Data Collection Devices and activate the impacted services again

------------
Procedure B
------------

Use this procedure to recover the corrupt data and repair the impacted indices

1. Deactivate the impacted service.
   
   see step #1 from Procedure A

2. Create a temporary elasticsearch index.

   - identify naming convention (including a date suffix) by observing output of below command

     # curl -s localhost:9200/_cat/indices?h=index

     ex: youraffectedindex1_2020-xx-xxtxx-xx-xx-xxxx


   - create a temporary index using name identified in above step

     # curl localhost:9200/youraffectedindex1_2020-xx-xxtxx-xx-xx-xxxx -X PUT -d {}

     ex: # curl localhost:9200/afmlogindex_2020-02-01t10-10-10-0100 -X PUT -d {}

3. Reindex data from corrupt index to the temporary elasticsearch index.

   - this may take time depending on size of data in affected index

   - begin the indexing process

      # curl -s localhost:9200/_reindex?wait_for_completion=false -d '{"source":{"index":"youraffectedindex1_writer"},"dest":{"index":"youraffectedindex1_2020-xx-xxtxx-xx-xx-xxxx"}}' | jq .

      ex: # curl -s localhost:9200/_reindex?wait_for_completion=false -d '{"source":{"index":"afmlogindex_writer"},"dest":{"index":"afmlogindex_2020-02-01t10-10-10-0100"}}' | jq .

  - above command will print a task-identifier

    ex: "task": "rur5BcBNTGqdtydEDjHMAA:22687961"

  - use above reported unique task-identifier and repeatedly query the progress of the task until it completes

    # curl -s localhost:9200/_tasks/rur5BcBNTGqdtydEDjHMAA:22687961 | jq .completed
true

   - verify the index status

     # curl -s localhost:9200/_cat/indices?v
     # curl -s localhost:9200/_cat/aliases?v

4. Remove the corrupted elasticsearch index.
   
   - see step #2 in Procedure A.

5. Create a new *_writer alias for the temporary elasticsearch index.

   - run below command to create an alias (for your newly created index) named as the index you just deleted

     # curl localhost:9200/_aliases -X POST -d '{"actions" : [ {"add" : { "index" : "youraffectedindex1_2020-xx-xxtxx-xx-xx-xxxx" , "alias" : "youraffectedindex1_writer" }} ] }'

     ex: # curl localhost:9200/_aliases -X POST -d '{"actions" : [ {"add" : { "index" : "afmlogindex_2020-02-01t10-10-10-0100" , "alias" : "afmlogindex_writer" }} ] }'

   - verify the index status

     # curl -s localhost:9200/_cat/indices?v
     # curl -s localhost:9200/_cat/aliases?v

6. Reactivate the impacted service.
 
   see step #3 in Procedure A.

7. Confirm newly initialized elasticsearch indices.

   - verify the index status

     # curl -s localhost:9200/_cat/indices?v
     # curl -s localhost:9200/_cat/aliases?v

   - confirm that you see a new index named yourimpactedindex1_YYYY_MM_DD and that yourimpactedindex1_writer points to this latest index instance
   - this happens because eventually index are rotated and removed (this also depends on the retention policy)

Fix:
Statistics and event data displays as expected.

906045 : License Manager role cannot assign a purchased pool license

Component: BIG-IQ Device Management

Symptoms:
Users assigned to the built-in License Manager role gets the following error when trying to assign a purchased pool license:

System Unavailable - The system was unable to complete one of the requested tasks (504 Gateway timeout). Try performing this task again.

Conditions:
When a user assigned to the License Manager role tried to assign a license from a purchased pool.

Impact:
Cannot assign a license to BIG-IP.

Workaround:
Use a Registration Key Pool or Utility Pool instead along with the License Manager role if possible. Otherwise, use Administrator role.

828873-4 : Unable to successfully deploy BIG-IP 15.0.0 on Nutanix AHV Hypervisor

Component: REST Framework and TMOS Platform

Symptoms:
In the deployment of BIG-IP 15.0.0 on Nutanix AHV Hypervisor, f5-label service is failing with inappropriate input device error.

Conditions:
Deployment of BIG-IP v15.0.0 on Nutanix AHV Hypervisor.

Impact:
Deployment of BIG-IP v15.0.0 is not stable to log into GUI or terminal on Nutanix AHV Hypervisor.

Workaround:
Steps:

1. Mount the drive:
mount -o rw,remount /usr

2. Add a comment below the line in the '/usr/lib/systemd/system/f5-label.service' service file:
#StandardInput=tty

3. Reload the daemon:
systemctl daemon-reload

4. Restart the service:
systemctl restart f5-label

Fix:
The I/O device has been changed to the default input device '/dev/null' to resolve the issue.

823565-2 : BIG-IQ REST framework JVM process fails intermittently

Component: REST Framework and TMOS Platform

Symptoms:
The BIG-IQ REST framework JVM process (restjavad) fails intermittently when RAM memory is low or during highly intensive memory operations.

Conditions:
This happens when there is not enough RAM available for the normal operation of the BIG-IQ REST framework, especially in environments with a relatively large number of managed BIG-IP devices and related configuration objects, and a high volume of collected statistical data.

Impact:
BIG-IQ management interface and device management workflows are temporarily unavailable while restjavad restarts.

Workaround:
Increase the physical memory (RAM) that is available for BIG-IQ (hardware or virtual).

Fix:
This issue is fixed and restjavad no longer fails under these conditions.

822225 : Cancelling a queued deployment evaluation task

Component: REST Framework and TMOS Platform

Symptoms:
If you're running several deployment evaluations at the same time (or very near in time) and cancel one of the queued evaluations, the cancelled evaluation will get stuck in 'Cancelling' status.

The REST request will deploy something similar to the following:

# restcurl -u admin: mgmt/cm/firewall/tasks/deploy-configuration
{
  "items": [
    {
      "createChildTasks": true,
      "currentStep": "CHECK_LICENSE",
[...]
      "startDateTime": "2019-08-28T15:20:15.962+0100",
      "status": "CANCEL_REQUESTED",
      "type": "Full",
[...]

Conditions:
Cancelling a queued deployment evaluation.

Impact:
This deployment evaluation and all evaluations after it will never be run.

Workaround:
Restart restjavad to change status to FAILED.

752929-4 : SNMP trap sent after standby BIG-IQ in HA configuration is rebooted

Component: BIG-IQ System User Interface

Symptoms:
If a standby BIG-IQ in a high availability (HA) configuration is rebooted, the active BIG-IQ incorrectly generates an SNMP trap OID .1.3.6.1.4.1.3375.20.1.0.5 [managed device unavailable] instead of what it should generate which is OID .1.3.6.1.4.1.3375.20.1.0.3 [peer not available].

Conditions:
Rebooting a standby BIG-IQ in an HA configuration.

Impact:
This incorrect SNMP trap issued can impact BIG-IQ monitoring systems.

Workaround:
None

Fix:
This issue is now fixed and BIG-IQ now sends the correct SNMP trap and message.

1026865 : Data format issues in security policy rules after upgrade to BIG-IQ v8.0★

Component: BIG-IQ SSL Orchestrator

Symptoms:
Some security policy rules get updated to null because of the upgrade scripts for BIG-IQ v8.0. These scripts run when the user upgrades to v8.0 from any previous version.

Conditions:
This issue occurs during an upgrade to v8.0 when there are rules in a security policy.

Impact:
Setting security policy rules to null causes data inconsistency in security policy rules.

If a user does not notice the null/empty values and deploys a configuration change to any of the managed BIG-IP devices, all security policies rules on the managed BIG-IP device will be lost.

Workaround:
After the BIG-IQ is upgraded to v8.0, if there are any security policy rules, use the following workaround:
1. Edit the security policy to remove the rules; then, add them back and redeploy the policy.
2. Deploy a new policy with the same configuration, then attach this policy to topologies that used the old policy.
3. For the impacted BIG-IP, remove the SSLO service from BIG-IQ, then re-discover and import the impacted BIG-IP.

Fix:
This issue is fixed in BIG-IQ v.8.1.0. Upgrading to v8.1.0 will not impact the security policy rules.

1026141 : The DCD can not handle the amount of statistic records sent by a single BIG-IP's avrd

Component: AppIQ

Symptoms:
The DCD can not handle the amount of statistic records sent by a single BIG-IP's avrd, if the amount of statistics exceeds system limit.

Conditions:
A single BIG-IP produces a high volume of statistic records by the avrd module. Adding more DCDs will not help as single BIG-IP can only be connected to single DCD.

Impact:
Agent Manager process on the DCD gets "high water mark" at its source buffer, and new statistics are lost.

Fix:
The DCD can now receive almost twice the amount of statistics rate than before, reducing the likelihood of data loss.

1024253-1 : BIG-IQ: elasticsearch.yml is not backing-up in Qkview

Component: BIG-IQ Device Management

Symptoms:
When attempting to generate and extract a quickview file, the yaml file does not include Elasticsearch configuration information.

Conditions:
Qkview generation is not containing elasticsearch.yml under /var/config/rest/elasticsearch/config/

Impact:
Elastic Search configuration information is missing as elasticsearch.yml is missing from Qkview file under /var/config/rest/elasticsearch/config/ .

Workaround:
Workaround: UCS Backup includes elasticsearch.yml under /var/config/rest/elasticsearch/config/

Fix:
Qkview extraction now includes elasticsearch.yaml as expected.

1023545 : Tenant information report should contain the Obfuscate Data option to remove Tenant information

Component: BIG-IQ Device Management

Symptoms:
Tenant information is not obfuscated in the Utility Billing reports under Devices > License management > Reports > Generate

Conditions:
When running the Utility Billing reports displays Tenant information even if the obfuscate data option is selected.

Impact:
Usage data is not accurately reported causing issues with F5 billing.

Fix:
Tenant information is now obfuscated when obfuscate data option is selected.

1021697 : Requests results from searchd may be out of sync with DB

Component: REST Framework and TMOS Platform

Symptoms:
When discovering devices and importing services in bulk, large configurations might not properly synchronize. If this happens, performing a global search might reveal the item was not updated in the BIG-IQ database.

Conditions:
Large configurations with bulk data insert/update/processing.

Impact:
Global search items might not be synchronized with what is in the BIG-IQ database.

Fix:
Search is no longer out of sync for long periods.

1021149-1 : Elasticsearch may be unhealthy due to too high keepalive tcp values

Solution Article: K68489751

Component: AppIQ

Symptoms:
Elasticsearch display as unhealthy and log messages similar to the following in /var/log/elasticsearch/eslognode.log

2020-07-14T00:04:13,292][INFO ][o.e.d.z.ZenDiscovery ] [xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx] master_left [{xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx}{xxxxxxxxxxx}{xxxxx}{192.0.2.3}{192.0.2.3:9300}{zone=default}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]
[2020-07-14T00:04:13,293][WARN ][o.e.d.z.ZenDiscovery ] [xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx] master left (reason = failed to ping, tried [3] times, each with maximum [30s] timeout), current nodes:

Conditions:
There are long-lived TCP connections on port 9300 between BIG-IQ and data collection devices (DCD). Those connections send keepalive probes after being idle for 299 seconds. If an intermediate device idle-times out this TCP connection in less than 299 seconds, the ES cluster will experience stability problems.

Impact:
ElasticSearch Cluster is not healthy.

Workaround:
See https://support.f5.com/csp/article/K68489751

1017125-5 : Audit Log Syslog Servers do not send messages to remote syslog servers

Component: BIG-IQ Monitoring - Logs

Symptoms:
Audit Log Syslog Server entries will not transmit messages to remote syslog servers despite their configuration and creation appearing successful and the remote syslog server being fully reachable.

/var/log/restjavad.X.log will log constraint messages like:
[INFO][06 May 2021 05:36:09 PDT][/cm/adc-core/audit-logger AdcAuditLoggerCollectionWorker] Did not find own device , continue...
[INFO][06 May 2021 05:36:09 PDT][/cm/global/audit-logger GlobalAuditLoggerCollectionWorker] Did not find own device , continue...
[INFO][06 May 2021 05:36:09 PDT][/cm/shared/audit-logger SystemAuditLoggerCollectionWorker] Did not find own device , continue...
[INFO][06 May 2021 05:36:09 PDT][/cm/firewall/audit-logger FWAuditLoggerCollectionWorker] Did not find own device , continue...
[INFO][06 May 2021 05:36:10 PDT][/cm/access/audit-logger AccessAuditLoggerCollectionWorker] Did not find own device , continue...

Conditions:
Software version containing ID945389 fix (8.0.0 and later or an earlier EHF).

Impact:
BIG-IQ does not transmit messages to remote syslog servers and constantly logs "Did not find own device , continue..." messages to /var/log/restjavad.x.log.

Workaround:
Issue "bigstart restart restjavad" once after observing condition. Messages will cease and syslog entries will successfully be transmitted to valid remote syslog servers.

Fix:
During startup, hostname retrieval is reattempted until it is successfully retrieved and syslog service is now started after that fetch has completed.

1016473 : collected-stats-internal-logging is disabled when BIG-IP is removed from BIG-IQ

Component: AppIQ

Symptoms:
collected-stats-internal-logging becomes disabled when BIG-IP is removed from BIG-IQ, of its status before the BIG-IP device was added.

Conditions:
1. Check what is the default setting on the BIG-IP AFM, and if the collected-stats-internal-logging is enabled.
2. Add BIG-IP to BIG-IQ, discover LTM and AFM services. After adding AFM device into BIG-IQ's managed devices, check if the setting still appears to be as follows:

   [root@C3565120-bigip1:Active:Standalone] config # tmsh list security analytics settings collected-stats-internal-logging
   security analytics settings {
       collected-stats-internal-logging enabled
   }
3.Remove services from BIG-IP in BIG-IQ, then remove BIG-IP from BIG-IQ.
4. Check security analytics settings 'collected-stats-internal-logging' and 'collect-all-dos-statistic'.

Impact:
collected-stats-internal-logging is disabled for BIG-IP AFM.

Fix:
collected-stats-internal-logging status remains as expected, even after BIG-IP removal for BIG-IQ.

1016385 : Adding a standby BIG-IQ for high availability (HA)

Component: REST Framework and TMOS Platform

Symptoms:
When adding a standby BIG-IQ for a HA configuration, it fails with a og messages message simlar to "The Primary and Secondary IP addresses are not of the same type" followed by a message "Reject reentrant transition to status FAILED"

Conditions:
This occurs when the discovery addresses of the high availability (HA) devices are not of the same IP version (for example, mixing IPv4 addresses with IPv6).

Impact:
The mismatched address versions cause the HA pairing to fail, as intended. However, an issue in the error handling code can prevent this failure from being communicated back, making it appear that the process is hung.

Workaround:
You must use either IPv4 or IPv6 for the discovery addresses of high availability (HA) devices, not a mix of both. As long as the versions are consistent, the bad error handling code won't run.

Fix:
The error handling code has been fixed and the error is now reported correctly.

1014061-1 : Cannot create a security policy rule that uses Client IP Subnet Match as a traffic match condition

Component: BIG-IQ SSL Orchestrator

Symptoms:
When you create a security policy rule that uses Client IP Subnet Match as a traffic match condition, BIG-IQ displays the error message: "Rule <XYZ> is missing required field subnet."

Conditions:
1) Configuration>SSL ORCHESTRATOR>Security Policies>Create.
2) Specify the policy name and target device, then click Create Rule.
3) For the traffic match condition, select Client IP Subnet Match, and then specify a static value for the subnet value to match (for example 15.10.0.0/16).
4) When you try to save the rule, BIG-IQ displays: "Rule <XYZ> is missing required field subnet."

Impact:
You won't be able to create the security policy rule using the Client IP Subnet Match condition.

Fix:
When you specify a security policy rule using the Client IP Subnet Match condition, BIG-IQ now saves the rule correctly.

1011941 : Admin users can run arbitrary commands on the BIG-IQ configuration utility

Solution Article: K06024431

1011321 : Refector stats on big-IQ to use more effective DcD API

Component: BIG-IQ Local Traffic & Management

Symptoms:
There is some performance issue between BIG-IQ and DCDs.

Conditions:
This happens when there is a large number of LTM objects (like thousands of VS, Pool, Node, pool-members).

Impact:
BIG-IQ is very slow to load the Device and Application pages.

Fix:
We have added a property "dcdPollerVersion" which is 1 by default. To adopt this change, you must change the property from 1 to 2 by performing the following steps:

1. Get the stats-refresh config (GET https://your-big-iq/mgmt/cm/adc-core/current-config/stats-refresh)
2. Copy the JSON except "generation" and "lastUpdateMicros"
3. Create a PUT query with "dcdPollerVersion" value as 2 (in the JSON saved in step 2)
(PUT https://your-big-iq/mgmt/cm/adc-core/current-config/stats-refresh)

BIG-IQ will now use the enhanced DCD stats-poller task.

1008397 : Grafana vulnerability CVE-2019-15043

Solution Article: K00843201

1005049-2 : DCD cluster stops working after adding a DCD or a standby BIG-IQ to a high availability configuration

Component: AppIQ

Symptoms:
After adding a standby BIG-IQ to a high availability (HA) configuration or while (less likely) adding a DCD, the DCD cluster no longer functions and the restjavad.0.logs shows a message similar to the following:

[HTTP/1.1 503 Service Unavailable]
Open Distro not initialized

Conditions:
Adding a standby BIG-IQ in a high availability (HA) configuration, or (less likely) when adding a DCD.

Impact:
BIG-IQ is unable to receive new analytics statistics and events and cannot display existing events.

Workaround:
After adding the standby BIG-IQ for a high availability (HA) configuration, or a DCD, run the following command on the active BIG-IQ:

export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64 ; /bin/bash -c "/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -h `netstat -nap | grep ':9300 ' | grep LISTEN | awk '{print $4}' | rev | cut -c 6- | rev` --accept-red-cluster -cd /var/config/appiq/elasticsearch/utils/ -icl -nhnv -cacert /var/config/rest/elasticsearch/config/es_root-ca.pem -cert /var/config/rest/elasticsearch/config/es_admin.pem -key /var/config/rest/elasticsearch/config/es_admin-key.pem"

Fix:
This issue is fixed and DCD clusters work as expected after adding a standby BIG-IQ or a DCD.

1004421-3 : ASM policy is removed after LTM virtual server deployment

Component: BIG-IQ Web Application Security (ASM)

Symptoms:
BIG-IQ removes any configured ASM policies related to a BIG-IP virtual server when you deploy a full LTM configuration or deploy a change for LTM on that virtual server with the ASM policies.

Conditions:
LTM and ASM services are provisioned and discovered by BIG-IQ while adding a BIG-IP device.

Impact:
Web application security is disabled on the affected virtual servers.

Workaround:
Redeploy BIG-IP virtual servers licensed for ASM and re-import LTM with the auto-generated ASM policy for the virtual server.

Fix:
BIG-IQ no longer removes ASM policies after you deploy an LTM virtual server.

1002965 : Change in ASM Event Logging format on BIG-IP causing incorrect value displayed in BIG-IQ

Component: BIG-IQ Web Application Security (ASM)

Symptoms:
In BIG-IP 14.0.0 and later the format of the applied blocking mask has changed slightly due to architectural changes. This results in ASM Event Logging logs from 14.0.0 and later BIG-IP devices not always showing the correct Applied Blocking Mask values of Block, Learn and Alarm when reviewing a specific entry.

Conditions:
-- BIG-IP 14.0.0 or later with ASM provisioned
-- Event Logging logs sent to BIG-IQ

Impact:
Block, Learn and Alarm values listed for an entry in BIG-IQ may be incorrect.

Workaround:
The Event Logging entries can be viewed locally on the BIG-IP to determine the correct Applied Blocking Mask values.

1000061-2 : Elasticsearch process in DCDs restarts every few hours

Solution Article: K61201515

Component: AppIQ

Symptoms:
Attempting to aggregate statistics data from BIG-IQ can cause the Elasticsearch (ES) process in the data collection devices (DCDs) to restart every few hours. When this happens, the following exception appears in the elasticsearch/esnode.log file for affected DCDs:

java.lang.NoClassDefFoundError: org/elasticsearch/client/RestClient

Conditions:
Many statistic dimension combinations are defined for a specific module, which might create more unique ES documents than the allowed document limit per BIG-IP (50000 / DCDs num).

For example, a managed BIG-IP device configured with an HTTP module might have traffic from multiple client IPs, with each client IP having multiple destination URLs.

Impact:
When BIG-IQ attempts to aggregate the statistics data under these conditions, the ES process fails and then restarts, resulting in the loss of that aggregated statistics data and pending module events.

Workaround:
Upgrade F5 Elasticsearch plugins.

Fix:
This issue is fixed and the Elasticsearch process no longer restarts every few hours.

• The F5 Networks Technical Support web site: http://www.f5.com/support/
• The AskF5 web site: https://support.f5.com/csp/#/home
• The F5 DevCentral web site: http://devcentral.f5.com/

Generated: Tue Jun 29 10:30:50 2021 PDT

Copyright F5 Networks (2021) - All Rights Reserved