1123685-2 : Occasionally Selinux modules are getting corrupted when the system reboots

Component: F5OS-A

Symptoms:
In rSeries appliances, if Selinux modules are corrupted
-> Virt-handler pod crashes continuously.
-> Tenant will be in pending state.
-> Semodule file size is 0 in dir "/etc/selinux/targeted/active/modules/400/"

Conditions:
If interruption happens during Selinux modules building on system bootup, the interruption can be an abrupt power off.

Impact:
-> Virt-handler pod is crashing continuously.
-> Tenant functionality is impacted.

Workaround:
None.

Fix:
Identify and remove the corrupted Selinux files and rebuild them while the system is booting up.

1121889-1 : ConfD encryption key can lock up the TPM module

Component: F5OS-A

Symptoms:
Due to an error that happens rarely in the HAL layer, the encryption key mechanism can misinterpret such an error as a valid identifier for the system. This causes the TPM to lock up, using that identifier, but then the actual identifier no longer unlocks the TPM.

Conditions:
This happens rarely but when it does, the system-manager cannot read the encryption keys and will not start ConfD.

This will manifest itself as unable to start up the configuration by attempting to become admin.

Impact:
The system is unusable. Installing a new ISO does not help.
The TPM must be cleared to become unlocked. Once the TPM is cleared, a new key is generated so existing encryptions need to be re-encrypted. This is will require that the ConfD system database be reset to default.

Workaround:
The workaround is to do the following:

 # docker exec system_platform-mgr tpm2_takeownership -c
 # docker restart system_manager
 # su admin
 # config
 # (config) system database reset-to-default proceed yes
 # exit; exit
 # docker restart system_api_svc_gateway

Fix:
The incorrect identifier is now ignored and the lockup is avoided.

Note that the fix does not unlock a locked system. The workaround will have to be applied first.

1117649 : rSeries Appliance inoperable after powering down from Linux while configured for Appliance mode

Component: F5OS-A

Symptoms:
If the rSeries device is powered down from Linux (for example, using 'halt -p', 'poweroff', or 'shutdown -h now') while in Appliance mode, the device becomes permanently disabled.

In this state, nothing external can be done to power on the Linux host, for example, cycling power, accessing the LCD Power on option, or pressing the Power button.

Trying to access the AOM menu from the serial console reports the following message:
 AOM Command Menu - disabled for security purposes.

Conditions:
-- Appliance mode is enabled (this is the state the 'appliance-setup-wizard' sets when it runs to completion).

-- The host is powered down (for example, using 'halt -p', 'poweroff', or 'shutdown -h now')

Impact:
The AOM command menu is not available to power on the host. A power cycle of the appliance does not power on the host.

The disabled appliance must be replaced.

Workaround:
***Important!***

If the BIG-IP rSeries appliance is configured for Appliance mode, do not power off the device using commands such as 'halt -p', 'poweroff', or 'shutdown -h now'.

Instead, run 'halt' and then remove power from the system (for example, unplug, remove power brick, remove power from rack).

Note: If you have already encountered this issue, contact F5 Support :: https://www.f5.com/services/support to request an RMA. For more information, refer to K12882: Overview of the F5 RMA process :: https://support.f5.com/csp/article/K12882 .

Fix:
Appliance mode no longer disables the AOM menu, allowing access to power on the host command with console access to the appliance.

1117621-1 : After an appliance upgrade from 1.0.1 to 1.1.1, a tenant in Provisioned state may show inconsistent CLI status★

Component: F5OS-A

Symptoms:
After an appliance upgrade from 1.0.1 to 1.1.1, if the running-state of a tenant is configured in the Provisioned state, the operational status of the tenant may oscillate between "Ready to deploy" and "Allocating resources to the tenant is in progress" state in the partition CLI status.

Conditions:
A race condition exists after an appliance upgrade from 1.0.1 to 1.1.1, that may display an inaccurate tenant operational state when the tenant is configured as Provisioned.

Impact:
The tenant state constantly changes.

Workaround:
Configure the running-state of the tenant to Deployed.

1117417 : Database config restore failed on rSeries appliance

Component: F5OS-A

Symptoms:
System database config-restore will fail when there is mismatch in the system images between when the backup is taken and the current images present on the system.

Conditions:
The current system images that are present on the system (show system image) do not match the list of images that are stored in the backup file.

Impact:
Config restore fails.

Workaround:
Edit the configuration backup file and delete the <image> stanza, from:

    <image xmlns="http://f5.com/yang/system/image">
to
    </image>

Fix:
Configuration restore on rSeries appliances now works regardless of differences in the set of available system software images.

1117277-1 : Occasional issue observed when tenant deployed on r2xxx/r4xxx series

Component: F5OS-A

Symptoms:
The r2xxx/r4xxx appliance interface drivers are not created in time and lead to tenant deployment failure after the PXE boot, live upgrade, reboot, and port profile change.

Conditions:
Live upgrade from any version to v1.1.1 and PXE and on reboot and on port profile change.

Impact:
Occasionally tenant deployment fails to come up.

Workaround:
None

1117237-1 : FPGA bit files are not updated to the latest version after a live upgrade

Component: F5OS-A

Symptoms:
FPGA bit files are not updated to the latest version after a live upgrade.

Conditions:
Live upgrade to an ISO file.

Impact:
Unexpected behavior with tenant and traffic.

Workaround:
Run the following commands from the bash prompt:

1. /bin/systemctl stop appliance_orchestration_manager_container.service

2. /bin/systemctl stop platform-services-deployment.service

3. reboot

Once the system is rebooted, the correct bit files will be installed.

Fix:
Cleaned up the stale/old container volumes before bringing up the new containers.

1116185 : Removing multiple images simultaneously from the webUI causes an error

Component: F5OS-A

Symptoms:
The image removal action handler takes the input and processes it one item at a time. From the CLI/RESTCONF interface, the user can provide one image at a time. But the webUI allows the user to select multiple images and click Delete to remove them in a single click. This was creating a backend handler issue that caused the image agent to crash.

Conditions:
When multiple images are selected and processed for removal from the webUI.

Impact:
In this situation, all subsequent image removal requests cause an error: "Error: application communication failure".

Workaround:
The issue is fixed in F5OS-A 1.2.0. To avoid this situation in other releases of F5OS-A, the user must select one image at a time for deletion from the webUI.

Restarting the image agent service recovers the system from this state.

Fix:
The issue is fixed by improving the image removal process.

1114485 : K3s cluster goes to unhealthy state when system is rebooted after changing hostname.

Component: F5OS-A

Symptoms:
When the system hostname is changed and the system is rebooted, all or some of the following symptoms may be encountered:
-- System-related pods in K3s are stuck in a failure state.
-- The K3s cluster shows more than one node.
-- OMD continuously cores.

Conditions:
The system is rebooted after the hostname is configured in confd.

Impact:
-- K3s cluster goes into an unhealthy state.
-- Tenant functionality is impacted.

Workaround:
None

Fix:
Changing the hostname via confd does not change the system hostname.

Configured hostname is reflected only in the bash and confd prompts.

When no hostname is configured, the bash prompt uses a default PS1 prompt.

1112533 : Status LED color always stays amber

Component: F5OS-A

Symptoms:
The status LED is always amber.

Conditions:
This occurs during normal operation when the status LED should be green.

Impact:
Status LED may not change to green when system is operational.

Workaround:
None

Fix:
Added a diagnostic task that periodically monitors and sets status LED color to green.

1112229 : File download API changes to support file download from the webUI

Component: F5OS-A

Symptoms:
Header information is not effective to download files from the webUI.

Conditions:
X-Auth token is required to download from the webUI.

Impact:
Downloading files from the webUI fails.

Workaround:
None

1112141 : 10G/25G/40G burst support in rSeries appliance

Component: F5OS-A

Symptoms:
When a burst of traffic at 100Gb/s is sent to a 10G/25G/40G port, the burst size supported by the rSeries appliance depends on the buffer size. Once the buffer is full, packets are dropped.

Conditions:
-- Use of 10G/25G/40G ports.
-- A 100Gb/s burst of traffic occurs.

Impact:
This results in loss of egress packets.

Workaround:
None

Fix:
Improved the burst capability on rSeries appliances when 10G/25G/40G ports are used.

1109525 : K3s cluster is unhealthy when the system date or time is changed

Component: F5OS-A

Symptoms:
When the system date is changed, some of the k3s cluster certificates becomes invalid, and pods enter into an unknown/non-operational state.

Once the system date and time are made current, most pods will be recovered.

Some of the virt-controller/virt-operator/virt-api kubevirt pods are in a failed state but tenant functionality is not affected.

Conditions:
System date and time is changed back and forth.

Impact:
Some of the k3s pods go into a failed/non-operational state.

Workaround:
Re-spinning the certificates will restore the pods.
Delete the pods to trigger a re-spin of certificates that are in a terminating or crashed state.
The orchestration manager will start the pod with a new certificate.

Command to delete the pod:

#kubectl delete pod <pod-name> -n <name-space>

Fix:
Only change the system date and time when necessary.

1109029 : Host Logs in F5OS-A not being rotated

Component: F5OS-A

Symptoms:
Log files under /var/log in host-os were able to grow in GBs.

Conditions:
Log files under /var/log not added in logrotate.

Impact:
Size of log files will grow in GBs, which will consume a significant amount of hard disk space.

Workaround:
N/A

Fix:
Host Logs in F5OS-A are now being rotated as expected.

1109021 : CLI commands are not logged in audit.log

Links to More Info: BT1109021

Component: F5OS-A

Symptoms:
CLI commands from ConfD are not getting logged in audit.log.

Conditions:
Execute commands using the ConfD CLI.

Impact:
CLI commands which are required for security compliance audit will not get logged in audiit.log file.

Workaround:
None

1105001-2 : Large tar/gz/iso file download via the restconf API fails.

Links to More Info: BT1105001

Component: F5OS-A

Symptoms:
Downloading large tar/gz/iso files using the restconf API results in a corrupted file.

Conditions:
Large tar/gz/iso file download via the restconf API.

Impact:
Download fails, the downloaded file is corrupted.

Workaround:
None

Fix:
Fixed the code to download large tar/gz/iso files.

1104569-1 : On upgrading, the correct webUI changes are not reflected

Component: F5OS-A

Symptoms:
On upgrading from one F5OS-A version to another, the appropriate webUI changes are not reflected, and the older changes still persist.

Conditions:
Upgrading from one F5OS-A version to another.

Impact:
Appropriate changes with respect to the version are not reflected on the webUI.

Workaround:
Refreshing the containers is a known workaround using /usr/libexec/platform-deployment stop and /usr/libexec/platform-deployment start.

Fix:
The fix is to clean up stale volumes so that new volumes are mounted after system reboot. A --remove-orphans flag was added to docker-compose down to remove volumes which were created in the previous run of docker-compose. Also, appliance_orchestration_manager was called to stop separately, as it was using a volume called config_vlogsev, which was also being used by other containers, because it is not a part of the platform.yml file. Also docker-compose down has been added before starting service in the beginning of the platform-deployment service to handle the scenario of upgrading from broken ISO to fix ISO.

1103001-1 : Tenants fail to come up after a live upgrade from pre-1.1.0 version to 1.1.0 on the r4xxx appliances★

Component: F5OS-A

Symptoms:
When a live upgrade is attempted from a pre-1.1.0 release to a 1.1.0 release on the r4xxx series of appliances, the tenants will not come up after the live upgrade.

The symptoms that will be seen are:
ICE driver may not load ( "lsmod | grep -i ice" will not show a line with 'ice' ), no VFs will be created, tenant deployment will fail.

Conditions:
-- An F5OS upgrade is performed on an r4xxx series appliance to version 1.1.0
-- The appliance was running pre-1.1.0 software
-- A license is installed
-- Tenants are attempted to be deployed

Impact:
Tenant deployment fails after live upgrade as the ICE driver is not loaded.

Workaround:
After the live upgrade, check that the tenant is failing to deploy.
Check that "lsmod | grep -i ice" does not show a line with 'ice'
reboot the system
Now rerun lsmod again. This should show the ice module line.

Fix:
Fixed in all versions after 1.1.0.

1101365-1 : Delay in tenant deployment with tenant image corruption error

Component: F5OS-A

Symptoms:
The system posts an intermediate error message:

Tenant image corrupted - Update the tenant config with proper image.

This error auto-recovers within 20 seconds.

Conditions:
Observed intermittently while bringing up the tenant.

Impact:
There is a delay in tenant deployment with an intermediate error on the CLI console.

Workaround:
None

1101237 : When configured for SNMP, the system does not properly report a sysObjectID for the F5OS system

Component: F5OS-A

Symptoms:
F5OS systems may not be detected by SolarWinds or other management systems due to the wrong sysObjectID configuration in SNMP.

Conditions:
SNMP

Impact:
F5OS systems may not be detected by SolarWinds or other management systems due to the wrong sysObjectID configuration in SNMP.

Fix:
The sysObjectIDs are correct now.

1100305-1 : Tcpdump capture of packets with interface-based filtering fails on r5000 and r10000 appliances

Component: F5OS-A

Symptoms:
On r5000 and r10000, running a tcpdump as follows:
appliance-1# system diagnostics tcpdump -nni 1.0

to filter packets traversing interface 1.0 only, will fail.

The error seen will be "errbuf ERROR:Interface configuration failed. Please retry tcpdump: pcap_loop: Interface configuration failed. Please retry."
and the client will terminate.

Retrying the client will not help, contrary to the message.

Conditions:
Tcpdump capture is started on an r5000 and r10000 device and the option to filter packets based on an interface ("-i" option) is chosen.

Impact:
Tcpdump cannot work in the interface filtering mode.
It will operate in the other modes; only the interface filtering option causes it to be unable to start.

Workaround:
1) Start a tcpdump capture with no interface filter
"system diagnostics tcpdump" or
"system diagnostics tcpdump -nni 0.0"

Packets will be captured from all interfaces, and further (non-interface) filters can be used to narrow down capture
For example:
"system diagnostics tcpdump host 1.1.1.1 and port 80" or
"system diagnostics tcpdump vlan 200"

2) Restart the tcpdump container. This would make the -i option available again.

1099437 : Nic-manager core file

Component: F5OS-A

Symptoms:
During a power down sequence the l2-agent may generate a core file. The system comes back up without any issue.

Conditions:
System power loss.

Impact:
Core file is generated.

Workaround:
None

Fix:
A fix has been added to detect and prevent creating an l2-agent core file during a power down.

1097925-2 : Resolving CVEs on F5OS-A 1.1.0

Component: F5OS-A

Symptoms:
F5OS-A 1.1.0 is vulnerable to the CVEs mentioned in the bug.
CVE-2021-27219
CVE-2021-43527
CVE-2022-23852
CVE-2020-10531
CVE-2022-24407
CVE-2018-1000805
CVE-2021-44142
CVE-2020-12321
CVE-2020-24489
CVE-2021-42574
CVE-2020-8625

Impact:
F5OS-A 1.1.0 is vulnerable to the CVEs mentioned in the bug.

1097833-2 : Debug messages logged in platform.log

Links to More Info: BT1097833

Component: F5OS-A

Symptoms:
When performing an ISO install on the hardware, some services log debug messages to platform.log until ConfD comes up.

Conditions:
This occurs during an ISO install.

Impact:
Unnecessary debug logs are logged to platform.log.

Workaround:
None

1092257-1 : Downloading files larger 500 megabytes via File Utilities in the webUI can result in a corrupted file.

Links to More Info: BT1092257

Component: F5OS-A

Symptoms:
When using the File Utilities feature in the webUI, if you select a file that is larger than 500MB and attempt to download it locally the file could become corrupted.

Conditions:
Selecting and attempting to download a file 500MB or greater when using File Utilities in the webUI.

Impact:
The downloaded file is corrupted.

Workaround:
Files 500 megabytes or larger in size can be selected and exported from the device using the "Export" option available in File Utilities that will export the file over HTTPS. Additionally, files can be exported from the device using the Secure Copy Protocol (SCP).

Fix:
Downloading files that are 500 megabytes or larger in size has been temporarily disabled in the webUI. You will receive a warning popup when you select a file that is 500MB or larger and click the Download button. The warning popup advises you to use another supported option to export the file.

1091641-1 : NTP (chrony) packet authentication is not fully implemented on VELOS

Links to More Info: BT1091641

Component: F5OS-A

Symptoms:
It is not possible to enable NTP packet authentication.

Conditions:
Running a version of F5OS-C earlier than 1.5.0.

Impact:
NTP packet authentication is not available.

Workaround:
None

Fix:
Added support for NTP packet authentication.

1090753-1 : NSO and ASW XBAR packet drops on 10G, 25G, and 40G interfaces.

Component: F5OS-A

Symptoms:
Unexpected egress packet drops can be seen in XBAR for 10G, 25G, and 40G ports.

This is a packet burst congestion issue that overflows XBAR egress buffers. The issue is seen mainly on 10G ports.

Conditions:
This issue can happen when customers are using the 10G, 25G, or 40G front panel interfaces.

Impact:
The impact is egress packets dropped at 10G, 25G, and 40G front panel interfaces.

Workaround:
The workaround for this issue is to upgrade to NSO bitfile version nso_1ST210EU2F50E2VG_v70.2.10.11_d22.06.23.00.bit and ASW bitfile version asw_1ST280EU2F50E2VG_v71.2.12.11_d22.06.15.00.bit. These bitfiles and newer include added packet buffer memory in the XBAR.

The added packet buffer memory greatly improves the packet drop issue, but does not resolve it completely. In testing, packet drops were still seen as system throughput approached 190Gb.

Fix:
The initial fix for this issue is to add memory to the 10G, 25G, and 40G output buffers. Memory was increased from 4Mb to 8Mb.

The added packet buffer memory greatly improves the packet drop issue, but does not resolve it completely. In testing, packet drops were still seen as system throughput approached 190Gb.

1090521-2 : Tenant deployment may fail if the memory configured is an odd number.

Links to More Info: BT1090521

Component: F5OS-A

Symptoms:
1. Tenant deployment fails.
2. System may go into bad state.

Conditions:
When memory configured for a tenant is set to an odd number.

Impact:
Tenant deployment fails.

Workaround:
This issue has been fixed in F5OS-A 1.2.0.

1090089-2 : NTP service does not work on rSeries appliances

Component: F5OS-A

Symptoms:
The NTP service does not work on rSeries appliances that run F5OS-A.

Running chronyc ntpdata returns "501 Not authorized"

Conditions:
-- rSeries appliance running F5OS-A
-- NTP configured

Impact:
NTP functionality does not work.

Workaround:
Change directory ownership to chrony using below command:

chown chrony:chrony /var/run/chrony

Fix:
Update ownership for "/var/run/chrony" directory and removed unwanted configuration from "chrony.conf".

1089721-1 : Prefix length support to allow multiple IP addresses

Component: F5OS-A

Symptoms:
Prefix length was not supported previously, and users had to configure one IP per command in order to support multiple source IPs.

Conditions:
Always

Impact:
Multiple source IP addresses cannot be allowed using a single command.

Workaround:
NA

Fix:
This improvement supports the configuration of prefix length to allow multiple IP addresses using a single command.
Prefix length is an additional parameter in the existing configuration command.

1083077-2 : LACP trunks are not configured automatically in BIG-IP tenant running on F5OS chassis/appliances

Component: F5OS-A

Symptoms:
When an LACP trunk is configured on an F5OS chassis/appliance and only the native VLAN is attached, the LACP trunk will not be automatically configured on the BIG-IP tenant.

Conditions:
This behavior is observed only when the LACP trunk is attached to a native VLAN.

Impact:
LACP trunk configuration will not be applied to the BIG-IP tenant automatically when only a native VLAN is attached to it on the platform.

Workaround:
Configure the LACP trunk in the BIG-IP tenant manually.

Fix:
LACP trunks are now configured automatically in BIG-IP tenant running on F5OS chassis/appliances, as expected.

1075361 : Messages log has a very high number of "error" and "fail" entries

Component: F5OS-A

Symptoms:
During system bring up/reboot, various fail and error logs are seen from multiple software components.

Conditions:
During system boot up or if we perform multiple reboots we may see various errors/failures in log messages.

Impact:
User will see error/fail messages, while System bring up/reboot.

Workaround:
N/A

Fix:
Fixed the error/fail logs for few components.

1072209-2 : Packets are dropped on VELOS when a masquerade MAC is on a shared VLAN

Links to More Info: BT1072209

Component: F5OS-A

Symptoms:
On the VELOS platform, any packets destined to a masquerade MAC address are dropped when the masquerade MAC is located on a shared VLAN (a VLAN shared between multiple F5OS tenants).

On rSeries hardware platforms, all traffic for this MAC is first handled by the software-rebroadcaster and is replicated to all tenants sharing that VLAN.

Conditions:
-- A masquerade MAC is configured on a shared VLAN.
-- Traffic to the MAC address is initiated, that is, ping a floating self-IP.
-- The packets are dropped on ingress.

Impact:
Connectivity issues.

Workaround:
Configure a static FDB entry at the partition level.

Fix:
Packets are no longer dropped when a masquerade MAC is on a shared VLAN.

1066185-1 : MIB files cannot be download or exported using file utilities.

Component: F5OS-A

Symptoms:
MIB directory is not available for download or export file utilities

Conditions:
-- VELOS chassis or rSeries appliance
-- You would like to download the MIB file(s) via the file utilities API

Impact:
You are unable to download the MIBs or export them.

Workaround:
None

1061149-1 : Libvirt core is generated on system reboot

Component: F5OS-A

Symptoms:
A flawed core file is generated on system reboot intermittently. But the tenant is actually healthy and functional after reboot.

Conditions:
Intermittently on system reboots.

Impact:
A libvirt core file is generated, but the tenant is actually healthy and functional.

Workaround:
None

Fix:
No impact on functionality. No user action is expected.

• The F5 Networks Technical Support web site: http://www.f5.com/support/
• The AskF5 web site: https://support.f5.com/csp/#/home
• The F5 DevCentral web site: http://devcentral.f5.com/

Generated: Tue Dec 6 22:27:54 2022 UTC

©2022 F5, Inc. All rights reserved.