995769 : CVE-2018-20060: python vulnerability

Component: F5OS-C

Symptoms:
urllib3 before version 1.23 does not remove the Authorization HTTP header when following a cross-origin redirect (i.e., a redirect that differs in host, port, or scheme). This can allow for credentials in the Authorization header to be exposed to unintended hosts or transmitted in cleartext.

Conditions:
urllib3 before version 1.23 does not remove the Authorization HTTP header when following a cross-origin redirect (i.e., a redirect that differs in host, port, or scheme). This can allow for credentials in the Authorization header to be exposed to unintended hosts or transmitted in cleartext.

Impact:
urllib3 before version 1.23 does not remove the Authorization HTTP header when following a cross-origin redirect (i.e., a redirect that differs in host, port, or scheme). This can allow for credentials in the Authorization header to be exposed to unintended hosts or transmitted in cleartext.

Workaround:
N/A

Fix:
N/A

995649 : CVE-2018-16402: libelf vulnerability

Component: F5OS-C

Symptoms:
libelf/elf_end.c in elfutils 0.173 allows remote attackers to cause a denial of service (double free and application crash) or possibly have unspecified other impact because it tries to decompress twice.

Conditions:
libelf/elf_end.c in elfutils 0.173 allows remote attackers to cause a denial of service (double free and application crash) or possibly have unspecified other impact because it tries to decompress twice.

Impact:
libelf/elf_end.c in elfutils 0.173 allows remote attackers to cause a denial of service (double free and application crash) or possibly have unspecified other impact because it tries to decompress twice.

Workaround:
N/A

995645 : CVE-2019-9636: python vulnerability

Links to More Info: K57542514, BT995645

995633 : CVE-2019-10160: Python vulnerability

Component: F5OS-C

Symptoms:
A security regression of CVE-2019-9636 was discovered in python, since commit d537ab0ff9767ef024f26246899728f0116b1ec3, which still allows an attacker to exploit CVE-2019-9636 by abusing the user and password parts of a URL. When an application parses user-supplied URLs to store cookies, authentication credentials, or other kind of information, it is possible for an attacker to provide specially crafted URLs to make the application locate host-related information (e.g. cookies, authentication data) and send them to a different host than where it should, unlike if the URLs had been correctly parsed. The result of an attack may vary based on the application.

Conditions:
A security regression of CVE-2019-9636 was discovered in python, since commit d537ab0ff9767ef024f26246899728f0116b1ec3, which still allows an attacker to exploit CVE-2019-9636 by abusing the user and password parts of a URL. When an application parses user-supplied URLs to store cookies, authentication credentials, or other kind of information, it is possible for an attacker to provide specially crafted URLs to make the application locate host-related information (e.g. cookies, authentication data) and send them to a different host than where it should, unlike if the URLs had been correctly parsed. The result of an attack may vary based on the application.

Impact:
A security regression of CVE-2019-9636 was discovered in python, since commit d537ab0ff9767ef024f26246899728f0116b1ec3, which still allows an attacker to exploit CVE-2019-9636 by abusing the user and password parts of a URL. When an application parses user-supplied URLs to store cookies, authentication credentials, or other kind of information, it is possible for an attacker to provide specially crafted URLs to make the application locate host-related information (e.g. cookies, authentication data) and send them to a different host than where it should, unlike if the URLs had been correctly parsed. The result of an attack may vary based on the application.

Workaround:
N/A

995597 : CVE-2018-15688: systemd Vulnerability

Component: F5OS-C

Symptoms:
It was discovered that systemd-network does not correctly keep track of a buffer size when constructing DHCPv6 packets. This flaw may lead to an integer underflow that can be used to produce a heap-based buffer overflow. A malicious host on the same network segment as the victim's one may advertise itself as a DHCPv6 server and exploit this flaw to cause a Denial of Service or potentially gain code execution on the victim's machine.

Conditions:
It was discovered that systemd-network does not correctly keep track of a buffer size when constructing DHCPv6 packets. This flaw may lead to an integer underflow that can be used to produce a heap-based buffer overflow. A malicious host on the same network segment as the victim's one may advertise itself as a DHCPv6 server and exploit this flaw to cause a Denial of Service or potentially gain code execution on the victim's machine.

Impact:
It was discovered that systemd-network does not correctly keep track of a buffer size when constructing DHCPv6 packets. This flaw may lead to an integer underflow that can be used to produce a heap-based buffer overflow. A malicious host on the same network segment as the victim's one may advertise itself as a DHCPv6 server and exploit this flaw to cause a Denial of Service or potentially gain code execution on the victim's machine.

Workaround:
N/A

991917 : F5OS: Controller/partition needs the ability to set and display a system hostname.

Component: F5OS-C

Symptoms:
System hostname is missing in operational data (state data).

For example: Even after configuring the system hostname, it is not visible when you submit the command: "show system state hostname"

syscon-2-active# show system state hostname
% No entries found.

Conditions:
1. Configure hostname in config mode using the CLI command: "system config hostname <name>".
2. Try to see the configured hostname using the CLI command: "show system state hostname".

Impact:
Hostname is not visible in state info.

Workaround:
Check for the configured hostname using the system controller's bash prompt or by checking running config of system ("show running-config system config hostname")

Fix:
Now hostname now displays when you use the CLI command: "show system state hostname."

Behavior Change:
"show system state hostanme" now gives a valid response and displays the current set hostname.

991061 : Admin cannot edit the tenant config in Deployed state from GUI if the tenants are created via CLI

Component: F5OS-C

Symptoms:
Tenant validations are not working when a tenant is created using the CLI and subsequently edited in the GUI.

Conditions:
Admin creates the tenant via CLI and subsequently edits the following tenant config when the running-state is set to Deployed:

-- Scale-up/Scale-down the tenant.
-- Add/Remove VLAN.

Impact:
Admin cannot scale up/scale down the cluster using the GUI if the tenant is initially deployed via CLI.

Workaround:
Use the CLI to scale-up/scale-down and add/remove the VLAN to the tenant.

989461 : CVE-2020-29573 glibc: stack-based buffer overflow if the input to any of the printf family of functions is an 80-bit long double with a non-canonical bit pattern

Links to More Info: K27238230, BT989461

989189 : CVE-2019-18282: Linux kernel vulnerability

Links to More Info: K32380005, BT989189

979249 : Nodes are displayed in the tenant instance IDs table even after removing them from the tenant

Links to More Info: BT979249

Component: F5OS-C

Symptoms:
When running the following command in confd:

 show running-config tenants tenant dag-tenant config nodes

One of the fields displayed is tenant-instance-ids. The ID is displayed even after deleting the tenant instance.

Conditions:
Add a tenant and then delete it.

Impact:
The tenant instance ID is still displayed. This is cosmetic and there is no functional impact.

Workaround:
None

Fix:
The active tenant list is now displayed properly.

951633 : qkview Hardening

Component: F5OS-C

Symptoms:
Under certain conditions, qkview does not follow current best practices.

Conditions:
Occurs while running qkview.

Impact:
Under certain conditions, qkview does not follow current best practices.

Workaround:
N/A

950477 : USB device presence causes errors in the blade log

Links to More Info: BT950477

Component: F5OS-C

Symptoms:
When a USB device is present in the blade, the VELOS.log contains a large number of errors from platform-hal related to the USB device and attempts to detect it.

Conditions:
USB device is present in the blade.

Impact:
Numerous unnecessary messages appear in the log.

Workaround:
These messages are benign, and you can safely ignore them.

950109 : Interface 'in-discards' counter not reset

Links to More Info: BT950109

Component: F5OS-C

Symptoms:
If you issue a reset counters command, the in-discards counter is not reset to 0.

Conditions:
Issue 'reset counters interfaces <interface>' or 'reset counters all' commands.

Impact:
Counter is not reset to 0.

Workaround:
None

1073017 : Downgrading controller software from 1.3.0 can sometimes leave platform services in degraded state

Links to More Info: BT1073017

Component: F5OS-C

Symptoms:
After downgrading the controller OS from 1.3.0 to 1.2.X, the system can end up in a state where previously imported software is no longer imported correctly and various platform services fail to start.

Conditions:
1. Upgrade controller OS to 1.3.0.
2. Downgrade controller OS back to an earlier version.

Impact:
System usability is impacted by a nominally supported downgrade path.

Workaround:
If a system is downgraded and encounters this issue, it may be necessary to rebuild the Openshift cluster on the affected controllers. If the system is still in a bad state after a cluster rebuild, further manual intervention by an SE may be required to restore the system to a healthy state.

Fix:
Downgrading Velos controllers from 1.3.X to earlier versions no longer results in missing imports and platform service start failures.

1072597 : Openshift cluster health can toggle between Ready and Not Ready when cluster health is not good.

Links to More Info: BT1072597

Component: F5OS-C

Symptoms:
orchestration_manager can report the cluster status as toggling between Ready and Not Ready when the cluster health is not good.

Conditions:
This can happen if bug 1071673 is encountered during upgrade. There may be other conditions that can cause the issues.

Impact:
The toggling will cause orchestration_manager to not label blades correctly if they are moved between partitions, or re-installed. It can also cause new partition namespaces to not be created in openshift.

Workaround:
N/A

Fix:
orchestration_manager has been updated so the cluster will correctly be marked as not ready when there are issues with the openshift pods.

1071805 : Removing controller images used for bare metal install can cause Openshift failures after upgrade

Links to More Info: BT1071805

Component: F5OS-C

Symptoms:
Some Openshift pods are stuck in an 'ImagePullBackOff' state, degrading system operation.

Conditions:
1. The ISO image used for bare metal install of controllers is removed.
2. The ISO image remove in step (1) is version 1.1.4 or earlier.
3. The currently running version of controller software is 1.2.X.
4. An upgrade is initiated to 1.3.0.

Impact:
System performance is degraded until manual intervention is taken.

Workaround:
To workaround this issue, the Openshift cluster on the affected controllers must be re-built.

Fix:
Removing controller images used for bare metal install no longer causes Openshift failures after upgrade in certain cases

1071693 : Kubevirt pods may not upgrade correctly on upgrade from 1.2.1 to 1.3.0★

Links to More Info: BT1071693

Component: F5OS-C

Symptoms:
During a controller upgrade from 1.2.1 to 1.3.0, the kubevirt pods fail to upgrade and produce an error that it cannot find the correct image in the repository.

Conditions:
The kubevirt images are not available on the controller registry.

Impact:
The tenant will not operate properly since it relies on kubevirt pods to be installed correctly.

Workaround:
Follow these steps on the active controller

cp omd-kubevirt-velos-install.sh /tmp

cd /tmp
vi omd-kubevirt-velos-install.sh

Add these lines at line 99 of that file

else
    echo "Using registry port of $official_port for kubevirt install"
    echo "Update registry port in kubevirt yml files"
    sed -i -e "s@:[0-9][0-9][0-9][0-9]/@:$official_port/@" $WORKDIR/kubevirt-velos.yaml

In one linux shell window:
oc delete -f /usr/share/omd/kubevirt/kubevirt-velos.yaml

In another linux shell window:
./tmp/omd-kubevirt-velos-install.sh /usr/share/omd/kubevirt/

1071673 : Openshift registry console pod cab gets stuck in ImagePullBackoff after upgrade to 1.3.0★

Links to More Info: BT1071673

Component: F5OS-C

Symptoms:
The openshift registry_console pod can get stuck in ImagePullBackoff after upgrading to version 1.3.0

This can be seen in the output of "oc get pods".

Conditions:
Rolling upgrade from 1.1.0->1.2.x->1.3.0

Impact:
Openshift will not work correctly because not all pods in the cluster will be running correctly. This will keep tenants from launching correctly.

Workaround:
On the active system controller:

touch /var/omd/CLUSTER_REINSTALL

This will cause the openshift cluster to be re-installed using the current image registry.

Fix:
1.3.1 release has been updated to include the old and new paths to the openshift pods in the registry.

1068517-1 : Software rebroadcaster is dropping all packets, 'rx_drops_no_producer'

Links to More Info: BT1068517

Component: F5OS-C

Symptoms:
Inbound ARP broadcasts on VLANs shared by the tenants are not received.

Conditions:
A high volume of DLF packets are handled by the software rebroadcaster.

Impact:
Loss of connectivity on VLANs shared among tenants.

Workaround:
Restart the sw_rbcast container on the affected blade:

# docker restart partition_sw_rbcast

Fix:
Use asynchronous messages to fpgamgr for DLF lookup to prevent the ZMQ socket from filling up.

1065085-1 : MD5 cipher is allowed on RESTCONF port 8888 with FIPS enabled license

Links to More Info: BT1065085

Component: F5OS-C

Symptoms:
When the System is installed with a FIPS enabled license, some of the MD5 ciphers are still allowed on RESTCONF port 8888 which is supposed to be disallowed.

Conditions:
The command "openssl s_client -connect <mgmt-ip>:8888 -cipher MD5" returns a valid certificate.

Impact:
MD5 SSLCipher continues to work on port 8888 on both system controller and partition mgmt-ips.

Workaround:
None

Fix:
Removed MD5 SSLCipherSuites from ssl.conf when FIPS enabled license is installed in the system.

1061757 : VLAN Listener for a VLAN shared between tenants may not upgrade properly★

Links to More Info: BT1061757

Component: F5OS-C

Symptoms:
After upgrading from 1.1.4 to a 1.2 release when there are tenants configured that share VLANs, the VLAN listener is not properly upgraded.

Conditions:
Tenants sharing VLANs in a configuration that is upgraded from 1.1.4 to 1.2.x.

Impact:
Traffic will not pass correctly.

Workaround:
Remove the VLAN from the interface(s) and then add it back (no changes to the tenant are necessary).

This re-creates the vlan-listener with the correct VTC value.

1061065 : After controller upgrade, tenant may not work correctly due to failed install of kubevirt Pods.★

Links to More Info: BT1061065

Component: F5OS-C

Symptoms:
After a controller upgrade, the kubevirt Pods that are part of that upgrade can fail to install correctly so the tenant will not deploy.

Conditions:
-- Controllers were recently upgraded.
-- Kubevirt Pods not installed correctly

Impact:
Tenant will not deploy correctly.

Workaround:
1. remove existing kubevirt Pods that are incorrectly installed.
2. Manually edit the kubevirt-velos-install.sh script to point to the correct registry port.
3. Rerun the install script to install the kubevirt Pods
 correctly.

Fix:
The kubevirt-velos-install.sh script is updated to the correct registry port, which allows the kubevirt Pods to be updated correctly.

1060417 : Tpm-integrity-status is "Unavailable" for standby controller, but tpm-status reports "Valid"

Links to More Info: BT1060417

Component: F5OS-C

Symptoms:
The tpm-integrity-status is "Unavailable" for the standby controller, but tpm-status reports "Valid".

Conditions:
This is encountered when checking TPM status:
syscon-2-active# show components component controller-* state tpm-integrity-status TPM
              INTEGRITY
NAME STATUS
---------------------------
controller-1 Unavailable
controller-2 Valid

Impact:
Wrong tpm-status will be displayed on confD.

Workaround:
Restart vcc-chassis-manager container.

From the root prompt of the system controller:
[root@controller-1 ~]# docker restart vcc-chassis-manager

Fix:
Issue is fixed in latest release. We are checking tpm-status in regular interval and updating correct information in confD.

1060405 : Management-address is incorrectly displayed in lldp neighbor information

Links to More Info: BT1060405

Component: F5OS-C

Symptoms:
The 'show lldp' command displays the management-address of the neighbor incorrectly.

Conditions:
-- lldp enabled
-- Run the 'show lldp' command

Impact:
Management-address of the neighbor is shown incorrectly. It is the display issue, there is no functional impact.

Workaround:
None

1059209 : No tenant config attributes are allowed after 'storage size'

Links to More Info: BT1059209

Component: F5OS-C

Symptoms:
While configuring the tenant in the one-line command, you are unable to give any other parameters after the storage size parameter. The storage size should be given at the end of the command only.

Conditions:
Preferring the storage command early while configuring the tenant in one-line command.

Impact:
Commands fail as invalid input if any other parameters are mentioned after storage size.

Workaround:
Place the storage size parameter at the end of command or split the config into multiple lines.

1058757 : Optical transceiver OPT-0043 reports unknown as media type

Links to More Info: BT1058757

Component: F5OS-C

Symptoms:
"show portgroups" reports unknown for the media type for an OPT-0043

Conditions:
OPT-0043 transceiver plugged into a system

Impact:
Cosmetic - this has no functional impact. The media field is not used by any software, it is reported as information for the user.

Workaround:
None

Fix:
OPT-0043 now reports media type as "40G BiDi"

1055841 : Chassis component alarm LED shows up on active controller

Links to More Info: BT1055841

Component: F5OS-C

Symptoms:
Chassis component alarm LED shows up on the active controller instead of the LCD module.

Conditions:
If a chassis component, such as a PSU, generates an alarm, the RED alarm LED would show up on the active controller instead of the LCD module

Impact:
A RED alarm LED could indicate a controller problem instead of a chassis component problem.

Workaround:
None.

Fix:
Chassis component alarms, such as from a PSU, now generate a RED alarm LED on the chassis instead of the active controller.

1055397 : Platform registry ports could become mismatched depending on import timing

Links to More Info: BT1055397

Component: F5OS-C

Symptoms:
Under certain conditions, it is possible for the platform registry port configuration to become mismatched between the two system controllers. This can lead to a number of cascading issues with tenant deployments later.

Conditions:
If a platform image import succeeds on one system controller and fails on the other, or a sync of multiple images leads to them being imported in a different order on the standby system controller compared to the active, it is possible to encounter this scenario.

Impact:
Tenants that reference a version of imported software with mismatched ports may attempt to pull images from the wrong registry port, resulting in tenant failure or starting up with the wrong version of platform software images.

Workaround:
It is possible to fix the port mismatch by removing and re-importing the images with mismatched port assignments.

Fix:
Fixed issue where platform registry ports could become mismatched depending on import timing

1055329 : VLAN shared between two tenants may not pass traffic to tenant with non-default CMP hash

Links to More Info: BT1055329

Component: F5OS-C

Symptoms:
If two tenants on a VELOS chassis are configured with a shared VLAN, one tenant may not pass traffic if it has a non-default CMP hash configured for that VLAN.

Conditions:
-- VELOS chassis
-- Configure a VLAN shared between two or more tenants
-- In one tenant, configure a non-default CMP hash for the VLAN

Impact:
No connectivity.

Workaround:
After configuring a non-default cmp hash, run
`docker restart partition_sw_rbcast`
on each blade.

Fix:
Fixed operation of shared vlan when cmp hash is not the default.

1055189-1 : Optical transceiver tuning values for OPT-0048 updated to reduce errors

Links to More Info: BT1055189

Component: F5OS-C

Symptoms:
OPT-0048 may show intermittent errors

Conditions:
OPT-0048 optical transceiver inserted into r10000 or r5000 appliance

Impact:
intermittent optical transceiver errors

Workaround:
None

1054837 : Vcc-ConfD may fail to start new child process

Links to More Info: BT1054837

Component: F5OS-C

Symptoms:
The error message

<err> Oct 13 13:41:40 vcc_install_versions_failed: Vcc-ConfD-RU: popen failed => Resource temporarily unavailable

occasionally appears in the /var/log_controller/cc-confd log.

Conditions:
Running system controller rolling upgrade.

Impact:
Presently the only known impact is the message appearing in the log.

Fix:
Vcc-ConfD processes started by popen are properly terminated with pclose.

1054021 : Tcpdump on VELOS chassis blade or rSeries appliance cores when line-dma agent layer below it fails

Links to More Info: BT1054021

Component: F5OS-C

Symptoms:
Line-dma agent is the underlying layer of tcpdump in the VELOS/rSeries family of chassis and appliance products
When it is not running, or if it cores or is otherwise not available and a client wants a tcpdump capture, tcpdump may core.

Conditions:
-- line-dma-agent is not functional at start, or at some later point in time during the tcpdump capture
-- a client requests a tcpdump capture

Impact:
Packet capture will be affected and will not work

Fix:
Tcpdump does not core anymore, and will retry line-dma-agent connection when clients ask for capture

1052941-2 : Hardware-fault alarm not cleared.

Links to More Info: BT1052941

Component: F5OS-C

Symptoms:
A hardware-fault alarm triggered by RAS unknown type errors is not cleared after the errors are resolved.

Conditions:
This occurs with hardware fault alarms due to RAS unknown type. The alarm is not cleared after the issue is resolved.

Impact:
Hardware-fault alarm with severity warning will be displayed and is not cleared.

Fix:
Fixed the issue that prevents RAS unknown errors from being cleared from the diagnostics report.

1051269-1 : Partition Confd cluster disk usage threshold feature not functioning as expected.

Links to More Info: BT1051269

Component: F5OS-C

Symptoms:
When there is an update in cluster disk usage threshold configuration, the change is not reflected in the state data.

default-1(config)# cluster disk-usage-threshold config critical-limit 91
default-1(config)# commit
default-1# show cluster disk-usage-threshold state critical-limit
cluster disk-usage-threshold state critical-limit 97

Conditions:
When you connect a cluster to Confd during a firmware update and the disk-usage-threshold is updated at the same time, updates will be missed.

Impact:
Some changes to the partition may not be performed, or they may not be reflected in the state data.

Fix:
Modified the cluster disk threshold subscriber to not use a shared access object in Confd.

1050761 : System logs the following error at startup: SDK error during device programming

Links to More Info: BT1050761

Component: F5OS-C

Symptoms:
During startup of the 'fpgamgr' container, the following error is logged in velos.log: "SDK error during device programming." API="f5sw_port_spn_state_get" code=-1 error="parameter error"."

Conditions:
System startup or fpgamgr restart.

Impact:
Error log message with no functional impact

Workaround:
None

Fix:
Fixed API call to prevent an error.

1050677 : Disk I/O stats inaccurate in snmpwalk for partition

Links to More Info: BT1050677

Component: F5OS-C

Symptoms:
Disk I/O stats is inaccurate in snmpwalk.

Conditions:
This occurs when running snmpwalk on a partition.

Impact:
Inaccurate disk I/O stats info in snmpwalk

Workaround:
None

Fix:
Disk I/O stats is now accurate in snmpwalk.

1047129 : Partition_tmstat_merged container core on shutdown

Links to More Info: BT1047129

Component: F5OS-C

Symptoms:
When the partition_tmstat_merged container is shutting down and it receives a message from the same container on another blade in the partition, it may crash with a core.

Conditions:
Container is shutting down and also receives a message from another blade.

Impact:
Crashes with a core file. No other impact. Core can be safely ignored and removed.

Workaround:
Remove core file.

Fix:
Race condition on shutdown fixed so that if message is received on shutdown it is properly handled.

1046765 : Tenant Data path will not work correctly on downgrade to controller version 1.1.x★

Links to More Info: BT1046765

Component: F5OS-C

Symptoms:
After a controller downgrade to version 1.1.x, the tenant datapath will not operate correctly.

Conditions:
The kubevirt software version does not downgrade to the correct kubevirt software version needed in the 1.1.x controller release.

Impact:
The tenant will launch correctly, but the datapath will be broken because of the a dma-agent protocol mismatch.

Workaround:
1. In one root command shell window, run this command to delete the current version of kubevirt softare pods.

[root@controller-2 ~]# oc delete -f /tmp/omd/scripts/kubevirt-velos.yaml

2. In another root command shell window, run this command to clear the kubevirt namespace and install the new version of kubevirt pods.

[root@controller-2 kubevirt]# /usr/share/omd/kubevirt/omd-kubevirt-velos-install.sh /usr/share/omd/kubevirt/

Fix:
The tenant datapath should work properly after a downgrade.

1046217 : Database import fails after database reset

Links to More Info: BT1046217

Component: F5OS-C

Symptoms:
Attempt to import database using file import in confd/config folder fails. During database reset operation config folder is deleted, due to which import fails

Conditions:
Database reset performed before import operation.

Impact:
Database import fails

Workaround:
None

Fix:
File import operation, to create configs folder if missing.

1045253 : Errors related to LCD module show up in logs

Links to More Info: BT1045253

Component: F5OS-C

Symptoms:
The system controller log file can contain errors related to failed communication with the LCD module.

controller-2 platform-monitor[1]: priority="Err" msg="Action Error" name="LCD Sensor Monitor" inputId="1f156c2b-0db1-11ec-bdd4-024264410634" index=0 message="unable to get LCD sensor info" interface="zmq-input"

Conditions:
The error message shows up when the LCD module is restarting due to initial system startup or a firmware update.

Impact:
The error message is not system critical and can be safely ignored.

1045177 : Stale interfaces are left behind upon portgroup mode change from 100GB to 40GB

Links to More Info: BT1045177

Component: F5OS-C

Symptoms:
There are situations when stale interfaces are left behind in the config cdb, when the portgroup mode changes from 100GB to 40GB, 4x25GB or 4x10GB. This causes l2-agent on the blade to exit.

Conditions:
-- reset-to-defaults/backup/restore
OR
-- live install

-- change the portgroup mode from 100GB to 40GB
-- commit

Impact:
The interfaces corresponding to portgroups are not present and stale interfaces are left behind.

Workaround:
Steps for mitigation:
1) verify the issue is caused by the lack of pgindex in cdb:

a) from config mode in partition, create a backup file
(config)# system database config-backup name test
b) look for pgindex in the /var/F5/partition{id}/configs/test:
grep pgindex /var/F5/partition{number}/configs/test

c) if no entries are found, this is the issue

2) remove the slots corresponding to the impacted partition from the system controller configuration and commit

3) re-add the slots corresponding to the impacted partition from the system controller configuration and commit

4) from the partition cli, ensure the system redundancy shows the blade is present and operational

5) from the partition cli, change the portgroup mode from 100GB to 40GB and commit (example below)
(config)# portgroups portgroup 1/1 config mode MODE_40GB; top
(config)# portgroups portgroup 1/2 config mode MODE_40GB; top
(config)# commit


6) wait for the blades to resync by monitoring 'show system redundancy'

At this point the interfaces should be republished matching the new 40GB mode.

Fix:
Proper interfaces will be published to match the portgroup modes that were changed.

1044557-1 : Output from the image removal command is confusing and reveals inappropriate, internal details.

Links to More Info: BT1044557

Component: F5OS-C

Symptoms:
When running commands such as "image <controller|partition> remove iso <version>", the error output contains the following message, among other details:

"Error: unexpected response back from API: 1"

Conditions:
The output occurs after you issue a command to remove the image controller or partition images that are in use. A typical example is when you are trying to remove an ISO that uses OS/service artifacts.

Impact:
The error message from these commands is unhelpful to the user and reveals internal implementation details.

Workaround:
None

Fix:
The fix is present in version 1.2.2. The error message is replaced by one of the following (or another more helpful message if more specific information is available):

"Error: failed to remove controller image; may be in use"
"Error: failed to remove partition image; may be in use"

1044317 : dagd core

Links to More Info: BT1044317

Component: F5OS-C

Symptoms:
Dagd crashes and leaves a core file.

Conditions:
The exact conditions, especially from user point view, are not identified.

Impact:
Traffic disrupted while dagd restarts.

Workaround:
None

Fix:
Make dagd more robust against system conditions.

1044257 : Removal of old chassis partition images might cause tenant issues after blade reboot★

Links to More Info: BT1044257

Component: F5OS-C

Symptoms:
After upgrading the system to version 1.1.4 and old chassis partition images are removed from the system, tenants might not start up correctly after a reboot of the blade hosting the chassis partition.

Conditions:
This might occur if the tenant was started after the system was upgraded to an interim release (such as 1.1.1, 1.1.2, 1.1.3), after originally running version 1.1.0.

Impact:
Tenants will not start correctly, will not pass traffic, or be accessible on their management interfaces.

Workaround:
To work around this issue:

1. Upgrade the system controller to 1.1.4.
2. Wait for the system controller upgrade to complete.
3. Upgrade the chassis partition(s) to 1.1.4.
4. Wait for chassis partition upgrade(s) to complete.
5. Configure all tenants to return to the "Provisioned" state.
6. Wait for all tenants to stop.
7. Configure all tenants back to the "Deployed" state.
8. Remove the old chassis partition and system controller software versions.

Fix:
N/A

1044249 : On initial installation, blades fail to PXE boot after chassis startup.

Links to More Info: BT1044249

Component: F5OS-C

Symptoms:
on initial installation, blades fail to PXE boot after chassis powers up.

Other symptoms:

1. When trying to deploy a tenant on a single blade or when multiple blades are bundled for the same partition in the Chassis Partition login (TENANT MANAGEMENT>Tenant Deployments), the "Running Version" remains "Unavailable" indefinitely.

2. Blades are not available for login or other activity from the CLI.

Conditions:
Multiple factory-fresh blades are powered up.

Impact:
Blades fail to PXE boot. This means they fail to load an initial image and cannot join a cluster.

Workaround:
On both controllers, reboot the system controller or restart the image server container.

Type the command to restart image server on each system controller:

docker restart vcc-image-server

Fix:
N/A

1044117-2 : Kubevirt pods are not reinstalled after recovering cluster using internal debug setting★

Links to More Info: BT1044117

Component: F5OS-C

Symptoms:
While reinstalling the openshift cluster by configuring an internal debug flag, the kubevirt pods were not reinstalled. Without these pods, the tenant will not operate.

Conditions:
When a cluster reinstall is initiated by configuring the internal debug flag, an internal variable was not being reset which prevented the kubevirt pods to be installed.

Impact:
The tenant will not operate.

Workaround:
In a bash console shell, execute the following command:
systemctl restart orchestration_manager_container.service

Fix:
Fix is in release V1.2.2

1043909 : Inconsistencies in disk threshold limits.★

Links to More Info: BT1043909

Component: F5OS-C

Symptoms:
Inconsistencies are being observed while configuring disk threshold limits.

default-2# show cluster disk-usage-threshold state
cluster disk-usage-threshold state warning-limit 85
cluster disk-usage-threshold state error-limit 90
cluster disk-usage-threshold state critical-limit 97
cluster disk-usage-threshold state growth-rate-limit 10
cluster disk-usage-threshold state interval 60

No checks are implemented to raise an exception if you attempt to set a critical limit to a value less than error/warning limit.

Conditions:
The problem is seen only while upgrading to 1.3.0 when you configure the disk threshold limits against the constraints.

Impact:
Upgrade can fail if the constraints introduced in version 1.3.0 are violated.

Workaround:
Configure the critical limit > error and warning limit
error limit > warning limit or set to default values before upgrading to 1.3.0

Partition Confd
-------------------------
default-2(config)# cluster disk-usage-threshold config critical-limit 90
default-2(config)# cluster disk-usage-threshold config error-limit 85
default-2(config)# cluster disk-usage-threshold config warning-limit 80
default-2(config)# commit
Commit complete.

1042845 : Unable to remove platform services versions that appear unused

Links to More Info: BT1042845

Component: F5OS-C

Symptoms:
Under certain circumstances, a version of controller or partition services may appear "not in use" in ConfD/GUI tables, but removal of that version is still blocked because other parts of the service package are still in use by other system components.

Conditions:
1. Attempt to remove an (apparently inactive) version of controller or partition services via ConfD or GUI.
2. Other components on the system still silently depend on that version of services, even though ConfD/GUI output does not reflect this.

Impact:
Unable to remove versions of software that appear unused, and the cause is unclear.

Workaround:
N/A

Fix:
Removal of platform services that appear "unused" is no longer blocked by hidden higher-level component dependencies

1042785-1 : Configuring spanning tree (stp) while disabled may display incorrect state

Links to More Info: BT1042785

Component: F5OS-C

Symptoms:
While stp is disabled, configuring a field such as MSTP max-hop causes the the enabled-protocol to display an incorrect value.

Conditions:
Delete enabled-protocol configuration field.
Delete another stp configuration field such as MSTP max-hop

Impact:
The stp enabled-protocol display is incorrect.

Workaround:
To mitigate, do not configure stp while not enabled.

Fix:
Configuring stp while disabled will not lead to incorrect display.

1042273 : ETCD-HA Instance may not initialize correctly after PXE-booting the system controller.

Links to More Info: BT1042273

Component: F5OS-C

Symptoms:
The ETCD-HA instance may not initialize correctly after PXE-booting a system controller and re-installing that system controller into the openshift cluster. When the instance initializes incorrectly and one of the system controllers is down, the openshift API does not operate correctly.

Conditions:
PXE boot of a system controller in a running openshift cluster.

Impact:
When the instance initializes incorrectly and one of the system controllers is down, the openshift API does not operate correctly.

Workaround:
None

Fix:
Fixed the Ochestration-manager to correctly re-initialize the ETCD-HA instance when a system controller is PXE booted and the system controller is re-added into the openshift cluster.

1042253-1 : System controller upgrade from 1.2.0-10357 to 1.2.1-10301 intermittently fails★

Links to More Info: BT1042253

Component: F5OS-C

Symptoms:
The upgrade proceeds to the point where both system controllers boot to the new image but neither system controller becomes active.

Conditions:
Whenever this issue is observed, show full-configuration system redundancy config mode is something other than the default (auto).

Impact:
Neither system controller becomes active. The ability to configure the System controllers is compromised.

Workaround:
Restarting both Vcc-ConfD containers (or a reboot of both system controllers) should clear the problem.

Fix:
Intermittent loss of active system controller when upgrading from 1.2.0-10357 to 1.2.1-10301 is fixed in 1.2.1.

1041381 : Tcpdump capture may not include broadcast and multicast egress (generated by the system and being sent out) when "--dls true" option is used

Links to More Info: BT1041381

Component: F5OS-C

Symptoms:
When DLS feature is turned on using "--dls true" option, broadcast and multicast packets generated by the host CPUs of the system and egressing out of the VELOS system will not be part of the capture.

The default mode when no "--dls" option is specified is "--dls false", which has no issue

Conditions:
The 'DLS' feature of tcpdump is turned on by explicitly invoking packet capture with the non-default mode "--dls true"

Impact:
Capture will not be complete and will not contain the egressing broadcast and multicast packets.

Workaround:
Use the default mode ( i.e no "--dls option specified) or explicitly turn off dls mode ("--dls false")

1039085 : Partition config restore operation can cause the system to stop processing fdbs.

Links to More Info: BT1039085

Component: F5OS-C

Symptoms:
In rare cases, a partition config-restore operation can cause a race condition that locks up a platform component. This causes fdbs to no longer be processed, and can affect traffic processing.

Conditions:
Issuing a config-restore operation on the partition cli. This issue is more likely to occur when the number of tenants increases.

Impact:
Fdbs will no longer be processed. Traffic processing can be impacted due to missing fdbs.

Workaround:
First, restart the network manager on both controllers:
    - "docker restart partition<partition_number>_network_manager"
Second, redeploy all tenants.

1038557-1 : Partition merged stats only reflect one blade when tmstat-rsync service moves to other blade

Links to More Info: BT1038557

Component: F5OS-C

Symptoms:
A few show stats commands such as 'show qos state' that report stats for all blades in a partition could report only the stats from a single blade when the tmstat-rsync service moves from the blade is was on initially to another blade.

Conditions:
The tmstat-rsync service has moved to a blade other than the initial blade it was running on and a show command that combines stats from all the blades in a partition is run.

Impact:
A few show stats commands will only report data from a single blade.

Workaround:
Restart the tmstat-rsync service so it runs back on the initial blade.

1037749 : Switch daemon crashes occasionally on shutdown.

Links to More Info: BT1037749

Component: F5OS-C

Symptoms:
Shutting down the system sometimes causes the switch daemon to crash.

Conditions:
This occurs rarely during system shutdown.

Impact:
A core file is saved to /var/shared/core/container/.

Workaround:
None.

Fix:
This has been fixed in 1.2.2 and 1.3.

1037673 : Vcc-lacpd on a system controller can crash and leave a core file while restarting.

Links to More Info: BT1037673

Component: F5OS-C

Symptoms:
Vcc-lacpd on a system controller crashes, leaving behind a core file and a system log indicating a crash occurred. After the crash, the daemon recovers within a few seconds.

Conditions:
The crash only occurs during a restart of vcc-lacpd. Most commonly, a restart will occur during a system controller software update, using the "go-standby" command, or from a fatal error.

Impact:
The internal mgmt network to all blades may go down for a few seconds. Traffic running on tenants will be unaffected.

Workaround:
Limit failover scenarios on the system controllers, like use of the system controller "go-standby" command or system controller software updates.

Fix:
Vcc-lacpd no longer leaves a core file under these conditions.

1037525 : Some of the PCie AER severity and types are incorrect in the diagnostic monitoring.

Links to More Info: BT1037525

Component: F5OS-C

Symptoms:
Some of AER error type and severity events are displayed incorrectly in the diagnostics monitoring.

Conditions:
If an AER (Advanced Error Reporting) error occurs the decoding of the error type and severity as reported in the diagnostic could be incorrect.

Impact:
AER errors in diagnostic monitoring could be interpreted incorrectly as a 'Fatal' error.

Workaround:
As there is not a complete mitigation for this, the AER errors are correctly logged in the system logs and can be confirmed by timestamp and device to obtain the correct information

Fix:
Fixed an issue with incorrect diagnostics reporting.

1035353-1 : Missing controller images in show image controller CLI operation

Links to More Info: BT1035353

Component: F5OS-C

Symptoms:
After software upgrade, the controller images in display of "show image controller" shows only active controller images. The standby controller images are missing in "show image controller" CLI command. This is very occasional and won't happen always.

Conditions:
Using CLI/RESTCONF command operations for show image controller

Impact:
User won't see the standby controller images in "show image controller"

Workaround:
The reboot of standby controller using the CLI operation "system reboot controllers controller standby" would resolve the issue and bring the controller images back into CLI display.

1034993-1 : Key-migrationd service can crash if server elements are incomplete

Links to More Info: BT1034993

Component: F5OS-C

Symptoms:
The key-migrationd service crashes after defining some server-group information for radius/ldap servers.

Conditions:
After defining system->aaa->server-groups->server-group but not fully defining the item, and then attempting to read the item.

Impact:
Core file is created and key-migration malfunctions.

Workaround:
Remove the partially-defined server group or fully define all server-group items.

Fix:
The key-migration works without crashing.

1034481 : When using IPv6 on mgmt-floating and dhcp, it is possible to get different ip addresses on failover

Links to More Info: BT1034481

Component: F5OS-C

Symptoms:
When running IPv6 and using dhcp to assign the mgmt-floating address, a chassis failover can cause the ip address to be changed.

Conditions:
Running IPv6, using dhcp for mgmt-floating and failing over a system controller. IPv4 is unaffected as is IPv6 statically assigned addresses.

Impact:
Services wont be available on mgmt-floating as expected until the user finds the interface on an unexpected IPv6 address

Workaround:
None

1034169 : Qkview reports status of "partial file recorded" when out of disk space

Links to More Info: BT1034169

Component: F5OS-C

Symptoms:
When qkview attempts to create a qkview file and there is insufficient disk space, the status recorded is "partial file recorded". The actual cause is low disk space, and no qkview is collected in this case. The recorded status should indicate so.

Conditions:
Run the qkview collection with less than 1 GB of available disk.

Impact:
Cosmetic.

Fix:
The status now indicates: Out-of-disk. Unable to create Qkview file.

1033817 : GUI effected due to /api/data/f5-cluster:cluster/nodes/node taking more than 25 seconds to complete

Links to More Info: BT1033817

Component: F5OS-C

Symptoms:
The 'show cluster nodes node' command takes more than 25 seconds to complete.

Conditions:
This happens on a chassis that is not fully populated.

Impact:
The get api /api/data/f5-cluster:cluster/ takes more time, resulting in slow page load times.

Workaround:
None

Fix:
Modified diag-agent partition to check the blade ready status before contacting it for disk-usage information. This is reduce the timeouts

1033813 : Partition 'show interfaces' command can be slow

Links to More Info: BT1033813

Component: F5OS-C

Symptoms:
A 'show interfaces' command or the corresponding RESTCONF API request that includes 'show interfaces interface state counters' or 'show interfaces interface ethernet state counters' can take a long time to execute.

Conditions:
If a blade was present in the partition, but is either physically removed or powered off, but the slot is not removed from the partition configuration.

If a 'show interfaces interface state counters' query is issued for an aggregate (trunk), a delay will also be observed.

Impact:
UI screen refresh is slow (2 to 8 seconds per missing blade), or the CLI 'show' command take a long time to return.

Workaround:
Use the system controller UI or CLI to remove the non-existent blade from the partition.

Fix:
Fixed an issue causing the show interfaces command to be slow when a blade is removed.

1032697 : File delete operation throws an improper message

Links to More Info: BT1032697

Component: F5OS-C

Symptoms:
A file delete operation has a confusing error message:

syscon-1-active# file delete file-name log/host/ansible.log

Only /mnt/var/confd/configs/ /var/shared/ configs/ diags/shared/ paths are allowed for Delete file operation on Controller
ConfD.

Conditions:
Attempting a file delete operation from a directory which does not have delete permission

Impact:
The error message lists the actual paths along with the virtual paths on which delete is supported.

Workaround:
None

Fix:
On file delete operation, it only list virtual paths

1032341 : Confd Encryption key gets rewritten intermittently

Links to More Info: BT1032341

Component: F5OS-C

Symptoms:
The key should always return the same value and hash, unless it is changed via key-migration.

The reading of memory (EEPROM) will sometimes return "resource temporarily unavailable" which is treated as an error instead of simply doing a retry.

Conditions:
The EEPROM might be busy because of use by other components.

Impact:
The encryption key changes, thus invalidating all currently encrypted items, thus requiring re-entry of these.

Workaround:
The only workaround is to re-enter all encrypted items and hope that the "resource temporarily unavailable" does not occur.

Fix:
Fixed an issue where the system no longer considers "resource temporarily unavailable" as an error unless it happens 10 times in a row. The system does a retry and if that works, the system avoids setting a new key.

1029561 : GNOME GLib vulnerability CVE-2021-27219

Links to More Info: K82112489, BT1029561

1022729 : Management port issues with instance names containing lacpd, lldpd, stpd, or tmstat-rsync

Links to More Info: BT1022729

Component: F5OS-C

Symptoms:
The management port stops working when instance names contain any of the following: lacpd, lldpd, stpd, or tmstat-rsync

Conditions:
Instances whose names include any of the following:
  lacpd
  lldpd
  stpd
  tmstat-rsync

Impact:
Management port no longer works.

Workaround:
Avoid naming instances using any of the following:
  lacpd
  lldpd
  stpd
  tmstat-rsync

Fix:
You can now successfully name instances using strings containing the following:
  lacpd
  lldpd
  stpd
  tmstat-rsync

1022589 : New blank blades inserted into system can wind up in a reboot loop and possibly be damaged

Component: F5OS-C

Symptoms:
Blades fresh from manufacturing do not contain an OS image. If not made part of a partition with an os image defined, when inserted, they will wind up in a continuous reboot loop. There is potential that this may cause damage to blade components if allowed to continue for an extensive period of time.

As systems shipped from factory include all slots in the default partition and that partition is set up with a partition image already configured, this condition should only be possible when blades are added in the field and the site has added partition definitions which do not have OS images set.

Conditions:
Freshly manufactured blade installed in a system slot which is not part of a partition with a defined iso image

Impact:
Potential drive media damage if the reboot loop is allowed to continue for an extended period of time.

Workaround:
Options to mitigate:
1--Install an OS image on the new blade
2--Power down the blade using AOM until ready to load an image

The simplest method to install an os image is to be sure the installation slot is part of a partition definition which includes a set os image. By default the blade will pxeboot that image.

1008549 : iHealth indicates multiple unhealthy and critical states for un-inserted PSUs.

Links to More Info: BT1008549

Component: F5OS-C

Symptoms:
The component health for PSUs that are not inserted in the VELOS chassis is shown as unhealthy along with an iHealth critical severity.

Conditions:
This issue occurs on a VELOS chassis that has one or more PSUs not populated.

Impact:
The chassis health of these non-populated PSUs are shown has unhealthy in iHealth.

Fix:
Modified diag-agent service so that it does not mark an unhealthy state for PSUs that are not present in the chassis.

1008433 : VQF hot signal asserted warnings

Links to More Info: BT1008433

Component: F5OS-C

Symptoms:
A PEL log entry occurs indicating an FPGA HOT signal asserted:

Warning | AOM | 5 | Na | VQF hot thermal event

Conditions:
This issue happens at system startup.

Impact:
If the issue occurs during system startup, it is an erroneous error message and can be safely ignored.

Workaround:
Fixed an erroneous FPGA HOT signal that occurs during system startup.

1005025 : Orchestration-manager core on standby controller during cluster bringup.

Links to More Info: BT1005025

Component: F5OS-C

Symptoms:
A core file from orchestration-manager may be created on the standby switch during cluster bringup.

Conditions:
This may occur intermittently during cluster bringup.

Impact:
A core file is generated, but orchestration-manager will restart and will not cause any issues with system function.

Workaround:
None

1004309 : NSS vulnerability CVE-2020-12403

Links to More Info: K61267093, BT1004309

1004305 : libxml2 2.9.10 vulnerability CVE-2020-7595

Links to More Info: K04460334, BT1004305

1004189 : libcroco vulnerability CVE-2020-12825

Links to More Info: K01074825, BT1004189

1004049 : Show system mgmt-ip displays "Application Timeout" on the Active system controller

Links to More Info: BT1004049

Component: F5OS-C

Symptoms:
On a system where the standby system controller is rebooting, running the 'show system mgmt-ip' command in confd can display an "Application Iimeout" error.

Conditions:
-- Standby system controller is rebooting.
-- The 'show system mgmt-ip' command is run in confd

Impact:
This problem is limited to system controller mgmt-ip only.

Workaround:
The command would fail very first time the command is executed while standby controller is rebooting. After it fails the first time, the command displays output in subsequent retries.

Fix:
The 'show system mgmt-ip' command now works while the standby system controller is rebooting.

1000453 : CVE-2019-25013: glibc vulnerability

Links to More Info: K68251873, BT1000453

• The F5 Networks Technical Support web site: http://www.f5.com/support/
• The AskF5 web site: https://support.f5.com/csp/#/home
• The F5 DevCentral web site: http://devcentral.f5.com/

Generated: Thu Jan 20 19:41:31 2022 PST

Copyright F5 Networks (2022) - All Rights Reserved