968881 : Creating a partition using the CLI, 'commit check' fails

Links to More Info: BT968881

Component: F5OS-C

Symptoms:
When creating a partition using the CLI, and trying to validate the changes with 'commit check', a validation error occurs:

partitions 'partition part1 uuid' is not configured.

Conditions:
-- Create a partition using the CLI.
-- Attempt to validate the changes using 'commit check'.

Impact:
The 'commit check' operation rejects this config change. This error is misleading, indicating that you need to specify a uuid value.

Note: Not only is uuid irrelevant, it is not possible for you to specify it.

Workaround:
None

950901 : Wrong chassis serial number chs599996s populates to VELOS tenants

Links to More Info: BT950901

Component: F5OS-C

Symptoms:
Under some corner cases, system controllers may have a blank serial number in the /etc/PLATFORM file. If partition software is started during this period, any tenants deployed on that partition will report an incorrect chassis serial number chs599996s.

Conditions:
- Restarting the system controllers
- Removing and adding the blades from the tenant

Impact:
VELOS tenants may report the wrong serial number, visible in "tmsh show sys hardware" or "tmsh list cm device".

If a VELOS tenant spans multiple blades, and the different blades pick up different serial numbers from F5OS, the tenant may fail to properly cluster; multiple tenant blades will function as cluster primary, competing over the cluster management IP.

Workaround:
The correct chassis serial number can be seen in the license file, which can be viewed from a tenant by running "tmsh show sys license".

If a tenant is currently bifurcated (multiple blades functioning as cluster primary), the immediate mitigation is to set the tenant to "provisioned" and then back to "deployed".

If a tenant is reporting the incorrect chassis serial number (chs599996s), then the following should restore the correct serial number:

1. Determine which controller has a blank serial number for the partition, by looking at "grep CHASSIS_SERIAL_NO /var/log/sw-util.log | tail -1" run on each controller, e.g.:

    [root@controller-2 ~]# for i in controller-{1,2}; do echo -n "$i: "; ssh $i grep CHASSIS_SERIAL_NO /var/log/sw-util.log | tail -1; done
    controller-1: ++ CHASSIS_SERIAL_NO=chs700144s
    controller-2: ++ CHASSIS_SERIAL_NO=
    [root@controller-2 ~]#

    In this example, controller-2 is affected.

2. Reboot the affected controller. After it reboots, check whether it has a blank serial number. Repeat this step until it boots, and reports a non-blank serial number.
3. From the partition, set the tenant to "provisioned" and then back to "deployed".
4. After the tenant reboots, confirm the serial numbers is now correct (not chs599996s) in the output of "tmsh show sys hardware"

1125505-1 : LOP communication may stop working on a chassis controller after failover

Component: F5OS-C

Symptoms:
Various services may become unresponsive or not work correctly when communicating with the LOP.

Conditions:
This can happen rarely during failover of a chassis controller.

Impact:
Any functionality that interacts with the LOP could be impacted.

Workaround:
Reboot the affected chassis controller.

Fix:
Resolve a platform-hal LOP communication lockup that can occur due to a race condition.

1117561 : Vcc-terminal-server gets stuck in unsuccessful restart loop

Component: F5OS-C

Symptoms:
Terminal services will fail to function and the vcc-terminal-server-main component container will appear to be stuck with a status of restarting.

Conditions:
When the vcc-terminal-server-main component container is restarted without first recreating the container from scratch, internal filesystem changes persist which prevent it from starting up successfully.

Impact:
This should generally not be seen in normal usage, although error conditions can trigger a basic container reset.

Workaround:
Restarting the system should perform a complete bringup of all component containers, including vcc-terminal-server-main, resulting in normal operation.

Fix:
This fix modifies the internal filesystem operations such that they do not prevent the container from initializing properly after a basic restart.

1114861 : Local controller etcd instance may become unreachable after a rolling upgrade.★

Component: F5OS-C

Symptoms:
The local etcd instance on a system controller may be unreachable after a rolling upgrade. This can be seen via the output of /etc/etcd/dump_etcd.sh script on the controller.

Conditions:
This can happen during a rolling upgrade.

Impact:
The local etcd instance will become unreachable. If the local instance on the active controller is unreachable, and the standby controller fails, etcd and the openshift cluster will go offline.

Workaround:
None

Fix:
The unreachable etcd instance will be restored once the rolling update is completed.

1113233 : External access for BIG-IP Next pods can be lost after tenant redeployment

Component: F5OS-C

Symptoms:
Some of the pods for a BIG-IP Next tenant can lose external access after the tenant is redeployed.

Conditions:
Stale iptables rules assigned to the tenant are not being cleared after a tenant redeployment.

Impact:
The tenant loses external access on one or more of its pods.

Fix:
The iptables rules entries are cleared on a tenant redeploy.

1112229-2 : File download API Changes to support file download from the GUI

Component: F5OS-C

Symptoms:
Header information is not effective to download files from the GUI

Conditions:
X-Auth token is required to download from the GUI.

Impact:
Downloading files from the GUI fails.

Workaround:
None

1111549 : System import functionality is unstable if PXE install source is not imported

Component: F5OS-C

Symptoms:
If a VELOS controller is PXE installed with a given ISO, and that ISO is not imported manually on the controller after the installation, future imports may fail or be left in an inconsistent state.

Conditions:
1. PXE install VELOS controller
2. Fail to manually import ISO used for PXE install
3. Import other software

Impact:
Confusing import and upgrade failures under conditions that seem like they shouldn't produce issues.

Workaround:
After PXE installing a VELOS controller, make sure to manually import the ISO used for PXE install before importing any other platform software components.

Fix:
Better handling for case where ISO used for PXE install of VELOS controllers is not imported after install

1111237 : Logrotate parameters do not get updated by software upgrade

Component: F5OS-C

Symptoms:
If the parameters (frequency/size) for log file rotation are updated in a new software release, they are not updated on the target system during upgrade. The result is that the size of retained log messages depends on the upgrade history, not on the software version.

Conditions:
System that is live upgraded from any version to any other version prior to F5OS-C-1.5.0.

Impact:
When logfiles are collected by qkview, differing amounts of data may be gathered, perhaps omitting information that was intended to be collected.

Workaround:
None.

Fix:
The system updates the logrotate parameters during software install, so that the setting correspond to the software version, not the upgrade history.

1110429 : Duplicate service-instance entries in chassis partition

Component: F5OS-C

Symptoms:
In rare circumstances, when viewing the partition service-instance entries, duplicate entries will exist for system level daemons like LACPD, L2FwdSvc, and SwRbcaster. The issue occurs rarely, and user should only notice a cosmetic difference.

Conditions:
Adding blades to and removing blades from a partition may trigger the issue.

Impact:
Display is not correct.

Workaround:
Delete and recreate the affected partition.

Fix:
Duplicate service-instance entries will be removed in cases of a blade rebooting and a blade being added to a partition.

1109021-1 : CLI commands are not logged in audit.log

Component: F5OS-C

Symptoms:
CLI commands from confd are getting logged in audit.log.

Conditions:
Execute commands using the confd CLI.

Impact:
CLI commands which are required for security compliance audit will not get logged in audiit.log file.

Workaround:
None

1107433 : BIG-IP Next floating IPs do not properly issue GARPs on failover

Component: F5OS-C

Symptoms:
The floating management IP of a BIG-IP Next high availability (HA) pair may not function correctly after an high availability (HA) failover when running on F5OS-C-1.4.0.

Conditions:
When the BIG-IP Next active instance changes location, the mapping of IP to MAC address and switch ports needs to be updated in the network, or packets will continue to be routed to the "old" active instance.

Impact:
The floating management IP will not be usable for some period of time after failover. If a client is actively connected and sending packets, this condition can persist forever.

Workaround:
None.

Fix:
The high availability (HA) failover logic now correctly issues unsolicited ARP replies (GARP) in order to flush/update the IP address mappings.

1106881 : F5OS with an AFM license provisioned may provide incorrect AFM stats to a BIG-IP tenant

Component: F5OS-C

Symptoms:
This is an intermittent problem where the affected BIG-IP tenant may receive incorrect statistics from the F5OS platorm. This can cause the BIG-IP tenant to drop DNS traffic that should not be dropped.

Typically, the BIG-IP tenant will have periods of time where it receives the correct stats, and periods where it receives incorrect stats.

Conditions:
All of the below must be true:

-- Two or more BIG-IP tenants are deployed either on the same node in a partition or on the same appliance.
-- An AFM license is installed on the F5OS platform.
-- At least one tenant is receiving malformed DNS traffic.

Impact:
Clients that send DNS traffic to the affected BIG-IP tenant will not receive DNS responses when they should.

Workaround:
When AFM is provisioned for the system, deploying tenants on different nodes on a chassis based system or one tenant per appliance avoids the issue.

Fix:
BIG-IP tenants receive the correct platform statistics regardless of the node in which they are deployed.

1106093 : After a node is removed from a running tenant, the operational status of that node remains

Component: F5OS-C

Symptoms:
When a node is removed from the configuration of a running tenant, the node that was removed remains in the CLI operational status of the tenant.

Conditions:
-- A tenant spans multiple nodes and is running.
-- One of the nodes is removed

Impact:
The operational status of the tenant for the node that was removed is incorrect.

Workaround:
None

Fix:
The correct operational status of the tenant is displayed.

1105001 : Large tar/gz/iso file download via the restconf API fails.

Component: F5OS-C

Symptoms:
Downloading large tar/gz/iso files using the restconf API resulting in a corrupted file.

Conditions:
Large tar/gz/iso file download via the restconf API

Impact:
Download fails, the downloaded file is corrupted.

Workaround:
None

Fix:
Fixed the code to download large tar/gz/iso files.

1104769 : After a node is removed from a tenant in provisioned mode, the operational status of that node remains

Component: F5OS-C

Symptoms:
When a node is removed from the configuration of a tenant that is in provisioned mode, the node that was removed remains in the CLI operational status of the tenant.

Conditions:
-- A tenant spans multiple nodes and is in a state of Provisioned.
-- One of the nodes is removed

Impact:
The operational status of the tenant for the node that was removed is incorrect.

Workaround:
None

Fix:
The correct operational status of the tenant is displayed.

1103105 : The naming convention of core files has changed

Component: F5OS-C

Symptoms:
Core files were previously named <app>-1.core.gz or <app>-2.core.gz where <app> was the first 12 characters of the failing executable. New core file are named as follows: core.<app>.<pid>.<timestamp>.core.gz.

Conditions:
A core file is created.

Impact:
The name of the core file is now different.

Workaround:
None

Fix:
The core files will now have a name that is consistent with other F5 formatting for core-files.

1102137 : Diagnostics ihealth upload qkview-file does not auto-complete with available qkview file names

Component: F5OS-C

Symptoms:
The confd command

system diagnostics ihealth upload qkview-file ?

Is not tab-expandable, and you are not presented with the list of available qkview files.

Conditions:
Running 'system diagnostics ihealth upload qkview-file <TAB>' to see the list of available qkview files.

Impact:
The available qkview files are not presented using tab autocomplete.

Workaround:
Run "system diagnostics qkview list" to obtain the list of available qkview files, and then manually type the desired qkview file name in when using the 'system diagnostics ihealth upload qkview-file' command.

Fix:
Hitting <TAB> after system diagnostics ihealth upload qkview-file will produce a list of available files. Entering part of the name and <TAB> will auto-complete selecting a valid and available qkview file name.

1100861 : System aaa primary-key state not returning both hash and status

Component: F5OS-C

Symptoms:
Requesting both hash and status by the query command "system aaa primary-key state" fails.

Conditions:
When no key-migration has been performed.

Impact:
Requesting the status fails.

Workaround:
Perform a key migration.

Fix:
The fix allows the query of the system state and there is no failure, returning "NONE" if no key-migration was known to the system.

1097833-1 : Debug messages logged in platform.log

Component: F5OS-C

Symptoms:
When performing an ISO install on the hardware, some services log debug messages to platform.log until confd comes up.

Conditions:
This occurs during an ISO install.

Impact:
Unnecessary debug logs are logged to platform.log.

Workaround:
None

1095977 : Tenant disk image not removed from blade on scale-down of tenant deployment

Component: F5OS-C

Symptoms:
If a tenant is scaled down to run on fewer blades, the tenant disk image will be left behind on the blade that the tenant was scaled down from.

Conditions:
BIG-IP tenant image running on one or more blades where one of the running blades is removed from the tenant configuration.

Impact:
Tenant image file will be left on blade where the tenant is scale down from:

/var/F5/partition<#>/cbip-disks/<tenant_name>.raw

Workaround:
The tenant disk image can be removed directly from the blade via the shell using the rm command. This can be done by the chassis administrator.

Fix:
Bug has been fixed to cause removal of tenant disk image when tenant deployment is scaled down.

1093301 : GUI keeps showing "Firmware updates are currently in progress" after upgrade

Component: F5OS-C

Symptoms:
- GUI keeps showing "Firmware updates are currently in progress"
- Both system controllers report that they are active
- Chassis firmware updates can fail

Conditions:
The active system controller is updating its AOM, and while that update is occurring the peer system controller assumes the active role. When the initially active system controller's AOM boots again it will not report that the arbitration state has changed.

Impact:
There seems to be no operational impact.

Workaround:
Reset the initially active system controller.

Fix:
Fix for GUI continuously showing "Firmware updates are currently in progress" after upgrade

1092913 : Tenant CPU pinning can fail when a blade is moved to a new partition and the blade was previously running a deployed tenant

Component: F5OS-C

Symptoms:
Newly deployed tenants on a blade just moved to a new partition will not perform optimally and confd tenant status may show the cpu allocations to the tenant failed.

Conditions:
A blade is running a deployed tenant

The blade is moved into a new partition

   or

The blade tenant is deleted, and less than about 2 minutes after that, the blade is moved to a new partition

Impact:
It's possible one or more new tenants deployed on the blade in the new partition will fail to be assigned to appropriate cpus. This could affect the performance of those tenants.

Workaround:
The main issue is that before the blade moved to the new partition, it never had the chance to release the cpus assigned to any tenants that had been deployed on the blade.

The first thing is to try and avoid this problem from happening at all be ensuring that tenants are not deployed on a blade before moving it to a new partition. Changing a tenant's deployed state to any other state (provisioned, configured, deleted) is not synchronous, so wait at least 2 minutes before moving the blade to a new partiiton.

But if avoidance fails, it is still possible to clean up the tenant cpu allocator database. The simplest steps are:

1. If any tenants are deployed on the blade in the new partition, set them to provisioned, and wait more than 2 minutes.
2. Manually remove the tenant cpu database file on the blade: "rm /opt/f5/cpumgr/cpu_users"
3. Reboot the blade (this will recreate the above file, but with all tenant records cleared)
4. Redeploy any tenants from step 1.

1092257 : Downloading files larger 500 megabytes via File Utilities in the webUI can result in a corrupted file.

Component: F5OS-C

Symptoms:
When using the File Utilities feature in the webUI, if you select a file that is larger than 500MB and attempt to download it locally the file could become corrupted.

Conditions:
Selecting and attempting to download a file 500MB or greater when using File Utilities in the webUI.

Impact:
The downloaded file is corrupted.

Workaround:
Files 500 megabytes or larger in size can be selected and exported from the device using the "Export" option available in File Utilities that will export the file over HTTPS. Additionally, files can be exported from the device using the Secure Copy Protocol (SCP).

Fix:
Downloading files that are 500 megabytes or larger in size has been temporarily disabled in the webUI. You will receive a warning popup when you select a file that is 500MB or larger and click the Download button. The warning popup advises you to use another supported option to export the file.

1091933 : After a partition upgrade, tenant health and traffic statistics are no longer presented★

Component: F5OS-C

Symptoms:
Health and traffic statistics for a tenant are not available after a partition upgrade.

Conditions:
This occurs after upgrading a partition.

Impact:
You are unable to see updated health and traffic statistics for the tenant.

Workaround:
The tenant running-state can be toggled from Running to Configured and then back to Running.

1091641 : NTP (chrony) packet authentication is not fully implemented on VELOS

Component: F5OS-C

Symptoms:
It is not possible to enable NTP packet authentication.

Conditions:
Running a version of F5OS-C earlier than 1.5.0.

Impact:
NTP packet authentication is not available.

Workaround:
None

Fix:
Added support for NTP packet authentication

1091313 : Rapid transition of BIG-IP Next tenant from Standby-to-Active-to-Standby can leave floating addresses active

Component: F5OS-C

Symptoms:
A rapid transition of a BIG-IP Next tenant from Standby-to-Active-to-Standby can leave the floating IP address active even though the tenant is in Standby.

Conditions:
This happens only if the transition to Active and back to Standby happens in under 40 seconds or so.

Impact:
The high availability (HA) floating address remains active on both instances of the BIG-IP Next tenant, causing high availability (HA) issues with the tenant.

Workaround:
If this condition occurs, effect a failover on the BIG-IP Next tenant.

This causes the address to be removed on the previously Active instance.

If failover is unsuccessful, wait for 60 seconds and issue one more failover.

1090521-1 : Tenant Deployment may fail if the memory configured is an odd number.

Component: F5OS-C

Symptoms:
1. Tenant Deployment Fails.
2. System may go into bad state.

Conditions:
When memory configured for a tenant is set to an odd number.

Impact:
Tenant Deployment Fails.

Workaround:
This issue has been fixed in F5OS-A 1.2.0.

1090145 : VLAN-Listener incorrectly updated on Network Manager component restart

Links to More Info: BT1090145

Component: F5OS-C

Symptoms:
When the Network Manager component is restarted, VLAN Listener entries can be incorrectly updated to each tenant's default Service ID.

Conditions:
Network Manager restarts can happen due to system controller restarts, partition upgrades, or a manual restart.

Impact:
Some traffic could incorrectly follow the default Port Hash disaggregation algorithm. For example, if a VLAN has been set to use the IPPORT disaggregation algorithm, this reset can cause some of the traffic to revert to using the default Port Hash algorithm.

Workaround:
Inside the affected tenants, the cmp-hash field can be changed back to default, then changed back to the desired setting.

1089037-1 : Dnsmasq configuration blocks resolution of names in .local domains

Links to More Info: BT1089037

Component: F5OS-C

Symptoms:
DNS resolution of names in .local domains will be blocked by the dnsmasq configuration.

Conditions:
Some domain names are in the .local domain.

Impact:
Name resolution of hostnames in the .local domain will not work correctly.

Workaround:
None

Fix:
Dnsmasq configuration has been updates to remove overly restrictive local=/local/ entry.

1088565 : Various services may stop working on a system controller if the LCD is malfunctioning

Component: F5OS-C

Symptoms:
Various services may become unresponsive or not work correctly.

Conditions:
LCD is not working, or host cannot communicate with the LCD.

Impact:
Any functionality that interacts with platform-hal could be impacted.

Workaround:
Recover or repair the LCD. Rebooting the affected system controller can also help temporarily.

Fix:
Fixed a leak that occurs when platform-hal cannot communicate with the LCD.

1085005 : 'cluster nodes node blade-N reboot' failure message is incorrect

Links to More Info: BT1085005

Component: F5OS-C

Symptoms:
Attempting to reboot a blade that is not assigned to the partition generates a cryptic error message:

default-2(config)# cluster nodes node blade-2 reboot
Error: Node is Not Assigned false

Conditions:
Attempt to reboot a blade that is not assigned to the partition.

Impact:
Inaccurate information provided to the user.

Workaround:
None

Fix:
Fixed the error message:

default-1(config)# cluster nodes node blade-2 reboot
Error: Node is not assigned to this partition.

1084817-2 : Container api-svc-gateway crashes due to certificate issues partition database

Component: F5OS-C

Symptoms:
The api-svc-gateway container crashes when a bad self-signed certificate or key is published to partition database.

Conditions:
A corrupted certificate/key causes the issue

Impact:
The api-svc-gateway service crashes.

Workaround:
Run the following command:

(config) # system database reset-to-default proceed

Fix:
In the scenario this happens, api-svc-gateway now:

 * detects when it cannot set up an SSL connection using these credentials
 * logs an error
 * sets health status to unhealthy with appropriate error and severity
 * tries to start a grpc server with only insecure credentials

1084581 : Log files collected by QKView are truncated with the newest entries removed

Component: F5OS-C

Symptoms:
If log files are exceedingly large, they may be truncated when collected by QKView from the 'bottom-up', meaning that the most recent log entries are clipped.

Conditions:
Log files exceed the maximum file size (default 500 MB) specified during QKView creation.

Impact:
Most recent log entries are clipped, making diagnosis difficult.

Workaround:
Collect the log files manually.

Fix:
QKView log files are now truncated 'top-down', preserving the most recent log entries.

1083993 : File import should check the target doesnt exist

Component: F5OS-C

Symptoms:
File import will fail if the same file name already exists.

Conditions:
Importing a file that already exists on the file system.

Impact:
An error occurs if the file already exists.

Workaround:
None

1081333 : Local file path in file transfer-status for remote file import operation does not show appropriately

Component: F5OS-C

Symptoms:
Local file path does not show appropriately when file import operation is done from a remote URL.

Conditions:
File transfer from a remote URL with query params.

Impact:
Inappropriate file name shown in local file path.

Fix:
Modified file name in utils-agent code to remove unnecessary query params after file name.

1080421-2 : LACP does not transmit PDU's when creating a LAG

Component: F5OS-C

Symptoms:
The LAG interface creation will not be successful and tx packet count in 'show lacp' will be zero.

Conditions:
This issue occurs due to a race condition while creating a LAG interface and is not reproducible every time.

Impact:
Link aggregation of the front panel ports will not work as expected.

Workaround:
1) clear newly added lag configurations
   a) remove lacp interface
      no lacp interfaces interface <lag-name>
   b) remove interfaces from lag
      no interfaces interface <interface> ethernet config aggregate-id
   c) remove lag interface
      no interfaces interface <lag-interface>
2) create Lag interface and add interfaces to the lag

Fix:
Fix code to remove the race condition and read lag-type as LACP

1080417-1 : List of running containers are not captured in host qkview

Component: F5OS-C

Symptoms:
List of running containers are not captured in host qkview

Conditions:
Collect qkview and look for list of containers running on the system from qkview file.

Impact:
Unable to get the list of running containers from qkview

Workaround:
Administrator needs to run 'docker ps' command on the system and share the output with support.

Fix:
Qkview includes list of running containers on the system

1079857 : Orchestration-agent logs spurious "Warn" severity messages

Component: F5OS-C

Symptoms:
Orchestration-agent can issue Warning-level log messages about "unknown tags" in velos.log, similar to this:

2022-02-09T19:57:07.444093+00:00 controller-1(p1) orchestration-agent[1]: priority="Warn" version=1.0 msgid=0x503000000000003 msg="unknown tag in operation" TAG=1013977200 OP=4.

Conditions:
Schema changes can result in new items being added that the code does not care about, but can result in log messages.

Impact:
No functional impact, these warnings an be safely ignored.

Workaround:
None

Fix:
The orchestration-agent no longer logs warnings about unknown tags.

1079809 : Alert manager status is occasionally reported as unhealthy on startup

Component: F5OS-C

Symptoms:
When Alert manager health is reported as unhealthy, it displays system controller health as unhealthy and causes failover

Conditions:
On startup, if any one of the analog sensors reports a sensor fault while reading its initial state, it is causing alert manager go unhealthy.

Impact:
It causes controller health fault/failover

Workaround:
None

Fix:
Added diagnostic tasks to monitor the health of sensors, VFC and VPC periodically and raise alerts.

1079037 : Tenant deployment fails when tenant name ends with hyphen

Component: F5OS-C

Symptoms:
A tenant fails to deploy and reports "Tenant deployment failed - Server is not responding" if the user-configured tenant name ends with a hyphen.

Conditions:
Tenant name ends with a hyphen.

Impact:
Tenant will not deploy.

Workaround:
Delete the tenant and recreate it with a valid name, consisting only of alphanumeric characters with embedded hyphens.

Fix:
The configuration validation code rejects attempts to create a tenant with an invalid name.

1078433-1 : BIG-IP Next Tenant Management IP address is not reachable after a chassis power cycle.

Links to More Info: BT1078433

Component: F5OS-C

Symptoms:
After a chassis is restarted during a system reboot or power cycle, the tenant cannot be reached using the management IP address.

Conditions:
Primarily during a chassis power cycle, tenant IP addresses cannot be reached.

Impact:
Tenant admins will not be able to perform management operations such as network creation/Application configuration.

Workaround:
If the tenant is recovered within a certain time period, toggling its state from Partition will fix it.

CLI:

tenants tenant <name> config running-state configured; commit; exit
tenants tenant <name> config running-state deployed; commit; exit

1076705-1 : Etcd instance might not start correctly after upgrade★

Component: F5OS-C

Symptoms:
/etc/etcd/dump_etcd.sh might show that the etcd instance native to system controller #1 or #2 does not come up after an upgrade.

This displays in the output of /etc/etcd/dump_etcd.sh and might occur for the .3.51 or .3.52 node:

failed to check the health of member 25fa6669d235caa6 on https://100.65.3.52:2379: Get https://100.65.3.52:2379/health: dial tcp 100.65.3.52:2379: connect: connection refused
member 25fa6669d235caa6 is unreachable: [https://100.65.3.52:2379] are all unreachable

This can cause a longer OpenShift outage if the system controller containing the healthy instance is rebooted, and complete outage if the system controller containing the healthy instance is lost.

Conditions:
This is caused by a previous mount failure of the drbd file system, which causes a corruption of the etcd instance on the standby system controller. This is seen very infrequently.

Impact:
The local etcd instance on the affected system controller will not work correctly, compromising the high availability (HA) availability of the OpenShift cluster. The cluster will continue to work correctly while both system controllers are up.

Workaround:
Rebuild the OpenShift cluster by running "touch /var/omd/CLUSTER_REINSTALL" from the CLI on the active system controller. This will cause all running tenants to be taken down during the cluster reinstall, which takes 50+ minutes.

Fix:
This is fixed in F5OS-C 1.4.0 and later.

1075693 : CVE-2021-22543 Linux Kernel Vulnerability

Component: F5OS-C

Symptoms:
A flaw was found in the Linux kernel’s KVM implementation, where improper handing of the VM_IO|VM_PFNMAP VMAs in KVM bypasses RO checks and leads to pages being freed, while still accessible by the VMM and guest.

Impact:
This flaw allows users who can start and control a VM to read/write random pages of memory, resulting in local privilege escalation. The highest threat from this vulnerability is to confidentiality, integrity, and system availability.

Workaround:
None

Fix:
Updated the kernel version to kernel-3.10.0-1160.45.1.el7.rpm .

1073777 : LACP interface goes down after the partition is disabled/enabled

Component: F5OS-C

Symptoms:
If LACP interfaces are configured and used by the tenant, after the partition is disabled/enabled, the LACP interface may not work properly due to wrong interface LACP state, which is caused by the disable/enable process.

Symptoms: interface state lacp_state is LACP_DEFAULTED
For example: interface 2/1.0 is one of member of an LACP interface, which works fine before the partition is disabled/enabled. After the partition is disabled/enabled, run "show interfaces interface 2/1.0 state" command; the output is lacp_state LACP_DEFAULTED.

lacp_state LACP_DEFAULTED is wrong which will cause vlan-listeners missing on the blade. vlan-listeners missing on the blade will cause tenant interface failure.

Conditions:
1. LACP interface is used.
2. LACP interface is configured correctly and up before the partition is disabled/enabled.
3. Disable/enable the partition.

Impact:
Tenant will not be able to send/receive user traffic and it will not recover automatically.

Workaround:
Remove the ethernet interface from the aggregation and re-attach it to the aggregation. This will recover the interface state and resolve the problem.
Example: commands to recover the interface (interface 2/1.0 belongs to the LACP interface lag1 and 2/1.0 has a wrong lacp_state).
1. In the partition CLI, remove the aggregate-id for the interface.

Entering configuration mode terminal
default-1(config)# no interfaces interface 2/1.0 ethernet config aggregate-id ;commit
Commit complete.
default-1(config)#

2. Re-add the aggregate-id.
default-1(config)# interfaces interface 2/1.0 ethernet config aggregate-id lag1 ;commit

3. Reboot the blade (optional step -- use only if step 1 and 2 did not fix the problem).

1073581 : Removing a 'patch' version of services might remove the associated 'base' version as well

Component: F5OS-C

Symptoms:
Removing a 'patch' version (X.Y.Z, Z>0) of a platform ISO or services might, under certain conditions, lead to the unexpected removal of the 'base' version (X.Y.0) associated with that patch.

Conditions:
1. A 'patch' ISO is imported when the 'base' associated with the patch is not already imported (Ex. An F5OS-C 1.2.2 ISO is imported, and F5OS-C1.2.0 is not already imported).
2. Some time later, the F5OS-C 1.2.2 ISO is removed. This also removes the 1.2.0 services.

Impact:
F5OS-C removes software that wasn't explicitly chosen to be removed.

Workaround:
To work around this issue, import the 'base' version ISO (X.Y.0) before importing any patches. If this is done, removal of a 'patch' will not remove the 'base'. If a 'base' was already removed accidentally, re-importing the 'base' ISO will also make it available again.

Fix:
N/A

1073305-1 : Upgrade to F5OS-C 1.3.0 failed to upgrade chassis partition

Component: F5OS-C

Symptoms:
Upgrading VELOS from F5OS-C 1.2.2 to 1.3.0 caused partition containers to go in crashbackoffloop back. This can be checked by running this command:

oc get pods --all-namespaces |grep -i crash

Conditions:
After upgrading to F5OS-C 1.3.0, tenant datapath interfaces do not come up.

Impact:
Traffic is impacted.

Workaround:
Restarting the chassis partition, that is, disabling and enabling the chassis partition fixes the issue.

Fix:
N/A

1072209 : Packets are dropped on VELOS when a masquerade MAC is on a shared VLAN

Component: F5OS-C

Symptoms:
On the VELOS platform, any packets destined to a masquerade MAC address are dropped when the masquerade MAC is located on a shared VLAN (a VLAN shared between multiple F5OS tenants).

On rSeries hardware platforms, all traffic for this MAC is first handled by the software-rebroadcaster and is replicated to all tenants sharing that VLAN.

Conditions:
-- A masquerade MAC is configured on a shared VLAN.
-- Traffic to the MAC address is initiated, i.e. ping a floating self-ip
-- The packets are dropped on ingress

Impact:
Connectivity issues.

Workaround:
Configure a static fdb entry at the partition level.

Fix:
Packets are no longer dropped when a masquerade MAC is on a shared VLAN.

1069917 : Platform registry ports can become de-synchronized, impacting Openshift deployment

Component: F5OS-C

Symptoms:
Under certain circumstances, platform services on the active and standby controllers can end up residing in docker registries that use different port numbers. This can result in Openshift pods failing to start, because they expect these port assignments to be synchronized.

Conditions:
Varied causes, which lead to inconsistent registry port assignments on active and standby controllers.

Impact:
Openshift pods cannot start.

Fix:
Added more checks to ensure platform registry ports are synchronized between controllers.

1067077 : NSS: Memory corruption in decodeECorDsaSignature with DSA

Links to More Info: K54450124

1066185 : MIB files cannot be download or exported using file utilities.

Component: F5OS-C

Symptoms:
MIB directory is not available for download or export file utilities

Conditions:
-- BIG-IP Next system
-- You would like to download the MIB file(s) via the file utilities API

Impact:
You are unable to download the MIBs or export them.

Workaround:
None

1064225 : HTTP response status codes do not match the result of the file import/export operations

Links to More Info: BT1064225

Component: F5OS-C

Symptoms:
HTTP response status codes do not match the result of the file import/export operations.

Conditions:
When the unallowed path is given in file import CLI

Impact:
User experience

Workaround:
None

Fix:
Return the proper HTTP codes in error scenarios as well.

1060097 : Chassis admin user cannot use SCP to import controller/chassis partition ISO images

Component: F5OS-C

Symptoms:
The controller admin user cannot copy ISO images to the chassis for import.

Conditions:
SCP is not allowed for the admin user.

Impact:
User must either use root shell access or a different upload protocol that may not be convenient based on the user network configuration.

Workaround:
None

Fix:
The controller admin use can now use the SCP command on a client machine to upload controller or chassis partition ISOs to the chassis active or floating management IP by specifying a target directory of "IMAGES/" in the same fashion that the partition admin user can use for importing tenant images.

scp F5OS-C-1.5.0-4444.PARTITION.DEV.iso admin@activeccip:IMAGES/

1059073 : QKView upload to iHealth is not supported by web proxy

Component: F5OS-C

Symptoms:
The upload to iHealth feature for sending QKView data to the external F5 service is not supported by web proxy. This means that QKView files must be copied to a local computer on your intranet before the files can be uploaded.

Conditions:
Uploading a QKView file to the iHealth service.

Impact:
Several steps are required to move QKView files around before uploading to iHealth, which is inconvenient.

Workaround:
Copy the QKView file to a computer on the intranet.

Then you can upload the QKView file to the iHealth service from that secondary computer when it has internet access.

Fix:
Web proxy now supports uploading QKView files to iHealth.

1053873 : Partition CLI command 'show cluster nodes node' may time out

Component: F5OS-C

Symptoms:
Running the command 'show cluster nodes node' in the chassis partition CLI may cause a time-out.

Conditions:
Internal problem within the CLI design can trigger this situation where the CLI command fails and times out.

Impact:
User is unable to access the output of this command.

Fix:
CLI command infrastructure was redesigned.

1051241 : LAG and interface names should not contain special characters such as whitespace, asterisk, slashes, and curly braces.

Links to More Info: BT1051241

Component: F5OS-C

Symptoms:
-- LAGs (trunks) not functional in system.
-- LACPD daemon in a restart loop
-- The "LAGs" and "Interfaces" pages in the F5OS partition GUI fail to load and report "Something went wrong. Check the web browser console for more details or contact technical support for assistance."

Conditions:
-- A LAG is defined that has a space in the name.

When a user creates an LAG interface that contains whitespace characters, the LACPD daemon fails to read the interface information, and keeps restarting until the interface is removed or the name is changed to a regular name.

Impact:
All trunks in the partition are down.

Workaround:
Use the the F5OS partition CLI to remove the LAG, for example:

ottersPart-1(config)# no interfaces interface "test lag"
ottersPart-1(config)# no interfaces interface 2/2.0 ethernet config aggregate-id
ottersPart-1(config)# commit and-quit
Commit complete.
ottersPart-1#

Fix:
N/A

1043701 : Allow configurable partition hostname

Component: F5OS-C

Symptoms:
The partition name is set by the chassis administrator at creation time and cannot be changed. Every chassis starts with a partition named "default", so it can be difficult to determine what partition the user is connected to.

Conditions:
Users who have multiple partitions on multiple chassis, and need to easily tell them apart.

Impact:
User confusion, possible unintended configuration changes when connected to the wrong partition.

Workaround:
Create all partitions with unique names, and remove/do not use the "default" partition.

Fix:
The partition now has a configurable hostname under /system/config/hostname. This hostname accepts either a simple label or a fully qualified domain name, following DNS name rules.

The first label is used in the partition prompt in preference to the partition name.

The partition hostname can be set by the chassis administrator, or by partition administrator if not configured at the chassis level.

Behavior Change:
Partition hostname can now be configured, and is used in the partition CLI prompt if present.

1040461 : Permissions of some qkview control files do not follow standards

Links to More Info: BT1040461

Component: F5OS-C

Symptoms:
Permissions of some qkview control files do not follow standard.

Conditions:
Viewing permissions of qkview files.

Impact:
Some do not follow standards.

Workaround:
None

Fix:
Permissions of all qkview control files now follow the standards.

1039721 : The system image displays data for only one VELOS controller★

Component: F5OS-C

Symptoms:
The 'show system image' command displays image details for only one VELOS controller, even though there is more than one.

Conditions:
Running the 'show system image' command after uploading VELOS software.

Impact:
No image information is displayed for a VELOS controller on the user interface.

Workaround:
Reboot the VELOS controllers (specifically vcc-partition-software-manager container).

Fix:
This issue has been fixed.

1034093 : protobuf vulnerability: CVE-2021-3121

Component: F5OS-C

Symptoms:
A flaw was found in github.com/gogo/protobuf before 1.3.2 that allows an out-of-bounds access when unmarshalling certain protobuf objects.

Conditions:
- Unmarshalling protobuf objects

Impact:
This flaw allows a remote attacker to send crafted protobuf messages, causing panic and resulting in a denial of service. The highest threat from this vulnerability is to availability.

Workaround:
N/A

Fix:
Protobuf updated to mitigate CVE-2021-3121

• The F5 Networks Technical Support web site: http://www.f5.com/support/
• The AskF5 web site: https://support.f5.com/csp/#/home
• The F5 DevCentral web site: http://devcentral.f5.com/

Generated: Tue Aug 9 08:39:48 2022 PDT

©2022 F5, Inc. All rights reserved.