999833 : Backplane connectivity failures upon blade boot

Solution Article: K74379524

Component: F5OS

Symptoms:
After powering on or rebooting one or more blades in a partition, blades can fail to establish data plane connectivity between each other across the backplane, resulting in complete loss of traffic in between blades.

This can exhibit itself with symptoms such as an inability for multi-blade tenants to cluster, or traffic received by one blade being unable to route that traffic to another blade, among other possibilities.

The problem can be identified more specifically by inspection of the tmstat 'vqf_cfg' table in the partition_fpga container. On each blade perform the following:

docker exec -it partition_fpga tmctl -d blade -w 120 vqf_cfg

If the 'active_blades' column does not have the same value for all assigned and enabled blades within a partition, and are otherwise expected to be running with good states, then this issue has likely been encountered.

Conditions:
A blade is booted for the first time, or rebooted.

Impact:
Loss of data plane traffic in between blades within a partition.

Workaround:
Attempt to reboot the blade until the condition clears.

999405 : Partition high availability (HA) framework core on shutdown

Component: F5OS

Symptoms:
Corefile named "underdog-1.core.gz" is found.

Conditions:
If the partition "system database config-backup" or "system database config-restore" commands are used the high availability (HA) framework daemon (process named "underdog") may produce a corefile the next time it is shut down.

Impact:
A corefile is produced.

Fix:
The config-backup/config-restore commands no longer corrupt memory and produce a corefile.

999229-1 : Management session may be unresponsive after rebooting the standby System Controller.

Component: F5OS

Symptoms:
When the standby system controller is rebooted, the management session may freeze for 10-15 seconds.

Conditions:
The standby system controller is rebooted

Impact:
The management session is frozen briefly, then resumes normal operation.

Workaround:
None

997821 : Bi-directional optics part is not recognized and interface remains in the down state.

Component: F5OS

Symptoms:
Interface remains in down state even with optics and fiber inserted, and the optics type is not identified

Conditions:
Bi-Directional optics is present in the system

Impact:
Interface will not be operational

Workaround:
None

Fix:
Bi-Directional optics type should be recognized and interface should be operational

997237-1 : Changing chassis-id in system network config causes the OpenShift cluster not to install correctly.

Component: F5OS

Symptoms:
If the chassis-id is changed from the default of 1 in the 'system network' configuration on the system controller, the OpenShift cluster does not reinstall correctly once the change takes effect.

Conditions:
Chassis-id is changed from default of 1 to any other value.

Impact:
OpenShift cluster does not re-install correctly, which means tenants cannot be launched on the system.

Workaround:
None

995841 : IPv6 addresses defined in RADIUS servers fail

Component: F5OS

Symptoms:
Setting up a IPv6 RADIUS server fails.

Conditions:
-- RADIUS server being defined
-- An IPv6 address is used

Impact:
Radius servers configured with IPv6 cannot be used.

Workaround:
Use only IPv4 addresses.

Fix:
The fixed version now allows both ipv4 or ipv6 addresses.
Note that the radius server might need to have a separate entry for each type of address.

Behavior Change:
ipv6 addresses are now supported which wasn't the case before.

995801 : CVE-2018-18074: python vulnerability

Component: F5OS

Symptoms:
A credentials-exposure flaw was found in python-requests, where if a request with authentication is redirected (302) from an HTTPS endpoint to an HTTP endpoint on the same host, the Authorization header is not stripped and the credentials can be read in plain text. A man-in-the-middle attacker could exploit this flaw to obtain a user's valid credentials.

Conditions:
A credentials-exposure flaw was found in python-requests, where if a request with authentication is redirected (302) from an HTTPS endpoint to an HTTP endpoint on the same host, the Authorization header is not stripped and the credentials can be read in plain text. A man-in-the-middle attacker could exploit this flaw to obtain a user's valid credentials.

Impact:
A credentials-exposure flaw was found in python-requests, where if a request with authentication is redirected (302) from an HTTPS endpoint to an HTTP endpoint on the same host, the Authorization header is not stripped and the credentials can be read in plain text. A man-in-the-middle attacker could exploit this flaw to obtain a user's valid credentials.

Workaround:
N/A

995785 : CVE-2019-11068: libxlst vulnerability

Component: F5OS

Symptoms:
libxslt through 1.1.33 allows bypass of a protection mechanism because callers of xsltCheckRead and xsltCheckWrite permit access even upon receiving a -1 error code. xsltCheckRead can return -1 for a crafted URL that is not actually invalid and is subsequently loaded.

Conditions:
libxslt through 1.1.33 allows bypass of a protection mechanism because callers of xsltCheckRead and xsltCheckWrite permit access even upon receiving a -1 error code. xsltCheckRead can return -1 for a crafted URL that is not actually invalid and is subsequently loaded.

Impact:
libxslt through 1.1.33 allows bypass of a protection mechanism because callers of xsltCheckRead and xsltCheckWrite permit access even upon receiving a -1 error code. xsltCheckRead can return -1 for a crafted URL that is not actually invalid and is subsequently loaded.

Workaround:
N/A

995781 : CVE-2019-3861: libssh2 vulnerability

Component: F5OS

Symptoms:
An out of bounds read flaw was discovered in libssh2 in the way SSH packets with a padding length value greater than the packet length are parsed. A remote attacker who compromises a SSH server may be able to cause a denial of service or read data in the client memory.

Conditions:
An out of bounds read flaw was discovered in libssh2 in the way SSH packets with a padding length value greater than the packet length are parsed. A remote attacker who compromises a SSH server may be able to cause a denial of service or read data in the client memory.

Impact:
An out of bounds read flaw was discovered in libssh2 in the way SSH packets with a padding length value greater than the packet length are parsed. A remote attacker who compromises a SSH server may be able to cause a denial of service or read data in the client memory.

Workaround:
N/A

995777 : CVE-2016-4448: libxml2 vulnerability

Solution Article: K14614344

995733-1 : The LACPD process crashes and writes a core file following physical interface deletion.

Component: F5OS

Symptoms:
The lacpd process crashes and writes a core file following deletion of the physical interface if the interface is a working member of an LACP trunk. Physical interface deletes are commonly caused by either removing an associated blade from the partition or changing the port bifurcation configuration (e.g., from 1x100G to 4x25G).

Conditions:
Delete a physical interface that is a working member of an LACP trunk.

Impact:
Lacpd crashes and writes a core file; traffic disrupted while lacpd restarts.

Workaround:
Delete the aggregate-id for any physical interface before the interface is deleted.

Fix:
The system now checks whether the working member is deleted and does not core.

995061 : CVE-2019-17006: NSS Vulnerability

Component: F5OS

Symptoms:
A vulnerability was discovered in nss where input text length was not checked when using certain cryptographic primitives. This could lead to a heap-buffer overflow resulting in a crash and data leak. The highest threat is to confidentiality and integrity of data as well as system availability.

Conditions:
A vulnerability was discovered in nss where input text length was not checked when using certain cryptographic primitives. This could lead to a heap-buffer overflow resulting in a crash and data leak. The highest threat is to confidentiality and integrity of data as well as system availability.

Impact:
A vulnerability was discovered in nss where input text length was not checked when using certain cryptographic primitives. This could lead to a heap-buffer overflow resulting in a crash and data leak. The highest threat is to confidentiality and integrity of data as well as system availability.

Workaround:
N/A

994429-1 : Lost traffic on previously active system controller.

Component: F5OS

Symptoms:
Traffic is lost on the previously active system controller, so half of the traffic is lost.

Conditions:
If active system controller does not function, here are some examples:
-- The system controller is physically non-functional (no electrical activity, etc.).
-- System software results in system controller failures for a long time.

Impact:
Traffic is lost on the previously active system controller, so half of the traffic is lost.

Workaround:
Reboot the newly active system controller.

Fix:
The system does not lose traffic when the active system controller fails and does not come back.

994321 : Partition shows "failed" state when enabled for brief period

Component: F5OS

Symptoms:
After enabling a partition, the "show partitions" command will show a status of "failed" while the partition instance is in the process of starting up, but has not yet started. Once the startup completes, the status will transition to "running", "running-active", or "running-standby".

Conditions:
Partition is starting after being enabled.

Impact:
The user may think the partition software has encountered a failure condition.

Workaround:
Wait for 20 to 30 seconds for the status to be reported correctly.

Fix:
The partition status reporting logic now uses additional state information to correctly distinguish between "starting" and "failed".

993985-1 : Image import is not re-attempted if controller reboots while importing★

Component: F5OS

Symptoms:
If a system controller reboots in the middle of importing platform software, the import does not automatically restart on the next boot. Additionally, it is not possible to overwrite it via a new file transfer to try again.

Conditions:
-- Valid F5OS software has begun importing on a system controller, but the import is not complete yet.
-- A reboot is issued by either the admin user or the software.

Impact:
The software is not imported.

Workaround:
Follow this procedure:

1. Download the software again, but using a different destination file name.

2. Try the import operation again.

993325-1 : System controller does not have remote method to power on after being remotely powered off.

Component: F5OS

Symptoms:
If the system controller is powered off from the Linux bash shell using 'shutdown -P' or an equivalent command, there is no method available in Always On Always-On Management (AOM) menu, or any other method, to remotely power back on the system controller.

Once the system controller is powered off, it can be powered on only by either reseating the system controller or performing a full chassis power cycle.

Conditions:
The system controller is powered off using a bash command.

Impact:
You are unable to remotely power on a system controller after it has been powered off.

Workaround:
Reseat the controller or power cycle the chassis.

991309-1 : 'TPM Randomization failed' message displays in the log.★

Component: F5OS

Symptoms:
When the BIOS on a blade is freshly updated and the blade is booted, there may be a 'TPM Randomization failed' message observed in the log output. This message occurs only once (if at all) and is then followed by a successful boot of the system. This is caused by the TPM randomization step of the boot not being completed. The next boot of the system, initiated by the BIOS when this condition is encountered, allows the TPM randomization to complete. For security purposes, the system does not boot to the OS until TPM randomization has successfully completed.

Conditions:
This particular instance of this message may occur after a fresh BIOS update to the system. If the initial boot after a BIOS install is interrupted, the subsequent boot may display this message in the log. The BIOS then causes the blade to reboot, allowing the TPM randomization step to complete.

Impact:
There is no impact to the functionality of the system. The message is for informational purposes only in this situation. If the TPM randomization step of the BIOS execution is not able to complete successfully, the BIOS causes the blade to reboot until the TPM randomization is successful.

Note: If the system continually fails to boot due to this issue, you may have a hardware issue that requires F5 response to correct.

Workaround:
Allow the blade to continue to boot. Once the BIOS has caused the blade to reboot, the condition clears itself and the message no longer appears as part of the boot process.

991029 : Primary-Key-Migration set response message references incorrect command.

Component: F5OS

Symptoms:
After setting the primary key, the system reports the following message:

-- Response Info: Key migration is initiated. Use 'show system primary-key state status' to get status.

This is the wrong command.

Conditions:
This is encountered when running the following command:

system aaa primary key set

Impact:
The prompt for the set command is misleading.

Workaround:
Use the command:
show system aaa primary-key state status

990897-1 : Tenant operational state has incorrect data when it has no nodes.

Component: F5OS

Symptoms:
When the admin removes nodes from tenant configuration and upgrades the system controller software, the tenant status comes up with an error state.

Conditions:
Admin removes nodes from the tenant configuration using either of the following commands:

no tenants tenant <name> config nodes

tenants tenant <name> config nodes [ ]

Impact:
This has no effect on system functionality. The system is simply reporting an incorrect state of the tenant.

Workaround:
To process any traffic through the tenant, you must have a minimum of one node in the tenant config, for example:

show running-config tenants tenant <name> config nodes
tenants tenant example1
 config nodes [ 1 ]
!

990757 : File extension validation on URL input fields for F5OS File Utilities and Software Management are not case insensitive.

Component: F5OS

Symptoms:
The input validation on the URL field when you import a file, checks that the URL ends in .iso or .img, but it does not accept mixed case or uppercase entries, for example, .ISO / .IMG / .iMg / .Iso, etc.

Conditions:
-- Importing a file, either with File Utilities or the Software Management import utilities for System Controller and Chassis Partition software images.
-- The URL has a file extension that is not lowercase.

Impact:
The URL field reports a validation warning.

Workaround:
Make sure the file extension appended on the end of the URL is all lowercase.

Fix:
File extension validation on URL input fields for F5OS File Utilities and Software Management features is now case insensitive.

990753 : Importing a file can result in errors when parameters are correct.

Component: F5OS

Symptoms:
Importing a file using File Utilities, or using the Software Management import utilities for System Controller and Chassis Partition software images can result in an error, even when the URL and other parameters are correct.

Conditions:
-- Importing a file.
-- The first attempt has an invalid URL.
-- Subsequent import attempts contain similarities in the URL paths used on the first attempt.

Impact:
The import fails and the error is ambiguous.

Workaround:
Refresh the screen and initiate a new file import attempt, making sure the URL and all other parameters entered are correct and valid.

Fix:
Importing a file no longer reports errors when parameters are correct.

990749 : The URL field on file import does not validate a HTTPS protocol.

Component: F5OS

Symptoms:
A URL that does not specify HTTPS as the protocol is accepted as valid input, but results in a failed file transfer.

Conditions:
-- Importing a file to the device.
-- Enter a URL that does not specify HTTPS as the protocol.

Impact:
The URL field accepts the input, but the file transfer fails. Lack of protocol validation on the URL input does not prevent you from entering a URL that can result in a failed file import.

Workaround:
All file import URLs must use HTTPS protocol.

Fix:
URL field on file import now validates HTTPS protocol.

988669-1 : Interface config TPID leaf is exposed but it is not supported.

Component: F5OS

Symptoms:
Interface config tpid leaf is exposed but it is not supported.

Conditions:
-- In CLI config mode, the system presents tpid as a possible config option.
-- In CLI show mode, tpid is shown as oper data.

Impact:
Interface config tpid leaf is exposed but it is not supported.

Workaround:
Ignore the tpid from interface/config or interface/state.

987509 : CVE-2020-1971: OpenSSL vulnerability

Solution Article: K42910051

987225 : Viewing the content of restricted directories

Component: F5OS

Symptoms:
The file list/show command provides unrestricted access to all the directories.

Conditions:
Viewing the content of directories.

Impact:
User can view all the directories.

Workaround:
N/A

Fix:
Fixed the File list/show commands to restrict access to whitelisted directories.

986061 : Partition config-backup silently overwrites a previous config backup.

Component: F5OS

Symptoms:
The partition 'system database config-backup' command overwrites an existing backup file.

Conditions:
Running the partition 'system database config-backup' command.

Impact:
If a config backup exists, the operation silently overwrites it.

Workaround:
Specify a unique filename for each backup operation:

system database config-backup name unique-filename

985009 : PSU data missing in the show components component.

Component: F5OS

Symptoms:
After a system controller failover, the PSU information may not be populated in the command: show component component.

Conditions:
Active system controller fails over to the standby.

Impact:
PSU data is not listed. There is no operational impact to the system.

Workaround:
A total system reboot is the only option for listing the PSU data in the 'show components component' output.

Fix:
PSU data is now present in show components component results.

984753 : Unable to read key from storage.★

Component: F5OS

Symptoms:
The checksum changed from size 256 to 512. Systems that created the encryption keys using 256 would fail when upgrading.

Conditions:
System must have been configured with old software that wrote 256-bit hashes and upgraded to a newer version that expected 512-bit hashes.

Impact:
The upgrade does not accept the key since the hash does not match.

Workaround:
Going into the vcc-confd container and requesting the key twice would change the key and store a new 512 hash.

From the System Controller:

# docker exec -it vcc-confd bash
# getConfdMasterKey.sh --storage eeprom
# exit

Fix:
With the fix, either 256 or 512 hash keys are accepted so upgrading from an older system does not fail as before.

984721 : CLI commands for DNS and NTP could be simplified

Component: F5OS

Symptoms:
The CLI commands to configure DNS and NTP require specifying addresses twice.

For example, specifying a DNS server:
 config
 system dns servers server 10.10.10.10 config address 10.10.10.10
 commit

Conditions:
Configuring a DNS or NTP server using the CLI commands.

Impact:
There is no operational impact; however, it is preferable to enter the IP address only once.

Workaround:
None. You have to specify the IP address twice.

984073 : Slow system controller operations for CLI image commands.

Component: F5OS

Symptoms:
A number of system controller operations related to images suffer from an approximately 5-second delay. This includes CLI commands such as 'show image partition' and 'show image controller', but there is also a delay when using tab-completion to view suggested completions for such commands. The delay appears to be mostly constant, independent of the number of partitions in the system. The delay is always present for these commands.

Conditions:
Using CLI commands related to images.

Impact:
In addition to the 5-second delays experienced directly on the CLI, there is likely similar impact to related GUI functionality, such as the partition management page. These delays do not appear to negatively impact any functionality.

Workaround:
None

Fix:
Previously, there was a delay when using CLI commands such as 'show image partition' and 'show image controller', or when using tab completion with these commands. This has been fixed and there is no longer a delay.

982189 : Partition status displayed by system controller may not be accurate

Component: F5OS

Symptoms:
The system controller may display a partition status of "running" rather than "running-active" or "running-standby".

Conditions:
If the system controller component responsible for reporting the partition state restarts, or misses a state-transition message, the state will not be updated until the next state-transition.

Impact:
Incomplete and possibly misleading status information is reported to the user.

Fix:
The state-reporting mechanism has been updated to account for missed messages or restarts, so that displayed status will be accurate.

981605 : Qkview truncates lines of top command at 80 characters.

Component: F5OS

Symptoms:
The top command defaults to 80 characters, if run outside of a console, as it is in qkview.

When top is executed in this fashion, the -w parameter must be used, in order to specify output width.

Conditions:
Run qkview.

Impact:
Output of top command is truncated at 80 characters. Some contents of the top command may be missing in qkview files.

Workaround:
Run top separately from qkview.

Fix:
Previously, qkviews were not always complete. This issue has been fixed.

980653 : Transient system error for AOM/LOP fault due to not receiving health reports from all installed fantrays or PSU controllers.

Component: F5OS

Symptoms:
The system may report a transient error for a fault in the AOM/LOP due to not receiving health reports from all installed VFC (fantray) or VPC (PSU controller) cards, and then the error condition is resolved only a couple of seconds later.

For example, the error condition occurring:

65543 controller-1 aom-fault ASSERT ERROR "Fault detected in the AOM" "2020-12-30 01:16:32.522027731 UTC"
65543 controller-1 aom-fault EVENT Network Access "Asserted: LOP is not receiving health reports from all installed VPC cards" "2020-12-30 01:16:32.522051183 UTC"

The resolution of the error two seconds later:

65543 controller-1 aom-fault CLEAR ERROR "Fault detected in the AOM" "2020-12-30 01:16:34.184866455 UTC"
65543 controller-1 aom-fault EVENT Network Access "Deasserted: LOP is not receiving health reports from all installed VPC cards" "2020-12-30 01:16:34.184891732 UTC"

These transient error reports do not indicate an AOM/LOP fault and can be ignored.

Conditions:
No special conditions cause the transient error report for an AOM/LOP fault caused by not receiving health reports from all installed fantrays and PSU controllers. It may occur at any time.

Impact:
The transient error report for an AOM/LOP fault caused by not receiving health reports from all installed fantrays and PSU controllers can be ignored.

Workaround:
The transient error report for an AOM/LOP fault caused by not receiving health reports from all installed VFC(fantray) and VPC(PSU controller)cards can be ignored.

Fix:
Fixed in CC-LOP firmware v1.00.1085.00.1 and later available with F5OS-C 1.2.0.

977849 : tcpdump Hardening

Component: F5OS

Symptoms:
Under certain conditions, tcpdump does not follow current best practices.

Conditions:
Under certain conditions, tcpdump does not follow current best practices.

Impact:
Under certain conditions, tcpdump does not follow current best practices.

Workaround:
N/A

974297 : Certificate import interface does not provide correct actionable error.

Component: F5OS

Symptoms:
Confd CLI gives an irrelevant error message when an expired cert is set to a TLS certificate element.

Aborted: 'system aaa tls config certificate': Key public_key does not match Certificate public_key.

Conditions:
# system aaa tls config key
<key>
# system aaa tls config certificate
<expired cert content>

Impact:
A misleading error message is printed.

Workaround:
None

Fix:
CLI is fixed to give correct error message in all the following cases:

'Public Key does not match certificate'
'Not valid Certificate/Key Contents'
'Certificate is expired.'
'Certificate is not yet valid.'
'Not an OpenSSL Key'
'Bad public key from private key'

974269-2 : Fields on file import and export popups are reset after a failed file import attempt.

Component: F5OS

Symptoms:
All previously filled in fields are reset and cleared on the file import/export popup upon a failed transfer attempt.

Conditions:
A file import or export operation fails.

Impact:
You must fully reenter the information in all of the fields in order to resubmit the file transfer request.

Workaround:
Reenter the data into the fields and reattempt the file transfer.

973573 : Import and sync of platform software on both controllers takes up to 15 minutes.

Component: F5OS

Symptoms:
When importing platform software on a controller, it can take up to 15 minutes to fully import and synchronize between both controllers.

Conditions:
Transferring an ISO image to a VELOS controller for import.

Impact:
Import takes a long time, and the import may appear to be stuck or failed when it is not.

Fix:
Platform software import and synchronization times have been significantly reduced.

973469 : The ed25519 certificate and key are not accepted.

Component: F5OS

Symptoms:
The GUI stops working without any warning or error if an ed25519 crt/key is imported.

Conditions:
Import an ed25519 crt/key.

Impact:
GUI stops working. In the system controller log you see errors:

-- controller-1 /usr/bin/authd[7]: priority="Err" version=1.0 msgid=0x3901000000000026 msg="OpenSSL PEM_read_bio_PrivateKey failed read key" file="server.key".
-- controller-1 /usr/bin/authd[7]: priority="Err" version=1.0 msgid=0x3901000000000022 msg="OpenSSL X509_PUBKEY_get failed to get key.".

Workaround:
Do not use ed25519 crts/keys.

973449 : System date/time do not display in the GUI, cannot be set.

Component: F5OS

Symptoms:
The system does not display the current date/time and timezone in the GUI, which can make it difficult to review alerts or logs without knowing/remembering how the system is configured.

The system does not provide a mechanism to update the system clock directly, without NTP.

Conditions:
Attempting to view or set system time and date via the GUI.

Impact:
You cannot view or set system time and date through the GUI.

Workaround:
None

973217 : Qkview generation mishandles filename with space in it.

Component: F5OS

Symptoms:
If the qkview filename contains a space, the system uses only first word for the qkview filename.

Conditions:
Using filenames with spaces in them while generating a qkview.

Impact:
Only the first word is used in the generated qkview filename.

Workaround:
Do not use space in qkview filenames.

965353 : The 'show image' report can have staggered output.

Component: F5OS

Symptoms:
The 'show image' report can conflate subtable output into a single, long set of columns. In some instances, the report begins to stagger row output. Although the information presented is accurate, the format is suboptimal.

Conditions:
Very wide screen widths cause table conflation to take place for the 'show image' report.

Impact:
Some rows (right-hand side of report) wrap around to the next line, making the report more difficult to interpret.

Workaround:
Adjust the screen width before running 'show image', as in:

syscon-1-active# screen-width 60
syscon-1-active# show image

Fix:
System images report no longer produces conflated output with staggered rows.

962645 : 'limited' user and role are visible

Component: F5OS

Symptoms:
'limited' user and role are for internal use but the role is not hidden.

Conditions:
Viewing the list of available users and roles.

Impact:
The 'limited' user is visible when it should not be.

Fix:
The 'limited' user and role is now hidden.

960893 : The tenant deployment fails if the tenant name exceeds 49 characters.

Component: F5OS

Symptoms:
If an admin configures a tenant with a name that is longer than 49 characters, tenant deployment fails.

Conditions:
Partition created and enabled on VELOS hardware for admin to login and create a tenant configuration.

Impact:
A configured tenant fails to schedule on the VELOS cluster. If the tenant name has more than 49 characters, the server rejects the deployment request.

Workaround:
Delete the existing tenant and create a new tenant deployment using a name with 49 or fewer characters.

Note: The system might not prevent you from using more characters, but the recommendation is 49 or fewer.

957129 : Qkview collection running on peer does not cancel when main qkview is canceled.

Component: F5OS

Symptoms:
Qkview collection is distributed, and there is a main process for collecting qkview information from peer devices. The main qkview process (running on the active system controller) or the partition manager, spawns processes to collect from its peers.

Peers are not aware of whether the main qkview operation has been canceled.

Conditions:
A qkview is canceled, and then immediately restarted.

Impact:
Partial qkview collection.

When a qkview is canceled on the main collection system, the peers are not aware of this, and continue to collect. The peer qkviews may not be collected if the peers are still processing the last qkview request.

Workaround:
Wait 5 minutes after canceling, and then run qkview again.

957093 : Switch-related events with Notice severity found in confd event log during blade reboot.

Component: F5OS

Symptoms:
Several switch-related events sometimes occur when a blade is rebooted. These events are generated if the switch port to which the blade is connected reports an FEC Uncorrected Error, and posts error messages similar to the following:

-- NOTICE 'Switch Port in fault state'.

The errors usually clear soon after the blade boots up.

Conditions:
This occurs upon system startup.

Impact:
The system generates a few unwanted events. If the blade boots successfully and networking is functioning normally, you can safely ignore the 'Switch Port in fault state' events.

Workaround:
None

956909-1 : Status led may be left off after LCD test.

Component: F5OS

Symptoms:
Status LED may be left in the off state after an LCD test.

Conditions:
Issue occurs after the LCD test is executed.

Impact:
Status LED may not reflect actual state of the system after an LCD test.

Workaround:
Power-cycling the chassis resolves the Status LED state.

945517 : Root and Admin users need to have their passwords changed from default simultaneously

Component: F5OS

Symptoms:
When the root or admin password is changed for the first time, the other account password is changed if it is still set to the default.

Conditions:
Initial password change

Impact:
The initial password change will change both the root and admin passwords.

Behavior Change:
When the root or admin password is changed for the first time, the other account password is changed if it is still set to the default.

939893-1 : The CLI does not include firmware version information for sirr or ssd.

Component: F5OS

Symptoms:
The show components components information does not include the firmware version information for the sirr or ssd.

Conditions:
Running the show components command.

Impact:
The show components command does not report the firmware version information for the sirr or ssd data fields.

Workaround:
None

1042845-3 : Unable to remove platform services versions that appear unused

Component: F5OS

Symptoms:
Under certain circumstances, a version of controller or partition services may appear "not in use" in ConfD/GUI tables, but removal of that version is still blocked because other parts of the service package are still in use by other system components.

Conditions:
1. Attempt to remove an (apparently inactive) version of controller or partition services via ConfD or GUI.
2. Other components on the system still silently depend on that version of services, even though ConfD/GUI output does not reflect this.

Impact:
Unable to remove versions of software that appear unused, and the cause is unclear.

Workaround:
N/A

Fix:
Removal of platform services that appear "unused" is no longer blocked by hidden higher-level component dependencies

1042253 : System controller upgrade from 1.2.0-10357 to 1.2.1-10301 intermittently fails★

Component: F5OS

Symptoms:
The upgrade proceeds to the point where both system controllers boot to the new image but neither system controller becomes active.

Conditions:
Whenever this issue is observed, show full-configuration system redundancy config mode is something other than the default (auto).

Impact:
Neither system controller becomes active. The ability to configure the System controllers is compromised.

Workaround:
Restarting both Vcc-ConfD containers (or a reboot of both system controllers) should clear the problem.

Fix:
Intermittent loss of active system controller when upgrading from 1.2.0-10357 to 1.2.1-10301 is fixed in 1.2.1.

1041853 : QOS and Software install status screens missing online help

Component: F5OS

Symptoms:
The QOS and Software Install webUI screens should have online help, but they do not.

Conditions:
This is specific to the QOS and Software Install screens in the GUI.

Impact:
Online help is missing for these screens.

Fix:
Help screens are available starting in F5OS-C 1.2.1.

1039085-1 : Partition config restore operation can cause the system to stop processing fdbs.

Component: F5OS

Symptoms:
In rare cases, a partition config-restore operation can cause a race condition that locks up a platform component. This causes fdbs to no longer be processed, and can affect traffic processing.

Conditions:
Issuing a config-restore operation on the partition cli. This issue is more likely to occur when the number of tenants increases.

Impact:
Fdbs will no longer be processed. Traffic processing can be impacted due to missing fdbs.

Workaround:
First, restart the network manager on both controllers:
    - "docker restart partition<partition_number>_network_manager"
Second, redeploy all tenants.

1033953 : When a large number of VLANs are configured, a blade might run out of memory

Component: F5OS

Symptoms:
When a large number of VLANs are configured, the system might run out of memory.

The api_svc_gateway service's memory usage might grow without bounds.

Conditions:
A large number of VLANs are configured.

Impact:
System might become slow and go down.

Workaround:
None

Fix:
Fixed unbounded memory usage by api_svc_gateway.

1032341-1 : Confd Encryption key gets rewritten intermittently

Component: F5OS

Symptoms:
The key should always return the same value and hash, unless it is changed via key-migration.

The reading of memory (EEPROM) will sometimes return "resource temporarily unavailable" which is treated as an error instead of simply doing a retry.

Conditions:
The EEPROM might be busy because of use by other components.

Impact:
The encryption key changes, thus invalidating all currently encrypted items, thus requiring re-entry of these.

Workaround:
The only workaround is to re-enter all encrypted items and hope that the "resource temporarily unavailable" does not occur.

Fix:
Fixed an issue where the system no longer considers "resource temporarily unavailable" as an error unless it happens 10 times in a row. The system does a retry and if that works, the system avoids setting a new key.

1029205 : The upgrade fails when there are many partitions.★

Component: F5OS

Symptoms:
Upgrade fails with an error

syscon-2-active(config)# system image check-version iso-version 1.2.0-8089
Error: Compatibility verification failed.

Conditions:
8 or so partitions

Impact:
Upgrade fails

Workaround:
$ echo 256 > /proc/sys/fs/inotify/max_user_instances

Fix:
Increase inotify tunable.

1028873 : Colon character is not allowed in the password.

Component: F5OS

Symptoms:
Password change fails when the password has colon character

Conditions:
Colon character in the password

Impact:
Password change fails.

Fix:
Handle colon in the password properly

1028381 : Both system controllers may report as active.

Component: F5OS

Symptoms:
Vcc-ConfD may hang, causing a watchdog timer to expire. This causes both controllers to appear as 'active.'

Conditions:
It is unknown which conditions trigger this.

Impact:
When it occurs, the system is degraded. System upgrades will fail, other management operations may fail, and traffic could be disrupted.

Workaround:
None

Fix:
Vcc-ConfD should no longer hang and let the watchdog timer expire.

1028033 : Tcpdump capture file can fill the /var/F5/partition.

Component: F5OS

Symptoms:
Tcpdump capture files are located at /var/F5/partition. As captures grow in size, these may use up all the space on /var/F5/partition, starving other components that want to use this storage.

Conditions:
Tcpdump has to be started with a capture to file ( "-w") option

Impact:
/var/F5/partition may run out of space, and other components that need space might error

Fix:
This fix does the following:

- If started with the "-w" option, the pcap file size is limited to: available space in /var/F5/partition - 2GB
- If there is less than 2GB, tcpdump will throw an error ("ERROR: Only ... 1K blocks available in /var/F5/partition. Need atleast ... 1K blocks of free space.") and not start the tcpdump session
- If there is less than 100MB, tcpdump will print a "WARNING__Capture_file_size_will_be_limited_to_97216_1K_blocks)"
- If the space is less than 1 MB, tcpdump will fail. Explicit -C option can be used to override

1027929 : Adding a VLAN to a LAG that is already configured on a tenant may not configure the VLAN correctly.

Component: F5OS

Symptoms:
Traffic egressing the VELOS system does not reach the external destination.

Conditions:
A VLAN is configured on a tenant and the VLAN is added to a LAG which does not have members from all blades in the partition.

Impact:
Traffic is disrupted.

Workaround:
Remove VLANs from the tenant, then add them to the lag, then re-add them to the tenant.

Fix:
When a VLAN is added to a LAG, program the host VLAN table
for blades that do not contain LAG members.

1027837 : Media type of optics with part number OPT-0047 reports as unknown.

Component: F5OS

Symptoms:
Optics media types is displayed as unknown

Conditions:
Optics with part number OPT-0047 is present in the system

Impact:
Media type will not be known

Workaround:
NA

Fix:
Media type should be reported as 100G PAM4 BiDi

1026237 : Partition high availability (HA) framework can fail to report 'failed' state if the node crashes immediately when becoming 'active'

Component: F5OS

Symptoms:
The system controller "show partitions partition" status output shows both controller 1 and controller 2 as "running-active".

The partition "show system redundancy" also reports incorrect active/active status.

Conditions:
Normally, the partition instance runs on both system controllers in an Active/Standby configuration. If the Active yields or fails, the Standby will become Active.

If the partition instance crashes immediately after reporting Active status, it does not update the status reported to the system controller software, so will appear to be active. The other partition instance will detect the crash, and reclaim the active role if possible.

The observed instance of this problem occurred when the partition volume ran out of space due to an large number of qkviews and tcpdumps.

Impact:
The system controller and partition CLI/GUI will display both partition instances as "active", even though only one of them is.

Workaround:
To prevent the crash, limit the number of partition qkviews stored on the partition. Once taken, the qkview should be promptly copied to another system and deleted before taking another.

Fix:
The partition high availability (HA) framework no correctly updates status to when a database crash is detected, and now shows a "failed" when the database is inoperable.

1025949 : The partition storage is limited.

Component: F5OS

Symptoms:
Tcpdump or qkview dump fails due to lack of volume free space.

Conditions:
When tcpdump log is too big or qkview is captured multiple times, it can run out of partition volume space.

Impact:
Partition will will unstable.

Workaround:
-- Limit tcpdump logging.
-- Don't capture qkview multiple times before removing old ones.

Fix:
Increase partition volume size.

1025937 : Observing genError when SNMPWalk performed after removing blade from the partition.

Component: F5OS

Symptoms:
After removing blade from partition, snmpwalk fails

Conditions:
1. Remove blade from partition
2. After removing blade, perform snmpwalk for same partition

Impact:
An error occurs while running snmpwalk on the partition.

Workaround:
None

Fix:
Updated attribute's range in snmp platform-stats mib.

1024333 : iHealth related show commands fail when run individually or as part of "show system."

Component: F5OS

Symptoms:
Either when " show system" runs, or when individually run as in "show system diagnostics ihealth state username", these commands fail with ""error-tag": "malformed-message".
/var/confd/log/devel.log will show error messages as follows:
 devel-c get_elem error {external_timeout, ""}

Conditions:
show system" command is run , or when individually ihealth show commands are run as in "show system diagnostics ihealth state username

Impact:
Show commands will fail

Fix:
" show system" command and ihealth show commands such as "show system diagnostics ihealth state username" will succeed

1023937 : L2 Listener entries are not created for tenants sharing the same VLAN

Component: F5OS

Symptoms:
After creating two tenants that share a VLAN, the L2 Listener entries for these tenants were not created. This results in all tenant traffic matching the RBCAST-LISTENER vlan-listener where it is handled by the Software Rebroadcaster.

Conditions:
STEPS TO REPRODUCE:

- Here used the default partition

- Create a vlan

  default-2(config)# vlans vlan range 1155 config name lab-vlan

- Assign vlan to an interface

  default-2(config)# interfaces interface 1/1.0 ethernet switched-vlan config trunk-vlans 1155

- Create two tenants that share vlan 1155.

  default-2(config)# tenants tenant foo-tenant config image BIGIP-14.1.4.3-0.0.8.ALL-VELOS.qcow2.zip.bundle cryptos enabled nodes 1 mgmt-ip 10.10.10.1 prefix-length 24 gateway 10.10.10.254 running-state deployed vlans 1155
  default-2(config-tenant-foo-tenant)#
  default-2(config)# tenants tenant bar-tenant config image BIGIP-14.1.4.3-0.0.8.ALL-VELOS.qcow2.zip.bundle cryptos enabled nodes 1 mgmt-ip 10.10.10.2 prefix-length 24 gateway 10.10.10.254 running-state deployed vlans 1155
  default-2(config-tenant-bar-tenant)# commit

- Run "show fdb" to see if L2 listener entries were created for the tenants.

Impact:
This issue results in all tenant traffic matching the RBCAST-LISTENER vlan-listener where it is handled by the Software Rebroadcaster.

Workaround:
NA

Fix:
Fixed an issue where L2 Listener entries were not created after creating two tenants that share a VLAN.

1023837 : Tcpdump captures to a file include FILEINFO with incorrect information.

Component: F5OS

Symptoms:
When tcpdump captures are done to a file, the pcap file includes FILEINFO which has version, platform and other such information but the information is incorrect.

Conditions:
Tcpdump capture to a file (-w optin) is performed

Impact:
Correct information about version and platform cannot be obtained from captured file

Fix:
FILEINFO will have correct version, platform and other information

1023729 : After a system controller reboot, the tenant operational status is set to a failed state in confd even though tenant stays running.

Component: F5OS

Symptoms:
After the active system controller reboots, the state of the tenant in the 'show tenants' confd output displays a failed state even though the tenant remained in the running state.

Conditions:
-- The active system controller has been recently rebooted.
-- The Openshift API server has not yet come online.

Impact:
You will see an incorrect tenant state of failed when the actual tenant state is running.

Workaround:
The tenant status will be updated correctly once the openshift API server is back online. This could take up to 5 minutes.

Fix:
During the system controller reboot when the Openshift API server is offline, the state of the tenant is no longer updated to a failed state since the tenant is still in a running state.

1023577 : Unable to delete the tenant images.

Component: F5OS

Symptoms:
You are unable to delete the tenant images using the CLI.

Conditions:
When an image is downloaded using "file import ..." CLI and then trying to delete the same image using "images remove .. " CLI, the removal fails and throws an error.

Impact:
The partition disk volume fills up and may cause the system to behave in an unexpected way.

Workaround:
None

Fix:
Images are given correct permission so that the delete is possible.

1023561 : Restoring from a chassis partition backup restores out-of-date tables

Component: F5OS

Symptoms:
When running "system database config-restore" on the chassis partition CLI, the fdb and vlan-listeners tables are erroneously restored. These tables can contain stale data that interferes with datapath configuration.

Conditions:
This issue occurs when running "system database config-restore" on the partition CLI.

Impact:
The datapath to tenants may not work until the tenants are re-deployed.

Workaround:
Delete the fdb and vlan-listeners tables immediately after restoring the configuration.

Follow this procedure:
1. system database reset-to-default proceed yes
2. system database config-restore name <backup_filename>
3. no fdb ; no vlan-listeners ; commit

Fix:
Deleting the "fdb" and "vlan-listeners" tables is no longer necessary after running system database config-restore" on the chassis partition CLI.

1022741 : Restoring config saved with different portgroup mode than what is running leaves interfaces without a hardware MAC address.

Component: F5OS

Symptoms:
Restoring configuration file saved with different portgroup mode(s) than the running configuration leaves interfaces without a hardware MAC address.

Conditions:
The issue occurs when the user does a restore of a backup file which has portgroup mode configuration different than what is currently running on the box.

Impact:
The lack of hw-mac-address per interface impacts the overall functionality of L2 protocols (e.g. LACP members cannot become operationally UP because LACP BPDUs exchange require port hw-mac-address).

Workaround:
On a chassis after backup/restore with different portgroup modes, follow the steps to address the issue:
- change the portgroup modes of the impacted blades to default MODE_100GB
- commit and allow blade(s) to reboot
- verify the blade ports are now matching the 100GB mode
- restore the portgroup mode of the impacted blades to desired mode (matching what is in the backup file)
- commit and allow blade(s) to reboot
- verify the ports are published to match the desired mode and the state contains hw-mac-address (issue command: show interfaces interface ethernet state hw-mac-address)
- reapply the configuration that references the interfaces (e.g. add back interface members into the LACP LAG)

Fix:
Restoring configuration files now works when the restored configuration has different portgroup modes.

1017001 : Live Up/Downgrade to non-patch version on Controller image with patch version produces instability.★

Component: F5OS

Symptoms:
Under certain Controller installation/upgrade conditions, a live upgrade of Controller Platform Services produces transient instability in running container versions after rebooting the affected Controller. This instability is fixed automatically within several minutes of controller reboot, but can have negative side effects.

Conditions:
1. A Controller was imaged using a 'patch version' ISO (Ex. 1.1.1, 1.1.2).
2. The Controller Platform Services are live up/downgraded to a 'non-patch' version of controller services (Ex. 1.0.0, 1.1.0, 1.2.0).
3. The Controller is subsequently rebooted, either automatically or by the user.

Impact:
The Controller will temporarily run the wrong version of platform containers after rebooting (it will run the container versions defined in the 'patch' version of services plus the container versions defined in the 'non-patch' version). If the patch version of services includes changes to the ConfD container, this can result in ConfD database corruption/instability.

Fix:
Fixed issue with Controller live upgrade instability affecting systems that were initially imaged with particular versions of Controller SW.

1016629 : System allows creation of VLAN names that are too long★

Component: F5OS

Symptoms:
The /vlans/vlan/config/name value is a free format string. Creating long VLAN names can violate common naming rules.

Conditions:
Creating VLANs whose names are longer than 56 characters (encountered at the /vlans/vlan/config/name endpoint).

Impact:
The F5OS software does not prevent you from creating VLAN names that are too long, however, the BIG-IP system cannot use them.

Note: When this issue is fixed, VLAN names in configurations and scripts will no longer behave as expected. Before upgrading, make sure to follow the instructions in Behavior Change to ensure your upgrade succeeds.

Workaround:
Create shorter VLAN names.

Fix:
VLAN names now have the following constraints:

- May start with an alphabetic character (Aa-Zz).
- Cannot exceed 56 characters in length.
- May contain alpha-numeric characters, periods (.), hyphens (-), and underscores (_).
- Must be unique among VLANs.

!Important! Before upgrading:

-- Ensure that all VLAN names meet these constraints.

-- Update any scripts that create VLANs whose names violate these constraints.

Behavior Change:
VLAN names now have the following constraints:

- May start with an alphabetic character (Aa-Zz).
- Cannot exceed 56 characters in length.
- May contain alpha-numeric characters, periods (.), hyphens (-), and underscores (_).
- Must be unique among VLANs.

Important upgrade information:

Before upgrading:

-- Ensure that all VLAN names meet these constraints.

-- Update any scripts that create VLANs whose names violate these constraints.

-- Configurations from previous versions containing /vlans/vlan/config/name strings that do not meet the new validation rules will fail to load after upgrade.

-- Configuration scripts with /vlans/vlan/config/name strings that do not meet the new validation rules will fail after upgrade.

1016621 : VLAN name validation changes★

Component: F5OS

Symptoms:
Previously, the /vlans/vlan/config/name was a free-format string.

Now, the name has the following constraints:
-- May start with just a letter
-- Cannot exceed 56 characters in length
-- May contain alpha characters, numbers from 0 through 9, period (.), hyphen (-), and underscore (_)
-- Must be unique among VLAN names

Conditions:
When you configure /vlans/vlan/config/name leaf, which is an optional leaf.

Impact:
Previous configuration with /vlans/vlan/config/name strings that do not meet the new validation rules will not load.

Previous configuration scripts with /vlans/vlan/config/name strings that do not meet the new validation rules will fail.

Workaround:
Before upgrading (ideally) or after upgrading and before saving the configuration or exercising scripts, adjust all /vlans/vlan/config/names so they meet the validation requirements.

Fix:
Additional validations were added to VLAN names. You must adjust existing configuration's /vlans/vlan/config/name strings and scripts to meet the new validation rules.

Behavior Change:
Previously, the /vlans/vlan/config/name was a free-format string.

Now, the name has the following constraints:
-- May start with just a letter
-- Cannot exceed 56 characters in length
-- May contain alpha characters, numbers from 0 through 9, period (.), hyphen (-), and underscore (_)
-- Must be unique among VLAN names

1016509 : System allows creation of duplicate VLAN names★

Component: F5OS

Symptoms:
The /vlans/vlan/config/name value is a free format string and allows duplicate names to be created.

Conditions:
Creating a VLAN using a name that already exists (encountered at the /vlans/vlan/config/name endpoint).

Impact:
Duplicate VLANs are created without error. Which VLAN the system uses is not predictable.

Workaround:
Ensure VLAN names are unique.

Fix:
VLAN names now have the following constraints:

- May start with an alphabetic character (Aa-Zz).
- Cannot exceed 56 characters in length.
- May contain alpha-numeric characters, periods (.), hyphens (-), and underscores (_).
- Must be unique among VLANs.

!Important! Before upgrading:

-- Ensure that all VLAN names meet these constraints.

-- Update any scripts that create VLANs whose names violate these constraints.

Behavior Change:
VLAN names now have the following constraints:

- May start with an alphabetic character (Aa-Zz).
- Cannot exceed 56 characters in length.
- May contain alpha-numeric characters, periods (.), hyphens (-), and underscores (_).
- Must be unique among VLANs.

Important upgrade information:

Before upgrading:

-- Ensure that all VLAN names meet these constraints.

-- Update any scripts that create VLANs whose names violate these constraints.

-- Configurations from previous versions containing /vlans/vlan/config/name strings that do not meet the new validation rules will fail to load after upgrade.

-- Configuration scripts with /vlans/vlan/config/name strings that do not meet the new validation rules will fail after upgrade.

1015497 : In rare cases, the blade software can disconnect from the system controller and never recover.

Component: F5OS

Symptoms:
In very rare scenarios, blade software components may be unable to communicate with the database on the system controller. LACP and STP daemons hang at startup, and it could cause other issues in a partition.

Conditions:
The issue can occur when both system controllers are rebooted at once.

Impact:
The LACP, LLDP, and STP daemons may be indefinitely unusable. It is suspected there could be other impacts depending on which blade software component is affected, though no other issue has been observed.

Workaround:
1. Reboot the affected blade.
2. Disable then re-enable the affected partition.

Fix:
The affected blade software component can now detect the connection issue and will re-establish the connection to the system controller's database.

1014153 : Timeout/failure on blade/controller pxeboot.

Component: F5OS

Symptoms:
When attempting to pxeboot blades or system controllers, it fails with a timeout.

Conditions:
This happens in the following scenarios:

-- The steps to configure pxeboot have not been performed.
-- The services provided by the image-server container have stopped working properly.

Impact:
Pxeboot fails.

Workaround:
To configure for pxeboot, be sure that a partition iso file has been imported, a partition has been configured and enabled, and the slots to boot are assigned to that partition.

If those steps are performed properly, then it is possible the image-server services are no longer properly functioning. In that case, you can recover using either of the following workarounds:
-- Restart the vcc-image-server container.
-- Reboot the system controller.

1014009-1 : Blade out of memory condition when using a large number of VLANs.

Component: F5OS

Symptoms:
If a tenant or tenants are assigned a large number of vlans, an out of memory condition can be triggered on the blade after several days.

Conditions:
A large number of vlans is assigned to a single tenant.

Impact:
Tenants may die and new tenants may fail to launch on the affected blade.

Workaround:
Reduce the number of vlans assigned to a single tenant.

1013977 : Timestamps not consistent across chassis.

Component: F5OS

Symptoms:
Blade logs use the PDT (Pacific Daylight Time) or PST (Pacific Standard Time) timezone. Other logs use the timezone local to the device (i.e., on a properly configured BIG-IP), and some controller logs record in UTC (Coordinated Universal Time).

Conditions:
You can see the differences when viewing the chassis logs across.

Impact:
Timezones are not consistent across chassis. This can impact troubleshooting.

Workaround:
None

1012437 : Tenant virtual disk is deleted when the tenant running-state is set to "configured."

Component: F5OS

Symptoms:
A VELOS tenant's virtual-disk is deleted when the tenant running-state is set back to "configured" after having been "deployed." This behavior differs from vCMP.

It is recommended that you set the tenant running-state to "provisioned" in order to stop the running tenant.

Conditions:
Tenant running-state changed from "deployed" to "configured."

Impact:
Virtual disk is deleted, resulting in loss of tenant configuration.

Workaround:
Fixed. No workaround needed.

Fix:
The system no longer deletes the virtual disk when the state is changed from deployed to configured.

1010529-1 : Erratic VLAN assignment during tenant start-up.

Component: F5OS

Symptoms:
After rebooting a tenant, the tenant might receive some erratic VLAN assignment requests, but the VLANs are not associated with this tenant. This spurious VLAN assignment may impact traffic.

Conditions:
1. A tenant is created without an assigned VLAN
2. Reboot the blade that tenant is assigned to

The second step is not required every time, but it is triggered due to the first step.

Impact:
Application traffic through the chassis could be impacted.

Workaround:
There are 2 workarounds:
1. Assign VLAN(s) to the tenant which does not have assigned VLANs and reboot all the blades of this tenant
2. Assign VLAN(s) to the tenant which does not have assigned VLAN and run 'docker restart of API_SERVICE_GW container' on all blades

Fix:
When a tenant is not configured with any VLANs, the tenant will no longer receive incorrect VLANs when a blade is rebooted.

1009685-1 : 1.2.1 platform software cannot be imported on Controller OS versions below 1.2.0

Component: F5OS

Symptoms:
It is not possible to import 1.2.1 platform software (Controller or Partition OS, services, or ISOs) on Controller OS versions lower than 1.2.0.

Conditions:
1. Running a version of Controller OS <1.2.0
2. Try to import 1.2.1 platform software.

Impact:
You are unable to import platform software version 1.2.1 if the Controller OS version is lower than version 1.2.0.

Fix:
It is now possible to import platform software version 1.2.1 while running version 1.1.4 of the Controller OS (but still not prior 1.1.X releases).

1009237 : Config backup files cannot be deleted.

Component: F5OS

Symptoms:
The config backup files that reside in /mnt/var/confd/configs on the System Controller and on /var/f5/partition/configs on the Partition, can not be deleted. When you attempt to delete these files using "file delete ..", the CLI throws an error.

Conditions:
Attempting to delete config backup files using "file delete ..." CLI.

Impact:
The backup files will take the disk volume and the system may work in an unexpected way.

Workaround:
None

Fix:
Backend support is added to allow deletion of the config backup files.

1009189 : SwitchD may fail to start due to insufficient memory.★

Component: F5OS

Symptoms:
The system controller is unable to communicate with one or more of the running blades. The 'show cluster' command indicates a blade(s) is NotReady even though the blade is fully booted. The other system controller may show similar mis-reporting of blades, but on different blades.

An assert in the switchd vendor SDK code is triggered if a 16MB allocation cannot be satisfied during switchd startup. SwitchD will not provide required switch configuration when this occurs resulting in a management port and chassis internal network outage.

Conditions:
The problem can manifest any time SwitchD starts/restarts. It is most likely to manifest during installation or live update of the system controller software.

Impact:
The problem persists until the system controller on which the condition occurs is rebooted.

Workaround:
None.

1008701 : Using curl to access 'scp:' URIs on the partition management IP does not work

Component: F5OS

Symptoms:
Attempting to upload a tenant image via

"curl filename scp:IMAGES"

would fail, even though

"scp filename admin@mgmt-ip:IMAGES"

works.

Conditions:
Accessing ssh/scp via curl rather that the scp application.

Impact:
Cannot use curl to copy files.

Workaround:
Use scp directly rather than curl.

Fix:
The ssh/scp server has been fixed to correctly interpret the file/directory names supplied by the 'curl' command.

1008585-2 : L2 Table corruption results in a traffic loss.

Component: F5OS

Symptoms:
The Layer 2 (L2) table on a blade can become corrupted under certain conditions. When this happens, traffic to the affected destination (either a tenant or a external interface) do not flow properly.

Conditions:
-- VELOS system with more than one blade installed.
-- A packet for a tenant associated with one blade arrives on a different blade that is encountering the L2 table corruption.

Impact:
Traffic loss to a tenant or the front-panel interfaces. This may include partial or full packet loss to the tenant.

Workaround:
None

Fix:
FPGA Manager now detects the corruption so incorrect entries are not written to the L2 table. This prevents traffic loss from occurring.

1008433-2 : VQF hot signal asserted warnings

Component: F5OS

Symptoms:
A PEL log entry occurs indicating an FPGA HOT signal asserted:

Warning | AOM | 5 | Na | VQF hot thermal event

Conditions:
This issue happens at system startup.

Impact:
If the issue occurs during system startup, it is an erroneous error message and can be safely ignored.

Workaround:
Fixed an erroneous FPGA HOT signal that occurs during system startup.

1007797 : Partition set-version command should warn user about reboot/traffic disruption when used on CLI

Component: F5OS

Symptoms:
The "set-version" command causes the partition control-plane software to be restarted, and reboots blades and tenant instances. This will disrupt dataplane traffic. The user should be warned, and acknowledge the intent to interrupt network traffic processing.

Conditions:
The chassis administrator changes the partition software version with the partition set-version command.

Impact:
Chassis administrator may inadvertently cause network traffic interruption.

Fix:
The "set-version" command now issues a warning and requires confirmation from the user before beginning the software installation. The prompt may be optionally bypassed by adding the "process yes" argument to the command.

1007089 : System controller 'show partitions' output can be inaccurate.

Component: F5OS

Symptoms:
The output of the 'show partitions' command should display the current operational status (disabled, running, starting, stopping, failed, etc.) for each partition instance.

In some cases, the displayed status is incorrect.

Conditions:
When the chassis has multiple partitions, and there are gaps in the partition IDs, the internal state tracking can fail to record and report the partition state, typically showing that a partition is 'failed', when it is actually operating normally.

Impact:
The user will be misled about the system state.

Fix:
The operational state monitoring has been fixed.

1004605 : Under rare conditions, blades may fail to start partition platform containers

Component: F5OS

Symptoms:
After encountering a rare import bug, subsequent upgrades of partition software may result in a state where blades in a partition fail to start partition platform services after a system controller failover event.

Conditions:
1. A rare issue leads to the conditions of ID 984977 occurring during an import of some version of the partition ISO or services.
2. That version of partition services is configured for use by an active partition.
3. The system controllers fail over and switch active/standby roles.
4. Blades in partition are rebooted.
5. Blades fail to pull and start partition services, or pull and start the wrong version of services.

Impact:
Partition performance is degraded or disrupted.

Workaround:
1. Remove all software configuration for the affected partition and disable the partition.
2. Re-enable the partition and re-configure the software versions you wish to use.

or

1. In bash on the affected blade(s), run 'cp /var/export/chassis/partition/<partition ID>/blade/* /var/docker/config/'
2. Reboot the affected blade(s).

Both workarounds will only persist until the next time the system controllers fail over, at which point they would need to be performed again. A more persistent workaround will require more involved, case-by-case modification of import structure by a support engineer.

Fix:
Blades no longer fail to start partition platform services under certain rare conditions

1004205 : Controller sshd service dies under rare circumstances.

Component: F5OS

Symptoms:
Under rare circumstances, the 'sshd' service on controllers will die and not automatically restart in response to rapid modification of host ssh keys.

Conditions:
Running a pre-1.2.0 version of the system controller OS.

Impact:
Under rare circumstances, 'sshd' dies and does not automatically restart, restricting ssh access to controllers.

Workaround:
1. Connect to the system controller serial console.
2. In bash, execute the command 'systemctl restart sshd'.

OR

1. Reboot the affected system controller.

Fix:
Fix intermittent issue that would cause 'sshd' on system controllers to die and not restart under certain circumstances.

1003965-1 : Crypto/Compression Acceleration not enabled by default when creating a tenant

Component: F5OS

Symptoms:
Prior to version 1.1.2, high-performance crypto processing and compression had to be explicitly enabled using the Crypto/Compression Acceleration setting on the Add Tenant Deployments screen.

Conditions:
When creating a new tenant deployment on a chassis partition.

Impact:
Performance can be impacted when Crypto/Compression Acceleration is disabled.

Workaround:
Manually enable Crypto/Compression Acceleration for the tenant instances that require it.

Fix:
Beginning with F5OS version 1.1.2, Crypto/Compression Acceleration is enabled by default when deploying a new tenant. Note that you may need to explicitly enable the setting on tenants deployed on a previous F5OS version.

1003461 : If the initial partition database startup is interrupted, it can leave an empty database★

Component: F5OS

Symptoms:
After the partition initially starts up, the partition cannot be used because there are no users or roles defined.

Conditions:
A problem occurs when the partition is starting up for the first time. This could happen if the partition were disabled shortly after being enabled, or if a controller failover occurs at the wrong time.

The problem will not occur after the database has initialized correctly.

Impact:
The partition cannot be used, since it is not possible to log in and set a configuration.

Workaround:
After a partition is created, if it is not possible to log in, the chassis administrator can remove all slots from the partition and delete it. After the transaction commits, re-create the partition with the desired slots.

Fix:
The database software correctly detects and recovers from this condition.

1002585 : Partition manager qkview data collection inadequate

Component: F5OS

Symptoms:
The partition manager container does not capture sufficient status information for problem diagnosis.

Conditions:
Capturing a qkview to diagnose system problems.

Impact:
It is difficult to determine the conditions that may have caused failures.

Fix:
Qkview now contains the complete partition running-config, and additional internal and operational state.

1001865 : No platform trunk information passed to tenant

Component: F5OS

Symptoms:
Trunk information is not being published to BIG-IP tenants for use in high availability (HA) group definitions.

Conditions:
When defining HA groups.

Impact:
No trunk or trunk member information is reported. This reduces the usefulness of information used to compare the relative health of HA peers and potentially initiating a tenant failover, depending on that output.

Workaround:
None

Fix:
Trunk information is now synchronized between the VELOS system and tenants, enhancing the tenant HA health check.

Behavior Change:
Trunk information is now synchronized between the VELOS system and tenants, which increases the usefulness of information used to compare the relative health of HA peers and potentially initiating a tenant failover, depending on that output.

1001145-2 : System controller config backup and restore causes the system not to function properly.

Component: F5OS

Symptoms:
After restoring a config backup, the system controller software assigns auto-generated IDs for the partitions. These auto-generated IDs are not mapped to blades and OpenShift cluster namespaces. This behavior causes the system to not work properly.

Conditions:
Reset the system to factory default settings using the system controller CLI, and then restore the backup.

Impact:
System goes into an Inoperative state for tenant deployments.

Workaround:
The steps outlined below walk you through the entire process of taking a backup of the system, and then restoring the system to the factory default settings. The steps also include sample commands.

Note: You must have console access to the system to complete this task. Running the reset-to-default command removes the management network.


Backup
=======
1.) Backup tenant configs by logging into the tenant. Save the config, and then copy the config file backup to a safe, external location:

tmsh save sys ucs /var/tmp/config.ucs


2.) Backup partition configs by logging into each partition. Backup the database, and then copy the database backup to a safe, external location:

part2-1(config)# system database config-backup name <partition-backup-filename>
result = Database backup successful.

part2-1(config)#
scp admin@@partitionIP:configs/<partition-backup-filename> <external-device-ip>


3.) Backup the controller config by logging into the system controller using a floating IP address, and then copy the backup config file to a safe, external location:

syscon-2-active(config)# system database config-backup name <controller-backup-filename>
response Succeeded.

syscon-2-active(config)#
scp root@floatingIP:/var/confd/configs/<controller-backup-filename> <external-machine-ip>


Important: Once you execute the reset-to-default command, the system controller deletes the associated filesystems and backup files. So make sure to backup the system controller, tenant, and partition configs before continuing.


4.) Delete partition configs:

part2-1(config)# system database reset-to-default proceed yes
result Database reset-to-default successful.

part2-1(config)#
System message at 2021-03-11 00:02:21...

Commit performed by admin via tcp using cli.


5.) Put all slots in the none-partition:

syscon-2-active(config)# slots slot 1 partition none
syscon-2-active(config)# slots slot 2 partition none

syscon-2-active(config-slot-2)# commit
Commit complete.


6.) Remove the partitions from the system controller:

syscon-2-active(config)# no partitions partition part2
syscon-2-active(config)# no partitions partition part3

syscon-2-active(config)# commit
Commit complete.


7.) Using a console connection, reset the controller config to factory defaults from the system controller CLI:

syscon-2-active(config)# system database config reset-default-config true

syscon-2-active(config)# commit
Commit complete.


8.) Reboot the controllers. Once controllers are fully rebooted, proceed to the restore task.



Restore
=======

1.) Set up the system controller mgmt network using the wizard/CLI.

2.) Import the backup from the external device to the system controller:

ssh root@floatingip
mkdir -p /var/confd/configs/

scp <external-machine>:<backup> /var/confd/configs/


3.) Restore the controller config:

syscon-1-active(config)# system database config-restore name <controller-backup>
response Succeeded.


4.) Reboot the blades.

5.) To restore the partition config (including VLANs/tenants/interfaces/etc.), import the partition config to the active partition that is running on the controller (running 'show partitions' from the system controller CLI displays which controller is running the active partition):

part2-2(config)# system database config-restore name <partition-backup>
result Database restore successful.

part2-2(config)#
System message at 2021-03-11 01:10:34...
Commit performed by admin via tcp using cli.


6.) Restore the tenant config once the tenant mgmt-ip is reachable. To do so, copy the config after the partition restores (where config.ucs is your .UCS filename):

tmsh load sys ucs /var/tmp/config.ucs

Fix:
Config backup and restore to factory default settings now works properly.

• The F5 Networks Technical Support web site: http://www.f5.com/support/
• The AskF5 web site: https://support.f5.com/csp/#/home
• The F5 DevCentral web site: http://devcentral.f5.com/

Generated: Tue Sep 21 12:09:24 2021 PDT

Copyright F5 Networks (2021) - All Rights Reserved