1933721-2 : Interface remain down in F5OS after removing and reinserting SFP modules

Links to More Info: BT1933721

Component: F5OS-C

Symptoms:
After SFPs are removed and reinserted in a VELOS blade, the interface will remain down in F5OS until the blade is rebooted. The peer switch may report the interfaces as having a link.

Conditions:
- VELOS chassis running F5OS-C 1.8.0
- SFPs in blade are removed and reinserted.

Impact:
F5OS interfaces remain reported as operationally down until the blade is rebooted.

Workaround:
After SFP modules are removed and reinserted on a blade, reboot the blade.

1926829-2 : When attributes are added under exporters for Open Telemetry, the keys are not visible on webUI.

Component: F5OS-C

Symptoms:
When attributes are added under exporters for Open Telemetry, the keys are not visible on webGUI.
If any exporters have existing attributes and we try to edit the exporter from webUI, the attributes get deleted.

Conditions:
Adding or updating attributes to an open telemetry exporter through webUI.

Impact:
New attributes created under exporters don not have their keys visible on the webUI.
Editing the exporter from the webUI will delete existing attributes.

Workaround:
Adding attributes to exporters or updating existing exporters can be done from the CLI.

1921793-1 : Health summary is not reported for some nodes in controller and partition ConfD

Component: F5OS-C

Symptoms:
System health summary is missing for some nodes.

Conditions:
It is observed when iso is upgraded to 1.8.1 branch

Impact:
System health summary is not reported for some nodes. It throws error while fetching summary.

Workaround:
None

Fix:
Updated Node tag in components properly. Since GET:health api is fixed in diag-agent, Show system health summary reports etails properly for all nodes.

1920325-1 : The network-manager container crashes when it fails to create FDB entry in database

Links to More Info: BT1920325

Component: F5OS-C

Symptoms:
Network-manager container crashes.

Conditions:
The issue may occur when there is an upgrade/downgrade, tenant creation/deletion, or reset/restore the database.

Impact:
The network-manager container will restart.

Workaround:
None

Fix:
The network-manager will not crash when it fails to create FDB entry in database.

1891301-2 : CVE 2020-27743: pam_tacplus through 1.5.1 lacks a check for a failure of RAND_bytes()/RAND_pseudo_bytes().

Component: F5OS-C

Symptoms:
libtac in pam_tacplus through 1.5.1 lacks a check for a failure of RAND_bytes()/RAND_pseudo_bytes(). This could lead to use of a non-random/predictable session_id.

Conditions:
The current version pam_tacplus from version 1.6.0 doesn't have the fix as this was added in version 1.6.1 source package.

Impact:
This could lead to use of a non-random/predictable session_id which means an adversary could gain access.

Workaround:
N/A

Fix:
By updating the pam_tacplus source code to 1.7.0 where the vulnerability was fixed in 1.6.1, the new code does not have this issue.

1890297-1 : Memory leak in l2_agent daemon on F5OS

Links to More Info: BT1890297

Component: F5OS-C

Symptoms:
- Large memory consumption by the l2_agent.
- Tenant disruption on rSeries appliance

Conditions:
- An F5OS system with SNMP configured and a LAG (Link Aggregation Group) with more than 1 member.

- SNMP monitoring in use.

We can check the l2_agent memory consumption by using `top` command.

Ex: Top output showed a 15GB l2_agent process:

   PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19454 root 20 0 17.3g 14.9g 1788 S 0.0 5.9 174:04.57 /confd/bin/l2_agent -s appliance-1

Impact:
Eventually the system experience OOM (Out of Memory).

On an rSeries appliance, a tenant might experience disruptions and slowness, up to and including a TMM SIGABRT core.

Workaround:
Restart the L2_agent process on the F5OS host.

On an rSeries appliance, one of the following:

- as root, run the command 'docker restart system_L2'
- as an administrative user from the F5OS CLI, enter config mode and then run 'system diagnostics os-utils docker restart node platform service system_L2'

On a VELOS system, restart the partition<ID>_L2 instance on each controller. Log into the system controller as an administrative user, enter config mode, and then run:

system diagnostics os-utils docker restart node controller-1 service partition1_L2
system diagnostics os-utils docker restart node controller-2 service partition1_L2

(Adjust 'partition1' to the partition ID for the partition in question)

Fix:
The l2_agent process no longer leaks memory.

1889913-2 : VELOS partition Allowed IP rule restrictions

Component: F5OS-C

Symptoms:
Configuring a "system allowed-ips allowed-ip" rule in a VELOS partition without limiting it to a specific port (22, 80, 443, 7001-7032, 8888, or SNMP) allows access to more than expected.

Conditions:
- VELOS partition
- Allowed IP Address rules configured to use port 'All' in the GUI (or that do not specify a specific port when configured via the CLI)

Impact:
Permits more traffic than it should.

Workaround:
Instead of creating an 'All' port allowed IP address rule, create individual rules for each specific port.

Fix:
Expected behavior is performed when using port 'All'.

1871517 : CVE-2017-18342 PyYaml arbitrary code execution from untrusted data

Links to More Info: K000139901

1850481 : Standby tenant is unreachable after F5OS partition upgrade to 1.7.x or higher.

Links to More Info: BT1850481

Component: F5OS-C

Symptoms:
- The `tmsh show net arp` may show arps with an unknown status.
- The confd CLI `show dag-states` command shows dag tables consisting of only zeros.

Conditions:
* Multi-slot tenant in a device group
* Connection mirroring enabled
* Upgrade F5OS partition from 1.6.x to 1.7.x or greater

Impact:
Standby tenant is inaccessible.

Workaround:
None

Fix:
This issue has been fixed in F5OS partition upgrades to 1.7.x or higher.

1850165-1 : Missing internal interface pgindex field causes l2-agent to restart★

Component: F5OS-C

Symptoms:
Upon upgrade from 1.1 -> 1.6 -> 1.8, l2-agent on blade will exit due to interface data mismatch. This mismatch happens because the pgindex hidden leaf is missing from cdb, but the l2-agent on blade expects it.

Conditions:
Chain upgraded from 1.1 -> 1.6 -> 1.8. Version 1.8 is the version where l2-agent added more logic to check interface data inconsistency.

Impact:
Dataplane is not functioning.

Workaround:
Work around is to delete the blades from the partition and re-add them. This will require user to reconfigure interface data (vlans, lag members).

Fix:
With this fix, the upgrade into 1.8 will work as expected, and l2-agent on the blade will find matching interface data.

1826209-1 : Error log does not contain all needed information.

Links to More Info: BT1826209

Component: F5OS-C

Symptoms:
An "Interface data differ" log is logged by l2-agent, but all of the compared fields in the log message are identical.

Conditions:
L2-agent logs an error message that the interface data differs.

Impact:
The lack of some data such as interface type and slot ID in the log entry makes troubleshooting more complex.

Workaround:
Save the backup configuration file, and inspect the file for hidden fields. For example, search for pgindex under the interface entry.

Fix:
With this change, the log ERROR will display all required data.

1819289-2 : Zero is not allowed as Prefix Length for allowed IPs

Links to More Info: BT1819289

Component: F5OS-C

Symptoms:
It is not possible to save a prefix length with a value of ‘0’.

Conditions:
Prefix Length value is configured to '0'.

Impact:
Allowed IPs cannot be created with prefix value '0'.

Workaround:
Works from CLI.

Fix:
Fixed to accept '0' as prefix length value.

1818185 : The meaning of the interface phyport internal field changed from phyport to DID. This will break functionality that is using phyport★

Links to More Info: BT1818185

Component: F5OS-C

Symptoms:
Two different symptoms surfaced after the meaning of phyport changed:
- reset-counters for an specific interface
- lag members association with a lag in fpgamgr

Conditions:
You configure LAGs while running 1.6.2, with LAG members spanning multiple blades (e.g. blade-1 and blade-3), and then perform a live upgrade to 1.8.0 and above.

You attempt to reset counters for a specific interface (e.g. 1/1.0) after a live upgrade from 1.6.2 into 1.8.0 and above.

Impact:
For the lag members mismatch, the data packets will be forwarded to the wrong port.

For the reset counters, the counters are not reset for the specified interface.

Workaround:
For the reset-counters issue, execute reset counters for all interfaces. This will clean up the counters.

For the LAG member mismatch, reboot the blades after live install ends successfully.

Fix:
The fix addresses the mismatch in interface phyport values.
This will allow reset counters per interface to work.
This will allow the lag members to be properly handled after live upgrade.

1817669-1 : Timeout for the Ansible playbook during cluster install cannot be retried.

Component: F5OS-C

Symptoms:
If there are other issues on the chassis that cause the ansible playbooks to run slowly during Kubernetes cluster install, the playbook cannot be retried correctly if it reaches timeout.

Conditions:
This can occur, if there are other issues on the chassis that cause the ansible playbooks to run slowly, such as DNS or remote auth issues when a Kubernetes cluster rebuild is executed.

Impact:
The Kubernetes cluster install may fail repeatedly because it will not correctly recognize the timeout, and raise the amount of time it will wait.

Workaround:
Mitigation is resolve the issue(s) causing the playbooks to run slowly. This may involving removing bad DNS servers or remote auth servers that may be causing the slow down.

Fix:
The orchestration-manager code has been updated to correctly recognize the timeout error, and handle it correctly.

1814073-1 : F5OS chassis switchd core dump

Component: F5OS-C

Symptoms:
The switchd process experiences crashes that generate core dumps.

Conditions:
These crashes are typically observed during certain interface queries or other operations involving statistics updates.

Impact:
The switchd process crashes and generates core files. Temporary service disruptions may occur for functionalities reliant on the switchd process.

Workaround:
None

Fix:
This issue has been fixed, ensuring switchd includes proper handling for TMSTAT query.

1814053-2 : Orchestration Agent process may core

Component: F5OS-C

Symptoms:
User may see the Orchestration Agent process on the active controller core.

Users may find logs with message ID 0x401000000000027. They shall look like the following.

network-manager[1]: priority="Err" version=1.0 msgid=0x401000000000027 msg="Failed to parse ZMQ message header" session_id="<sessionid>".

Conditions:
N/A

Impact:
Tenants that are currently running are not affected. New tenants that are deployed when this condition is occurring may be delayed in coming to a running state.

Workaround:
None

Fix:
The Orchestration Agent behaves correctly.

1814045-2 : Daemons that handle ZMQ messages may crash under certain conditions.

Component: F5OS-C

Symptoms:
Certain allowed IP configurations on partitions may cause crash scenarios when handling zmq messages

Conditions:
Configurations that allow all IPs as part of Allowed IP settings.

Impact:
With the said condition, handling certain ZMQ messages may cause daemon crash scenarios.

Workaround:
Avoid using allow all IPs as part of Allowed IP configuration.

Fix:
Daemons no longer crash in partitions while handling zmq messages

1789417-1 : Component fpgamgr in restart loop with segmentation fault after failed FPGA firmware update

Links to More Info: BT1789417

Component: F5OS-C

Symptoms:
Component fpgamgr experiences segmentation fault after failed FPGA firmware update and persists in a reboot loop. The CLI command "show cluster nodes node state platform fpga-state" indicates that FPGA_STATE persists in FPGA_INIT and never reaches the state FPGA_RDY.

Conditions:
FPGA firmware update fails and one or more of the FPGA devices does not show up on the PCI bus. This causes a FPGA SDK segmentation fault upon the fpgamgr component startup, and perpetual reboot loop so long as the FPGA issue persists.

Impact:
A failure in the FPGA firmware update process results in one or more FPGA devices not being detected on the PCI bus. This, in turn, causes a segmentation fault in the FPGA SDK upon the startup of the fpgamgr component, leading to a continuous reboot cycle until the FPGA issue is resolved.

Workaround:
None. Perpetual reboot loops after trying to load FPGA firmware that do not recover typically indicates a hardware error and requires an RMA.

Fix:
One of the BARs fails to initialize when the PCIE speed does not load at the intended Generation. This issue was causing a segmentation fault in the SDK, but it has now been resolved by having the SDK notify the fpgamgr of the missing BAR instead. While the device may still fail to load, the fpgamgr will no longer experience a crashing loop as a result.

1789141-2 : If 'ldap-group is configured for a role but LDAP search fails, users with the default GID for the role can still get those privileges

Component: F5OS-C

Symptoms:
When an 'ldap-group' mapping is configured for a F5OS role, and the mapping fails (because the filter is invalid or the LDAP query of remote groups fails for some other reason), the default mapping for the role (or, what is configured in 'remote-gid' for the role) is still used.

For example, if you were attempting to map the F5OS role 'admin' (default GID 9000) to an LDAP group 'CN=my-ldapgroup', and the LDAP search for that group failed (because the provided filter was invalid, the group does not exist, etc.), users with GID 9000 would still be able to authenticate and login with 'admin' privileges.

Conditions:
1. LDAP authentication is enabled.
2. A role mapping is applied via the 'ldap-group' configuration for a F5OS role.
3. The provided 'ldap-group' filter is invalid or another unexpected issue is encountered when querying the LDAP server.

Impact:
Users can login with privileges in excess of what one might expect given the system configuration.

Workaround:
If the LDAP group/users have Posix attributes ('gidNumber'), it is possible to map the F5OS role using this GID number by specifying it in the 'remote-gid' configuration under the role.

If this is not feasible, it is possible to directly validate the 'ldap-group' mapping was successful by inspecting this file from a bash shell:

[root@appliance-1(test):Active] ~ # cat /etc/ldap-gid-map.txt
1108:=9000

If there is an entry that has the default GID for the role on the right-hand side of ':=' in this file, it means the mapping was applied successfully and users with the default or 'remote-gid' GID will not be able to obtain the role permissions. If such an entry is missing, you will need to fix the 'ldap-group' filter so an LDAP query of the group can be successful.

Fix:
If a configured 'ldap-group' mapping fails, deny all role-based access for the mapped role until it is fixed or de-configured.

1789125-1 : VQF VOQ entries missing for the functional blades in the show fpga-tables output

Component: F5OS-C

Symptoms:
Blade 13 is in faulty state due to a different issue related to memory DIMMs.

For the show FPGA tables command there is output for VOQs corresponding to blades 1 and 11.

And in the vqf_voq_stat table output, the remaining VOQ stat requests starting from 13 do not return data although the tmstat table for some other blades are intact.

Conditions:
One of the intermediate blades from the list of show components is faulty and leads to skipping of processing the vqf voq stat requests for rest of the blades that are properly functional.

Impact:
Improper output for the 'show fpga-tables vqf_voq_stat' command.

Workaround:
None

Fix:
Added a code change to get the stats completion for rest of the functional blades when one of the blades is faulty.

1785621-1 : Tenant deployed with Max Memory available on system results in Resource allocation failed - Node is up but Platform services not responding

Component: F5OS-C

Symptoms:
Tenant fails to come to running state when deployed with max memory on system.

Conditions:
Tenant should be deployed with max-available memory on the blade in prior releases of F5OS-C 1.8.1 version.

Impact:
Tenant fails to come to running state.

Workaround:
Since the max memory available for tenants on blade is corrected in F5OS-C-1.8.1, the tenant memory should be configured accordingly.

Step 1. Move failed tenant to configured state and adjust the memory to the new max-available memory of the tenant.

Step 2. Move the tenant back to the deployed state.

Fix:
Max memory available on system for tenant deployment has been corrected with right value.

1783781 : Bash history file containing "PRIVATE KEY" may block qkview

Component: F5OS-C

Symptoms:
Qkview file generation gets stuck at zero percent complete:

# system diagnostics qkview status
result {"Busy":true,"Percent":0,"Status":"collecting","Message":"Collecting Data","Filename":"controller1.qkview.tar.gz"}

Subsequent attempts to generate a qkview fail with the result "Qkview capture can not be initiated. Another Qkview capture is already in progress"

Conditions:
-- Generating qkview
-- The bash history file is large and contains the text "PRIVATE KEY"

Impact:
Qkview files are not able to be collected

Workaround:
1. Run system diagnostics qkview cancel
2. mv ~/.bash_history ~/.bash_history.bak
3. Re-run qkview

Fix:
TBD

1782925-3 : Active Directory LDAP integration without uidNumber/gidNumber does not work after system reboot

Links to More Info: BT1782925

Component: F5OS-C

Symptoms:
After an rSeries appliance reboot, Active Directory LDAP authentication configured with "Unix Attributes" set to false does not work and users from Active Directory are unable to authenticate with the F5OS system.

There will be messages similar to the following logged in platform.log shortly after the reboot:

authd[8]: priority="Err" version=1.0 msgid=0x3901000000000101 msg="LDAP API error during : -" oper="SASL bind" code=-1 msg="Can't contact LDAP server".
authd[8]: priority="Warn" version=1.0 msgid=0x3901000000000098 msg="Unable to retrieve domain Sid for supplied servers and domains; server will be treated as if it has unix attributes present.".

Conditions:
- F5OS device configured with Active Directory LDAP authentication, and the "Unix Attributes" setting configured as false.
- System reboots

Impact:
LDAP remote authentication does not work.

Workaround:
To workaround this issue on an rSeries appliance, create a cron task to restart the system_user_manager and authentication-mgr docker containers after a system reboot:

1. Log into the system as root and create /etc/cron.d/ldap-post-reboot with these contents (not including the '==='):
===
# Workaround for post-reboot issue with LDAP auth (ID1782925)
#
# In the the first five minutes after the system reboots, assume the first
# instance of the following log message that we see is a result of the management
# port lack of connectivity when the docker containers start up, and restart both
# system_user_manager and authentication-mgr once.
#
# authd[8]: priority="Err" version=1.0 msgid=0x3901000000000101 msg="LDAP API error during : -" oper="SASL bind" code=-1 msg="Can't contact LDAP server".

@reboot root timeout 5m sh -c 'tail -n0 -F /var/F5/system/log/platform.log | grep -a -m1 authd.*0x3901000000000101 && sleep 20s && echo Restarting authd and user-manager && docker restart system_user_manager authentication-mgr' || echo "Timed out"
===

This mitigation may fail under some corner cases, e.g. potentially after an upgrade or if something goes wrong with the platform services such that they don't start up within the first five minutes after system boot. In those circumstances, log into the system as root and restart the system_user_manager and authentication-mgr containers:

    docker restart system_user_manager authentication-mgr

1779677-1 : Multiple docker containers can get assigned the same bridge IP during rolling upgrade★

Component: F5OS-C

Symptoms:
Multiple containers can get the same bridge IP during a rolling upgrade or docker restart

[root@controller-2 ~]# docker inspect controller-services-registry-2502 | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "100.64.0.2",
                    "IPAddress": "100.64.0.2",
[root@controller-2 ~]# docker inspect partition-services-registry-2202 | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "100.64.0.2",
                    "IPAddress": "100.64.0.2",

Conditions:
When multiple containers start at the same time.

Impact:
This causes one of the two containers to answer requests depending on which container last refreshed the arp cache.
The other container does not work properly.

Workaround:
Reboot the system.

Fix:
Docker address allocator uses bit map to manage IP address pool but it's not thread safe.

Now, set/unset bitmap operations are protected by a lock.

1779465-1 : SwitchD core file observed after live upgrade

Component: F5OS-C

Symptoms:
Users may observe core files being generated on both controllers after a system live upgrade.

Conditions:
The occurrence of the core is non-deterministic, but it can happen after the live upgrade.

Impact:
When this issue occurs, the SwitchD process generates a core file on the controller.

Workaround:
Reboot the controllers after observing SwitchD core file on the controller.

Fix:
This issue has been resolved to ensure proper process initialization during SwitchD initialization.

1778689-1 : Duplicate OMD alerts during Inaccessible Memory incident

Component: F5OS-C

Symptoms:
During certain conditions where an “Inaccessible Memory” issue occurs, duplicate OMD alerts may accidentally be triggered at the same time due to overlapping OID/alert IDs associated with the same root cause.

Conditions:
This issue arises when an “Inaccessible Memory” incident occurs, resulting in OMD generating redundant alerts “openshiftCertsExpWithinNinetyDays” for the same event, which is causing confusion and unnecessary noise in alert tracking systems.

Impact:
False-positive or duplicate alerts for OMD.

Workaround:
To verify and troubleshoot the issue, you can:

1. Use the confD command 'show cluster cluster-status' to check the cluster's current status.
2. Analyze the openshift.log/velos.log file for any errors or abnormalities related to the incident or cluster health.

Fix:
The issue has been addressed by implementing enhanced logic in OMD alert generation to eliminate duplicate alerts resulting from overlapping OID/alert IDs. The system now ensures each alert is uniquely identified and mapped to its respective event, preventing redundancy during “Inaccessible Memory” incidents. All configurations have been updated to maintain integrity and consistency.

1772433-2 : Config restore fails after upgrade★

Links to More Info: BT1772433

Component: F5OS-C

Symptoms:
1. Bare metal to: 1.6.1-19136
2. Upgrade to: 1.8.0-19115
3. Take controller backup
4. Reset dabase: system database config reset-default-config
5. Attempt to apply backup from step 3, this fail.

Conditions:
-- Upgrade from 1.6.1 to 1.8.0+
-- Perform config-restore

Impact:
Unable to perform config-restore after upgrade.

Workaround:
None

Fix:
With the fix for ID1917841, you can now perform the config-restore.

1772305-1 : Unable to deploy a tenant to both BX110 and BX520 blade in same partition

Component: F5OS-C

Symptoms:
A tenant can only be deployed to a partition if it is deployed to a node that is the same type as the other nodes that are running tenants. Deploying a multi-bladed tenant that includes both BX110 and BX520 blades is not supported.

Conditions:
Deploying a tenant to a partition that contains a mix of BX110 and BX520 blades.

Impact:
If a partition contains both BX110 and BX520 blades, you must choose to deploy tenants to one blade type or the other but not both.

Workaround:
Deploy tenants to nodes that are of the same blade type.

Fix:
None

1772053-1 : High memory usage due to log flood when one controller is in FIPS error state

Links to More Info: BT1772053

Component: F5OS-C

Symptoms:
In FIPS error state, the active controller triggers a sync to the errored controller which results into an infinite loop of waiting as the peer is unreachable. This dumps an enormous amount of logs in ccsync.log and consumes excessive memory.

Conditions:
One active controller and one FIPS errored out controller.

Impact:
Consumes high system memory and log files are rotated in no time leaving a huge dump of logs in ccsync.log

Workaround:
- stop ccswatch.service
- Recover FIPS errored controller
- restart ccswatch.service

Fix:
Added retries to wait for a finite time period before exiting to reduce log flood and memory usage.

1759733-1 : Controller reboot during a controller RMA can cause openshift cluster to fail.

Links to More Info: BT1759733

Component: F5OS-C

Symptoms:
If a system controller is rebooted after it's ETCD instance has been started, but before the controller has been fully added to cluster, it can cause a failure that will not automatically recover. The controller will not be able to join the cluster after this failure.

Conditions:
A system controller is rebooted after it's local ETCD instance has been started, but before the controller is fully added into the openshift cluster.

Impact:
The rebooted controller will persistently fail to join the cluster after this failure. As such the cluster will not be redundant between the 2 system controllers.

Workaround:
Rebuild the openshift cluster to recover the affected system controller.

Fix:
The fix cleans any stale ETCD state when the process of adding the controller to the cluster after the reboot. This allows the controller to be re-added to the cluster correctly.

1757729-2 : Default port for LDAP server does not match default server type

Links to More Info: BT1757729

Component: F5OS-C

Symptoms:
On Server Groups screen, when adding an LDAP server, the default value for LDAP Over TCP type is set to 636 port by default, which is used for LDAP over SSL. This behavior is causing confusion.

Conditions:
When configuring an LDAP server.

Impact:
This issue can be confusing because the default setting for LDAP over TCP type is set to 636 port (instead of 389, which is the port used for LDAP over TCP).

Workaround:
None

Fix:
The default value for the ‘Port’ field has been changed to 389 to align with the default value for LDAP over TCP type.

1753469 : Add notification to set-version when downgrading the system from F5OS-A/C-1.8.0

Links to More Info: BT1753469

Component: F5OS-C

Symptoms:
A downgrade to an earlier version of F5OS from F5OS-A/C 1.8.0 can leave the system inoperable. Refer to ID1712009 for more information.

Conditions:
Perform a config-restore or config reset-to-default operation to an earlier version of F5OS.

Impact:
A downgraded system may be inoperable.

Workaround:
Refer to ID1712009 for workaround.

Fix:
There is an issue with performing a config-restore after downgrading from F5OS-A/C 1.8.0 (ID1712009). If you intend to perform a config-restore or config reset-to-default operation, please refer to the F5OS-A/C 1.8.0 release notes for information on avoiding this issue.

1752821-1 : Cluster re-install with missing system controller does not complete★

Links to More Info: BT1752821

Component: F5OS-C

Symptoms:
If a cluster re-install is issued when only one system controller is installed in the chassis, the cluster re-install will not complete and the system will not be functional.

Conditions:
-- Only one system controller is in a chassis, or one of the system controllers is broken.
-- Re-installing the cluster via 'touch /var/omd/CLUSTER_REINSTALL'

Impact:
System will not be able to launch tenants or pass traffic.

Workaround:
None

Fix:
The cluster orchestration layer has been update to allow K8S cluster install when one system controller is missing from the system. If the system controller is broken, but still inserted into the system the "/var/omd/FORCE_PEER_CC_MISSING" can be created on the remaining controller, and it will behave as if the broken CC has been removed from the chassis. Once the broken controller is replaced, the /var/omd/FORCE_PEER_CC_MISSING file should be removed.

1750613-1 : If a system controller PXE boots and reimages, partitions may not start correctly, and cause data loss★

Links to More Info: BT1750613

Component: F5OS-C

Symptoms:
If a system controller PXE boots, the partition instance restart on that controller may not work and the partition instance will be left in the "failed"/not running state with no configuration database. If that instance later becomes "active" it will overwrite the correct partition configuration database with the empty database.

Example failed partition instance state:

syscon-1-active# show partitions
                                                                   RUNNING
             BLADE OS SERVICE PARTITION SERVICE STATUS
NAME ID VERSION VERSION CONTROLLER STATUS VERSION AGE
----------------------------------------------------------------------------------------
none - - -
default 1 1.6.2-22734 1.6.2-22734 1 running-active 1.6.2-22734 40m
                                       2 failed - 11m


Normally following a controller reimage, the partitions will complete restart after all the ISOs are replicated to the controller and reimported. This may take 15 to 30 minutes depending on how many images are present. The partitions will show as "failed" while this resync occurs, and then they will start up normally. In the failure case, the instance stays "failed" indefinitely.

Do NOT attempt to enable/disable the partition while it is in this "failed" state, or perform a software upgrade (set-version). If that happens, the "wiped" partition instance may start up and become Active, and all partition configuration will be lost.

Conditions:
This problem occurs when the partition is running a "patch" version of partition-services rather than a "base" version. Patch versions have a version number (major.minor.patch) that ends in a number other than “0” (zero).
A race condition may occur between the completion of the partition ISO import and the initiation of the partition, resulting in a potential declaration of success despite failure. In such cases, the operation will not be retried.

In this scenario, the partition might never get started, so it has no opportunity to form an HA pair with the other partition instance and synchronize the configuration database and tenant images. If it does eventually become Active it will erase all partition configurations.

Impact:
All partition and tenant configuration in that partition is lost.

Workaround:
Following a PXE boot or reimage of the controller, check the status of all partition ISOs using the "show image partition" command. For patch versions, the partitions may stay in the "failed" state. However, for base versions, the partition should automatically restart and become running-standby within approximately 5 minutes after the ISOs have been imported. No further corrective action is necessary in this scenario.

To recover force the partition instance startup code to retry by changing the partition configuration in a minimally disruptive way. Recommend toggling the partition mgmt-ip to 'none' and then back, as this will force the retry but not permanently change any configuration.

Example:
syscon-1-active(config)# partitions partition default config mgmt-ip ipv4 address 0.0.0.0 ; exit
syscon-1-active(config)# commit
Commit complete.
syscon-1-active(config)# partitions partition default config mgmt-ip ipv4 address <ip address>; exit
syscon-1-active(config)# commit
Commit complete.
syscon-1-active(config)#

Do NOT attempt to enable/disable the partition while an instance is in this "failed" state following a reimage or perform a software upgrade (set-version). If that happens, the "wiped" partition instance may become Active, and all partition configuration will be lost.

Fix:
Partitions restart and form an HA pair correctly following system controller reimage/replacement, regardless of partition services version.

1737677-1 : Reboot of both system controllers results in dataplane issues

Component: F5OS-C

Symptoms:
Traffic outage after simultaneously rebooting both system controllers.

Conditions:
With a multi-blade partition configured, reboot both system controllers simultaneously.

Impact:
Traffic outage

Workaround:
Reboot blades in affected partition.

1737517-1 : Rare partition startup conditions can cause persistent application-communication error on that partition

Links to More Info: BT1737517

Component: F5OS-C

Symptoms:
While executing partition commands related to tenants. Commands include but not limited to commits related to VLANs, tenants, and interfaces, or, showing data related to VLANs, tenants, and interfaces. Persistent error logging in the partition's confd.log and devel.log about an unregistered lac_mac_hook/write_all callpoint.

Conditions:
Secific cases, where a partition failover occurs, when the partition starts up, or reset to its default settings

Impact:
The partition is effectively inoperable, as very few commands are related to VLANs, and tenants. Additionally, VLANs are functional.

Workaround:
Reboot active partition's system controller or toggle the partition's enabled state.

1730833-4 : Tmm may egress broadcast traffic even when VLANs are disabled in F5OS

Links to More Info: BT1730833

Component: F5OS-C

Symptoms:
In certain scenarios such as restoring a UCS on an F5OS tenant, if the VLANs in F5OS are disabled, tmm may egress broadcast traffic such as gratuitous ARPs onto the disabled VLANs.

Conditions:
-- An F5OS tenant where VLANs were assigned and then removed.
-- An F5OS tenant where tmm is not in forced-offline mode.
-- An action occurs on the tenant (such as restoring a UCS or restarting tmm, or loading the config) that results in gratuitous ARPs.

Impact:
This could cause IP address conflicts on the network or other issues related to unexpected broadcast traffic such as gratuitous ARPs on the network.

Workaround:
- In F5OS, remove the affected VLANs from the LAG or interface.

- On the tenant use forced offline to prevent traffic egress.

- If you are restoring a UCS from another BIG-IP such as for a platform migration, put the source BIG-IP into forcedoffline state before taking the UCS.

- delete the tenant, and recreate without any VLANs assigned.

Fix:
A single tenant with a vlan that was configured and then removed via F5OS will no longer leak broadcast traffic onto the network on the removed vlan.

This fix does not address the issue when multiple tenants are attached to the same vlan. F5 has created ID1758957 for that issue.

1710765-2 : The node number fetched by the SNMP disk stats handler from the disk operational handler has the wrong blade value.★

Links to More Info: BT1710765

Component: F5OS-C

Symptoms:
Rarely, SNMP command output may not show up the disk stats for a particular blade. This could happen because of incorrect blade value of the blade fetched from the backend.

The partition "velos.log" file may show below logs:

1. <Timestamp> default platform-stats-bridge[8]: nodename=controller-2(p1) priority="Err" version=1.0 msgid=0x4305000000000007 msg="" msg="Invalid slot value." value=761491247.

2. <Timestamp> default platform-stats-bridge[8]: nodename=controller-2(p1) priority="Err" version=1.0 msgid=0x4305000000000007 msg="" msg="Failed to assign blade instance" value=761491247.

Conditions:
1. Upgrade the partition
2. Configure SNMP community of any version
3. Execute SNMPWalk command on the disk stats table MIB.

Impact:
SNMPWalk will miss the disk utilisation stats of problematic blade.

Workaround:
As a workaround, either restart the platform-stats-bridge container of the partition or disable/enable the partition from Confd.

Fix:
As a workaround, either restart the platform-stats-bridge container of the partition or disable/enable the partition from Confd.

1710453-1 : Partition configuration wiped out during Controller reboot

Links to More Info: BT1710453

Component: F5OS-C

Symptoms:
In rare cases the partition configuration volume can be wiped during a system controller reboot when partitions are disabled, resulting in partition configuration loss.

Conditions:
When partitions are disabled and a system controller is rebooted there can be a shutdown race between a (spurious) resize request and LVM shutdown that can cause one of the partition volumes to get removed.

When the partition is subsequently enabled, whichever controller instance starts first will establish the current configuration. If the instance that was removed starts first, the partition is reinitialized to a clean configuration.

If the partition is running when the system controller reboots it will automatically resync itself from the other system controller as soon as it restarts. Configuration loss is not observed, though there may be missing logfiles on one of the system controller partition instances.

Impact:
Partition and tenant configuration is lost, and must be restored from backup before continuing.

Workaround:
Partitions should be left enabled. As long as at least one partition instance is running, the high availability subsystem will ensure that no configuration is lost.

Chassis power loss won't trigger this problem since there won't be a "race" between the stopping components.

Fix:
The spurious resizes no longer occur, and the error paths in partition volume resize and partition enablement can no longer result in removing the volumes.

1710405-1 : MAC exhausted error can occur even though there are available MACs

Links to More Info: BT1710405

Component: F5OS-C

Symptoms:
MAC address processing during tenant configuration can result in a "MAC exhausted" error even though there are available MAC addresses.

Conditions:
If the processing of a tenant's configuration releases MAC addresses to the partition's free list then this can erroneously cause a MAC exhaustion error. In this case there may be error logs in velos.log as well indicating failure to update or modify the MAC address pool.

Impact:
This can disrupt tenant configuration.

Workaround:
Modifying the tenant in the CLI when adding VLANs to a tenant is less likely to run into this issue.

Fix:
The code has been modified to log the error but not cause the misleading MAC exhaustion error and not block tenant configuration.

1709665-2 : Blade NotReady after liveupgrade★

Links to More Info: BT1709665

Component: F5OS-C

Symptoms:
A blade is stuck in the NotReady state after an upgrade.

Conditions:
-- The VELOS system is being upgraded.
-- A reboot is triggered before the grub config update is complete.

Impact:
Blade stuck in NotReady state.

Workaround:
Perform a clean install of the blade by PXE installing it. Connect to the serial console of the blade and interrupt the boot process by selecting 'b' when the boot process displays "Press <c> to enter setup".

1709121-5 : Unable to create a tenant as the Network Manager start-up or failover may result in a looping process

Links to More Info: BT1709121

Component: F5OS-C

Symptoms:
While creating a new tenant, an error occurs:

"Failure for data/f5-tenants:tenants API. The server or an underlying service is unreachable."

The network-manager service seems to hang, or it might be in a restart loop.

In confd, the 'show system mac-allocation state' command indicates that no MAC addresses have been allocated.

$ show system mac-allocation state
system mac-allocation state free-single-macs 16
system mac-allocation state allocated-single-macs 0
system mac-allocation state free-large-blocks 2
system mac-allocation state allocated-large-blocks 0
system mac-allocation state free-medium-blocks 0
system mac-allocation state allocated-medium-blocks 0
system mac-allocation state free-small-blocks 0
system mac-allocation state allocated-small-blocks 0
system mac-allocation state total-free-mac-count 80
system mac-allocation state total-allocated-mac-count 0 <---
system mac-allocation state total-mac-count 80

Conditions:
This can occur with combinations of tenants using MAC blocks greater the size 1. The specific combinations are somewhat unpredictable.

Impact:
Tenants cannot be created.

Workaround:
None

Fix:
The code will be updated to prevent the hang condition.

1699821-2 : Partition data missing

Links to More Info: BT1699821

Component: F5OS-C

Symptoms:
The system controller can be rebooted while a partition is being created. This can cause the partition to not function correctly.

Conditions:
A system controller is rebooted while the partition is being created.

Impact:
Partition doesnt work as expected. /config, /shared, /images paths (one or more) will be missing.

Workaround:
Disable and delete the defective partition, then re-create the partition.

Fix:
Controller reboot during partition creation completes correctly after the controller returns to service.

1696325-1 : Unresolved VQF IMM watchdogs after system controller failover, VoQ Window Errors, and extensive disconnect to confd

Component: F5OS-C

Symptoms:
The VoQ IMM Enabled status in the fpga-tables vqf-voq-stats output from the CLI remains 0 indefinitely resulting in traffic loss between blades.

Example:
show fpga-tables vqf-voq-stats
                                                                    COS MEM COS WIN
             EMM IMM SMS FILL FULL HI COS LO SMS EMM IMM ERR
SLOT NAME ENABLED ENABLED DRPLVL PKT CNT BYTE CNT DROP DROP DROP DROP DROP DROP DROP CNT
--------------------------------------------------------------------------------------------------------------------------

3 13.12 1 0 32767 1819895878 2330473381038 200121 0 0 86532 0 14 9 0
3 13.13 1 0 32767 1815815755 2322725261469 251277 0 0 58031 0 14 9 0
3 13.14 1 0 32767 1824204787 2337092078111 211707 0 0 1528 0 14 9 0
3 13.15 1 0 32767 1839939128 2357633747305 208636 0 0 0 0 14 9 0
3 13.4 1 0 32767 0 0 0 0 0 0 0 14 9 0
3 13.9 1 0 5427 0 0 0 0 0 0 0 14 9 0

Conditions:
A temporary loss of the dataplane links between the system controller and a blade on a system, followed by an extensive outage for that blade to the confD database.

Impact:
Traffic loss from the blade reporting the zero values for IMM Enabled towards the destination blade. The destination blade is indicated by the first number in the decimal of the "NAME" column.

For instance, if the IMM ENABLED values are 0 for "Slot 3 and NAME "13.12", this indicates that traffic from slot 3 towards slot 13 will be lost.

Workaround:
Reboot the blades reporting the IMM Enabled values of 0.

1696269-1 : If partition confd initiates a failover due to a health fault, it may incorrectly attempt to fail over repeatedly

Links to More Info: BT1696269

Component: F5OS-C

Symptoms:
In some conditions, when the partition confd initiates a failover to the other controller, it fails to complete the failover in a timely fashion and the original instance reclaims the active role. If the failover was due to a controller fault and is still present, it will immediately fail over again.

Conditions:
If a controller health fault is present on system controller-1, and the partition redundancy mode is set to either "auto" or "prefer-1".

Impact:
While the partition instance is failing back and forth, the control-plane functions are unavailable or degraded, and this can impact dataplane operations.

Workaround:
Set the partition "system redundancy config mode" to "active-controller". When a controller fault exists, and the controller fails over, the partition will automatically prefer to follow the active controller location.

1696157-4 : Container api-svc-gateway crashes after enabling a tenant

Links to More Info: BT1696157

Component: F5OS-C

Symptoms:
The api-svc-gateway container crashes intermittently.

The logs contain the following entries

appliance-1.chassis.local tcpdumpd-manager[8]: priority="Info" version=1.0 msgid=0x5401000000000095 msg="Interfaces/VLANs were removed. No change to hardware programming needed.".
appliance-1.chassis.local Core-helper.Appliance: priority="Err" msgid="0x6501000000000001" msg="Core dumped on Appliance" process="api_svc_gateway" location="/var/shared/core/container/core.system_api_svc.api_svc_gateway.25499.1728690599.core.gz"
appliance-1.chassis.local alert-service[9]: priority="Notice" version=1.0 msgid=0x2201000000000029 msg="Received event." event="327680 appliance core-dump EVENT NA 'Core dumped on appliance. process=api_svc_gateway, location=/var/shared/core/container/core.system_api_svc.api_svc_gateway.25499.1728690599.core.gz'

Conditions:
1. Enabling a tenant by changing it's running-state to deployed.
2. Enabling a tenant followed by deleting the tenant from the CLI promptly.

Impact:
The api-svc-gateway container crashes.

Workaround:
None. The api-svc-gateway will restart immediately and tenants will be recovered automatically.

Fix:
The api-svc-gateway will not crash and tenant will be in the expected state after performing the operations.

1695589-1 : Data-plane links are bounced on HA failover

Links to More Info: BT1695589

Component: F5OS-C

Symptoms:
If the active management port link is cycled down and up, a system controller and partition HA failover will occur. When the system controller failover occurs, a slot state change event is generated causing switchd to "link bounce" all data plane ports even though the slot state on those ports has not changed.

Any act performed on the chassis that would cause a slot state change event will trigger this behavior. That includes inserting or removing a blade.


The impact of the link bounce can be observed by 'IMM watchdog events' reported in the partitions velos.log (/var/F5/partition<id>/velos.log:

fpgamgr[14]: nodename=controller-1(p4) nodename=blade-3(p4) priority="Warn" version=1.0 msgid=0x305000000000008 msg="VQF IMM Watchdog." slot=5 port=9.

Conditions:
This occurs when the active system controller management link is marked down, resulting in an HA switchover or any other act performed on the chassis that can lead to a slot state change event (ie removing/inserting a blade).

Impact:
The data plane links are bounced (brought down and immediately back up), this will trigger the VQF IMM watchdogs.

Workaround:
None.

1691557-1 : CVE-2020-8037: tcpdump memory leak.

Links to More Info: K000149929

1682425-1 : Rate limiting does not work on BX520 front panel interfaces

Component: F5OS-C

Symptoms:
Broadcast and DLF traffic on BX520 front-panel interfaces is not rate-limited.

Conditions:
Excessive broadcast or DLF traffic is present at the front panel interfaces.

Impact:
Excessive broadcast or DLF traffic can cause traffic loss.

Workaround:
None

Fix:
This issue has been fixed by configuring the BX520 rate-limiter hardware correctly.

1681533 : F5 VELOS ATSE firmware v7.10.7.12

Component: F5OS-C

Symptoms:
F5 VELOS ATSE firmware v7.10.7.12

Conditions:
F5 VELOS system

Impact:
Not applicable.

Workaround:
None

Fix:
Fixes intermittent register access. See ID1624057 for more information.

1681529 : F5 VELOS ATSE firmware v7.10.7.02

Component: F5OS-C

Symptoms:
F5 VELOS ATSE firmware v7.10.7.02

Conditions:
F5 VELOS system

Impact:
Not applicable.

Workaround:
None

Fix:
Fixes intermittent register access. See ID1624057 for more information.

1681525 : F5 VELOS ATSE firmware v7.10.7.22

Component: F5OS-C

Symptoms:
F5 VELOS ATSE firmware v7.10.7.22

Conditions:
F5 VELOS system

Impact:
Not applicable.

Workaround:
None

Fix:
Fixes intermittent register access. See ID1624057 for more information.

1681521 : F5 VELOS ATSE firmware v7.10.7.11

Component: F5OS-C

Symptoms:
F5 VELOS ATSE firmware v7.10.7.11

Conditions:
F5 VELOS system

Impact:
Not applicable.

Workaround:
None

Fix:
Fixes intermittent register access. See ID1624057 for more information.

1681501 : F5 VELOS ATSE firmware v7.10.7.00

Component: F5OS-C

Symptoms:
F5 VELOS ATSE firmware v7.10.7.00

Conditions:
F5 VELOS system

Impact:
Not applicable.

Workaround:
None

Fix:
Fixes intermittent register access. See ID1624057 for more information.

1680105-2 : Using 'iburst' option is preferred when adding NTP servers.

Links to More Info: BT1680105

Component: F5OS-C

Symptoms:
It's reported that sometimes system time drifts even with NTP server configured.

Conditions:
This is a common occurrence among specific NTP servers.

Impact:
System time drift.

Workaround:
Use 'iburst' option.
It helps making more reliable synchronization and initial accuracy with the server.

Fix:
From 1.8.1 and later, If the default settings are not specified, the settings will automatically change to iburst=true and association-type=pool.
The old NTP configurations, which have the default settings, will be updated to new default settings after the upgrade..
This change is relatively secure and is not likely to result in any problems.

1679941-2 : "gen error" while running snmpget/snmpbulkget commands

Links to More Info: BT1679941

Component: F5OS-C

Symptoms:
Triggered shell script which does the snmpget/snmpbulkget in a loop with 50sec delay in each loop reports genError for hrStorageAllocationunits

Conditions:
Snmpwalk is fetching the value for any index. No validation for the key passed.

Impact:
Some OIDs report an error, for example

Error in packet
Reason: (genError) A general failure occured
Failed object: HOST-RESOURCES-MIB::hrStorageAllocationUnits.131080

Workaround:
None

Fix:
Need to validate the index/key

1677797-1 : OMD on Active CC hung due to 'oc delete project' command hang, after delete and recreate a partition and move slots

Links to More Info: BT1677797

Component: F5OS-C

Symptoms:
After deleting and recreating a partition and then moving slots in to the new partition, as a result:
* Blades scheduling is disabled
* multus and/or kubevirt are unhealthy
* Pods pending in the new partition
* Controller-manager pods CrashLoopBackOff
* New partition namespace is terminating

Conditions:
This issue occurs when you delete and recreate a partition.
During this operation, slots are moved to the new partition.
The ‘oc delete project’ command hangs, causing OMD Active CC to hang.

Impact:
This leads to system instability due to blade scheduling issues. Unhealthy pods impacting functionality and service availability.

Workaround:
Restart OMD services on Active CC.

Fix:
The issue has been resolved by adding timeouts to the ‘oc delete project’ command. This ensures the operation will not hang indefinitely, preventing the OMD Active CC from locking up and allowing the system to recover cleanly after partition and slot changes. You should now experience improved reliability during these operations.

1673925-4 : Missing masquerade MAC FDB entry causes excessive DLFs following tenant failover.

Links to More Info: BT1673925

Component: F5OS-C

Symptoms:
The FDB entry for the tenants masquerade MAC is missing from a blades internal L2 table after a tenant failover.

The output of

[root@blade-1 ~]# docker exec -i partition_fpga tmctl -d blade -w 180 nse_l2 -s mac,l2_tag
mac l2_tag
--- ------

[root@blade-1 ~]

where MAC and L2_tag match the masquerade MAC and VLAN from the output of 'show FDB'

Conditions:
During tenant failover, the system will delete the masquerade MAC from the old active and add it to the new active. In parallel, the system will detect a port-motion event when the tenant issues a GARP for the new MAC.

This introduces a race condition between the static ADD from the system and the dynamic port-motion event from the H/W. If the port-motion event is processed last, the new static entry can be deleted erroneously.

Impact:
All front-panel traffic towards the tenant will encounter a DLF, causing excessive DLF traffic to the tenant.

Workaround:
From the tenant, remove and then re-add the masquerade MAC to the traffic group.

Fix:
For port-motion events, don't delete the existing entry if it's a static system entry.

1672269-1 : Blades missing L2 entries causing excessive DLFs.

Links to More Info: BT1672269

Component: F5OS-C

Symptoms:
Excessive DLFs from certain blades due to missing L2 entries.

The 'l2fs_stat' tmstat table shows the IDs of the blades to which L2 entries will be forwarded to:

[root@blade-1 ~]# docker exec -i partition_fpga tmctl -d blade -w 180 l2fs_stat -s svc_ids
svc_ids
---------------------------------
[ 0x2c 0x4c 0x6c 0x8c 0xac 0xcc ]

[root@blade-1 ~]#

In this example, blade-1 will forward to blades 3, 5,7,9,11 and 13.

A blade should have an entry for all other blades in the partition.

Conditions:
Reboot of a tenant or changing the tenant from deployed to configured back to deployed.

Impact:
L2 entries learned on the affected blade are not forwarded to other blades causing missing L2 entries on those blades.

Workaround:
Reboot the blade that's missing the entries for other blades.

For example, blade-1 is missing IDs for all blades in the partition:

[root@blade-1 ~]# docker exec -i partition_fpga tmctl -d blade -w 180 l2fs_stat -s svc_ids
svc_ids
---------------------------------
[ ]

[root@blade-1 ~]#

Fix:
On tenant deletion, don't remove service IDs belonging to the L2FwdSvc.

1670437-1 : Jumbo frames with an IP length greater than 9174 bytes may be dropped

Links to More Info: BT1670437

Component: F5OS-C

Symptoms:
Jumbo frames with an IP total length greater than 9174 bytes are dropped when traversing the VELOS inter-blade backplane.

Conditions:
This issue may occur for VELOS tenants with a VLAN MTU set to 9175 or higher.

Impact:
Data transfers between a VELOS tenant and another host configured with the same MTU may be disrupted. Individual packets may be dropped, or some flows may be permanently dropped.

Workaround:
Do not set the VLAN MTU higher than 9174 on a VELOS tenant.

Fix:
The MTU limit of the inter-blade backplane has been increased to align with the maximum supported size of jumbo frames, ensuring that jumbo frame communication is reliably transmitted without packet drops.

1670029-1 : Reset counter functionality not working properly on rSeries platforms

Links to More Info: BT1670029

Component: F5OS-C

Symptoms:
On rSeries appliances, interface counters will be reset briefly but then revert to the previous values. This behavior occurs within both the Link Aggregation Group (LAG) and individual interfaces, affecting the accuracy of network statistics and troubleshooting efforts.

Conditions:
Execute the “reset counters all” or equivalent command. The counters briefly reset before reverting to their previous values.

Impact:
The issue impacts the accuracy of interface statistics displayed in the GUI section under “Network -> Network Details.” When you reset counters for a specific interface, only the “Out” counters are successfully reset to 0, while the “In” counters remain unchanged or continue increasing. This causes confusion or incorrect reporting during network diagnostics or performance monitoring.

Workaround:
None

1660961-4 : Active Directory LDAP integration without uidNumber/gidNumber does not work with LDAP over TLS

Links to More Info: BT1660961

Component: F5OS-C

Symptoms:
Configuring an F5OS device to integrate with Active Directory using group names to map to roles rather than requiring unix attributes (uidNumber/gidNumber) in the directory will not work if the LDAP servers are configured to use encryption (TLS/SSL).

Log messages similar to the following in platform.log / velos.log:

authd[8]: priority="Err" version=1.0 msgid=0x3901000000000101 msg="LDAP API error during : -" oper="bind" code=-1 msg="Can't contact LDAP server".
authd[8]: priority="Warn" version=1.0 msgid=0x3901000000000098 msg="Unable to retrieve domain Sid for supplied servers and domains; server will be treated as if it has unix attributes present.".

Conditions:
- LDAP system authentication configured to authenticate against an Active Directory Server
- Under the system Authentication Settings configuration in the Common LDAP Configuration section, "Authenticate with Active Directory" set to True and "Unix Attributes" set to False
- LDAP group filters specified for one or more roles

Impact:
LDAP authentication functions based on unix attributes in the directory (uidNumber/gidNumber)

Workaround:
None

1644293 : Interface status alert and SNMP trap is not sent immediately after interface is disabled

Links to More Info: BT1644293

Component: F5OS-C

Symptoms:
When an interface is disabled, the alert or SNMP trap is not sent immediately.

Conditions:
-- Disable an interface.

Impact:
No alert or SNMP trap is sent when an interface is disabled. The trap is sent when the interface is re-enabled.

Workaround:
None

Fix:
Add a new "Interface disabled" event triggered when an interface is disabled. The "Interface up" and "Interface down" alerts changed to events.

1644221-3 : Log file grows to gigabytes (GBs) under /var/log

Links to More Info: BT1644221

Component: F5OS-C

Symptoms:
The default setting for logrotation on host-os is once per day. This can be troublesome if a problem arises and causes an excessive amount of log files to be generated. In such cases, the log files will grow to several GBs within a day.

Conditions:
If any service floods the logfiles under /var/log then file starts to grow in GBs.

Impact:
System disk gets full and becomes unusable.

Workaround:
None

Fix:
This issue has been fixed and the Log files will no longer grow in GBs.

1644185-1 : DAG State table is not cleaned when a tenant is deleted or moved to configured/provisioned

Links to More Info: BT1644185

Component: F5OS-C

Symptoms:
DAG State table is not cleared when a tenant is deleted, or moved to configured or provisioned state

Conditions:
1. Deploy a tenant and confirm the sDAG state table is present in partition ConfD.
2. Delete the tenant

Impact:
DAG State table is not deleted. The stale table is no longer functional.

Workaround:
The stale table can be manually deleted.

Fix:
DAG State table is now cleaned when a tenant is deleted.

1642081 : "default" partition key sometimes initialized improperly★

Links to More Info: BT1642081

Component: F5OS-C

Symptoms:
There is a potential for the default partition to incorrectly initialize the partition primary key at initial startup.

If this happens the API gateway on the blades will log this error message and secure tenants will be unable to connect.

2024-09-05T17:05:18.626737+00:00 default api-svc-gateway[12]: nodename=blade-1(p1) priority="Err" version=1.0 msgid=0x5803000000000010 msg="Key header check failed" HEADER="????xg?A????j?8?????p?}=?ajT".

Once the database & key are mismatched, the partition database is non-recoverable.

Conditions:
This issue only affects the "default" partition, and only during initial database creation following either a USB install or resetting the system controller database using "system database config reset-default-config true".

It does not affect any other partition. It does not occur if the controller database is reinitialized using "system database reset-to-default".

Impact:
Tenant will be unable to connect to the API Gateway and start up correctly.

Other encrypted fields will also be unable to be decoded.

Workaround:
Before configuring and enabling the default partition, recreate the default partition using the following command sequence.


syscon-2-active# config
Entering configuration mode terminal
syscon-2-active(config)# no partitions partition default
syscon-2-active(config)# validate
Failed: illegal reference 'slots slot 1 partition'
syscon-2-active(config)# partitions partition default ; exit
syscon-2-active(config)# validate
Validation complete
syscon-2-active(config)# commit
Commit complete.

If the partition has ever been enabled, this sequence will not have the desired effect, and will not repair the partition.

Fix:
The database startup initialization is fixed to ensure that the default partition primary key is correctly initialized.

1638629-1 : "Unhealthy" kubevirt pod due to internal networking issue with blade★

Links to More Info: BT1638629

Component: F5OS-C

Symptoms:
Some kubevirt pods are in a "CrashLoopBackOff" state following a live upgrade. The output of the 'show cluster' command shows that kubevirt status is unhealthy.

Conditions:
Exact conditions are unknown and this occurs rarely.

It was encountered during internal testing after a live upgrade.

Impact:
Might affect tenant deployment & traffic on the issued blade.

Workaround:
There are 2 workarounds for this issue:

1. Reboot the affected blade
2. Unschedule & reschedule the affected node

Steps for workaround #2:

'oc adm cordon <node>' ------> Mark <node> as unschedulable.

'oc adm drain <node> --delete-local-data --ignore-daemonsets' -----> safely evicts all pods from the specified node,preparing it for maintenance or decommissioning.

'oc adm uncordon <node>' -------> mark the node as schedulable again. After the maintenance is complete, can use this command to allow new pods to be scheduled onto the node.

Fix:
Please follow the work around steps and contact f5 support if need further assistance.

1634545 : OpenShift cluster may fail to install if no management IP's are configured★

Links to More Info: BT1634545

Component: F5OS-C

Symptoms:
The OpenShift cluster may fail to install after a bare-metal install or cluster rebuild if not management IP's have been configured on the system controller management ports. The output of the 'show cluster' command reports that 'MasterInstall' is in a state of Failed.

syscon-1-standby# show cluster
STAGE NAME STATUS
--------------------------------------
AddingBlade Not Started
HealthCheck Done
HostedInstall Not Started
MasterAdditionalInstall Not Started
MasterInstall Failed <===========
NodeBootstrap Done
NodeJoin Not Started
Prerequisites Done
RemoveBlade InProgress
ServiceCatalogInstall Not Started
etcdInstall Done

Conditions:
No management IP's configured on the system controller management ports while an OpenShift cluster install is initiated, either via a bare-metal install or a manual cluster rebuild.

Impact:
OpenShift cluster install will fail until management IP's are configured on the the system controller management ports.

Workaround:
Configure management IP's on the system controller management ports.

Fix:
F5OS will now add a default route on both system controllers that will allow the OpenShift cluster install to complete even when no management addresses have been configured on the system controller management ports.

1633681-1 : Dynamic FDB entries may not be flushed from all blades when a vlan tag is removed from a LAG.

Component: F5OS-C

Symptoms:
When a vlan tag is removed from a LAG in a VELOS partition, existing FDB entries for that vlan that were learned on that LAG may not be flushed out on each blade.

If that vlan is then added to a different interface or LAG, the old FDB entries may get updated via L2 learning. But if that fails to happen (e.g. due to ID1620077), the old entries may persist.

Conditions:
Remove a vlan tag from a LAG on VELOS, and add the vlan to another.

Old FDB entries may persist when moving a vlan tag from a LAG to another LAG. If moving a vlan tag from a LAG to an interface, L2 learning seems to correct the situation.

Impact:
Since the old FDB entries are not flushed, if the system fails to update them via L2 learning also, egress traffic that matches these old entries is dropped.

This depends on which blades have the old entries and where the tenants are assigned to run. Tenant instances running on those blades are impacted, for the MAC address and vlan matching the old entry.

Workaround:
If old L2 entries persist, a reboot of the blade is required to clear them out.

1633073-4 : A core can occur in a forked process with an Orchestration Agent

Links to More Info: BT1633073

Component: F5OS-C

Symptoms:
You may occasionally notice a core file from a forked process of the orchestration agent.

Conditions:
This can occur in orchestration agent during normal operation.

Impact:
There’s a minimal impact. The core occurs rarely. It happens in a forked process during a read of the partition token. It doesn’t core the overall orchestration agent, only the forked process. There are no error logs. If the read fails, there will be a retry.

Workaround:
None

1629257-2 : Diag-agent service memory utilization increases because of heartbeat probe

Links to More Info: BT1629257

Component: F5OS-C

Symptoms:
Diag-agent service memory utilization rises if not controlled which can lead to OOM.

Conditions:
Diag-agent service generates heartbeat events which are sometimes creating a deadlock in the service. Once deadlock is hit the memory queue of diag-agent service in increasing because of heartbeat probes and eventually diag-agent service memory utilization also rises.

Impact:
Diag-agent service memory utilization rises if not controlled which can lead to OOM.

Workaround:
None

Fix:
Updated diag-agent service handle event locking in a better way so that a deadlock does not occur.

1628557-3 : F5OS high memory usage when using snmp

Component: F5OS-C

Symptoms:
Excess memory usage by snmpd while running commands.

Conditions:
Excess memory usage by snmpd on the F5OS Chassis or Appliance system.

Impact:
Potential system crash with out of memory errors.

Workaround:
Excess memory used can be released by restarting the snmpd service.

The CLI commands to restart the service are given below,

In 1.8.0:
==========
F5OS-A:
appliance-1(config)# system diagnostics os-utils docker restart node platform service snmpd
appliance-1(config)# system diagnostics os-utils docker restart node platform service system_platform-stats-bridge


Releases earlier to 1.8.0
==========================
F5OS-A:
docker restart snmpd
docker restart system_platform-stats-bridge

F5OS-C (Controller):
docker restart snmpd
docker restart platform-stats-bridge-cc

F5OS-C (Partition):
docker restart partition<n>_snmpd
docker restart partition<n>_platform-stats-bridge

Fix:
Excess memory usage no long occurs.

1627541-1 : System Controller unexpected failover in auto mode due to unhealthy SwitchD

Links to More Info: BT1627541

Component: F5OS-C

Symptoms:
A issue was identified where an unhealthy status reported by switchd was causing a system controller failover.

Conditions:
This issue occurs when switchd experiences a transient connection problem with ConfD and as a result reports it is unhealthy.

Impact:
The reporting of a transient ConfD connection problem as unhealthy triggers an unexpected system controller failover.

Workaround:
None.

Fix:
Switchd no longer reports an unhealthy condition because of a transient ConfD connection interruption thus removing this as a trigger of system controller Failover.

1624853-3 : ETCD consumes a high amount of CPU time

Links to More Info: BT1624853

Component: F5OS-C

Symptoms:
ETCD may consume a significant amount of CPU time after a controller failover, or when tenants are being deployed or removed.

Conditions:
Conditions causing extended high CPU time are unknown at the moment.

Impact:
This may slow down other F5OS control plane processes while ETCD is consuming a high amount of CPU.

Workaround:
If the ETCD CPU usage is continually high, it is possible to restrict the CPU's that ETCD is allowed to run on.

This can be done from the system controller shell, and needs to be done on both system controllers. This will need to re-done on system controller reboot or failover.

for x in $(pgrep 'etcd$'); do taskset -cp 4-7 $x; done

1624777-1 : Tenants will not deploy since Orchestration Agent process is continuously generating a core

Links to More Info: BT1624777

Component: F5OS-C

Symptoms:
When attempting to deploy a tenant an error occurs:

tenants tenant my-bigip-1 config type BIG-IP (fill out all prompts)
default-1(config-tenant-my-bigip-1)# commit
Aborted: application communication failure

Core files are found in the partition's /shared/core/container/ directory.

Conditions:
-- Creating a BIG-IP tenant
-- Orchestration agent is crashing

Impact:
Tenants cannot be deployed if Orchestration Agent is crashing. User will not be able to deploy a tenant successfully.

Workaround:
None

1624665-4 : ConfD state data shows key and certificate configured for secure (mTLS) even after deleting from config

Links to More Info: BT1624665

Component: F5OS-C

Symptoms:
ConfD operational state data shows key and certificate configured for mutual transport layer security (mTLS) even after deleting them from configuration.

Conditions:
When the exporter is configured with mutual TLS. And then the key and certificate are deleted from the configuration. ConfD operational state data displays the deleted key and certificate for the exporter.

Impact:
No functional impact.

Workaround:
Delete the exporter and reconfigure it again.

Command to delete the exporter from ConfD CLI:

no system telemetry exporters exporter <exporter-name>

1624449-2 : SNMP polling of coreTotal5minAvg causing timeouts and genErrors

Links to More Info: BT1624449

Component: F5OS-C

Symptoms:
While running an snmpwalk that includes coreTotal5minAvg, you may get a timeout or a general error:

 Timeout: No Response from 10.170.9.16

The general error occurs less frequently:

 Error in packet
 Reason: (genError) A general failure occured

Conditions:
-- snmpwalk a MIB that includes coreTotal5minAvg
-- The polling is done for CPUs that are not present

Impact:
Error in packet

 Reason: (genError) A general failure occurred

 Failed object: iso.3.6.1.4.1.12276.1.2.1.1.3.1.6.8.112.108.97.116.102.111.114.109.0

Workaround:
After the system starts, after about two minutes, platform-stats-bridge will log this log message:

msg="DB ready check done" NAME="SnmpCpuStatsHandler".

After that log message, you will be able to check coreTotal5minAvg.

Fix:
Modified code such that snmpwalk will not be executed for offline cpus

1624057-2 : BX110 Port Flapping or interface/connectivity issues

Links to More Info: BT1624057

Component: F5OS-C

Symptoms:
F5OS-C v1.8.0 has a fix for an issue "VELOS interfaces flapping if an interface is disabled"; however a corner case remains that could still cause port flapping or have ATSE register reads return 0xebade001 instead of the correct value.

Conditions:
VELOS system

Impact:
Interfaces are intermittently marked DOWN and then UP. Traffic is disrupted while the interface is marked DOWN.

There may be other intermittent issues with interfaces or general connectivity issues.

Workaround:
Upgrade to F5OS-C 1.8.0 EHF-1

1623761 : After cleaning up disk due to disk space full error, tcpdump program still detects the disk as full and aborts

Links to More Info: BT1623761

Component: F5OS-C

Symptoms:
Tcpdump program detects the disk and aborts if the disk does not have enough space. However, even after cleaning up the disk, tcpdump does not recover from the abort state.

Conditions:
Fill up the disk space in /var/F5/partition/shared/. Then, run tcpdump from confd. An abort error will show up. After cleaning up the disk space, the system will still show abort errors when running tcpdump in confd.

Impact:
Can not run tcpdump after the disk space have been full at one point in time.

Workaround:
Restart the tcpdumpd_manager container on the controller that is running-active for the partition.

1623101-2 : External OTEL server receives log data for both the platform and event logs, even if only one of them has been configured

Links to More Info: BT1623101

Component: F5OS-C

Symptoms:
The configured OTEL exporter receives log data from both platform-log and event log, even when only one of them is configured.

Conditions:
This occurs when you configure one telemetry exporter with only either of “platform-log” or “event-log” instruments and another telemetry exporter with “all” or “logs” or both “[platform-log event-log]” instruments.

Impact:
The telemetry exporter configured to receive only platform-log or event-log instrument data will receive data from both log instruments.

Workaround:
None

1622869-5 : Might see TPOB core after HA disassembly

Links to More Info: BT1622869

Component: F5OS-C

Symptoms:
TPOB container might crash after performing BIG-IP Next-HA disassembly operation.

Conditions:
-- BIG-IP Next in a HA pair
-- The HA pair is disassembled and factory reset

Impact:
No impact, as the container gets re-created

Workaround:
None

Fix:
No Fix needed

1620513-1 : CVE-2024-38477 httpd: NULL pointer dereference in mod_proxy

Links to More Info: K000140784, BT1620513

1620077-4 : FDB entry port motion not working if new interface is a trunk/LAG

Links to More Info: BT1620077

Component: F5OS-C

Symptoms:
Immediately after a fail-over of traffic from one trunk/LAG to another, outbound traffic from the appliance or chassis to certain addresses may be interrupted for up to five minutes before recovering.

Conditions:
Switching traffic from one LAG to another on an appliance or chassis.

Impact:
Temporary disruption of tenant’s outbound traffic on an appliance or chassis system.

Workaround:
None

Fix:
Updated handling of FDB entry port motion to include cases with a trunk/LAG as the new interface.

1615969-4 : Tenant operational data is not getting updated properly after upgrade

Links to More Info: BT1615969

Component: F5OS-C

Symptoms:
Tenant pods are up and running but not all details are updated.
Intermittently after upgrade to F5OS-A 1.8.0 version, Tenant operation data in confD not getting updated

Conditions:
Occasionally, the tenant's operational data is not completely updated.

Impact:
Operational data for tenant is not updated properly after system upgrades to F5OS-A 1.8.0 intermittently.

Workaround:
Toggle tenant running-state to configured and deployed, then verify the tenant details again.

Fix:
Handled tenant operational map data updates properly.

1615917-1 : L2_agent crashed due to SNMP★

Links to More Info: BT1615917

Component: F5OS-C

Symptoms:
After upgrading system to 1.8.0, L2-agent crashes.

Conditions:
1. Create system with older version (earlier then 1.8.0)
2. Configure SNMP
3. Upgrade system to 1.8.0 version
4. L2-agent will start crashing.

Impact:
L2-agent crashes and you are unable to do get/set operations for interfaces using ConfD interfaces.

Workaround:
None

Fix:
Fixed an issue causing l2-agent to crash after upgrade.

1614821-3 : CVE-2024-3596 - Blast-RADIUS

Links to More Info: K000141008, BT1614821

1614429-1 : iHealth upload is failing with error "certificate signed by unknown authority"

Links to More Info: K000140362, BT1614429

Component: F5OS-C

Symptoms:
When attempting to use the QKView upload feature, the upload may fail with the message "certificate signed by unknown authority". This is due to a recent change in certificate authority that is inconsistent between F5OS and iHealth.

Conditions:
Always, after mid-July 2024.

Impact:
Unable to upload QKView files to iHealth with a single click.

Workaround:
You can use the File Export feature to download QKView files, and then upload these files to iHealth.

You can find the QKView files in the GUI at System Settings > File Utilities, then choose "diags/shared" as the base directory, then select "qkview".

Fix:
Certificate authorities used by the iHealth upload feature in F5OS will be updated.

1612557-1 : Dma-agent service health warnings appears in show system summary

Component: F5OS-C

Symptoms:
Dma-agent service health warnings shown in show system health summary even when dma-agent service is reporting healthy.

Conditions:
When the health file is not deleted by any means and created again making it untracked.

Impact:
When dma-agent sevice health file reports dma-agent to be healthy, stale data (including warnings) might be seen in show system health summary.

Workaround:
SSH to the impacted blade and restart the platform-monitor service. E.g.

  ssh blade-1
  docker restart platform-monitor

Fix:
Show system health won't show stale data (warnings) when dma-agent service health file reports dma-agent to be healthy.

1612405-5 : LACP status shows UP in BIG-IP tenant even if its down on F5OS.

Links to More Info: BT1612405

Component: F5OS-C

Symptoms:
LACP Trunk is UP in BIG-IP tenant even when it’s DOWN on F5OS.

Conditions:
Condition 1:
1. Setup a rSeries or VELOS system.
2. Configure LACP LAG with interfaces operationally down.
3. Make sure LACP Trunk is DOWN on F5OS.
4. Upgrade the software.
5. Launch a BIG-IP tenant.
6. Check LACP trunk status inside tenant.

Condition 2:
1. Setup a rSeries or VELOS system.
2. Configure STATIC LAG with interfaces operationally down.
3. Ensure STATIC Trunk is DOWN on F5OS.
4. Launch a BIG-IP tenant.
5. Check the Trunk status inside the tenant. It will be DOWN.
6. Convert LAG type to LACP
7. Check the Trunk status inside the tenant. It will be UP even though it is down on F5OS.

Impact:
LACP Trunk members are shown as working members even though they are DOWN.

Workaround:
Check the interface config. If the admin is disabled, enable it.

Fix:
The status of LACP members is read whenever an LACP member is added as an operational member.

1612217-1 : A large amount of SPVA DoS allow list entries can overload DMA-Agent causing a tenant to fail to pass traffic

Links to More Info: BT1612217

Component: F5OS-C

Symptoms:
If the DMA-Agent receives a high volume of SPVA allow list entries at once, it may become overwhelmed and stop working. As a result, no traffic will be able to exit the tenant. This can be identified by observing the DMA-Agent using 100% of the cpu.

Conditions:
This is usually seen in configurations where there are many virtual servers configured with a dos profile that contains an IP-based allow list.

The problem does not arise when VIPs are added individually, but it often happens after TMM is restarted following a tenant reboot.

Impact:
Tenant will fail to pass any traffic on the data-plane.

The TMSTAT sep_stats.tx_send_drops3 will be incremented.

Workaround:
Perform the following on the tenant:
tmsh modify sys db dos.forceswdos value true
tmsh save sys conf

To recover the DMA-Agent in F5OS, set the tenant state to “configured” and then set it back to “deployed.

Fix:
The DMA-Agent now handles a high volume of SPVA allow list entries.

1612101-2 : When vCPU cores configuration changed for BIG-IP Next tenant, RRD stats shows both the old and new CPU data stats

Links to More Info: BT1612101

Component: F5OS-C

Symptoms:
The RRD stats display the data for old and new CPU cores. You can match the new CPU cores and validate the data. The old CPU cores data is invalid and should not be displayed.

Conditions:
When user configures BIG-IP Next tenant and changes the vCPU cores.

Impact:
No Functional Impact. Both old and new data stats appear for cpu-stats in RRD. However, data streaming works as expected.

Workaround:
None

Fix:
None

1607745-3 : Apache HTTPD vulnerabilities CVE-2024-38476, 2024-38474 and CVE-2024-38475

Links to More Info: K000140618

1603509 : No alarm sent when front panel management link is down

Links to More Info: BT1603509

Component: F5OS-C

Symptoms:
When the front panel management port is down, no alarm is sent

Conditions:
Happens only when chassis is power cycled or blades are inserted/removed in slot 0 and 1.

Impact:
No alarm sent when front panel management link is down and switch stats displayed will not have accurate entries in "show system health".

Workaround:
None

Fix:
Diag-agent will not remove switch port entries when it receives module present events for slot 0 and 1.

1600693-1 : F5OS - BIG-IP Tenant does not display VELOS Chassis slot serial number

Links to More Info: BT1600693

Component: F5OS-C

Symptoms:
F5OS BIG-IP Tenant does not display the serial number for the slot ("Host Board Serial") under "System Information"

Conditions:
BIG-IP tenant is running on a chassis, and command "tmsh show sys hardware" is run from the tenant

Impact:
The slot serial number is not immediately visible to the user

Workaround:
For CLI, login to the partition and run command "show components component state serial-no". For GUI, login to the active controller, then go to System Settings -> System Inventory. The blade serial number will be shown.

Fix:
F5OS was updated to provide the blade serial number to the tenant for display. The tenant was updated to populate the blade serial number into "show sys hardware" command output, so it is now visible to the user. This fix requires a version 17.5 tenant.

1598937 : SNMP traps are not always sent★

Links to More Info: BT1598937

Component: F5OS-C

Symptoms:
After upgrading to 1.8.0 version SNMP traps may stop working.

Conditions:
Upgrade system to 1.8.0 from previous version.

Impact:
SNMP trap functionality does not work

Workaround:
Reconfigure the SNMP configuration.

Fix:
Correct the SNMP configuration in the upgrade case. So, issue is resolved.

1598509-2 : iHealth client can occasionally throw a core file

Links to More Info: BT1598509

Component: F5OS-C

Symptoms:
The iHealth client, accessible with the command line,
system diagnostics ihealth can be used for uploading QKView files to the iHealth service. If this client loses connection to the system database for any reason, it may throw a core file, in the host system's /var/shared/core directory.

Conditions:
System has been up for a long time, and there is a problem with the ConfD database causing the iHealth client to disconnect.

Impact:
A core file may be thrown. The iHealth client will restart if this happens, so functionality is not affected.

Workaround:
Retry the ihealth client operation.

Fix:
The iHealth client will only access the ConfD database when it needs to query information, and not maintain an open connection.

1596149-1 : Monitor rSeries ATSE to BE2 links and Raise Alarms in the Event of Failures

Links to More Info: BT1596149

Component: F5OS-C

Symptoms:
Monitor rSeries ATSE to BE2 links and Raise Alarms in the Event of Failures

Conditions:
F5 rSeries r5000, r10000, or r12000-series appliance.

This update is not applicable to r2000 or r4000-series appliances.

Impact:
In cases where errors are detected between the ATSE and BE2 links, alarms and events will be reported.

Workaround:
None

Fix:
Monitor ATSE to BE2 links and raise alarms and report events when errors are detected.

1595113-4 : Interface state enabled value stale due to timeout to reach confd

Component: F5OS-C

Symptoms:
When trying to modify the interface admin status to disabled across five different interfaces on five blades in a VELOS partition in a single commit message, the CLI operation to update the state interface enabled field fails with an error "system call failed". "Failed to write 68 bytes to ConfD: Connection timed out".

Conditions:
This can occur when a failover of chassis-controller and partition occurs, right before the interface enabled field changes.

Impact:
Stale value for interface/state/enabled field.

Workaround:
Enable and re-disable the interfaces.

Fix:
With the fix, the interface/state/enabled field will reflect accurately the configuration admin status of the interface.

1594125 : GUI fails to modify interfaces on F5OS-C

Links to More Info: BT1594125

Component: F5OS-C

Symptoms:
Interface-related operations from the GUI fail.

Conditions:
-- Interface-related operations like LAG creation or deletion.
-- F5OS build prior to 1.8.0-15246

Impact:
You are unable to perform interface operations from the GUI

Workaround:
None

Fix:
GUI is able to modify the interfaces on F5OS-C

1593385 : F5OS Tenant Throughput (bits/packets) and TMM CPU usage higher than expected until VLAN is added or removed

Links to More Info: BT1593385

Component: F5OS-C

Symptoms:
Higher CPU usage and throughput from the tenant than expected. Traffic being directed to a single blade in a multi-blade system.

Conditions:
Repeated deletes/adds of a VLAN from/to a tenant. After approximately 130 deletes, the issue occurs.

Impact:
Traffic imbalance, higher than normal CPU usage.

Workaround:
Re-add the recently deleted VLAN to the tenant.

Fix:
Properly clean up internal storage when a VLAN is deleted from a tenant.

1592221 : A partition's internal bridge IP address is not detected correctly if there is a missing partition ID in the list of partitions.

Links to More Info: BT1592221

Component: F5OS-C

Symptoms:
The system controller logs will include the msg "Floating IP is not present for enabled partition; do not change controller state" on system controller failover.

Conditions:
When the list of partition IDs includes "holes" in the list. For example, there are partition IDs 1 and 3 (but no 2) on the chassis. This can happen if a partition is deleted.

Impact:
System controller failover is impacted.

Workaround:
Recreate a partition (no need to enable it). It will use the missing ID in the list.

Fix:
The code has been fixed to correctly check the partition ID when detecting presence of the partition bridge IP address.

1591645-3 : EPVA related dma-agent crash

Links to More Info: BT1591645

Component: F5OS-C

Symptoms:
A dma-agent seg_fault occurs when there is a conflict between special EPVA allow-list entries.

Conditions:
A conflict between two entries on the allow-list triggers a code path in the dma-agent and resulting in a seg_fault.

Impact:
Traffic loss as the dma-agent needs to be restarted by its watchdog/start up script. Tenants need to re-register with the datapath.

Workaround:
None

Fix:
This issue has been fixed by setting a THREAD local variable in the epva_tbl_mgmt thread, preventing a seg_fault when the edge case method is triggered.

1591585 : Sshd, httpd, rsync crashes with bunch of whitespaces in /etc/hosts file

Links to More Info: BT1591585

Component: F5OS-C

Symptoms:
When VELOS system controllers fail over, OMD rewrites /etc/hosts on each controller to move around where the 'etcd3.chassis.local' name is assigned.

When this occurs an extra space character is added to the controller-1.chassis.local and controller-2.chassis.local lines. If you add enough whitespace to /etc/hosts (uncertain how much, but megabytes will do it), it starts causing daemons to crash in getnameinfo() calls as they try to resolve the local system IP to a hostname.

Conditions:
VELOS System controllers fails over. Extra space characters are added after controller-X.chassis.local.

Impact:
Sshd, httpd, rsync crashes when the whitespace in /etc/hosts becomes excessive.

Workaround:
Run below command in bash to remove extra space in etc/hosts file.
sed -i 's/[[:space:]]＼+$//' /etc/hosts

Fix:
Fixed C-1.8.0

1591553 : Including /etc/resolv.conf and /etc/hosts files in QKView capture

Links to More Info: BT1591553

Component: F5OS-C

Symptoms:
The /etc/resolv.conf and /etc/hosts files are included to check the configured parameters in host QKView from the affected device.

Conditions:
F5OS-A 1.7.0 and lower versions QKView capture does not include the /etc/resolv.conf and /etc/hosts files.

Impact:
The /etc/resolv.conf and /etc/hosts files are not captured in F5OS-A 1.7.0 and lower versions.

Workaround:
None

Fix:
The /etc/resolv.conf and /etc/hosts files are included in QKView capture as part of F5OS-A 1.8.0 release.

1591549-1 : Support for case-insensitive LDAP username lookup

Links to More Info: BT1591549

Component: F5OS-C

Symptoms:
Previously, username lookup for LDAP-authenticated users was always case-sensitive.

Conditions:
Third-party authentication is configured with LDAP or Active Directory; user(s) in question reside in LDAP directory.

Impact:
Username lookups for authentication/authorization against LDAP directory were always conducted in a case-sensitive fashion, even for directories where case-insensitive was the default for the organization (e.g. Windows AD).

Case-insensitive default is considered a safer security posture. It prevents username masking and cache injection when multiple users that only differ by case, with differing authorization privileges, exist in the same directory.

Workaround:
Always use correct case for case-sensitive searches.

Fix:
A new option was added which allows the admin to enable case-insensitive searches for LDAP username lookups. Note that case-sensitive remains the default for security reasons.

1591069 : Blades may fail to get marked as InCluster in "show cluster" output after rolling upgrade

Links to More Info: BT1591069

Component: F5OS-C

Symptoms:
After a rolling upgrade, one or more blades may be marked as "Not In Cluster" in the "show cluster" output.

Conditions:
Perform rolling upgrades from a manufacturing-installed F5OS C v1.7.0 to F5OS C v1.7.1.

Impact:
System will function correctly, but "show cluster" output will show the blade is not being marked "In Cluster".

Workaround:
To workaround the issue, the orchestration-manager daemon can be restarted, which will result in the "In Cluster" status being updated. This action needs to be performed from the shell on both controllers.

systemctl restart orchestration_manager_container.service

Fix:
Fixed issue in the orchestration-manager daemon causing the "In Cluster" status not being updated.

1590617-1 : Partition Network Manager is crashing when turning up.

Links to More Info: BT1590617

Component: F5OS-C

Symptoms:
Upon Partition turn up, the Network Manager component crashes.

Conditions:
The Partition is turning up. This can happen due to partition creation, partition enable, or controller reboot.

Impact:
No impact. The Network Manager will successfully start after a retry.

Workaround:
None

Fix:
None

1590425 : Adding blade to openshift cluster can fail with ansible error

Links to More Info: BT1590425

Component: F5OS-C

Symptoms:
Adding or re-Adding a blade to the OpenShift cluster can fail with the following ansible error:

fatal: [blade-2.chassis.local -> controller-1.chassis.local]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: {{ hostvars[groups.oo_first_master.0].openshift.master.api_url }}: 'dict object' has no attribute 'master'＼n＼nThe error appears to be in '/usr/share/ansible/openshift-ansible/roles/openshift_manage_node/tasks/main.yml': line 5, column 3, but may＼nbe elsewhere in the file depending on the exact syntax problem.＼n＼nThe offending line appears to be:＼n＼n# systemd to start the master again＼n- name: Wait for master API to become available before proceeding＼n ^ here＼n"}

Conditions:
Adding or re-Adding a blade to blade the OpenShift cluster after the etcd instance been rebuilt.

Impact:
New blade will not join the cluster correctly.

Workaround:
The workaround is to rebuild the OpenShift cluster which will regenerate the openshift.fact file that had been corrupted.

Fix:
The fix checks the openshift.fact file before running any ansible playbooks to make sure it is correct.

1588093-1 : Forwarding host log files to remote targets

Links to More Info: BT1588093

Component: F5OS-C

Symptoms:
/var/log/messages growing quickly, consuming the disk space, making the box unusable.

Conditions:
Having /var/log/messages as a host-logs files entry to forward the file lines to a remote destination.

Impact:
When syslog generated files are configured to be forwarded as files, forwarding efficiency can be affected compared to utilizing selectors.

The /var/log/messages being in this list can lead to a cyclical logging issue, where the disk space is consumed faster than the logs can be rotated out, potentially resulting in a full disk.

Workaround:
Use selectors instead for any file that is syslog generated.

The host-logs files configuration is meant for text files that cannot be forwarded through selectors configuration.

Fix:
To prevent filling the disk, files that are forwarded out line by line would not be processed locally. This will prevent having entries in /var/log/messages.

1587925-1 : Modifying a RADIUS server from the web UI requires the Secret to be configured or re-entered

Links to More Info: BT1587925

Component: F5OS-C

Symptoms:
Modifying a RADIUS server from the webUI always requires the Secret to be configured or re-entered.

Conditions:
Modifying a RADIUS server from the webUI.

Impact:
It requires the Secret to be entered, even if it is already configured.

Workaround:
If secret configuration is not required, edit the RADIUS server from the CLI.

Fix:
Create a Radius server and edit it. Editing the port or timeout fields no longer requires the Secret to enable saving.

1587837 : Memory leak in multiple components

Links to More Info: BT1587837

Component: F5OS-C

Symptoms:
A mishandling of memory allocation in the data provider callback library can cause memory allocation to grow over time. This memory usage growth can cause poor performance and the Out Of Memory (OOM) killer may kill components, causing outages.

Conditions:
If a data provider processes overlapping requests it can leak memory. The components most affected by this are the platform-stats, snmp-service, an L2 agent.

Impact:
Components may crash or get killed.

Workaround:
Monitor memory usage and periodically restart daemons that experience excessive memory growth. On a chassis system, a manual failover and the rebooting the standby controller will restart all daemons.

To minimize the occurrence of this leak, do not constantly poll for statistics, especially from multiple monitoring stations.

Fix:
The library has been fixed to no longer leak session data.

1586965-1 : No active instance of ConfD after failover

Links to More Info: BT1586965

Component: F5OS-C

Symptoms:
Unable to configure VELOS system, ConfD CLI commands fail.

Conditions:
Rarely, after failover newly active system controller silently transitions to none.

Impact:
Unable to configure VELOS system, ConfD CLI commands fail.

Workaround:
Reboot chassis.

Fix:
In releases with this fix in place, after failover there will be always be an Active instance of ConfD.

1586893 : Metrics server pod on system controller can exit and not be restarted

Links to More Info: BT1586893

Component: F5OS-C

Symptoms:
The openshift cluster does not operate properly and the controller-manager pods are in a crash loop.

Conditions:
When the metrics server pod exits it causes the controller-manager pods to go into a crash loop.

Impact:
Openshift cluster will not operate correctly since controller-manger pods are in running state.

Workaround:
If you see the metrics-server pod that is in the exit state, the following commands can be run at the root shell prompt.

1. Run oc get pods -n kube-system |grep -i metrics and get the pod name.

2. Run oc delete -n kube-system pod/<pod-name>

1586773 : BX520 Internal FPGA links can fail to come UP during initialization

Links to More Info: BT1586773

Component: F5OS-C

Symptoms:
On the BX520, internal FPGA links fail to initialize with the following error:

fpgamgr[9]: nodename=blade-11(p1) priority="Err" version=1.0 msgid=0x301000000000006 msg="SDK error during programming." API="f5sw_xilinx_cmac_datapath_reset" port=18 error="Waiting for the Xilinx MAC RX_ALIGN to be acheived has failed. Number of retries have exceeded".

Conditions:
Reboot of a BX520 blade.

Impact:
Traffic outage between FPGAs.

Workaround:
Reboot the affected blade.

Fix:
Implement updated initialization procedure for internal FPGA links.

1586661-2 : First login for a remote user fails

Links to More Info: BT1586661

Component: F5OS-C

Symptoms:
The first time a remote user attempts to login to a system, the access is denied despite providing the correct credentials. This is true for both TACACS or RADIUS remote users.

Conditions:
This happens always. A way to simulate the first login is to delete the file /etc/libnss-udr/passwd.

Impact:
The first login fails. Subsequent remote login attempts succeed with proper credentials.

Workaround:
Attempt renote login again.

Fix:
The user now can login with proper credentials from the first attempt. Note that the fix involves having the following version of openssh (or newer):

# rpm -q openssh
openssh-7.4p1-21.F5.6.2.7.el7.x86_64

1586641 : OPT-0063 400G-FR4 periodically has non-zero RMON_RX_BAD_FCS

Component: F5OS-C

Symptoms:
Small number of FCS errors, up to approximately 100 per second, may be seen in FPGA 400G MAC RMON stats.

Conditions:
No special conditions known.

Impact:
Small number of packets will be dropped for FCS failure.

Workaround:
Disable and re-enable the 400G link.

1586089-2 : Resource-admin is unable to perform SCP.

Component: F5OS-C

Symptoms:
Resource-admin is unable to perform SCP.

Conditions:
When trying to use SCP with resource-admin for the available virtual paths.

Impact:
Resource-admin cannot perform SCP file transfers.

Workaround:
Though SCP fails, the file upload/download API works for file upload/downloads.

Fix:
Permissions for resource-admin to perform the SCP file transfer were added.

1586057-1 : F5OS displays an incorrect error if the admin tries to set a password before committing a new user

Component: F5OS-C

Symptoms:
F5OS reports that a password was rejected and displays the configured password policy if the admin tries to set a new user’s password before the new user has been added to the system.

Conditions:
The admin tries to set a password for a user that has just been configured but not yet committed.

Impact:
The administrator could mistakenly think that the selected password is inadequate. But the actual problem is that the user has not been committed to the system yet.

Workaround:
When creating a new user, admins must commit the new user before setting a user’s password.

Fix:
None

1585853 : Telemetry streaming pauses if mgmt-ip gets updated

Links to More Info: BT1585853

Component: F5OS-C

Symptoms:
Telemetry streaming to an external OTEL server is paused for some time if mgmt-ip of the F5OS device is updated.

Conditions:
There should be a telemetry exporter configured to receive data and the mgmt-ip of the F5OS device will be updated at a later time..

Impact:
The external server won’t receive the telemetry data for some time after updating mgmt-ip.

Workaround:
Disable and enable the exporters from ConfD using below commands to re-establish the connection after updating mgmt-ip.

system telemetry exporters exporter <exporter-name> config disabled

system telemetry exporters exporter <exporter-name> config enabled

Fix:
Updated the otel-collector service in F5OS to re-establish the connection with the external server in the event of a lost connection caused by mgmt-ip updates.

1585749-1 : Including lspci commands in QKView capture

Links to More Info: BT1585749

Component: F5OS-C

Symptoms:
The lspci command helps in analyzing the system's faults by evaluating PCI busses. This command is not captured in the QKView file.

Conditions:
Running QKView.

Impact:
The lspci command output is not included in the QKView.

Workaround:
None

Fix:
The lspci command is added in QKView capture.

1585237-2 : When telemetry exporter is not reachable, logs to enable send_queue or retry will be printed in platform.log

Links to More Info: BT1585237

Component: F5OS-C

Symptoms:
When telemetry exporter is not reachable, logs to enable send_queue or retry will be printed in platform.log.

Conditions:
Logs will be printed only when configured telemetry exporter is not reachable.

Impact:
No functional impact.

Workaround:
Ensure the exporter is reachable.

Fix:
OTEL service will not be logging retry and send queue logs when exporter is not reachable.

1585001 : Radius authentication does not work when the shared secret key in the radius configuration is more than or equal to 32 characters

Links to More Info: BT1585001

Component: F5OS-C

Symptoms:
The remote radius users authentication fails when the radius shared secret has more than 31 characters.

Conditions:
The radius shared secret having more than 31 characters

Impact:
The remote radius users will not access to the system.

Workaround:
Log in as an admin into the system and change the radius 'secret' field to have characters less than or equal to 31.

system aaa server-groups server-group <server-group-name>servers server <server-address> radius config secret-key <number-of-characters-should-be<=31>

Then commit the changes.

Fix:
When the radius secret key is longer than 31, the radius users will not have access to the system.

1584469-1 : BX520 TCPDUMP throughput improvement

Component: F5OS-C

Symptoms:
The BX520 blades had more throughput than the BX110, but the TCPDUMP utility could not keep up with the amount of TCPDUMP traffic the smaller blade could do.

Conditions:
BX520 TCPDUMP throughput was quite low compared to BX110 blades, about half of BX110 when it should be double to 3x that of BX110 since BX520 has 4x throughput as BX110.

Impact:
Slower TCPDUMP from dropping TCPDUMP packets when customer uses system diagnostic TCPDUMP in the confD cli.

Workaround:
None

Fix:
Now the line-dma-agent is servicing the DMs on NSO/TAM fast enough for the TCPDUMP higher-throughput traffic on BX520.

1583233-1 : The 'show portgroups' command may not display DDM statistics, or may display stale/out-of-date DDM statistics

Links to More Info: BT1583233

Component: F5OS-C

Symptoms:
An F5OS system (rSeries appliance or VELOS partition) may display stale/out-of-date DDM statistics or no DDM statistics if there are interface in the system that do not have SFP modules inserted.

Conditions:
- r5000, r10000, or r12000-series appliance
- VELOS partition
- Previous interfaces in the system that do not have an SFP module inserted.

Impact:
System does not report correct DDM statistics in 'show portgroups' command output.

Workaround:
Run the ‘show portgroups’ command for each interface that has an SFP module inserted, that is, ‘show portgroups portgroup 5’.

Fix:
Fixed the display issue in ‘show portgroups portgroup state ddm data’.

1582553-1 : The 'components component state' data is not displayed in ConfD.

Links to More Info: BT1582553

Component: F5OS-C

Symptoms:
- No data will be displayed as part of “show components component” in ConfD.
- In the absence of component platform information, GUI features default to r5xxx platform, leading to some functional issues for other platforms.

Conditions:
Intermittently occurs when initializing the state data.

Impact:
You cannot view the hardware information, which is updated under “show components component”.

GUI functional issues for other platform:
For r10xxx - Raid Configuration will not be visible.
For r4xxx/r2xxx - Port Groups may not function as expected. STP screens and Port Mappings will show up, which are not applicable to the platform and will be non-functional.

Workaround:
Log into the appliance as root and restart the platform-mgr docker container:

docker restart platform-mgr

Fix:
The functionalities disrupted on the GUI can be accessed via the CLI.

1582105-1 : Partition RESTCONF may return an incomplete response for f5-cluster:cluster/nodes/node

Component: F5OS-C

Symptoms:
When querying f5-cluster:cluster/nodes/node in a partition, it succeeds for 1000 calls, but then starts returning an incomplete response.

Conditions:
This only happens on chassis with at least one empty slot. Each time that cluster/nodes/node/<blade>/state/tenant-memory is requested on an empty slot, an internal queue will hold on to that request. When the queue is full, requests will stop working.

Impact:
After the symptom starts, cluster/nodes/node cannot be queried successfully until partition services are restarted.

Workaround:
Modify queries to avoid requesting tenant-memory on empty slots. For example, do not use the top-level cluster/nodes/node, but instead use cluster/nodes/node/blade-1.

Fix:
Fixed platform-stats-bridge to no longer query blades that are not present or ready.

1581589 : Lack of IPv4 management address causes OpenShift Ansible playbooks to fail

Links to More Info: BT1581589

Component: F5OS-C

Symptoms:
If there are no IPv4 addresses defined, ansible playbook executions will fail to look up a default route, causing the playbook to fail.

Conditions:
VELOS chassis with no IPv4 management addresses configured.

Impact:
This will fail the addition of new blades to the cluster, as well as a failure in the return merchandise authorization (RMA) situation for both blades and controllers.

Workaround:
The workaround is to add an IPv4 default route to both controllers from the bash shell.

nmcli conn modify team0 ipv4.gateway 192.6.3.254 ipv4.route-metric 32768
nmcli conn up team0

Fix:
Added a default route to allow the ansible playbooks to lookup the route and interface it requires.

1580489-1 : BE2 GCI interface training issue results in failure to process networking traffic

Links to More Info: BT1580489

Component: F5OS-C

Symptoms:
Some particular rSeries systems fail to process networking traffic due to the BE2 GCI interfaces not training properly, resulting in an FPGA datapath lockup.

One potential indication of this is the DMA agent detecting a DM Tx Action ring hang, which can be observed in velos.log / platform.log:

dma-agent[13]: priority="Alert" version=1.0 msgid=0x4201000000000130 msg="Health monitor detected DM Tx Action ring hung." ATSE=0 DM=0 OQS=3

Conditions:
RSeries r5000, r10000, or r12000-series appliance

This issue does not affect r2000 or r4000 series appliances.

Impact:
The system stops delivering traffic from front-panel ports to the host, although egress traffic may continue to work. If an LACP LAG is configured, ports will be unable to join the LAG.

Workaround:
None, and F5 continues tracking the BE2 issue via ID1596625.

Fix:
During system startup, FPGA manager now ensures that the BE2 GCI interfaces are brought up and trained properly.

1580349-1 : Loading backup file with partition ID 1 that is not named "default", throws an error★

Links to More Info: BT1580349

Component: F5OS-C

Symptoms:
Loading an F5OS-C system controller backup file with partition ID 1 that is not named "default" throws an error following a reset-to-default.

Conditions:
If the chassis admin deletes the partition named "default", and then creates a new partition, it will be assigned partition ID 1.

The reset-to-default operation re-creates the default partition with an ID of 1.

Impact:
Saved configuration cannot be restored after reset-to-default.

Workaround:
None

Fix:
Config-restore has been changed to allow restoring a saved configuration that contains a partition with ID 1 that is not named "default".

1580165-1 : Removing a failed patch ISO can remove base services imported from a different ISO★

Links to More Info: BT1580165

Component: F5OS-C

Symptoms:
Removing a failed patch ISO also removes the base services ISO imported by another ISO. Further upgrade will fail even though importing the patch version is successful. You may observe the below log.

appliance-1(config)# system image check-version iso-version 1.5.2-21056
response Compatibility verification succeeded.

Conditions:
-- Base services are already imported by another ISO.
-- Same version patch ISO import failed.
-- Delete the failed patch ISO.

Impact:
Upgrade to a new successful import of patch ISO of the same version will fail.

Workaround:
Rebooting the device will resolve the issue.

Fix:
While removing the failed patch ISO, added a check that if the base services are imported by another ISO, do not delete the base services ISO.

1579453-1 : SAN Validation Mismatch: Key/Cert virtual server No Key Configured

Links to More Info: BT1579453

Component: F5OS-C

Symptoms:
When TLS key/cert is set in confd, create-csr accepts invalid SAN values without generating a CSR or errors. Without a key/cert, confdcli correctly validates the CSR.

2: Run create-csr with various san values
appliance-1(config)# system aaa tls create-csr name namesan san ""
----------------------------------------------------------------^
syntax error: "" has a bad length/size. <======== EXPECTED

appliance-1(config)# system aaa tls create-csr name namesan san ''
appliance-1(config)# <===== should give error

appliance-1(config)# system aaa tls create-csr name namesan san "IP"
appliance-1(config)# <======= should give error

appliance-1(config)# system aaa tls create-csr name namesan san "DNS"
appliance-1(config)# <==== should give error

appliance-1(config)# system aaa tls create-csr name namesan san "f5best"
appliance-1(config)# <==== should give error

appliance-1(config)# system aaa tls create-csr name namesan san IP:1.1.1.1
response <====== EXPECTED

Conditions:
Invalid SAN values are accepted

Impact:
Confd accepting invalid SAN values

Workaround:
None

Fix:
Fixed in F5OS-A 1.8.0

1577049-1 : CVE-2024-1086 - Linux kernel vulnerability

Links to More Info: K000139430, BT1577049

1576545-2 : After upgrade, BIG-IP Next tenant os unable to export toda-otel (event logs) data to Central Manager★

Links to More Info: BT1576545

Component: F5OS-C

Symptoms:
After upgrade, the BIG-IP Next tenant is unable to export toda-otel (event logs) data to CM in VELOS

Conditions:
Upgrading BIG-IP Next tenant from 20.1 to 20.2 on a VELOS system.

Impact:
After upgrade, the BIG-IP Next tenant is unable to export toda-otel (event logs) data to CM

Workaround:
For VELOS Standalone
====================
After upgrade, if the f5-toda-otel-collector cannot connect to host change the tenant status from "DEPLOYED" TO "CONFIGURED" TO "DEPLOYED" to fix the issue. Please note that it will take 5 to 10 min for tenant status to change and it might impact the traffic.

For VELOS HA follow the following steps
=======================================
1. Setup CM on Mango build
2. Add 2 BIG-IP Next instances(Mango build) on the CM
3. Bring up HA on CM with the Enable Auto Failover option unchecked
4. Add a license to the HA instance.
5. Deploy a basic HTTP app in FAST mode with WAF policy attached (Enforcement mode - Blocking, Log Events - all)
6. Send the traffic and verify the WAF Dashboard under the Security section, should be able to see the Total Requests and Blocked response fields with non-zero values
7. Upgrade standby instance to latest nectarine build with the "auto-failover" button switched off.
8. We will observe the instances goes into an unhealthy state on CM.
9. Change the status of the standby instance from Deployed to Configure Mode and save it through partition GUI/CLI.
10. After confirming the status of the pods, change the state of the standby instance back to the Deployed state from the configured state. There should be no impact on the traffic flow during this step.
11. Now do the force failover and check the health status of instances, it will still show unhealthy as instances are in between upgrades.(one instance with Mango build (standby node) and other with Nectarine build(Active node))
12. Now Upgrade the standby instance to the latest nectarine build with the "auto-failover" button switched off.
13. HA should look healthy in this state and traffic should continue to flow.
14. Change the state of the standby instance from Deployed to Configure Mode and save it using partition GUI/CLI
15. After confirming the status of the pods for the instance on partition CLI, change the state of the standby instance back to the Deployed state from the configured state.
16. We will observe the Event logs on the WAF Dashboard under the security section on CM.
17. We can also observe the logs on the "f5-toda-otel-collector" pod showing no Export failures.
18. Upgrade the CM. Systems should be Healthy.

1576241 : Duplicate MAC on different tenants

Links to More Info: K000139293, BT1576241

Component: F5OS-C

Symptoms:
VELOS system controller and chassis partition software may incorrectly start allocating the same MAC addresses to different objects in chassis partitions. In the worst case, this can result in multiple tenants using the same MAC addresses on the same VLAN, resulting in traffic disruptions for those tenants.

This issue occurs when the following conditions are met:

You are running F5OS-C 1.6.x software on the F5 VELOS system controllers.
The system controllers restarted simultaneously, such as during an out-of-service upgrade or power outage.
The F5 VELOS system controllers then fail over.

Conditions:
After this occurs, the VELOS system controller loses track of which MAC addresses have been allocated to chassis partitions, setting up a situation where creating new tenants or chassis partitions may re-use MAC addresses already allocated to objects on the system.

Impact:
Traffic disruption on tenants due to duplicated MAC address.

Workaround:
None

Fix:
Once a system is affected, upgrading to a version or engineering hotfix (EHF) that contains the fix for ID1576241 does not resolve the issue; manual intervention is also required to fix the issue.

1575925 : Running 'show system aaa primary-key state status' while a key migration is in progress can cause key migration errors

Links to More Info: BT1575925

Component: F5OS-C

Symptoms:
If a key migration is in progress (initiated via the ConfD action 'system aaa primary-key set'), and while it is in progress the status of the key migration is checked ('show system aaa primary-key state status'), this can intermittently cause the key migration to fail. Under these conditions, future attempts to 'show' this area of state will also return 'application communication failure'.

Conditions:
1. A ConfD primary key migration is initiated on a VELOS Controller or Appliance system.
2. While the key migration is in progress, the status of the migration is checked.

Impact:
Key migration fails, leaving encrypted ConfD elements in a corrupted state. Furthermore, all operational data callbacks for the 'system aaa primary-key' schema tree will fail indefinitely with 'application communication error'.

Workaround:
To workaround this issue, reboot the affected controller(s) or appliance. After the reboot, the user may re-attempt the key migration.

Fix:
Fixed issue where checking status of key migration could cause the migration to fail.

1575585 : Unable to add blade to Openshift cluster if newly-installed blade is not member of active partition

Links to More Info: BT1575585

Component: F5OS-C

Symptoms:
After a blade is clean-installed (PXE, USB, etc), if the blade is not a member of an enabled/functioning partition, the system is unable to add it to the Openshift cluster successfully.

If an administrator attempts to log into the blade via SSH, it will prompt them that root's password is expired and needs to be changed:

[root@controller-1(VELOS) ~]# ssh blade-1
You are required to change your password immediately (root enforced)
Changing password for root.
(current) UNIX password:
Connection to blade-1 closed.
[root@controller-1(VELOS) ~]#

The "show cluster" command output will report that a blade is reachable ("able to ping"), but will not be able to connect to it ("able to SSH"):

                                                          ABLE ABLE
                                        IN READY TO TO PARTITION
INDEX NAME INSERTED CLUSTER CLUSTER PING SSH STATE LABEL
--------------------------------------------------------------------------------------------------
1 blade-1.chassis.local true false false true false Not In Cluster
2 blade-2.chassis.local true false false true false Not In Cluster
3 blade-3.chassis.local true false false true false Not In Cluster

Conditions:
-- Blade is not a member of a VELOS partition, or is a member of a disabled partition.
-- A clean install is performed on blade (i.e. PXE install); this will be the case during an RMA replacement.

Impact:
- Blade will not join Openshift cluster.

Workaround:
Either configure the blade to be a member of an enabled partition, or manually log into the blade as root and go through the "change password" process.

1574861-1 : Incomplete API payload and CLI failure for openconfig interfaces when one controller node is not ready

Links to More Info: BT1574861

Component: F5OS-C

Symptoms:
When one of the system controller nodes transitions to a "NotReady" state:
The OpenConfig Interfaces API (/openconfig-interfaces:interfaces) returns incomplete or "unfinished chunk" payloads.
CLI commands such as 'show interfaces' fail, displaying an "application communication failure" error.

Conditions:
The problem might occur when one of the system controllers is not available.

Impact:
API users may experience incomplete data responses. Users might be temporarily unable to retrieve interface data from the CLI.

Workaround:
Minimize scenarios where one controller is not available.

Fix:
Modified the callpoint registration to ensure reliable data retrieval even when one of the system controllers is unavailable.

1573493-1 : Qkview does not collect the files gid-map.txt, /etc/libnss-udr/passwd, or /etc/libnss-udr/group

Links to More Info: BT1573493

Component: F5OS-C

Symptoms:
When a QKView is collected, the files gid-map.txt, /etc/libnss-udr/passwd, and /etc/libnss-udr/group are not present in the QKView.

Conditions:
A qkview is collected.

Impact:
It may not be possible to troubleshoot certain issues related to authentication.

Workaround:
None

Fix:
The files gid-map.txt, /etc/libnss-udr/passwd, and /etc/libnss-udr/group have been added to QKView collection. Whenever a QKView is collected, these files are present.

1572929-2 : Changing remote authentication methods from RADIUS/TACACS to LDAP may break remote-gid functionality.

Links to More Info: BT1572929

Component: F5OS-C

Symptoms:
If RADIUS or TACACS are utilized for authentication, the user’s ‘passwd’ details will be saved in /etc/libnss-udr/passwd. However, if the user switches to LDAP authentication and disables the previous method, their entry may not be removed from /etc/libnss-udr/passwd.

If a user is using GID remapping (by configuring remote-gid), the authentication will fail, at least when logging into the CLI.

Conditions:
- Enable RADIUS authentication and log into the system as a remote RADIUS-defined user.
- Change the authentication method to LDAP and disable RADIUS authentication.
- Configure remote-gid functionality for an LDAP-defined user. This LDAP-defined user should have the same name as the RADIUS-defined user.
- Log into the system as that remote LDAP-defined user.

Impact:
The authentication will fail for the LDAP-defined user. An error message will appear such as: “No valid role group found in user groups: 9002 123 5340”.

Workaround:
Log into the system as a ‘root’ user and clear the information in /etc/libnss-udr/passwd.

Fix:
The remote-gid functionality will no longer be affected by changing authentication methods from RADIUS/TACACS to LDAP. LDAP users with valid credentials will be allowed in.

1572493-2 : LAG Trunk Configuration is Missing Inside of Tenant

Links to More Info: BT1572493

Component: F5OS-C

Symptoms:
When creating a LACP LAG or Static LAG, the lag and its members will show as up on the F5OS and switch side (Arista and Cisco). However, on the tenant, tmsh will show that neither the trunk nor trunk members are present:
root@(localhost)(cfg-sync Standalone)(Active)(/Common)(tmos)# list net trunk
root@(localhost)(cfg-sync Standalone)(Active)(/Common)(tmos)#

Conditions:
BIG-IP tenant on F5OS system

Impact:
The trunk information will not be visible in the tenant.

- On high-end rSeries appliances (r5000, r10000, and r12000-series systems) and VELOS tenants, traffic will still work.

- On low-end rSeries appliances (r2000 and r4000-series systems), traffic will not flow.

Workaround:
NA

1572489-1 : User accounts with username which includes only numeric values or special characters like "." or ".." or starts with '-' are inactive

Links to More Info: BT1572489

Component: F5OS-C

Symptoms:
User accounts created with username that include only numeric values are inactive or non-functional. Also, usernames starting with dash ‘-’, contain only “.”, “..”, or any invalid characters (other than letters, digits, underscores, dashes and $ (at the end)) creates non-functional user accounts.

Conditions:
User account with username containing only numerics or starts with dash '-' or username like "." or ".." are non-functional.

Impact:
Non-functional user accounts are created. User functionalities like set-password, change-password, or other would not be working as expected.

Workaround:
None

Fix:
User account creation with invalid username will not be possible. An error will be displayed for invalid usernames.

Following is an example:
appliance-1(config)# system aaa authentication users user 12345676578 ?
Possible completions:
  Error: "12345676578" is an invalid value.

There wont be 'config' option available to create/configure new user account until you provide valid usernames.

1572137-1 : Upload/Download API should work with '/api' and '/restconf'

Links to More Info: BT1572137

Component: F5OS-C

Symptoms:
Upload/Download is not working with '/api' endpoint.

Conditions:
Use '/api' endpoint to upload/download a file.

Impact:
Fails to Upload/Download a file.

Workaround:
None

Fix:
Fixed an issue occurring with the Upload/Download API.

1560533 : Inconsistent case values (upper and lower case) for different F5OS-C SNMP OIDs

Links to More Info: BT1560533

Component: F5OS-C

Symptoms:
AlertSource in SNMP alert contains text as Controller starting with uppercase C instead of lower case in core alert events.
Similarly, for core alert events generated in blade, comes with Blade instead of blade.

Conditions:
Process crash generating core file and SNMP alerts are enabled.

Impact:
Tools processing SNMP alerts might get affected if tooling is case-sensitive.

Workaround:
None

Fix:
Fixed alertSource text for SNMP core alert events to send lower case.
Tools modified to read alertSource of SNMP core alert events require to update as per the correction.

1559509 : Incorrect displayed state of blade internal data link

Links to More Info: BT1559509

Component: F5OS-C

Symptoms:
The "ifcfg" TMSTAT table on VELOS blades displays an incorrect state for a blade internal link between FPGAs. The "av.1" link is shown as DOWN regardless of its actual state. This link carries tenant traffic on VELOS blades and its operating state may be relevant when performing diagnostics.

Conditions:
The issue is seen on all VELOS blades running a F5OS version that does not have the fix.

Impact:
This issue may incorrectly indicate a breakage in a blade's datapath when there is actually none.

Workaround:
It is possible to view the correct link state with a lower-level debugging command.
From the Linux CLI of a VELOS blade, run the following command to get the current state of the blade's data links.

[root@blade-4 ~]# docker exec partition_fpga fpgatool -c "linkscan show"

Fix:
Corrected a configuration field in fpgamgr code that updates link status.

1558505 : After restarting the fpgamgr service, the last service-instance is not processed

Links to More Info: BT1558505

Component: F5OS-C

Symptoms:
Traffic outage. One service-instance on the slot is missing.

Conditions:
The fpgamgr service restarting without a full system reboot.

Impact:
Traffic outage.

Workaround:
Reboot the device.

1556173 : Poor management backplane link performance on system controller failover

Links to More Info: BT1556173

Component: F5OS-C

Symptoms:
The connectivity of the chassis management backplane may be disrupted for a minimum of 1-5 seconds, and in specific situations, for up to 20 seconds. During this time, tenant instances are unable to communicate with each other over the chassis management backplane.

Conditions:
Failover of the system controller has been observed. Rebooting the active system controller may aggravate the symptoms.

Impact:
Since tenant instances cannot communicate with one another during this period, if the link downtime exceeds 10 seconds, it will trigger a BIG-IP tenant's clusterd timeout. If that BIG-IP tenant is active in an HA pair, a failover will tigger such that the standby BIG-IP is now active.

Additionally, a sod out-of-band mgmt timeout will be triggered for that BIG-IP tenant even if the system controller's management interfaces are configured in a trunk. In some scenarios, this can trigger temporary split brain behavior between BIG-IP tenants in an HA pair.

This can cause unexpected HA failovers if the downtime is long enough and the tenants are multi-slot despite a TMM self-ip being configured in the HA mesh.

Workaround:
No workaround, only mitigations.

1. Do not reboot the active system controller. Perform a system controller failover, then reboot the controller that was previously active.

2. To mitigate issues during an unplanned controller failover, for example health check failures, increase each BIG-IP tenant's clusterd timeout and/or sod timeout up to 30 seconds to reduce erroneous sod and clusterd timeouts.

clusterd timeout can be modified in each BIG-IP via 'tmsh' modify sys db clusterd.peermembertimeout value <int>.

sod timeout can be modified in each BIG-IP via tmsh modify sys db failover.nettimeoutsec value <int>.

3. To mitigate issues during planned controller failovers in a maintenance window, it is possible to prevent unwanted inter BIG-IP tenant failovers or split brain behavior altogether. One strategy includes for each BIG-IP HA pair, set the BIG-IP device failover offline on the chassis where controller failovers are to be executed. While the BIG-IP device is offline, health checks like the sod and clusterd timeouts will not trigger a failover to offline BIG-IP devices. Once the maintenance window is over, each BIG-IP device should have failover set back online. Reference the following article to set a BIG-IP traffic-group's device offline. https://my.f5.com/manage/s/article/K15122.

Fix:
System controller failover incurs no chassis management backplane link downtime.

1555457 : System controller failover may take up to 60 seconds

Links to More Info: BT1555457

Component: F5OS-C

Symptoms:
During an HA failover of system controllers, it was observed that an system controller failover may take up to one minute.

Conditions:
System controllers failovers that are initiated by termination/restart of the vcc-confd container on the currently active system controller.

Impact:
A delay in system controller switchover negatively impacts system controller LACP (LACPD will only send PDUs from the active SC). This can cause problems with tenant HA, which sends HA to keep alive messages over the system controller control plane network.

Workaround:
Execute the system redundancy go-standby command to perform an HA switchover prior to rebooting the active system controller.

Fix:
During an system controller failover initiated by rebooting the active system controller, it takes 3 to 5 seconds for the ConfD Active role to change to the other system controller.

1552945-1 : Tenant images renamed with bracket are not supported★

Links to More Info: BT1552945

Component: F5OS-C

Symptoms:
Live upgrades with prior releases with tenants that use images with brackets in their name will fail when going to a version that restricts the tenant image name character set.

Conditions:
Tenants using image filename with brackets won't allow upgrades to releases that validate the image filename character set.

Impact:
The tenant will have to be recreated or upgrade to a version that does not have the validation.

Workaround:
Tenant has to be recreated with the original image that didn't contain brackets.

Fix:
Brackets were included in accepted character set for tenant image filename.

1552721 : Partition ipv6 managent address is not reachable after a partition switchover

Links to More Info: BT1552721

Component: F5OS-C

Symptoms:
Partition ipv6 management address is not reachable after a partition switchover.

Conditions:
Partition configured with an IPv6 management address.

Partition fails over (due to either go-standby or a fault) from one controller to the other, and then back.

Impact:
Partition is not reachable

Workaround:
Configure the partition system redundancy mode to "active-controller".

When the condition occurs, reboot the system controller that is running the standby partition, and then execute "system redundancy go-standby" on the active system controller.

Fix:
Partition management address is reachable after failover.

1552369 : F5OS-C: Partition volume cannot be removed if an active shell in that directory

Links to More Info: BT1552369

Component: F5OS-C

Symptoms:
The following error will be seen if there is an active shell(session) with the current directory /var/F5/partition{n}


+ lvremove -f /dev/partition_config/partition1
Logical volume partition_config/partition1 contains a filesystem in use.

Conditions:
There is an active shell(session) with the current directory /var/F5/partition{n}

Impact:
Partition volume fails to remove.

Workaround:
Don't ssh login to the system or don't change directory to /var/F5/partition* in ssh session.

Fix:
Any ssh session in the directory will be killed.

1550413 : System events visible in the CLI may not be visible in the GUI

Links to More Info: BT1550413

Component: F5OS-C

Symptoms:
Running "show system events" on the F5OS CLI typically reveals many events that are not visible in the GUI under System Settings > Alarms & Events.

The GUI filters the display of events according to their assigned severity. But since many events are not assigned a severity, such events will be hidden from view.

Conditions:
Events that are not assigned a severity are instead marked "NA". Such events are not visible in the GUI and can only be seen via the CLI or API.

Impact:
The omission of events displayed in the GUI can be misleading. Administrators using the GUI may not be aware of important events that have occurred on the platform.

Workaround:
All system events can be seen by running 'show system events' on the F5OS CLI or by retrieving them via the REST API.

Fix:
On fixed versions, a new option called 'All' has been added to the Severity drop-down selector in the GUI. This displays all events, including ones without a severity assigned.

1549753-1 : System telemetry exporter send queue and retry settings are causing memory issues

Links to More Info: BT1549753

Component: F5OS-C

Symptoms:
Memory issues are seen in system when telemetry exporter is not reachable for a long time.

Conditions:
When exporter is not reachable for a long time.

Impact:
System can go out of memory.

Workaround:
User can disable the send queue and retry setting using ConfD. For example:

appliance-1(config)# system telemetry exporters exporter <<exporter name>> config options send-queue-enabled false

appliance-1(config)# system telemetry exporters exporter <<exporter name>> config options state options retry-enabled false

Fix:
Send queue and retry settings are removed for telemetry exporters.

1549549 : Blades in the "none" partition may cause kubernetes services to fail.

Links to More Info: BT1549549

Component: F5OS-C

Symptoms:
If blades in a chassis a assigned to the none partition, it is possible that kubernetes services may get scheduled on that blade, and fail because they cannot find the correct container version for the service. This can cause the kubernetes cluster to fail, and specific services in the cluster to fail.

Conditions:
This can happen when there are one or more blades assigned to the none partition, and other blades and controllers in the chassis are rebooted. These reboots can cause the kubernetes services to get re-assigned to the blade in the none partition.

Impact:
The kubernetes cluster may show as failed, or the kubevirt or multus services may not operate correctly if their services land on one of the blades assigned to the none partition. This can cause existing tenants to fail, and new tenant deployments to fail.

Workaround:
The workaround is to move the blades in the none partition into a dummy partition that has a valid software version and is enabled. This will allow the blades to correctly start the kubernetes services assigned to those blades.

Fix:
Blades moved to the none partition are now marked as Non-Schedulable to that kubernetes will not try to schedule any services on them.

1549521-1 : VQF and VoQs fail to synchronize after system controller reboot

Links to More Info: BT1549521

Component: F5OS-C

Symptoms:
VQF and VoQs are unable to synchronize between blades after a system controller reboot.

Conditions:
System controller reboot.

Impact:
Loss of traffic between blades.

Workaround:
Reboot affected blades.

1538277-1 : Duplicate Service-Instance IDs for L2FwdSvc causes L2 entries to not be forwarded to all blades

Links to More Info: BT1538277

Component: F5OS-C

Symptoms:
Excessive DLFs in multi-bladed system causing traffic instability.

Conditions:
Two `L2FwdSvc` entries in the service-instance table have duplicate 'instance IDs'

Impact:
L2 entries are not forwarded to the affected blades causing excessive DLFs.

Workaround:
Reboot the higher number blade having the duplicate instance ID.

Fix:
Don't use the instance ID as the key into a map, using the slot number instead which is guranteed to be unique.

1538217-1 : View fpgamgr core file after partition shutdown

Links to More Info: BT1538217

Component: F5OS-C

Symptoms:
fpgamgr core file.

Conditions:
Partition shutdown.

Impact:
No impact other than the core file. Likely a timing problem as the portions of the fpgamgr shut down.

Workaround:
None

Fix:
This fpgamgr corefile on shutdown can be ignored.

1536413-1 : Allowed-ips allowed-ip <name> is not accepting the '-' in the names

Links to More Info: BT1536413

Component: F5OS-C

Symptoms:
Allowed IP profile got deleted while upgrading to 1.7.0 from lower versions. allowed-ip profile names with '-' got erased out. which got fixed in 1.8.0

Conditions:
While upgrading to 1.8.0 from lower versions other than 1.7.0, all allowed IP profile names should have atleast one alphanumeric and it should have not have any other special character other than ('-', '_' and '.')

Impact:
Allowed IP profile gets deleted if it is not matching the pattern.

Workaround:
Re-apply the allowed-IP profile configuration without eiphen '-' in the name

Fix:
Fixed the schema such that allowed IP profile name accepts the '-' in profile name.

1519869-1 : BIG-IP tenant reports blank interface

Links to More Info: BT1519869

Component: F5OS-C

Symptoms:
BIG-IP tenant reports a blank ("") interface member in the trunk when removing one or more interfaces from an aggregation.

Conditions:
BIG-IP tenant reports a blank ("") interface member in the trunk when removing one or more interfaces from an aggregation.

Impact:
BIG-IP tenant has an empty member in the trunk.

Workaround:
No workaround.

Fix:
BIG-IP tenant does not reports a blank ("") interface member in the trunk when removing one or more interfaces from an aggregation.

1505589 : Subject-Alternative-Name (SAN) feature now supports client-side SSL Validation

Links to More Info: K000139300, BT1505589

Component: F5OS-C

Symptoms:
Since no SAN was allowed to be inserted into the http-server’s self-signed certificate, client-side SSL validation was not supported.

This impacts Central Manager's VELOS/rSeries provider. The missing SAN field causes the certificate to be rejected.

Conditions:
Using the default self-signed certificate.

Impact:
Client-side SSL validation is not supported.

Workaround:
To add an SAN, you need to edit the /etc/pki/tls/openssl.cnf file and add it. However, this may not be effective for certain software that does not accurately read the configuration file.

Fix:
A new SAN field has been implemented, which is mandatory, and allows users to enter a value in the field. However, if the value “none” is used, the field can be omitted. Additionally, to allow entry of the SAN, a default tls certificate is created in /etc/auth-config/default/f5os.cert that has the SAN populated with the hostname and management-ip values. In the absence of a user-provided self-signed certificate, the http-server will automatically use the default certificate.

1505293 : Partition image removal message is truncated

Links to More Info: BT1505293

Component: F5OS-C

Symptoms:
If a partition is enabled and then disabled while running version A, and then upgraded to version "B", attempting to deport partition image "A" fails, the CLI throws truncated error messages.

Conditions:
The partition is upgraded with the state is disabled.

Impact:
Incomplete error messages for the failure reason. The error that is reported is:

"Error: Failed to remove software: 1.5.1-14085, error message: Standby removal failed for following reason: OS version".

Workaround:
None

1505221-1 : If accidentally import bad ISO images, it may not removed automatically

Links to More Info: BT1505221

Component: F5OS-C

Symptoms:
When you accidentally import ISO images from a faulty URL, they cannot be removed or replaced with the correct URL.

Conditions:
User accidentally imports faulty ISO images to the system.

Impact:
Deleting and importing system ISO images might have an impact.

Workaround:
Login to the command line with root user access and remove the image via 'rm' under '/var/import/staging', and import the correct ISO.

Fix:
Please refer to the workaround and further detail.

1498009 : Learned L2 entries in data-plane L2 forwarding table may disrupt some traffic flows between tenants

Links to More Info: BT1498009

Component: F5OS-C

Symptoms:
While a tenant transitions from active to standby, an egress packet in flight may trigger a L2 learn event in the FPGA data-plane. This can occur for tenants that transmit using a different MAC address while active, such as when MAC masquerading is enabled. If so, a dynamic L2 entry is created from the source MAC address of the egress packet. These dynamic entries also enable the service DAG without setting a service ID, which causes matching packets to be dropped in the VOQ system due to an invalid service DAG lookup result.

This can disrupt egress traffic for another tenant on the same device, attempting to transmit to the destination MAC address that was recently relinquished by the standby tenant. These drops increment the 'ic_voq_drops' counter in the tmctl vqf_global table.

These L2 entries will not be corrected by subsequent L2 learn events for the same MAC address from a different location. Thus, traffic disruption may persist until entries age out.

Conditions:
- MAC masquerade configured on the traffic-group of an HA pair of tenants.

- A failover from tenant A to tenant B.

- Another tenant running alongside tenant 'A' attempts to transmit to the MAC masquerade address that is now owned by tenant 'B'.

Impact:
Traffic disruption from one tenant to another in specific directions.

Workaround:
None

Fix:
L2 entries that are created from host generated L2 learn events, no longer enable the service DAG for matching packets.

1497657-1 : First SSH login after editing remote RADIUS or TACACS+ user privileges will still apply old privileges

Links to More Info: BT1497657

Component: F5OS-C

Symptoms:
The first SSH login after editing role-based privileges for a remote RADIUS or TACACS+ user will still give the user their prior privileges (or, if the user is newly created, login will be rejected with a message saying "This account is currently not available"). Subsequent logins will apply the updated user privileges.

Conditions:
1. RADIUS or TACACS+ Authentication is enabled.
2. A new user is created in one of the above auth systems, or an existing user’s role-based access is modified.
3. The affected user SSHs into F5OS for the first time after the change in step #2.

Impact:
First login to system after creation fails, or first login after modification of user privileges gives the user incorrect privileges.

Workaround:
None

Fix:
Fix issue where first SSH login after editing remote RADIUS or TACACS+ user privileges will still apply old privileges.

1497349 : Support for SSH-RSA host key algorithm for partitions added in non-fips mode

Links to More Info: BT1497349

Component: F5OS-C

Symptoms:
Unable to establish an SSH connection to the partition using the SSH-RSA host key algorithm in non-FIPS mode.

Conditions:
Attempting to connect to the partition from an SSH client using the SSH-RSA host key algorithm while in non-FIPS mode.

Impact:
SSH connections to the partition cannot be established using the SSH-RSA host key algorithm in non-FIPS mode.

Workaround:
None

Fix:
Support for the SSH-RSA host key algorithm has been added in non-FIPS mode.

1496977-2 : Remote GID mappings to F5OS roles are disconnected for TACACS+/RADIUS authentication methods.

Links to More Info: BT1496977

Component: F5OS-C

Symptoms:
Remote GID mappings (on a TACACS+ or RADIUS server) to F5OS GIDs/roles are not working correctly. When attempting to configure a remote mapping, it results in the access rejection with a message similar to below:

[root@system ~]# ssh radius_or_tacacs_user@<F5OS system mgmt IP>
Password:
Last login: <date> from <source IP>
No valid role group found in user groups: '9000'
Connection to <mgmt IP> closed.

Conditions:
A remote GID mapping is configured for a role in F5OS and the authentication method used for remote users is RADIUS or TACACS+.

Impact:
Remote users cannot log in to the system.

Workaround:
Configure remote user's GIDs in a way that they correspond to the GIDs in F5OS for the desired role(s). Then, remove any remote GID mappings in the F5OS configuration.

Fix:
Fixed remote GID mapping to F5OS roles for TACACS+/RADIUS authentication methods.

1496893 : Third etcd instance can get into an error state on controller upgrade from 1.5.1 to 1.6.1

Links to More Info: BT1496893

Component: F5OS-C

Symptoms:
The internal datastore of third Openshift etcd process has become out of sync with the etcd processes on the other two controllers.

Conditions:
A split brain situation occurred in the lower level database of the third etcd instance on each controller and is unable to recover.

Impact:
The user may notice inconsistencies with the display of tenants due to this condition.

1496837-2 : User-manager's ConfD socket getting closed.

Links to More Info: BT1496837

Component: F5OS-C

Symptoms:
After repeating the change of network type and device reboot, the device goes into a state where the user-manager is not interacting with ConfD.

Conditions:
- Change remote GID role and check '/etc/gid-map.txt' file if the value is reflected.
- Switch network type and reboot the device.

Repeat the above process until '/etc/gid-map.txt' file is not been updated correctly.

Impact:
Any ConfD configuration change that goes through user-manager fails. This includes any of the user’s password changes, or remote GID changes.

Workaround:
Rebooting the system will get the correct GID value from the ConfD and update the '/etc/gid-map.txt' file.

Fix:
The user-manager has no reason to use NSS to lookup any PW/group info, as it deals exclusively with the local user database.

Additionally, there is a ZMQ service that belongs in authentication-mgr (which understands remote authentication) that is in the user-manager container. It forces user-manager to use an ‘/etc/resolv.conf’ that can reference remote sources.

If the user-manager trips over a lookup that goes to LDAP (usually a local-db miss), it can be very slow and time out. The ConfD->user-manager channel is sensitive of slow responses, and shuts down subscriber/callpoint handler/daemon that takes over 15 to 30 seconds to respond. When this happens, the user-manager is going to see an EOF on its ConfD sockets.

This fix forces the user-manager to only lookup on local databases.

1496397-2 : Allowing entry of a Subject-Alternative-Name (SAN) for certificate and CSR creation

Links to More Info: BT1496397

Component: F5OS-C

Symptoms:
There is no method available for inputting the SAN field during the creation of certificates or CSR.

Conditions:
While creating a CSR through system aaa tls create-csr in ConfD.

Impact:
The option to include the SAN field in certificates and/or certificate request is not available.

Workaround:
To add an SAN, you need to edit the /etc/pki/tls/openssl.cnf file and add it. However, this may not be effective for certain software that does not accurately read the configuration file.

Fix:
A new SAN field has been implemented, which is mandatory, and allows users to enter a value in the field. However, if the value “none” is used, the field can be omitted. Additionally, to allow entry of the SAN, a default tls certificate is created in /etc/auth-config/default/f5os.cert that has the SAN populated with the hostname and management-ip values. In the absence of a user-provided self-signed certificate, the http-server will automatically use the default certificate.

As this is a new feature, back-porting to older versions has not been implemented and would be difficult and complex.

1494945-2 : ConfD Application Error when tenant interface stats are not available

Links to More Info: BT1494945

Component: F5OS-C

Symptoms:
When attempting to get tenant interface stats, the system displays "Error: application error".

Conditions:
The creation or modification of tenants may result in inaccurate handling of historical data by the tenant interface-stats logic. This could lead to the display of an “Error: application error” message when queried.

For example:
appliance-1# tenants tenant cbip-tenant-b state interface-stats down-sample-to 10 average 10s-avg
Error: application error

Impact:
Confd reports the error on the command line and logs the error in platform logs.

2024-01-24T20:12:37.123437567Z: [Error]: confd: msg="Action Point reply error" error="confd error: 'Unknown error', last='Invalid confd_vtype value: 0', errno=5"

Workaround:
None

Fix:
The problem has been resolved in more recent versions of F5OS-A. To resolve it, upgrade to a more recent version of F5OS-A. It will resolve once all interfaces are enabled.

1494809-1 : Allowing user to configure HostKeyAlgorithms parameters

Component: F5OS-C

Symptoms:
A new config CLI (system security services service sshd config host-key-algorithm) is implemented to allow HostKeyAlgorithms configuration.

Conditions:
In non FIPS mode, to enable or disable ssh-rsa HostKeyAlgorithm, this newly implemented CLI can be used.

Impact:
HostKeyAlgorithm usage was not configurable.

Workaround:
None

Fix:
This is a new CLI that can be used to enable or disable ssh-rsa HostKeyAlgorithm

1492621-4 : Config-restore fails when backup file has expiry-status field for admin or root user

Links to More Info: BT1492621

Component: F5OS-C

Symptoms:
For a root or admin user, if the value for Expiry-status in the backup file is not set to enabled, then config-restore fails.

Conditions:
During backup, if the "Expiry-status" value for admin or root user is not set to enabled, then restore fails with the backup.

Impact:
Database config-restore fails.

Workaround:
For admin and root user, comment expiry-status, expiry-date in the backup file and try to restore.

Fix:
Added NACM rules in ConfD for successful config-restore.

1492401-1 : User with operator role is not having read-access to all pages

Links to More Info: BT1492401

Component: F5OS-C

Symptoms:
- User experiences unauthorized error when trying to access "Tenant Images", "Software Management", "File
Utilities", "Configuration Backup", and "System Report"

- User sees no items when trying to access "File Utilities", "Configuration Backup", and "System Report" pages

Conditions:
User has operator role.

Impact:
User is not able to view certain pages.

1490753-2 : A linkUp and linkDown traps are sent when an up interface is disabled, and vice versa

Links to More Info: BT1490753

Component: F5OS-C

Symptoms:
When F5OS system is configured with SNMP Targets for managing the Trap notifications, linkUp and linkDown traps will be sent when interface state is toggled.

Conditions:
Always two traps (linkUp and linkDown) will be sent even when the interface state is toggled from UP to DOWN or DOWN to UP.

Impact:
No functional impact, but when two traps are sent, the interface state over SNMP can be misleading.

Workaround:
None

Fix:
The appropriate trap, that is, linkDown trap when F5OS interface state is down and linkUp trap when F5OS interface state is up, will be sent.

1488225 : Partition dagd cores during system startup

Links to More Info: BT1488225

Component: F5OS-C

Symptoms:
Occasionally, the partition dagd component triggers an assert and cores due to loss of connectivity with the internal system database. The partition dagd component will automatically restart.

Conditions:
The system database experiences a loss of connectivity during the startup of the partition.

Impact:
No functional impact.

Workaround:
None

1486697-2 : Configuring Expiry-status of root and admin users should not be allowed

Links to More Info: BT1486697

Component: F5OS-C

Symptoms:
Expiry-status of root and admin users are allowed to be configured and there is a chance of locking out these users.

Conditions:
If Expiry-status of any root or admin user is marked as Locked, that root or admin user cannot log in to the system.

Impact:
There is a chance that default users, such as root and admin, become locked out.

Workaround:
None

Fix:
You cannot edit the ‘Expiry-status’ field in webUI for admin and root users. Thus, it cannot be configured. The 'Expiry-status' field for root and admin users will now always display the default value as 'Enabled'.

1474833 : Debug output is missing from qkview

Links to More Info: BT1474833

Component: F5OS-C

Symptoms:
NSE debug registers missing from qkview output.

Conditions:
-- VELOS system
-- A qkview is taken

Impact:
Qkview file is missing some desired component output.

Workaround:
Without this fix, manually read desired debug registers with existing tools under the guidance of F5 support.

Fix:
With the current fix in place, all NSE debug registers are included in the standard QKView output.

1472917-1 : LDAP authenticated admins logging in via the serial console may have trouble disabing appliance mode during system instability

Links to More Info: BT1472917

Component: F5OS-C

Symptoms:
If ConfD is not running, F5OS offers an emergency option to disable appliance mode when an administrator logs in successfully via the serial console.

Conditions:
The admin role has been configured with a remote-gid that is not 9000 and the admin successfully authenticates via LDAP on the serial console while ConfD is not running.

Impact:
Remotely-authenticated admin users cannot disable appliance mode if ConfD is offline.

Workaround:
None

Fix:
Remotely-authenticated admin users can disable appliance mode if ConfD is offline.

1472373 : Failure of BX110 10G Links to recover after going DOWN

Links to More Info: BT1472373

Component: F5OS-C

Symptoms:
If the 10G link on the BX110 experiences a disruption, such as a cable pull or peer device shutdown, it may occasionally fail to re-establish connectivity even after the issue is resolved.

Conditions:
The 10G link on the BX110 experiences a disruption, such as a cable pull or shutdown on the peer device, leading to a DOWN state.

Impact:
Loss of connectivity.

Workaround:
None

Fix:
Regularly reset the 'DOWN' link to clear the failure state and enable the establishment of the connection.

1469385-2 : GUI freezes during LDAP user authentication if no remote GID mapped locally.

Links to More Info: BT1469385

Component: F5OS-C

Symptoms:
The LDAP remote user authentication freezes for a long time (more than a minute).

Conditions:
When trying to authenticate a remote LDAP user through the GUI without mapping any of the remote user GIDs to the F5OS local roles.

Impact:
Authentication freezes for a long period before rejecting the user.

Workaround:
One of the remote GIDs should be mapped to the local F5OS roles.

Fix:
Map the remote GID(s) to the F5OS role(s) to authenticate remote LDAP users successfully.

1469333-1 : VELOS management LAG may bridge traffic between management interfaces during LACP negotiation

Links to More Info: BT1469333

Component: F5OS-C

Symptoms:
When the management interfaces of VELOS system controllers are configured in a LACP LAG, the VELOS system may incorrectly forward some ethernet frames ingressing one management interface out the other management interface.

This behavior occurs during the period between when an interface links up and when the system completes LACP negotiation and adds the interface to the LAG.

This can result in management switches incorrectly learning non-VELOS MAC addresses as being present on the VELOS management LAG interface.

Conditions:
- VELOS system
- Management interfaces configured in LACP LAG

Impact:
VELOS management interfaces incorrectly forward non-VELOS frames from one management interface out the other, causing upstream switches to learn non-VELOS MAC addresses as being present on the VELOS management LAG interface.

Workaround:
Configure the upstream switch to be an LACP lag first, then configure the VELOS system MGMT interfaces to use an LACP lag.

1466397 : LDAP authentication is consuming several minutes to authenticate via GUI and SSH.

Links to More Info: BT1466397

Component: F5OS-C

Symptoms:
LDAP authentication is working fine. However, authentication takes several minutes, which lacks a user-friendly experience.

Conditions:
- Configure LDAP server-group.
- Configure LDAP_ALL as an authentication-method.
- Log in using LDAP user via GUI or SSH.

Impact:
The user is forced to wait for several minutes to get the result of LDAP authentication.

Workaround:
None

Fix:
Removed unnecessary GID lookup to speed up LDAP authentication.

1462329 : CC takes time to come up after reboot is triggered in active CC.

Links to More Info: BT1462329

Component: F5OS-C

Symptoms:
Containers take time to come up after reboot when active CC is rebooted.

Conditions:
Reboot should be triggered in active CC.

Impact:
Current standby CC takes time to come up after reboot.

Workaround:
None

1461289 : On a rSeries appliance, config-backup proceed is broken

Links to More Info: BT1461289

Component: F5OS-C

Symptoms:
On a rSeries appliance, system database config-backup 'proceed' is broken. It is about overwriting an existing backup file, but it prompts you to proceed even if a file does not exist.

Conditions:
System database config-backup always prompts for the user to proceed even if a file does not exist.

Impact:
No functional impact. When you provide input 'yes', the backup file will be generated.

Workaround:
When prompted to 'proceed', you must respond with 'yes'.

Fix:
The system database config-backup prompts the user with ‘proceed’ option only when the file exists and the user is not provided ‘proceed yes’ in the input CLI command.

1455913-4 : Tcpdump on F5OS does not honor the -c flag

Links to More Info: BT1455913

Component: F5OS-C

Symptoms:
When using Tcpdump on F5OS with the -c flag, Tcpdump will not stop after receiving the given number of packets.

Conditions:
A Tcpdump session is started with the -c or --count flag.

Impact:
The Tcpdump session will not terminate after receiving the requested number of packets and will continue until manually terminated.

Workaround:
N/A

Fix:
Tcpdump now honors the -c flag and will terminate after receiving the given number of packets.

1455769 : Slow execution of ansible-playbooks on cluster reinstall caused timeouts and retries for many hours.

Links to More Info: BT1455769

Component: F5OS-C

Symptoms:
A openshift cluster rebuild kept failing and retrying do to timeouts while running the ansible-playbooks to rebuild the cluster. This caused the cluster rebuild to fail for 8 plus hours, during which time not tenants could be started.

Conditions:
An openshift cluster rebuild was issued after upgrade the system.

Unable to reproduce this issue locally.

Impact:
While the ansible-playbook runs were timing out, it was not possible to launch tenants on the chassis.

Workaround:
The playbooks stopped timing out after 8 plus hours, no workaround is known.

Fix:
1.) Enhanced code that generates and corrects the /etc/hosts file to make sure all the necessary entries are always present and correct.
2.) Enhanced the code the handles the SSH connection caching to make sure it always cleared during ansible-playbook runs, so it won't get affected by a stale connection
3.) Playbook timeouts will be increased after a timeout failure up to 3x to try and allow the system to complete it's work even if something is slowing down the playbook runs.

1455725-1 : Partition go-standby command sometimes fails to change active instance

Links to More Info: BT1455725

Component: F5OS-C

Symptoms:
The partition "go-standby" command is sometimes too slow to finish taking over. When this happens usually the system briefly goes active/active and then resolves to the preferred node.

Conditions:
Attempting to force the partition active instance location using the go-standby. Normal HA framework initiated failovers work properly.

Impact:
When the confd instances are failing back & forth, the control plane daemons will be disconnected.

Workaround:
Allow the HA framework to manage instance locations and don't use "go-standby" to attempt to force instance location. If necessary, the "mode" can be set temporarily to "prefer" the desired location.

Fix:
Performance of yield/takeover operation has been improved.

1436153-2 : F5OS upgrades fail when SNMP configuration contains special characters.

Links to More Info: BT1436153

Component: F5OS-C

Symptoms:
As part of some security fixes, added a special character restriction in SNMP configuration in F5OS-A 1.5.1. This resulted in an upgrade failure to 1.5.1. If an upgrade to 1.5.1 is successful, the SNMP configuration will get deleted implicitly.

Conditions:
Upgrade to 1.5.1 fails when the SNMP configuration contains any special characters. The restricted special characters are: /*!<>^,/

Impact:
If the user encounters this issue, the system will go to an inaccessible state and require a forced downgrade.

Workaround:
Delete the SNMP configuration (community, target, or user) containing special characters before performing an upgrade to 1.5.1.

Fix:
The special characters in the SNMP configuration do not inject any security issues and can have special characters. Hence, the special characters restriction is removed in F5OS-A 1.5.2 and F5OS-A 1.8.0.

1429741-3 : Appliance management plane egress traffic from F5OS-A host going via BIG-IP Next tenant management interface instead of host management when both are in same subnet

Links to More Info: BT1429741

Component: F5OS-C

Symptoms:
When BIG-IP Next tenant is installed, a default route rule is added on host. If tenant management and host management IPs are on same subnet, then two similar rules are created with destination as same subnet.

The tenant route rule is created with higher priority (metric 0) resulting any management egress traffic destination belonging to same subnet is going through tenant management interface instead of host management interface.

Conditions:
BIG-IP Next tenant is deployed on appliance.

Impact:
End users receiving traffic from appliance, will observe sender IP as tenant management interface instead of host management interface.
    Note:
        a. This issue will be observed only when host management & tenant management subnet is same and also destination to which data is sent is on same subnet.
        b. This impacts management plane traffic within the appliance's management subnets.

Workaround:
N/A

Fix:
N/A

1429721-2 : SCP as non-root user does not report errors correctly for bad/non-existent files.

Links to More Info: BT1429721

Component: F5OS-C

Symptoms:
Using SCP to retrieve files from F5OS as "admin" or other non-root users should report a proper error when attempting to access an invalid directory or non-existent file.

Instead, the SCP command does nothing, reports no error, and exits with an on-zero exit status.

Conditions:
Attempt to read a non-existent/inaccessible file via SCP.

Impact:
The user is not informed about the failed SCP operation and the reason for the failure.

Fix:
SCP server software now reports errors the invalid/inaccessible filenames.

1429713 : VELOS ATSE v7.10.4.12 firmware

Links to More Info: BT1429713

Component: F5OS-C

Symptoms:
VELOS ATSE v7.10.4.12 firmware

Conditions:
VELOS CX410 blades.

Impact:
Not applicable.

Workaround:
Not applicable.

Fix:
Fixes RRDAG issues. See ID1347997 or ID1785385 for more information.

1411137-2 : Audit log entries are missing when creating or deleting objects via UI or API

Links to More Info: BT1411137

Component: F5OS-C

Symptoms:
When creating or deleting multiple remote-server related objects via UI or API, multiple restart happens causing log message drop.

Conditions:
While creating or deleting multiple objects related to remote-server, rsyslog restart everytime to apply new configuration. Due to the restart, some log messages are dropped.

Impact:
Log messages are dropped due to multiple restarts of the rsyslog.

Workaround:
None

1410729 : VELOS backplane packet priority issue

Links to More Info: BT1410729

Component: F5OS-C

Symptoms:
A packet priority issue was discovered during internal testing.

Conditions:
No special conditions.

Impact:
No impact has been reported.

Workaround:
Fixed in VQF bitfiles v8.10.1.3 and newer.

Fix:
Updated priority of backplane traffic in VQF bitfile.

1410609 : Watchdog resets during PSU management may cause AOM/LOP to remain in bootloader mode

Links to More Info: BT1410609

Component: F5OS-C

Symptoms:
The system controller AOM/LOP may encounter a watchdog reset while doing PSU management. If multiple watchdog resets occur in succession, then the AOM/LOP may remain in bootloader mode and be unavailable.

When this occurs, the PEL log will indicate a LOP watchdog reset in the LopPsuManagement task, for example:

07/17/2024 06:27:14 | 36644 | AOM | 128 | Network Access | 5 | LopPsuManagement task 100% of watchdog period, resettin
07/17/2024 06:27:20 | 36645 | AOM | 190 | Network Access | 5 | watchdog reset, successive watchdog resets: 9

When there have been 10 successive watchdog resets then AOM/LOP remains in bootloader mode and needs to be reprogrammed.

Conditions:
- A VELOS system controller, in either the CX-410 or CX-1610 chassis.

Impact:
If multiple watchdog resets occur in succession, then the system controller's AOM/LOP may remain in bootloader mode and be unavailable.

Workaround:
At the system controller host prompt, verify that the AOM/LOP is in bootloader mode.

[root@controller-1 ~]# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID f5f5:df11
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

An AOM/LOP in bootloader mode will enumerate as USB device f5f5:df11 as shown above.

To reprogram the AOM/LOP firmware, first locate the latest firmware version provided by F5OS-C. For example:

[root@controller-1 ~]# ls $(docker container inspect platform-fwu -f '{{ range.Mounts}}{{.Source}}{{printf "＼n"}} {{end}}' | grep config_fw-volume) | grep ^lop-chassis-controller
lop-chassis-controller-v2.01.1238.0.1.dfu

Then reprogram the AOM/LOP using the firmware version located above, for example:

[root@controller-1 ~]# docker exec -it platform-fwu dfu-util -D /usr/lib/firmware/lop-chassis-controller-v2.01.1238.0.1.dfu

It will take approximately 2 minutes to reprogram the AOM/LOP firmware image. The AOM/LOP will enumerate as USB device f5f5:3000 after reprogramming, as shown:

[root@controller-1 ~]# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID f5f5:3000
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Fix:
Fixed in lop-chassis-controller-application-2.01.1276.0.1 and later.

1410229 : Display a GUI warning to let user know tenants might be affected/reboot★

Links to More Info: BT1410229

Component: F5OS-C

Symptoms:
User is not informed about tenants getting temporarily affected when an F5OS upgrade operation is performed.

Conditions:
Upgrading a F5OS system

Impact:
User can be unaware that tenants will be affected/rebooted by performing a system upgrade.

Workaround:
None

Fix:
Added a warning message which will be displayed in the confirmation popup before triggering system upgrade. This new warning message conveys upgrade operation may lead to temporary downgrade of tenants.

1410225 : Enhanced the upgrade prompt for better understanding the impacts of upgrade on tenants

Links to More Info: BT1410225

Component: F5OS-C

Symptoms:
Older upgrade prompt didn't include information about impacts of upgrade on tenants.

Conditions:
An upgrade is triggered.

Impact:
Upgrading does not warn you that tenants will be started and traffic will be disrupted.

Workaround:
None

Fix:
Fixed in 1.8.0

1408477-1 : When more than one PCIe AER error has occurred, diag-agent reports this as a "RAS AER 'unknown' error" instead of the individual AER errors.

Links to More Info: BT1408477

Component: F5OS-C

Symptoms:
When more than one PCIe AER errors are occurred simultaneously, diagnostics will not report the events.

Conditions:
This occurs when more than one PCIe AER errors occur simultaneously.

Impact:
You are unable to see the individual PCIe errors.

Workaround:
None

Fix:
Updated diagnostics to consider and report more than one PCIe AER errors when occurred simultaneously.

1408369-1 : The "MAC exhaustion" error message during tenant creation may be caused by configuration processed during startup initialization★

Links to More Info: BT1408369

Component: F5OS-C

Symptoms:
If tenant configuration is processed before startup initialization has completed then there may be a MAC exhaustion error issued.

Conditions:
If startup initialization has not completed then the available MAC addresses are not known and a MAC exhaustion error is issued.

Impact:
The tenant configuration fails.

Workaround:
Reconfigure the tenant once startup has completed.

Fix:
The configuration code is now gated to not run until startup initialization has completed.

1403817 : SNMP IF-MIB misreport the status and speed of LACP LAGs

Links to More Info: BT1403817

Component: F5OS-C

Symptoms:
SNMP polling on IF-MIB provides incorrect status and speed of LACP Lag interfaces.

Conditions:
The issue is seen only on SNMP interface. The correct status and speed display on CLI or GUI.

Impact:
The user will see inappropriate status and speed details when polled for IF-MIB details on SNMP for LACP LAG interfaces.

Workaround:
None

Fix:
Fixed the issue to display the correct values of LACP LAG interfaces in IF-MIB SNMP polling.

1401965 : Copying BIG-IP ISO to /var/import/staging/, leaves ISO loopback mounted

Links to More Info: BT1401965

Component: F5OS-C

Symptoms:
An error occurs:
ERROR: sw-mgmt: priority=error msgid=0x3501000000000154 msg=Unexpected error processing "import /var/export/chassis/import/iso/<image>.iso": [Errno 30] Read-only file system: 'ace-1.1.7-0.0.3.i686.rpm'

Conditions:
Copying a BIG-IP ISO to /var/import/staging/ (rather than /var/F5/system/IMAGES or /var/F5/partition<num>/images)

Impact:
An error occurs and the ISO loopback remains mounted

Workaround:
None

Fix:
Fixed in F5OS-A/C 1.8.0

1401621-1 : Modifying a remote server with multiple selectors from the web UI removes the AUTHPRIV configuration.

Links to More Info: BT1401621

Component: F5OS-C

Symptoms:
The AUTHPRIV option is not available on the webUI. Modifying a remote log server, which has multiple servers, from the webUI removes the AUTHPRIV configuration

Conditions:
Modifying a remote server with multiple selectors from the webUI.

Impact:
The AUTHPRIV selector has been removed from the configuration.

Workaround:
To modify the configuration of a remote server with more than one selector, use the CLI.

Fix:
Added AUTHPRIV option to the webUI. Modifying the configuration of a remote server with more than one selector from the web UI will not remove AUTHPRIV from the configuration

1400557-1 : Incorrest slot info may cause blade backplane link errors

Links to More Info: BT1400557

Component: F5OS-C

Symptoms:
In VELOS v1.7.1, the system controller relies on the slot width of each blade in order to configure backplane port, as it is required for the blade to occupy the system controller blade slot. If the slot width is one, the system controller configures for Bx110 and if it is two, the configures Bx520. If the ConfD slot info is not available, the resulting backplane port configuration may be no suitable for the blade occupying the given slot.

Conditions:
A Bx110 or Bx520 blade is present in the chassis.

Impact:
Blade will fail to send /receive traffic over the backplane.

Workaround:
This defect is addressed in 1.8.0 by changing chassis manager, so that it updates confD with slot info provided by platofmr-ha before processing other HA info that might result in an error which skips the processing of the slot info.

Fix:
Blade backplane port link errors no longer observed when suitable slot info is present in system controller ConfD.

1400221-2 : OpenTelemetry exporters may not produce data upon first tenant being added to system

Links to More Info: BT1400221

Component: F5OS-C

Symptoms:
Telemetry streaming stops when the first tenant is configured.

Conditions:
When OpenTelemetry exporters are configured before the first tenant is configured within F5OS, this can lead to a condition where the exporters stop streaming metrics and logs.

Impact:
OpenTelemetry exporters stop producing metrics and logs.

Workaround:
The work-around is to disable and re-enable all exporters from the ConfD CLI.

system telemetry exporters exporter <name> config disabled

system telemetry exporters exporter <name> config enabled

Fix:
N/A

1400125 : Non-patch version of orchestration may start on controller after RMA replacement or rolling upgrade.

Links to More Info: BT1400125

Component: F5OS-C

Symptoms:
In a patch release after an RMA or rolling upgrade, the orchestration-manager version that is started may be from the base build of the patch release, rather than the version from the patch release, if orchestration-manager was updated in the patch release.

Conditions:
An RMA of a system running a patch release, or a rolling upgrade to a patch release.

Impact:
Base version of orchestration-manager may run on either controller instead of the patch version, until the controller is rebooted. This means that the patch release will not take affect until the controller is restarted.

Workaround:
Reboot the affected controller.

Fix:
Orchestration manager code was updated to wait for the patch registry to be created and populated before launching orchestration-manager.

1399929 : F5OS permits non-existent ethernet interfaces to be configured

Links to More Info: BT1399929

Component: F5OS-C

Symptoms:
F5OS allows you to manually type in non-existent interfaces of type "ethernetCsmacd" when adding an interface component.
The system later prohibits you from deleting this non-existent interface while the type is ethernetCsmacd.

Conditions:
User-triggered command for non-exposed interface type.

Impact:
The configuration contains a non-existent ethernet interface with no actual activity.

Attempting to delete the interface from the Partition CLI will result in the following error :

Partition-1(config)# no interfaces interface 1/1.1

Warning: Some elements could not be removed due to NACM rules prohibiting access.

From the Partition GUI The Network-> interfaces Page will show blank

"There are no items to show in this view."

Workaround:
Delete the non-existent interface.

Depending on the type of the created interface , there are 2 ways to delete it .

In case the interface is not a valid interface for the blade such as 25.0 or even a fake name like "example" you can delete it using "no interfaces interface 25.0"

If the interface is for a real interface, for example you have created interface 1/1.1 while the portgroup is in 100G mode (only 1.0 is valid) you will need the following procedure to delete it :

From from the Controller running the Active Partition

Login to controller that is the active one for the needed partition (can be seen via "show partitions" command on the controller). Also from "show partitions" on the controller check the partition ID.
From controller bash: docker exec -it partition2_manager bash (assuming the partition ID is 2)

From inside the partition run the following:
confd_cmd -c "mdel /interfaces/interface{1/1.1}"

Fix:
With this fix, F5OS will reject the creation of ethernetCsmacd.

1399757 : SNMP ifTable data missing for some interfaces when ports unbundled

Links to More Info: BT1399757

Component: F5OS-C

Symptoms:
SNMP interface data is not returned for all interfaces on the system when the device is configured with unbundled interfaces (4x10Gb or 4x25Gb modes).

Conditions:
Configure device with unbundled interfaces.

default-1(config)# portgroups portgroup 1/2 config mode MODE_4x25GB
default-1(config-portgroup-1/2)# commit
The following warnings were generated:
  'portgroups portgroup': VLAN, LAG, FDB, L2 protocols configuration is lost for the interfaces corresponding to the changed portgroups. Blade(s) 1 will reboot.
Proceed? [yes,no] yes
Commit complete.
default-1(config-portgroup-1/2)# top
default-1(config)# portgroups portgroup 2/2 config mode MODE_4x25GB
default-1(config-portgroup-2/2)# commit
The following warnings were generated:
  'portgroups portgroup': VLAN, LAG, FDB, L2 protocols configuration is lost for the interfaces corresponding to the changed portgroups. Blade(s) 2 will reboot.
Proceed? [yes,no] yes
Commit complete.

Check the SNMP output. it will not list all the interfaces.

Impact:
SNMP will not list all the unbundled (subports) interfaces.

Workaround:
None

Fix:
Fixed an issue with SNMP not listing all unbundled interfaces.

1397145-3 : Unable to add blade to Openshift cluster if VELOS partition root password is expired or locked

Links to More Info: BT1397145

Component: F5OS-C

Symptoms:
If a VELOS partition root password is expired or locked, the system may be unable to add the blade to the Openshift cluster (or manage the cluster).

The "show cluster" command output will report that a blade is reachable ("able to ping"), but will not be able to connect to it ("able to SSH"):

                                                          ABLE ABLE
                                        IN READY TO TO PARTITION
INDEX NAME INSERTED CLUSTER CLUSTER PING SSH STATE LABEL
--------------------------------------------------------------------------------------------------
1 blade-1.chassis.local true false false true false Not In Cluster
2 blade-2.chassis.local true false false true false Not In Cluster
3 blade-3.chassis.local true false false true false Not In Cluster

Conditions:
-- VELOS partition
-- root account in partition is expired or locked

Impact:
- Blade will not join Openshift cluster.
- Unable to deploy Tenants to blade.

Workaround:
Re-enable the root user account for the partition:

system aaa authentication users user root config expiry-status enabled

1394993 : Upon configuration changes, the l2-agent container restarts with a core.

Links to More Info: BT1394993

Component: F5OS-C

Symptoms:
On systems running F5OS-A or F5OS-C, wen the owner field of the fdb entry is updated by the system, for L2_LISTENER entries, l2_agent crashes.

Conditions:
Configuration changes triggered by system for L2_LISTENER fdb entries. Note that this field is not used by STATIC fdb entries, but the problem can be reproduced easily with STATIC entries.

Impact:
When l2_agent crashes there is a potential disruption to configuration processing.

Workaround:
None

Fix:
The fix will avoid the crash, and the update of the owner leaf will be processed accordingly.

1394913 : Rare LACPD crash during process termination

Links to More Info: BT1394913

Component: F5OS-C

Symptoms:
LACPD crashes, generating a core file.

Conditions:
While the LACPD process terminates, it may crash. Operations such as a host reboot and software upgrade cause the process to terminate.

Impact:
A core file is generated. No functional impact to the system.

Workaround:
N/A

Fix:
LACPD no longer crashes during process termination.

1394201 : Vcc-lacpd can intermittently core dump when disconnected from system database

Links to More Info: BT1394201

Component: F5OS-C

Symptoms:
Vcc-lacpd unexpectedly restarts, leaving a core file on the related system controller.

Conditions:
Vcc-lacpd can disconnect from the system database while the process is running. A disconnect of this nature is hard to predict and is not typical. When the connection is reestablished, the process typically crashes.

Impact:
A core file for vcc-lacpd process is generated. Vcc-lacpd process restarts and recovers. Chassis backplane LACP aggregations may go down for a few seconds while the process restarts, briefly interrupting mgmt traffic to blades. User dataplane traffic is unaffected.

Workaround:
None

Fix:
Vcc-lacpd does not crash during this case.

1393441 : Partition fails over on link fault when mgmt ports are aggregated