Manual Chapter : Troubleshooting

Applies To:

Show Versions Show Versions

BIG-IP LTM

  • 15.0.1, 15.0.0, 14.1.3, 14.1.2, 14.1.0, 14.0.1, 14.0.0, 13.1.3, 13.1.1, 13.1.0
Manual Chapter

Troubleshooting

Can't find Generic Message Profile

The genericmessage profile is only available via tmsh.

Log Messages

Configuration Validation Errors

Table 1. Configuration Validation Errors
Configuration Failure Condition Error Message
Deleting a route which is in use by a router profile The route <name> is referenced by one or more router profiles
Peer refers to non-existent route Peer <name> refers to non-existing Static-Route <name>
Route refers to non-existent peer. Static Route <name> refers to non-existing Peer <name>

Common Configuration Problems

Moving router and/or virtual to different traffic group

The BIG-IP® system does not support changing the traffic group or a router and/or virtual server. MRF stores state that has a different lifetime than a connection in an internal in-memory database (known as session db). This includes persistence tables (SIP LB), call tables (SIP ALG), and registrations tables (SIP ALG), etc. Records stored in session db are auto replicated between the active and standby device. Part of the key for each entry in the session db is the identifier for a traffic-group. If the traffic-group of a virtual and/or router instance is changed all data stored in session db will be orphaned.

Config changes not loading, or stats don't show up on new router instance

Most changes to config are applied to existing connections. Changes to the set of profiles used by a connection only apply to new connections. Since many message routing protocols use long lived connections, some config changes will not effect existing connections. For example replacing the router profile used by a virtual server will not apply to existing connections. Thus all traffic on existing connections will still be routed through the previous router instance and the stats for that traffic will be included with the previous router instance. To apply the traffic to the new router instance, the existing connections will need to be closed forcing the clients to create new connections.

iRule changes not loading

Changes to iRule scripts attached to a virtual or transport-config do not change the scripts executing on existing connections. New connections will use the updated scripts. To cause the new script code to be applied, all existing connections (both client side and server side) will need to be closed and new connections created. This may be avoided by moving the business logic of the script to a procedure as follows:

ltm rule mylib {
    proc sip_ingress {} {
        if { [SIP::is_request] and [clientside] } {
            # do something
            # change here
        } else {
            # do something else
            # change here
        }
    }
}
ltm rule routing {
    when SIP_INGRESS {
        call mylib::sip_ingress
    }
}

MRF Debugging

Did the message reach the message router?

There are multiple places where processing can stop or a failure can occur. The stats of the profiles added to the virtual server (or transport-config) should be used to determine if the message reached the message router. From the transport profile's stats (TCP/UDP/SCTP), it can be determined if packets were received by the transport filter. From the protocol profile's stats (sipsession), it can be determined if the received packets were correctly parsed into messages. If an error was found in the message parsing this should be detectable using the protocol's stats.

The message router profile (siprouter) stats should increment with each message received. The result of each messages routing operation should also be represented in the stats.

Why did the message fail routing

The MR_INGRESS event is raised for each message before it enters routing . Once routing is complete either MR_EGRESS or MR_FAILED event is raised. The message metadata can be logged during these events to help debug the results of routing. Some fields and their usage follows:

Metadata Field Populated Purpose
lasthop before MR_INGRESS Contains the TMM and flow_id of the originating connection of the message
nexthop before MR_INGRESS (or during MR_INGRESS) Selects the destination connection for the message
route after routing (or during MR_INGRESS) The value of the selected route (peer list). If set during MR_INGRESS, this route will be used instead of performing route lookup
originator before MR_INGRESS The IP, port and rtdom_id of the originator of the connection. Also contains the transport type and name of the originating connection.
status after routing The results of the route lookup
attempted after routing (or during MR_INGRESS) The list of destination hosts attempted. This list of hosts will be treated as marked down when performing peer selection and load balanced pick.
retry_count after routing The number of times this message has been submitted for routing
MR:route_status: "queue full"

One reason for MR_FAILED would be when MR:route_status is set to "queue full". This result can happen when the following conditions are met:

  1. MRF-SIP profile with TCP for transport.
  2. SIP Peer has very few pool members.
  3. One of the pool member is down.
  4. Burst of SIP Traffic with message size > 2K Bytes.

There are 2 configurable items (Max-Pending-Messages and Max-Pending-Bytes) in the router config to define the queue capacity. If the incoming traffic is high with large messages then the possibility of filling up the queue increases significantly before the connection request timeout occurs on the pool member which is down.

If the message size is larger than 2k then try increasing the Max-Pending-Bytes first. Otherwise, increase Max-Pending-Messages. If neither increase works, then increase both values.

MRF Logging

During MR_INGRESS, the message’s route can be examined as follows:

Log local0. “route [MR::message route]”

The transport type and name can be inspected (in v12.0 and later) via an iRule command as follows:

Log local0. “transport [MR::transport]”