Manual Chapter : Troubleshooting

Applies To:

Show Versions

BIG-IP LTM

17.1.0, 17.0.0, 16.1.4, 16.1.3, 16.1.2, 16.1.1, 16.1.0, 16.0.1, 16.0.0, 15.1.9, 15.1.8, 15.1.7, 15.1.6, 15.1.5, 15.1.4, 15.1.3, 15.1.2, 15.1.1, 15.1.0, 15.0.1, 15.0.0, 14.1.5, 14.1.4, 14.1.3, 14.1.2, 14.1.0, 14.0.1, 14.0.0

Troubleshooting

Configuration Validation Errors

Configuration Validation Errors
Configuration Failure Condition	Error Message
Deleting a route which is in use by a router profile	The route <name> is referenced by one or more router profiles
Peer refers to non-existent route	Peer <name> refers to non-existing Static-Route <name>
Route refers to non-existent peer.	Static Route <name> refers to non-existing Peer <name>

Connection Termination Reasons

If logging of reset cause is enabled via the tm.rstcause.log db variable, the reason for connection termination is logged to /var/log/ltm.

Reset reason examples:

Connection Termination Reasons
Reason Why Text	Description
SIP Error	Unexpected internal signaling
SIP parser error	Unable to parse SIP message on a stream transport (like TCP)

MRF SIP Troubleshooting Logs

If MRF SIP diagnostic log events are enabled via the log.mrsip.level db variable, the following events will be logged to /var/log/tmm?.

MRF SIP Troubleshooting Logs
Event Text	Log Level	Description
MR SIP: Invalid config attribute <name> in profile <name>	Error	An unexpected configuration attribute was found. For example, an unsupported persist-key was used.
MR SIP: Missing header <name> in the message	Error	One of the mandatory SIP header attributes (To, From, Call-ID, Route, Via) was missing. Since the message will not be accepted without the required attributes, this error occurs when an iRule script removes all instances of one of the required scripts after parsing.
MR SIP: Decrypt branch parameter failed with error : <error_text>	Error	Unable to decrypt our generated Via header.
MR SIP: Encrypt branch parameter failed with error : <error_text>	Error	Unable to encrypt our generated Via header.
MR SIP: Generation of AES encryption key failed	Error	Unable to generate AES encryption key for Via header encryption.
MR SIP: Parse error reading number for <value> value near <offset> Status code <status code>	Notice	Unable to parse a number for the specified value near the specified offset of an input SIP message. If error response is configured, the specified status code response will be returned and the corresponding stats counter will be incremented.
MR SIP: Parse error bad sip protocol version in headline near <offset>. Status Code <status code>	Notice	Invalid sip protocol version near the specified offset of an input SIP message. If error response is configured, the specified status code response will be returned and the corresponding stats counter will be incremented.
MR SIP: Parse error invalid or malformed uri in headline near <offset>. Status Code <status code>	Notice	The SIP URI in the headline of an input SIP message is invalid or malformed. If error response is configured, the specified status code response will be returned and the corresponding stats counter will be incremented.
MR SIP: Parser error invalid headline near <offset>. Status Code <status code>	Notice	The headline of the incoming sip message is invalid near the specified offset. If error response is configured, the specified status code response will be returned and the corresponding stats counter will be incremented.
MR SIP: Parser error to many headers near <offset>. Status Code %d.	Notice	The incoming SIP message contains to many headers to be processed. The header near the specified offset should be the first header that exceeded the limit. If error response is configured, the specified status code response will be returned and the corresponding stats counter will be incremented.
MR_SIP: Parser error extraneous header field near <offset> Status code <status code>	Notice	The incoming SIP message contains a extra field in a header near the specified offset. If error response is configured, the specified status code response will be returned and the corresponding stats counter will be incremented.
MR_SIP: Parser error header to large near <offset>. Status Code <status code>.	Notice	The incoming SIP message has a header line that is too long near the specified offset. If error response is configured, the specified status code response will be returned and the corresponding stats counter will be incremented.
MR_SIP: Parser error missing header code <code>. Status Code <status code>.	Notice	The incoming SIP message is missing a required header. The displayed code is a bit-field that can be decoded with access to the internals of the sip parser. If error response is configured, the specified status code response will be returned and the corresponding stats counter will be incremented.
MR_SIP: Parser error CSEQ method does not match headline tag <tag> : <tag>. Status Code <status code>	Notice	The incoming SIP message has a mis-match between the headline tag. If error response is configured, the specified status code response will be returned and the corresponding stats counter will be incremented.
MR_SIP: Parser max-forwards value has reached zero. Status Code <status code>	Notice	The incoming sip message has been forwarded too many times while being routed, causing the max-forwards value to be decremented to zero. The BIG-IP® system will not process this message. If error response is configured, the specified status code response will be returned and the corresponding stats counter will be incremented.
MR_SIP: Server in maintenance mode. Status Code 503	Notice	The server has been placed in maintenance mode and will not process traffic. If error response is configured, status code 503 response will be returned and the corresponding stats counter will be incremented.
MR_SIP: Loop detected. Status code 482.	Notice	The incoming SIP message contains a SIP round loop. If error response is configured, status code 482 response will be returned and the corresponding stats counter will be incremented.
MR_SIP: Missing Media Connection attributes. Status Code 488.	Notice	The incoming SIP message is missing required Media Connection Attributes. If error response is configured, status code 488 response will be returned and the corresponding stats counter will be incremented.
MR_SIP: Too many media sessions <count> / <count limit>. Error Code <code>	Notice	The number of media sessions in <count> has exceeded the configured <limit count>. If error response is configured, the specified code response will be returned and the corresponding stats counter will be incremented.

SIP Troubleshooting Logs

If MRF SIP diagnostic log events are enabled via the log.mrsip.level db variable, the following events will be logged to /var/log/tmm?.

SIP Troubleshooting Logs
Event Text	Log Level	Description
Max Global Registration limit reached	Error	MR SIP: Subscriber registration failed %s, configured max global registration value :%u reached
Concurrent Session Per Subscriber limit reached	Error	MR SIP: concurrent session per subscriber limit %u reached, subscriber cannot make calls: %s
Non registered subscriber call out	Error	MR SIP: non registered subscriber %s, call dropped. Change SIP session configuration to allow non registered subscriber call out
Subscriber registration failed	Error	MR SIP: subscriber %s, unable to register, received non-2xx SIP response"
HUDEVT_SA_COOKIE_PICK event failed	Error	MR SIP: HUDEVT_SA_COOKIE PICKED event error
Listener creation failed	Error	MR SIP: Failed to create Listener %K for the subscriber %s
Listener deleted	Error	MR SIP: Listener deleted due translation lookup failure %K

sipdb Tool

The sipdb tool will be used to display or delete the persistence or media records from session database. The persistence records are created in LB mode when persistence is configured.

The media records are created in ALG mode.

Usage

sipdb [options]
sipdb --persist [--delete] [--router=name] [--key=value] [--type=persistence_type] [--ipproto=protocol] [–verbose]
sipdb --media [--delete] [--router=name] [-key=call-id]
sipdb --register [–delete] [--router=name] [-key=subscriber uri]
sipdb --help

Options

Option

Description

--persist

-s

Indicates persistence mode. This option should be used to display or delete persistence records. This is the default mode.

Each record displays the persistence type, persistence key, originating ip:port, destination ip:port, protocol and the time remaining.

The records are grouped by the SIP router profile.

To delete the persistence record the record key has to be specified. Details are given below in the example section

--media

-m

Indicates media mode. This option should be used to perform operations on media records.

The media mode displays the Callid, Origination IP:RTP Port, RTCP Port, Interface name Destination IP: RTP Port, RTCP Port, Interface name.

In ALG-Translation mode, the output displays the translated address for the subscriber.

--register

-g

Indicates register mode. This option should be used to perform operations on register records.

The register mode displays the subscriber private address and translated address and the lifetime of the registration.

--help

-h

Displays the help text.

--router = name

-r name

The sip router profile name. This option is used to filter the output matching records for the specified SIP router profile.

The default partition '/Common' should be specified. For example '/Common/siprouter'

The option can be used for both modes i.e. persist and media modes.

--key=value

-k value

Specifies the key for the session record. The option is used to filter the display with the specific key or delete a specific key.

For persistence mode the key is either a SIP Call-ID, Source Address or Custom value.

For Media mode the key is SIP Call-ID.

For register mode the key is the subscriber uri.

--delete

-d

To delete a particular record.

This option along with the mode and the key details specifies the record to be deleted.

For persistence mode to delete a record the router name, key, persistence type and ip proto values have to be specified.

To delete a media entry the router name and SIP Call-ID needs to be specified.

--type = value

-t value

Type of persistence entry.

The option is applicable when deleting a persistence record.

Following are the applicable values.

[C\|c]	For Call-ID id persistence
[S\|s]	For Source Address persistence
[O\|o]	For Custom type persistence

--ipproto = value

-p value

Either TCP or UDP.

The option is applicable when deleting a persistence record.

--verbose

-v

This option is applicable in persistence mode.

Displays the destination transport and pool name in addition to the default display.

Examples

Default Display of Persistence Entries

#sipdb
Router: /Common/siprouter       Number of entries: 1
Key                             Originator             Destination           Proto Timeout
--------------------------------------------------------------------------------------------------------------------------------
C 1-8834@10.10.20.7             10.10.20.2:35462       10.10.10.2:5060       TCP   175
 
Router: /Common/siprouter_alg      Number of entries: 1
Key                             Originator             Destination           Proto Timeout
--------------------------------------------------------------------------------------------------------------------------------
C 1-8835@10.10.20.7             10.10.20.2:35462       10.10.10.2:5060       TCP   175

Verbose Display of Persistence Entries

# sipdb -v
Router: /Common/siprouter       Number of entries: 1
Key                             Originator             Destination            Proto Timeout Transport    Pool Name
-------------------------------------------------------------------------------------------------------------------------------------
C 1-8872@10.10.20.7             10.10.20.2:56913       10.10.10.2:5060        TCP   175     vs:vs_sip    sip_pool
 
Router: /Common/siprouter_alg       Number of entries: 1
Key                             Originator             Destination            Proto Timeout Transport    Pool Name
-------------------------------------------------------------------------------------------------------------------------------------
C 1-8874@10.10.20.7             10.10.20.2:56913       10.10.10.2:5060        TCP   175     vs:vs_sip    sip_pool

To filter the above record for a particular SIP router profile name

#sipdb --persist –router /Common/siprouter --verbose
#sipdb --persist –router=/Common/siprouter --verbose
#sipdb --persist -r /Common/siprouter
 
Router: /Common/siprouter       Number of entries: 1
Key                    Originator             Destination            Proto Timeout Transport  Pool Name
-------------------------------------------------------------------------------------------------------------------------------------
C 1-8872@10.10.20.7    10.10.20.2:56913       10.10.10.2:5060        TCP   175     vs:vs_sip  sip_pool

To filter the record for a persistence key

#sipdb --persist –key 1-8872@10.10.20.7 --verbose
 
Router: /Common/siprouter       Number of entries: 1
Key                    Originator             Destination            Proto Timeout Transport  Pool Name
-------------------------------------------------------------------------------------------------------------------------------------
C 1-8872@10.10.20.7    10.10.20.2:56913       10.10.10.2:5060        TCP   175     vs:vs_sip  sip_pool

To delete the above record

sipdb persist --delete --key 1-8872@10.10.20.7   --router /Common/siprouter --type C --ipproto TCP
 
Record Successfully deleted

Moving router and/or virtual to different traffic group

The BIG-IP® system does not support changing the traffic group or a router and/or virtual server. MRF stores state that has a different lifetime than a connection in an internal in-memory database (known as session db). This includes persistence tables (SIP LB), call tables (SIP ALG), and registrations tables (SIP ALG), etc. Records stored in session db are auto replicated between the active and standby device. Part of the key for each entry in the session db is the identifier for a traffic-group. If the traffic-group of a virtual and/or router instance is changed all data stored in session db will be orphaned.

Config changes not loading, or stats don't show up on new router instance

Most changes to config are applied to existing connections. Changes to the set of profiles used by a connection only apply to new connections. Since many message routing protocols use long lived connections, some config changes will not effect existing connections. For example replacing the router profile used by a virtual server will not apply to existing connections. Thus all traffic on existing connections will still be routed through the previous router instance and the stats for that traffic will be included with the previous router instance. To apply the traffic to the new router instance, the existing connections will need to be closed forcing the clients to create new connections.

iRule changes not loading

Changes to iRule scripts attached to a virtual or transport-config do not change the scripts executing on existing connections. New connections will use the updated scripts. To cause the new script code to be applied, all existing connections (both client side and server side) will need to be closed and new connections created. This may be avoided by moving the business logic of the script to a procedure as follows:

ltm rule mylib {
    proc sip_ingress {} {
        if { [SIP::is_request] and [clientside] } {
            # do something
            # change here
        } else {
            # do something else
            # change here
        }
    }
}
ltm rule routing {
    when SIP_INGRESS {
        call mylib::sip_ingress
    }
}

Dropped UDP datagrams

Dropped UDP datagrams have been observed at very low traffic rates (100 calls per second). One cause has been MPI latency. Try making sure the 'scheduler.hsbpollpode.ltm' db var is set to "always". This has been show to reduce the MPI call latency.

MRF Debugging

Messages received on per-client created connections

All messages received on an outgoing connection created using the per-client connection mode, will automatically be forwarded to the connection that received the request which caused the outgoing connection to be created. This includes request messages received on this connection. This is because the connection acts as a direct connection for communication between the original client and the other device. This routing is done be setting the nexthop of all messages received to the last hop of the original request message.

For example:

A SIP INVITE request is received on a connection from 10.10.10.21 to 10.10.20.50. This message gets routed to proxy server 10.20.30.85 using a transport-config that does not configure SNAT and has a connection-mode of per-client. An outgoing connection will be created from 10.10.10.21 to 10.20.30.85. All messages (whether responses of requests) received on the outgoing connection will be automatically routed to the SIP endpoint at 10.10.10.21 using the original incoming connection.

To route a message received on a per-client created connection to another device, the nexthop field will have to be cleared using the MR::message nexthop none command as follows:

when MR_INGRESS {
  MR::message nexthop none
  MR::message route config /Common/other_tc host 10.20.30.40:1234
}

Did the message reach the message router?

There are multiple places where processing can stop or a failure can occur. The stats of the profiles added to the virtual server (or transport-config) should be used to determine if the message reached the message router. From the transport profile's stats (TCP/UDP/SCTP), it can be determined if packets were received by the transport filter. From the protocol profile's stats (sipsession), it can be determined if the received packets were correctly parsed into messages. If an error was found in the message parsing this should be detectable using the protocol's stats.

The message router profile (siprouter) stats should increment with each message received. The result of each messages routing operation should also be represented in the stats.

Why did the message fail routing

The MR_INGRESS event is raised for each message before it enters routing . Once routing is complete either MR_EGRESS or MR_FAILED event is raised. The message metadata can be logged during these events to help debug the results of routing. Some fields and their usage follows:

Metadata Field	Populated	Purpose
lasthop	before MR_INGRESS	Contains the TMM and flow_id of the originating connection of the message
nexthop	before MR_INGRESS (or during MR_INGRESS)	Selects the destination connection for the message
route	after routing (or during MR_INGRESS)	The value of the selected route (peer list). If set during MR_INGRESS, this route will be used instead of performing route lookup
originator	before MR_INGRESS	The IP, port and rtdom_id of the originator of the connection. Also contains the transport type and name of the originating connection.
status	after routing	The results of the route lookup
attempted	after routing (or during MR_INGRESS)	The list of destination hosts attempted. This list of hosts will be treated as marked down when performing peer selection and load balanced pick.
retry_count	after routing	The number of times this message has been submitted for routing

Debugging Request Routing

SIP Overview

Request Routing Debugging

iRules can be used for route debugging. Remember that the iRule needs to be on each transport in the system (virtual servers and transport-configs). MR_INGRESS event runs on the connection that received the request message. MR_EGRESS event runs on the connection that the message is being sent out. MR_FAILED event runs on the connection that received the request message when a message failed to be routed.

SIP iRule commands can be run in the MR iRule events. MRF communicates with the SIP parser to instruct it as to which message is currently used during the MR event.

To know if a message is a request or a response, the following conditional can be used:

If {[SIP::response code] eq “”} { # this is a request message

During MR_INGRESS, the message’s route can be examined as follows:

Log local0. “route [MR::message route]”

The transport type and name can be inspected (in v12.0 and later) via an iRule command as follows:

Log local0. “transport [MR::transport]”

An example script for MR_INGRESS is as follows:

when MR_INGRESS {
  log local0. “transport: [MR::transport] flow_id: [MR::flow_id]”
  if {[SIP::reponse code] eq “”} {
    log local0. “request [SIP::method] persist key [SIP::persist] route [MR::message route]”
  } else {
    log local0. “response [SIP::response code] nexthop [MR::message nexthop] route [MR::message route]"
  }
}

After routing has occurred, the messages route field will be populated with the value of the selected route and either MR_EGRESS will be executed or MR_FAILED. If routing succeeded, the route status will be set to “route found” and MR_EGRESS event will be raised on the outgoing connection. If routing failed, the route status will be set and MR_FAILED event will be raised on the incoming connection.

when MR_EGRESS {
  log local0. “transport: [MR::transport] flow_id: [MR::flow_id]”
  if {[SIP::reponse code] eq “”} {
    log local0. “request [SIP::method] persist key [SIP::persist] route [MR::message route]”
  } else {
    log local0. “response [SIP::response code] nexthop [MR::message nexthop] route [MR::message route]"
  }
}
 
when MR_FAILED {
  log local0. “transport: [MR::transport] flow_id: [MR::flow_id] route status [MR::message status]”
  if {[SIP::reponse code] eq “”} {
    log local0. “request [SIP::method] persist key [SIP::persist] route [MR::message route]”
  } else {
    log local0. “response [SIP::response code] nexthop [MR::message nexthop] route [MR::message route]"
  }
}

Subscriptions

Product Usage

Trials

Registration Keys

Downloads

Applies To:

BIG-IP LTM

Troubleshooting