Manual Chapter :
Troubleshooting Network Connections
Applies To:
Show Versions
ARX
- 6.3.0
From any mode, use the ping command to send an ICMP ECHO request to another interface:
ping ip-address
where ip-address (for example, 10.10.10.1) is the destination for the ping.
The ARX pings the IP address repeatedly until you stop it by pressing <Ctrl-C>. Once you stop it, a report summarizes the results of the ping test.
bstnA# ping 172.16.100.183
bstnA# ...
ip-address is the destination for the ping, and
number (1-10,000) is the number of pings to send.
bstnA# ping 172.16.100.183 count 4
bstnA# ...
| The network processors use proxy-IP addresses to communicate with servers. Use the show ip proxy-addresses command (see Showing all Proxy IPs, on page 4-8 of the ARX® CLI Network-Management Guide) to list all proxy-IP addresses. Each network processor is assigned its own proxy-IP. |
| The network processors use virtual-IP (VIP) addresses to communicate with clients. Use the show global server command (see Showing All Global Servers, on page 10-12 of the ARX® CLI Storage-Management Guide) to list all VIPs. |
| There are two types of management interface: one in-band interface for each VLAN and one out-of-band (OOB) management interface. Use the show interface vlan command to list all VLAN-management interfaces (see Listing all VLAN Management Interfaces, on page 4-6 of the ARX® CLI Network-Management Guide). Use show interface mgmt to find the source address for the OOB Mgmt interface (see Showing the Interface Configuration and Status, on page 4-21 of the ARX® CLI Network-Management Guide). |
Use the source clause with the ping command to send an alternate source IP in the ICMP ECHO request:
ip-address is the destination for the ping,
from slot.processor (optional, 1-2.1-12) chooses a source processor. If you choose the processor and the source-address, they may conflict: the ping may not return to the processor that sent it. If you omit this, the CLI chooses the correct processor. In general, this should be omitted.
source-address is the source-IP address to send in the ICMP ECHO.
count number (optional, 1-10,000) limits the number of pings, as explained above.
bstnA> ping 10.53.2.10 source 10.50.20.110 count 2
bstnA> ...
prtlndA> show interface mgmt
prtlndA> show ip route
prtlndA> ping 10.1.23.1 source 10.1.23.11 count 4
prtlndA> ...
You can use the expect traceroute command to show each IP-router hop between the NSM and a given IP address. Like ping, the expect traceroute command is accessible from any mode:
destination-address is the IP-address for the traceroute, and
timeout seconds (optional, 1-2096) limits the time for the traceroute process.
If a hop is unreachable, the command outputs asterisks (*) until you interrupt it or it times out. Use <Ctrl-C> to interrupt the expect traceroute command.
The packet starts at an inband (VLAN) management interface. Inband-management interfaces were discussed in Configuring an In-Band (VLAN) Management Interface, on page 4-4 of the ARX® CLI Network-Management Guide.
bstnA# expect traceroute 192.168.25.19
bstnA# ...
The Test TCP (TTCP) utility is a benchmarking tool to measure throughput over TCP or UDP connections. You can run a TTCP test from the SCM processor to any filer, client station, or other server that supports TTCP. (TTCP tools are freely available from the Internet.) After configuring a remote station to receive TTCP transmissions, use the expect ttcp transmit command to invoke a 10-second test:
ttcp-server-address is the IP-address for the pre-configured server, and
timeout seconds (optional, 1-2096) sets a time limit for the TTCP test.
bstnA# expect ttcp transmit 192.168.98.55
bstnA# expect ttcp transmit 192.168.97.55
The optional timeout argument also cancels the test.
To test the RON connection from one ARX to another, run expect ttcp server at the receiving switch and then expect ttcp transmit from the sending switch. (For information on creating a RON tunnel, refer back to Creating a Tunnel to Another ARX, on page 6-4 of the ARX® CLI Network-Management Guide.) Like the previous expect commands, you run expect ttcp server from any mode:
expect ttcp server [timeout seconds]
prtlndA# expect ttcp server
Go to the sending switch to run the test. The IP address to use for the server switch is the.1 address on the servers private subnet. Use the show ron route command (see Showing the RON Routes, on page 6-10 of the ARX® CLI Network-Management Guide) to find the private subnet for the server switch. To continue the example, this command sequence finds the private subnet for prtlndA, then runs a TTCP test over the RON tunnel:
bstnA# show ron route
bstnA# expect ttcp transmit 169.254.24.1
To debug a connectivity issue, you can trace the software processes on an NSM processor. A traced NSM processor sends its log messages to a file named fastpath, which is similar to the syslog file described in the previous chapter. From cfg mode, use the logging fastpath processor command to choose an NSM processor to trace:
slot (2 on an ARX-4000; 1 on an ARX-2500, ARX-2000, ARX-1500, or ARX-500) is the slot number of the desired NSM, and
processor (1-12) is the processor number. Use show processors for a complete list of processors (and their modules and slots) on the ARX.
The first time you issue any logging fastpath command, the CLI issues a warning about the performance impact. Enter yes to continue. This prepares the processor for minimal logging. You can increase the volume of logs for individual processes and process groups on the NSM, as shown below.
bstnA(cfg)# logging fastpath processor 2.1
bstnA(cfg)# ...
Every NSM-log message originates from a log component on the NSM. A log component is a source of fastpath-log messages, typically an internal NSM process or group of processes. They are similar to the log components that write to the syslog, described in Log Components. The table below is an alphabetical list of all NSM log components, along with a brief description.
From cfg mode, use the logging fastpath component command to change the logging level for one NSM-log component:
logging fastpath component name level
name (1-128 characters) is an NSM-log component from the list of NSM-Log Components, and
level (0-10) chooses the logging level for the component. A 0 (zero) stops all messages from the component. A 1 causes the component to log non-recoverable errors only; this is the default. A 2 or 3 adds warnings and recoverable errors. A 4 adds logs about internal configuration changes. Levels 5 through 10 add increasing densities of per-packet messages.
The first time you enter any logging fastpath command, the CLI warns you about the performance impact. If the warning appears, enter yes to proceed.
This logging level applies to all NSMs that have been enabled for logging with logging fastpath processor, described earlier.
bstnA(cfg)# logging fastpath component NSM_CIFS 6
bstnA(cfg)# ...
You can focus the log output by filtering for a particular subnet, IP address, or other search string. Add a filter clause to the end of the logging fastpath component command:
name (1-128 characters) is an NSM-log component from the list of NSM-Log Components.
match-string (1-80 characters) is a string to match against. Quote the string if it contains any spaces.
include | exclude is a required choice: include collects any log message that matches the match-string, while exclude omits all matching log messages and gathers all other messages.
You can repeat this command multiple times to expand or narrow the search further. A packet that matches any of the entered filters is included in the log file (or excluded from it, if you use the exclude keyword). Log messages are matched in the order that you enter them, so a command to include logs with 10.10.53.99 is ineffective if a previous filter excludes 10.10.
As above, the CLI may prompt with a performance warning before it sets the filter. If necessary, enter yes to continue.
bstnA(cfg)# logging fastpath component NSM_CIFS filter 192.168.25.15
bstnA(cfg)# logging fastpath component NSM_CIFS filter 172.16.100.183
bstnA(cfg)# ...
You can stop all log messages from a given NSM component; this applies to every instance of the component, on every NSM processor that is enabled for logging. Use the no form of the logging fastpath component command:
bstnA(cfg)# no logging fastpath component NSM_CIFS
bstnA(cfg)# ...
Use the show fastpath logging command to see the logging levels for all NSM-log components:
bstnA(cfg)# show fastpath logging
Remember to stop all NSM processors from logging as soon as you finish your tracing session; NSM logging degrades performance. Use no logging fastpath processor to disable logging from a certain processor:
slot (2 on an ARX-4000; 1 on an ARX-2500, ARX-2000, ARX-1500, or ARX-500) is the slot number of the desired NSM, and
processor (1-12) is the processor number. Use show processors for a complete list of processors (and their modules and slots) on the ARX.
bstnA(cfg)# no logging fastpath processor 2.1
bstnA(cfg)# no logging fastpath processor 2.2
bstnA(cfg)# no logging fastpath processor 2.3
bstnA(cfg)# ...
You can use show logs fastpath to view the fastpath file and tail logs fastpath follow to follow it as it grows. To search through the fastpath file for a pattern, use grep pattern logs fastpath. These are the same commands used to access the syslog file, as described in.Accessing the Syslog.
bstnA(cfg)# show logs fastpath
| utc (2006-11-22T08:43:30 in the example) is the time in UTC. |
| uuu (000) is the millisecond time fraction. |
| switch (bstnA) is the switch name. |
| slot-proc-board (3-2-NSM_TX) is the chassis slot, processor number, and board type (always NSM). Use the show processors command for a list of all processors and their associated modules. |
| pid (2124) is the Process ID (PID). |
| ins (0) is the processs instance number. |
| sev (7) is the message severity, from 7 (debug) to 2 (critical). |
| id (MSG7) is the message-catalog ID, a unique ID for each log message. MSGn is an ID for an uncatologued message. The n is the severity of the message, from 7 (debug) down to 2 (critical). |
| msg (cifs_proxy.c:415: cifss-conn: CIFS ...) is the message text itself. |
From priv-exec mode, you can use the capture session command to start capturing IP traffic and stream it into a file:
session-id (1-4; 1-2 on the ARX-500) identifies this capture session, so that you can close it later.
ip-address is the address to match against. The capture includes all packets whose source IP or destination IP matches this address.
and-ip ip2 (optional) is a second IP address. If you select this option, the capture includes only the packets between ip-address and ip2 in both directions.
vlan vlan-id (1-4095) specifies the VLAN.
prefix (1-255 characters) is the prefix you choose for the file name. Each file is named as follows:
prefix.cap
where prefix is what you choose here, nnnnn is a file number for this session (there can be multiple files for each capture session, as explained later), and yyyymmddHHMMSS is the date and time the file was created.
A capture session begins when you invoke the command and ends when the capture file reaches its maximum size, or when you stop it with no capture session. The maximum file size and the no capture session command are both described below.
bstnA(cfg)# end
bstnA# capture session 1 ip 172.16.100.183 and-ip 192.168.25.10 vlan 25 file clientCap
bstnA# ...
By default, the capture session stops when it fills a single file with 16,000 kilobytes of data. You can use the filesize and/or filecount options to change the size and number of capture files:
capture session session-id ip ip-address [and-ip ip2] vlan vlan-id file prefix [filesize kilobytes] [filecount count]
filesize kilobytes (optional, 1-50,000; 1-1,000,000 on an ARX-4000) limits the size of the file; once the file reaches this limit, the capture stops. (One kilobyte is 1,000 bytes, not 1,024.) The default is 16,000 kilobytes.
filecount count (optional, 1-10) sets up a series of capture files; when the capture reaches the filesize limit, it can close the current capture file and start a new one. The default is 1 file. If you set a count of 2 or more, the capture process rotates the capture files indefinitely, and does not stop capturing packets until you use no capture session.
bstnA(cfg)# end
bstnA# capture session 4 ip 192.168.25.19 vlan 25 file shadow_usage filesize 500
bstnA# ...
To focus only on traffic to or from the above ports, use the optional protocol cifs arguments at the end of the command:
protocol cifs (optional) filters out all traffic except CIFS-related traffic.
For example, the following command sequence starts another capture session for CIFS-related traffic:
bstnA(cfg)# end
bstnA# capture session 2 ip 192.168.25.27 vlan 25 file fsrvr protocol cifs
bstnA# ...
You can use protocol non-cifs to filter out all CIFS-related traffic from the capture file(s).
From priv-exec mode, use no capture session to stop the capture immediately or remove a capture session that has stopped:
no capture session session-id [no-merge]
session-id (1-4) identifies the capture session to close.
no-merge (optional) prevents the command from automatically merging the output files. This is only relevant if the capture session generated 2 or more files, as specified with the filecount option. If you omit this, the merged filename is prefix.cap and contains all captured packets in chronological order.
bstnA# no capture session 1
bstnA# ...
You can monitor all traffic to and from the back-end filers with the proxy-all option. This captures any packet whose source or destination IP address is any of the proxy-IP addresses on the ARX (recall Adding a Range of Proxy-IP Addresses, on page 4-7 of the ARX® CLI Network-Management Guide). This option does not require a VLAN ID:
bstnA(cfg)# end
bstnA# capture session 2 proxy-all file proxyTraffic filecount 2
bstnA# ...
A capture file is accessible from the capture directory: use show capture to see the file listing:
bstnA# show capture
bstnA# ...
show capture file-name
where file-name (1-1024 characters) is the name of the chosen capture file.
bstnA# show capture clientCap.cap
You can show a high-level summary of the IP traffic captured in one of the above files. To show the summary, specify the capture file and use an optional summary keyword in the show capture command:
show capture capture-file summary [cifs | non-cifs]
capture-file (1-1024 characters) is the name of the chosen capture file.
summary specifies the output format.
cifs | non-cifs (optional) is a filter for the output. The cifs flag shows only CIFS-related traffic (to and from the ports listed earlier), and the non-cifs flag filters out all CIFS-related traffic.
You can only use this summary option when the capture session is stopped. Use no capture session to stop the session, even if the session has stopped writing to its only output file.
The output is the same as that of TShark with the -z option. It shows a table of UDP conversations and another table of TCP conversations, followed by separate RTT tables for NFSv2, NFSv3, and SMB (or CIFS). The first two tables, UDP Conversations and TCP Conversations, show the numbers of UDP frames and bytes exchanged between each pairing of IP addresses. Each IP-address pair is typically a VIP or proxy-IP and some external address, and appears on one line. These are followed by three tables with RTT statistics for specific NFS-procedure calls and/or CIFS commands.
bstnA# show capture nasTraffic.cap summary
bstnA# ...
Use the show capture sessions command to show the current status of all capture sessions, if any are running:
bstnA# show capture sessions
If you created multiple files from a single capture session and preserved them as separate files, you can later use the capture merge command to merge them into a single file. The capture session command may produce multiple capture files if the ARX reboots in the middle of the session, or if someone stops the capture session with the no-merge option (described earlier). The merge operation sorts all of the captured packets into chronological order, starting with the earliest.
capture merge file prefix
where prefix (1-256 characters) is the prefix that is common to the files that you want to merge. This is typically the same prefix used to create the files in the capture session command. This is also the name of the output file created by the merge operation (prefix.cap).
bstnA(cfg)# end
bstnA# show capture
bstnA# capture merge file proxyTraffic
bstnA# show capture
bstnA# ...
You can use the collect command to collect all packet-capture files, along with other diagnostic information, and send them to a remote site through FTP, SCP, or e-mail. This is described in Collecting Diagnostic Information. To collect only the packet-capture files, use the collect capture command (from Collecting Partial Information).
You can also monitor all of the traffic on a specific port, using an external packet analyzer. Port mirroring captures all packets going through designated source interface(s) and copies them to the MIRROR interface. A network analyzer at the MIRROR interface can therefore see all traffic going through the source interface(s) in real time.
From cfg mode, use the monitor system command to monitor one source interface:
slot/port is the source port. Use the show interface summary command to find all ports configured on the ARX.
{rx | tx | both} chooses the direction of monitored packets. Choose rx for received (ingress) packets, tx for transmit (egress) packets, or both.
A notice appears, cautioning you against excessive traffic through the MIRROR port (similar to the Important note above). Enter yes to continue.
You can optionally use another external interface as the destination instead of the MIRROR interface. This is your only option with the ARX-2000 and ARX-4000, which have no separate MIRROR interface. From cfg mode, use the destination-interface clause at the end of the monitor module command:
source-interface slot/port is the source port (for example, source-interface 2/9 on an ARX-4000). Use the show interface summary command to find all ports configured on the ARX.
{rx | tx | both} chooses the direction of monitored packets. Choose rx for received (ingress) packets, tx for transmit (egress) packets, or both.
destination-interface slot/port is the destination port where the network analyzer is connected (for example, destination-interface 1/6). Choose a destination port with equal or greater bandwidth than the source port.
You can monitor one source port at a time with monitor module; use monitor system (above) to monitor multiple source ports. As with monitor system, a prompt warns you not to overwhelm the destination port; enter yes to continue.
bstnA(cfg)# monitor module source-interface 2/4 both destination-interface 2/6
Use the show monitor command to view the current state of any active monitor session:
bstnA(cfg)# show monitor
Remember to shut off port-mirroring sessions when you are not actively monitoring them. Port mirroring puts a strain on port performance. Use the no monitor command to shut down a monitor session:
{system | module} chooses the type of monitor session to shut down.
For example, the following command sequence shuts down a monitor module session:
bstnA(cfg)# no monitor module
Recall that you can use monitor system to monitor more than one port. For the multi-port case, you can use no monitor system to selectively disable one source port:
bstnA(cfg)# no monitor system source-interface 2/3
To see cumulative statistics for filer connections, you can use the show statistics filer connections command:
filer-name (optional, 1-64 characters) identifies an external filer by the name defined on the ARX. For a complete list of filer names, use the show external-filer command (see Listing External Filers, on page 6-18 of the ARX® CLI Storage-Management Guide).
processor slot.proc (optional) focuses the output on a particular network processor. Use show processors for a complete list of processors (and their modules and slots) on the ARX.
The output is two tables, one for CIFS connections and one for NFS connections. The CIFS table breaks down the statistics between data-plane connections (directly from ARX clients) and control-plane connections (which go through one or more processes on the ACM or ASM), and shows counters for client sessions within the data-plane connections. If you choose the processor option, the output is similar but with a more-limited scope.
bstnA# show statistics filer fs2 connections
bstnA# ...
You can use the clear statistics filer connections command to clear the filer-connection statistics from the ARX database. This reduces all of the Max statistics to the current counts. Rebooting the ARX has the same effect. You can clear these statistics from priv-exec mode:
clear statistics filer [filer-name] connections
where filer-name (optional, 1-64 characters) identifies a single external filer. This clears only the connection statistics that pertain to a single filer. Use the filer name defined on the ARX. For a complete list of filer names, use the show external-filer command (see Listing External Filers, on page 6-18 of the ARX® CLI Storage-Management Guide).
bstnA# clear statistics filer fs2 connections
bstnA# show statistics filer fs2 connections
If a back-end filer is overwhelmed with TCP connections, you can forcibly drop all connections to the filer at once. This can occur after you reduce the limit on CIFS connections to the filer (see Limiting CIFS Connections to the Filer, on page 6-10 of the ARX® CLI Storage-Management Guide); new clients cannot connect until the number of current connections drops below the new limit.
Dropping TCP connections may cause a noticeable interruption for your currently-connected clients. From priv-exec mode, you can use drop filer-connections to close all TCP sessions with a particular filer:
filer-name (1-64 characters) identifies a single external filer. Use the filer name defined on the ARX. For a complete list of filer names, use the show external-filer command (see Listing External Filers, on page 6-18 of the ARX® CLI Storage-Management Guide).
bstnA# drop filer-connections smb1
