Deep Dive: Debugging Transit Traffic on Switches and Routers

Sometimes when debugging connectivity issues it helps to know if certain traffic passes through a given network device. On a firewall this is easy: you just check the logs. In addition, many firewall products allow taking traffic captures which can then be analysed with a tool like Wireshark.

Disclaimer: Dear readers, please note that this blog is a bit older and therefore the content, insights and statements may have changed over time as products, services and technologies evolve.

On a router or a switch, the situation is different. In the case of Juniper and Arista, the utility tcpdump is available on the device. But it does not capture transit traffic that is just passing the device. It only captures traffic directed to or originating from the control plane like packets from a routing protocol. But there are several options to check if certain traffic is passing through a router or a switch.

The first option is an “access control list” (ACL) with a counter. Every time a packet passes the device that matches the ACL the counter increases. The following two examples for Junos and EOS demonstrate how you can check if packets with destination IP address 10.100.35.2 with TCP destination port 22 enter the device through a specific interface.

set firewall family inet filter TCP-22-MON interface-specific
set firewall family inet filter TCP-22-MON term TCP-22 from destination-address 10.100.35.2/32
set firewall family inet filter TCP-22-MON term TCP-22 from protocol tcp
set firewall family inet filter TCP-22-MON term TCP-22 from destination-port 22
set firewall family inet filter TCP-22-MON term TCP-22 then count TCP-22
set firewall family inet filter TCP-22-MON term TCP-22 then accept
set firewall family inet filter TCP-22-MON term ACCEPT then accept
 
 
set interfaces ge-0/0/0 unit 0 family inet filter input TCP-22-MON
set interfaces ge-0/0/1 unit 0 family inet filter input TCP-22-MON
set interfaces ge-0/0/2 unit 0 family inet filter input TCP-22-MON
set interfaces ge-0/0/4 unit 0 family inet filter input TCP-22-MON

Attention

The last term called ACCEPT is important. Without it all traffic not matching the filter would be dropped.

The statement “interface-specific” lets Junos generate a dedicated counter for each interface where the filter was applied. With “show firewall” the counters can then be inspected.

Junos Verification

umb@mx1> show firewall   
 
Filter: __default_bpdu_filter__                               
 
Filter: TCP-22-MON-ge-0/0/0.0-i                               
Counters:
Name                                                Bytes              Packets
TCP-22-ge-0/0/0.0-i                                  8193                   88
 
Filter: TCP-22-MON-ge-0/0/1.0-i                               
Counters:
Name                                                Bytes              Packets
TCP-22-ge-0/0/1.0-i                                     0                    0
 
Filter: TCP-22-MON-ge-0/0/2.0-i                               
Counters:
Name                                                Bytes              Packets
TCP-22-ge-0/0/2.0-i                                     0                    0
 
Filter: TCP-22-MON-ge-0/0/4.0-i                               
Counters:
Name                                                Bytes              Packets
TCP-22-ge-0/0/4.0-i                                     0                    0
 
umb@mx1>

On Arista’s EOS the same ACL looks like this:

Arista EOS Configuration

ip access-list TCP-22-MON
counters per-entry
permit tcp any host 10.100.35.2 eq 22
permit ip any any
 
 
interface ethernet 1
ip access-group TCP-22-MON in

Of course, there is also a show command to display the counters.

Arista EOS Verification

show ip access-list TCP-22-MON

If the method with ACLs is not sufficient because for example header fields need to be inspected, then port mirroring is an option. This requires a capturing device that is connected directly to the network device. Often this is not feasible. Fortunately, there is also the possibility to perform remote mirroring. The traffic is not mirrored to a local port but rather to a GRE tunnel. The tunnel can be configured with the remote tunnel end being the IP address of the capturing device which can also be your notebook running Wireshark (given enough bandwidth is available). This method is also called ERSPAN.

The following example shows how to configure a Juniper MX router to mirror SSH traffic traversing interface ge-0/0/0 to a GRE tunnel. The IP address on the tunnel interface gr-0/0/0.0 needs to be set but it does not matter what IP address is used. The tunnel destination IP address is where your analyzer listens.

Junos remote port mirror

# tunnel services are needed for GRE
set chassis fpc 0 pic 0 tunnel-services
 
# GRE tunnel interface
set interfaces gr-0/0/0 unit 0 tunnel source 10.100.100.3
set interfaces gr-0/0/0 unit 0 tunnel destination 10.100.100.1
set interfaces gr-0/0/0 unit 0 family inet address 10.99.99.1/30
 
# mirroring
set forwarding-options port-mirroring input rate 1
set forwarding-options port-mirroring input run-length 1
set forwarding-options port-mirroring family inet output interface gr-0/0/0.0
 
# firewall filter to match SSH traffic
# the ACCEPT term is needed to not drop all other traffic
set firewall family inet filter MIRROR-SSH term SSH-CLIENT from protocol tcp
set firewall family inet filter MIRROR-SSH term SSH-CLIENT from destination-port 22
set firewall family inet filter MIRROR-SSH term SSH-CLIENT then port-mirror
set firewall family inet filter MIRROR-SSH term SSH-CLIENT then accept
set firewall family inet filter MIRROR-SSH term SSH-SERVER from protocol tcp
set firewall family inet filter MIRROR-SSH term SSH-SERVER from source-port 22
set firewall family inet filter MIRROR-SSH term SSH-SERVER then port-mirror
set firewall family inet filter MIRROR-SSH term SSH-SERVER then accept
set firewall family inet filter MIRROR-SSH term ACCEPT then accept
 
# apply the filter on an interface in both directions
set interfaces ge-0/0/0 unit 0 family inet filter input MIRROR-SSH
set interfaces ge-0/0/0 unit 0 family inet filter output MIRROR-SSH

On an Arista switch, we can achieve the same with the few config lines below. Be aware that IP routing must be enabled for this to work.

Arista remote port mirror

ip access-list SSH
permit tcp any any eq 22
permit tcp any eq 22 any
 
monitor session 1 source Ethernet1 rx ip access-group SSH
monitor session 1 destination tunnel mode gre source 10.100.100.3 destination 10.100.100.1

More details and options for remote port mirroring on Arista EOS can be found in this comprehensive article on EOS Central: https://eos.arista.com/advanced-mirroring-features/#Mirroring_To_GRE (login required).

If remote mirroring is used in the context of VXLAN the GRE tunnel destination must be reachable via the underlay. At least this is the case on Juniper QFX5120 and Arista DCS-7050X3 switches which are both based on the Broadcom Trident 3 ASIC.

Arista’s EOS has another nice option that is very useful for ad-hoc debugging: using tcpdump which is available directly on the switch. Of course, the limitations mentioned at the beginning of the article still applies and tcpdump on the switch sees only control plane traffic. However, there is a little twist! The destination of a monitoring session can not only be an interface or a tunnel. The destination can also be set to “CPU”. With that, the tcpdump program local on the switch can be used to inspect the traffic. It is clear that this only makes sense for very low volume traffic.

Arista mirror to cpu

monitor session 1 destination CPU
 
tcpdump monitor 1

Using these techniques, problems can be debugged much quicker compared to asking server administrators on both ends of a connection to do packet captures for us. Of course, it is also quicker than having to go to the remote data centre in order to connect a notebook with Wireshark to a mirror port.