This week we had an issue at our main campus where apps that use the WAN were being reported as “slow”. Its right of passage being told that the “network is slow” and it usually ends up being up to the Network Engineer to prove otherwise.
In this specific instance the network was slow but it wasn’t our fault! Let me entertain you with my troubleshooting tale:

Our main campus WAN is a 1 gbps MPLS circuit that is managed by a third party. So once I was told that specific apps were slow (a records system and a student management system both of which connect to a datacentre across the WAN) I went and checked them for myself, yep valid performance issues. OK so whats next? Check the WAN uplink, yep its up/up @ 1gbps, no runts, CRC errors and the interface has not been reset or flapped recently. Check the Riverbed, service running no errors in the logs, it’s working. OK try to download some files off the Internet, hmm only getting around 5-10mbit using fast.com. Now where does the issue lie? We can try a real time data transfer in both directions with iperf3 to work this out.

1. You will need two hosts one on your campus LAN and one on a subnet that is traversable across the WAN. I am using two Ubuntu Linux VMs for this exercise. LAN host is 192.168.10.1 and the remote WAN host is 192.168.20.1
2. On both your Linux hosts install iperf3:
sudo apt install iperf3
3. First off lets try an outbound transfer (upload):
Set up the iperf3 server on the far end (across the WAN):
sudo iperf3 -s
iperf3 will listen for connections on tcp port 5201 by default. You can change this with the -p switch.
4. Now setup the client side:
sudo iperf3 -c 192.168.20.1
Connecting to host 192.168.20.1, port 5201
[ 4] local 192.168.10.1 port 62323 connected to 192.168.20.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 47.4 MBytes 397 Mbits/sec
[ 4] 1.00-2.00 sec 45.1 MBytes 379 Mbits/sec
[ 4] 2.00-3.00 sec 46.1 MBytes 387 Mbits/sec
[ 4] 3.00-4.00 sec 53.1 MBytes 446 Mbits/sec
[ 4] 4.00-5.01 sec 47.6 MBytes 394 Mbits/sec
[ 4] 5.01-6.00 sec 51.4 MBytes 437 Mbits/sec
[ 4] 6.00-7.03 sec 42.0 MBytes 342 Mbits/sec
[ 4] 7.03-8.00 sec 46.0 MBytes 397 Mbits/sec
[ 4] 8.00-9.00 sec 48.4 MBytes 406 Mbits/sec
[ 4] 9.00-10.00 sec 26.6 MBytes 223 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 454 MBytes 381 Mbits/sec sender
[ 4] 0.00-10.00 sec 454 MBytes 380 Mbits/sec receiver

OK so we can see here that the upload (outbound) is working fine and we are getting the throughput as expected.

5. Now lets switch the roles around and try the download (inbound). On the LAN side change to server:
ctrl+c
sudo iperf3 -s
6. Set the WAN host as the client side to simulate a download (inbound traffic):
sudo iperf3 -c 192.168.10.1
Connecting to host 192.168.10.1, port 5201
[ 4] local 192.168.20.1 port 56880 connected 192.168.10.1 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.33 MBytes 11.1 Mbits/sec 91 1.34 KBytes
[ 4] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 1 1.34 KBytes
[ 4] 2.00-3.00 sec 216 KBytes 1.77 Mbits/sec 35 4.01 KBytes
[ 4] 3.00-4.00 sec 148 KBytes 1.21 Mbits/sec 14 9.35 KBytes
[ 4] 4.00-5.00 sec 319 KBytes 2.62 Mbits/sec 43 4.01 KBytes
[ 4] 5.00-6.00 sec 882 KBytes 7.22 Mbits/sec 73 8.02 KBytes
[ 4] 6.00-7.00 sec 963 KBytes 7.89 Mbits/sec 75 5.34 KBytes
[ 4] 7.00-8.00 sec 625 KBytes 5.12 Mbits/sec 50 4.01 KBytes
[ 4] 8.00-9.00 sec 647 KBytes 5.30 Mbits/sec 62 2.67 KBytes
[ 4] 9.00-10.00 sec 465 KBytes 3.81 Mbits/sec 24 4.01 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 5.49 MBytes 4.61 Mbits/sec 468 sender
[ 4] 0.00-10.00 sec 5.37 MBytes 4.51 Mbits/sec receiver

And we have a winner! The inbound side of the WAN connection had a throughput issue. In this particiluar instance the WAN carriers last mile provider had accidently messed up a port config limiting the inbound of the 1gbps link to > 100mbit. By using iperf3 I was able to provide our ISP support valid troubleshooting to raise with the carrier. I try to go through the OSI layers when troubleshooting an issue like this. I checked the physical link and convergence then go on to routed connectivity and layer 4.

As a side note you can also merge the above with one connection using the -R flag in iperf3: sudo iperf3 -c 192.168.20.1 -R
This will initate the data upload from client to server as previously described. Then it will do the reverse connection from server towards the client and print the results.