TCP retransmissions: when you should start worrying

Message

Vieri · Post by **Vieri** » Thu Nov 24, 2022 2:09 pm

Hi,

I have a large wired network, and some applications/hosts are experiencing some intermittent issues which *could* be network-related.
I'm trying to determine if it's the case or not.

If I tcpdump traffic from and to a specific host on a Gentoo router and view the pcap in wireshark, I can see quite a few TCP Retransmission messages. These are not necessarily a problem unless there are a lot, I suppose. They usually tell that there might be network congestion at a given point.

What should I be looking for to actually see if there is a case of serious network congestion?

I might want to tcpdump broadcast and ARP and check how much that amounts to because I believe too many broadcasts might lead to congestion.

Any command-line examples to actually do this?

I have a lot of interfaces (vlans) in the Gentoo router, and a command lilke this does not work:

Code: Select all

tcpdump -n -i any arp or ether broadcast or ether multicast

I need to specify an interface at a time:

Code: Select all

tcpdump -n -i lan arp or ether broadcast or ether multicast

Still, what should I do next?
Or should I dump everything on one interface with:

Code: Select all

tcpdump -n -i lan

and then use a program like Wireshark to see the ratio of broadcasts vs. rest of traffic? (not that I know if wireshark can do that)

Or should I be looking for something totally different?

Thanks

[EDIT]
I also see messages in Wireshark such as:

Code: Select all

Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)

but I don't know if this is just a Wireshark "thing" or if I should really worry about it.

mike155 · Post by **mike155** » Thu Nov 24, 2022 4:03 pm

Code: Select all

# netstat -s
...
Tcp:
    1153 active connection openings
    0 passive connection openings
    120 failed connection attempts
    52 connection resets received
    4 connections established
    164657 segments received
    158647 segments sent out
    15 segments retransmitted     <=====
    0 bad segments received
    702 resets sent
...

Run the command below on one of the computers connected to the network and generate some network traffic to another computer on the network:

Code: Select all

while [ 1 ]; do netstat -s | grep "retransmitted"; sleep 10; done

Does the counter increase if there's load on the network or on the router?

Post by **pietinger** » Thu Nov 24, 2022 6:29 pm

In addition to the very good recommendation from @mike155:

Every station in a network can discard packets if it is overloaded - even a simple switch where you connect 20 1-gbit/s-clients to a 10-gbit/s uplink (seldom! but maybe you have many 2,5-gbit/s clients ?). What I would do at first is checking net statistic (dropped packets) of all my layer-2 devices; then all my routers. Mostly it is a server which drops packets because it is overloaded ... but it can be a router also. You will see many retransmits of your server if your router has many dropped packets ...

Vieri · Post by **Vieri** » Fri Nov 25, 2022 3:08 pm

On the Gentoo router (default gateway) I have this:

Code: Select all

# netstat -s
Ip:
    Forwarding: 1
    340622918108 total packets received
    782168 with invalid headers
    319435421108 forwarded
    0 incoming packets discarded
    18461424204 incoming packets delivered
    338920370435 requests sent out
    153122819 outgoing packets dropped
    1711 dropped because of missing route
    21960 fragments dropped after timeout
    593721714 reassemblies required
    241094008 packets reassembled ok
    21960 packet reassemblies failed
    241265510 outgoing packets fragmented ok
    7 outgoing packets failed fragmentation
    594921321 fragments created
Icmp:
    30302568 ICMP messages received
    1064337 input ICMP message failed
    InCsumErrors: 484
    ICMP input histogram:
        destination unreachable: 15934018
        timeout in transit: 2526
        redirects: 4090
        echo requests: 10106034
        echo replies: 4255373
        timestamp reply: 4
        address mask request: 39
    656125390 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 369032178
        time exceeded: 154734
        redirect: 259141718
        echo requests: 18748977
        echo replies: 9047778
        timestamp requests: 5
IcmpMsg:
        InType0: 4255373
        InType3: 15934018
        InType5: 4090
        InType8: 10106034
        InType11: 2526
        InType14: 4
        InType17: 39
        OutType0: 9047778
        OutType3: 369032178
        OutType5: 259141718
        OutType8: 18748977
        OutType11: 154734
        OutType13: 5
Tcp:
    708226141 active connection openings
    712176862 passive connection openings
    435204 failed connection attempts
    473348564 connection resets received
    68 connections established
    17704266919 segments received
    21290409184 segments sent out
    26050675 segments retransmitted
    513294 bad segments received
    122752926 resets sent
    InCsumErrors: 727
Udp:
    1016787415 packets received
    990014 packets to unknown port received
    95556 packet receive errors
    710875521 packets sent
    92369 receive buffer errors
    0 send buffer errors
    InCsumErrors: 3173
    IgnoredMulti: 241510
UdpLite:
TcpExt:
    441 SYN cookies sent
    9 SYN cookies received
    123 invalid SYN cookies received
    424605 resets received for embryonic SYN_RECV sockets
    290655 packets pruned from receive queue because of socket buffer overrun
    8000 packets dropped from out-of-order queue because of socket buffer overrun
    170 ICMP packets dropped because they were out-of-window
    ArpFilter: 5163794
    628375764 TCP sockets finished time wait in fast timer
    135 time wait sockets recycled by time stamp
    11089 packets rejected in established connections because of timestamp
    15279320 delayed acks sent
    57298 delayed acks further delayed because of locked socket
    Quick ack mode was activated 160775870 times
    2977 times the listen queue of a socket overflowed
    3336 SYNs to LISTEN sockets dropped
    6776959392 packet headers predicted
    2673706502 acknowledgments not containing data payload received
    3024407425 predicted acknowledgments
    TCPSackRecovery: 1844055
    TCPSACKReneging: 472
    Detected reordering 1997742 times using SACK
    Detected reordering 74 times using reno fast retransmit
    Detected reordering 55370 times using time stamp
    56225 congestion windows fully recovered without slow start
    51806 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 694632
    28907 congestion windows recovered without slow start after partial ack
    TCPLostRetransmit: 1202349
    TCPSackFailures: 38328
    4790 timeouts in loss state
    5262732 fast retransmits
    344812 retransmits in slow start
    TCPTimeouts: 16353199
    TCPLossProbes: 6581136
    TCPLossProbeRecovery: 246467
    TCPSackRecoveryFail: 14586
    3255 packets collapsed in receive queue due to low socket buffer
    TCPBacklogCoalesce: 190441994
    TCPDSACKOldSent: 160814198
    TCPDSACKOfoSent: 7149
    TCPDSACKRecv: 3029747
    TCPDSACKOfoRecv: 6830
    15736882 connections reset due to unexpected data
    71942239 connections reset due to early user close
    364741 connections aborted due to timeout
    TCP ran low on memory 16 times
    TCPMemoryPressuresChrono: 200503487
    TCPSACKDiscard: 4374
    TCPDSACKIgnoredOld: 8976
    TCPDSACKIgnoredNoUndo: 1665298
    TCPSpuriousRTOs: 14769
    TCPSackShifted: 4252824
    TCPSackMerged: 4083854
    TCPSackShiftFallback: 5019340
    TCPDeferAcceptDrop: 11835890
    TCPReqQFullDoCookies: 441
    TCPRetransFail: 11888472
    TCPRcvCoalesce: 2555536310
    TCPOFOQueue: 5723614
    TCPOFODrop: 144062
    TCPOFOMerge: 6896
    TCPChallengeACK: 545874
    TCPSYNChallenge: 512688
    TCPFastOpenCookieReqd: 2
    TCPSpuriousRtxHostQueues: 63756
    TCPAutoCorking: 140202401
    TCPFromZeroWindowAdv: 307478
    TCPToZeroWindowAdv: 307480
    TCPWantZeroWindowAdv: 5163533
    TCPSynRetrans: 13733423
    TCPOrigDataSent: 11278752122
    TCPHystartTrainDetect: 8932636
    TCPHystartTrainCwnd: 182197280
    TCPHystartDelayDetect: 342347
    TCPHystartDelayCwnd: 8224143
    TCPACKSkippedSynRecv: 544
    TCPACKSkippedPAWS: 59
    TCPACKSkippedSeq: 240473
    TCPACKSkippedTimeWait: 21
    TCPACKSkippedChallenge: 13596
    TCPWinProbe: 2388351
    TCPKeepAlive: 10596
    TCPDelivered: 11807908890
    TCPAckCompressed: 230931
    TCPZeroWindowDrop: 64
    TCPRcvQDrop: 107199
    TcpTimeoutRehash: 13608283
    TcpDuplicateDataRehash: 130972909
    TCPDSACKRecvSegs: 3036784
    TCPDSACKIgnoredDubious: 10806
IpExt:
    InNoRoutes: 884
    InTruncatedPkts: 40
    InMcastPkts: 16547
    InBcastPkts: 239723311
    OutBcastPkts: 4134794
    InOctets: 417732533460804
    OutOctets: 820191038732917
    InMcastOctets: 543856
    InBcastOctets: 58763383622
    OutBcastOctets: 895762011
    InCsumErrors: 88
    InNoECTPkts: 457506382357
    InECT1Pkts: 12415218
    InECT0Pkts: 35748768951
    InCEPkts: 5692
MPTcpExt:

Still on the same Gentoo router I that ther are

26050677 segments retransmitted

, and about 10 minutes later (very active LAN) I have:

26050687 segments retransmitted

So, if I understand right you are suggesting I run something like on, say, my Gentoo router (or does it have to be a host within the LAN?):

watch -n 10 'netstat -s | grep "retransmitted"'

and from the same host where I'm monitoring the retransmissions (eg. same host where I'm running the above command) I should send data to another host in the LAN.

Correct?

Most of the LAN hosts are Windows, and I see that there's a netstat -s command there too. So I first ran it and got a reading of 42127 retransmissions (TCP IPv4).
I then generated traffic from this Windows host to a Windows server in the network (sent 3.5GB of data via SMB/CIFS) and finally read the stats again and got 50643 retransmissions (TCP IPv4).

So, 50643 - 42127 = 8516 retransmissions. Is that "a lot"?

If it's "bad" what should I be doing next?

Say I wanted to search for "dropped packets" in my Gentoo router... There are lots of dropped packets because it's also a firewall.

On the Gentoo router/gateway I ran ths if it's of any use:

Code: Select all

# ifconfig | grep -i drop
        RX errors 0  dropped 3319  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 3  dropped 9893  overruns 0  frame 11
        TX errors 0  dropped 9 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 6 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 87808  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 3 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 8
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 1  dropped 15882  overruns 0  frame 1
        TX errors 0  dropped 20 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 658074  overruns 0  frame 0
        TX errors 0  dropped 19 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 420362  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 11882  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 30  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 15134  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 3  dropped 0  overruns 0  frame 3
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 6277  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 1  dropped 3957  overruns 0  frame 1
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 5928  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 0  dropped 0  overruns 0  frame 0
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        RX errors 3  dropped 0  overruns 0  frame 3
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lan.1 being the interface with the most traffic:

Code: Select all

# ifconfig lan | grep -i drop
        RX errors 1  dropped 15882  overruns 0  frame 1
        TX errors 0  dropped 20 overruns 0  carrier 0  collisions 0
# ifconfig lan.1 | grep -i drop
        RX errors 0  dropped 658074  overruns 0  frame 0
        TX errors 0  dropped 19 overruns 0  carrier 0  collisions 0

Does anyone here know how to search for dropped packets in Cisco C9300 switches or in a C9500 or Nexus core?