Network Interrupts & Packet retransmitted

[PTameris]

Hi,

I have encountered a lot of network issues on our networkcard.

We have a IBM P-series server & AIX 4.3.3
The networkcard is a Gigabit Ethernet-SX PCI Adapter (14100401)

When we look at "netstat -p tcp"; and calculate packet retransmitted/data packacts sent it is more than 17% and according to the performance guide it's far too high (must be <5-10%)

And when we look at "entstat ent0"; i see a lot of interupt errors.....

Questions:
- What is wrong in this picture....
- How can we see with iptrace&ipreport to which host the retransmissions occur...
- and how to solve this problem....

Greetings
Peter Tameris

# netstat -p tcp
tcp:
955077047 packets sent
!!! 48820851 data packets (1495810456 bytes)
!!! 8278043 data packets (774221955 bytes) retransmitted
192305895 ack-only packets (105170593 delayed)
2 URG only packets
0 window probe packets
643526915 window update packets
61935862 control packets
3246753075 packets received
713606831 acks (for 1594673645 bytes)
51974866 duplicate acks
0 acks for unsent data
2075173390 packets (1904858712 bytes) received in-sequence
310072 completely duplicate packets (7547440 bytes)
1 old duplicate packet
2675 packets with some dup. data (139617 bytes duped)
24007438 out-of-order packets (72816315 bytes)
14 packet (1 bytes) of data after window
1 window probe
47288047 window update packets
625 packets received after close
0 packets with bad hardware assisted checksum
13594 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
18495075 connection requests
25901727 connection accepts
43208798 connections established (including accepts)
47935689 connections closed (including 81559 drops)
1187426 embryonic connections dropped
4145695374 segments updated rtt (of 3139196528 attempts)
0 resends due to path MTU discovery
0 path MTU discovery terminations due to retransmits
1451593 retransmit timeouts
253 connections dropped by rexmit timeout
212583 persist timeouts
533468 keepalive timeouts
8647 keepalive probes sent
377276 connections dropped by keepalive
0 time SACK blocks array is extended
0 time SACK holes array is extended
0 packet dropped due to memory allocation failure
220547 connections in timewait reused
0 delayed ACKs for SYN
0 delayed ACKs for FIN
0 send_and_disconnects

###########################################################

# entstat ent0
-------------------------------------------------------------
ETHERNET STATISTICS (ent0) :
Device Type: Gigabit Ethernet-SX PCI Adapter (14100401)
Hardware Address: 00:02:55:9a:32:f4
Elapsed Time: 149 days 16 hours 59 minutes 3 seconds

Transmit Statistics: Receive Statistics:
-------------------- -------------------
Packets: 35185646630 Packets: 3157007546
Bytes: 22904465733062 Bytes: 35948964629104
Interrupts: 364210321 Interrupts: 18941823101
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 0 Packets Dropped: 0
Bad Packets: 0
Max Packets on S/W Transmit Queue: 267
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 34

Broadcast Packets: 42534 Broadcast Packets: 32002446
Multicast Packets: 2 Multicast Packets: 0
No Carrier Sense: 0 CRC Errors: 0
DMA Underrun: 0 DMA Overrun: 0
Lost CTS Errors: 0 Alignment Errors: 0
Max Collision Errors: 0 No Resource Errors: 0
Late Collision Errors: 0 Receive Collision Errors: 0
Deferred: 0 Packet Too Short Errors: 0
SQE Test: 0 Packet Too Long Errors: 0
Timeout Errors: 0 Packets Discarded by Adapter: 0
Single Collision Count: 0 Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 34

General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 2000
Driver Flags: Up Broadcast Running
Simplex AlternateAddress 64BitSupport
PrivateSegment DataRateSet

 

[bjverzal]

The 'Errors' and 'Dropped' counters are all zero or near zero. Are you seeing entries in the errpt ?

Remember that the performance guide is often a 'best case' scenario. Are you using 'Jumbo Frame' support? Is that even offered in AIX 4.3.3 ?

Also, consider that this is over 149 days. Divide the number of packets by 149 days and I think you won't be so alarmed. Also, look at changes over the next 7-days. Perhaps all of the counters indicate issues from a 'long' time ago.

BV

 

[PTameris]

Hi Bjverzal,

No we see no errors in the errpt...
No we are not using Jumbo Frames.. We tried it but with our network cards the network went down if we down/uploaded very big file's... we never discovered why....

No we have still retransmitted packages...
# netstat -p tcp
tcp:
1222660811 packets sent
273087217 data packets (1875971642 bytes)
8282201 data packets (779115865 bytes) retransmitted

so in one day we have again 4158 packages retransmitted...
or is it not very much....?


# entstat ent0
-------------------------------------------------------------
ETHERNET STATISTICS (ent0) :
Device Type: Gigabit Ethernet-SX PCI Adapter (14100401)
Hardware Address: 00:02:55:9a:32:f4
Elapsed Time: 150 days 13 hours 57 minutes 11 seconds

Transmit Statistics: Receive Statistics:
-------------------- -------------------
Packets: 35432215988 Packets: 3465000365
Bytes: 23048130493996 Bytes: 36224747233723
Interrupts: 366801144 Interrupts: 19077368763
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 0 Packets Dropped: 0
Bad Packets: 0
Max Packets on S/W Transmit Queue: 267
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 12

and so 2590823 Sent interrupts
and so 135545662 Receive interrupts

Is this high? for me it is because it's more than 0...

Greetings
Peter

 

[bjverzal]

I don't think you have a problem on your hands at all. Let it go. IMHO.

BV