Too many TCP RETRANSMITS!

[dfusion]

I am monitoring traffic to/from our AS400 server and there is a huge amount of TCP retransmits. The AS400 is plugged into a Foundry switch and I've hard-fixed the port for 100-full and the NIC is 100-full. I've even tried them at 100-half, but the same results occur. The majority of traffic to this server is from a Stratus (VOS) machine, although other devices are re-transmitting to this machine as well.

I'm not looking for anyone to wave a magic wand to solve my problems, but any advice at trouble shooting would be great. Interface utilization on the switch port is only 1% and never jumps higher than 1.5%. I cannot understand why there are so many re-transmits. It's crazy!

I appreciate the help.


Sr. Network Engineer
ArcLight Systems, LLC

http://www.markholloway.com

 

[Whoheard]

Have you checked the stats on the box in question? It could be slow...yes? Have you tried another port on the switch to determine if there are other retransmits comingfrom them as well?

Also, have you tried adjusting the Retransmit threshold on the sniffer?


Just checking.

bob


I know what I know and that's all I know.

 

[dfusion]

I have tried alternate switch ports and get the same results. I have not adjusted the thresholds yet. I thought that Sniffer knew whether the packet was a re-transmit or not. I've looked at the utilization of the interface on the AS400 using SNMP, and utilization is 1Mbit.

Sr. Network Engineer
ArcLight Systems, LLC

http://www.markholloway.com

 

[laobancha]

I've experience a case. When its duplex & speed mismatch occurs, especially for switch-switch connection, you may find many errors on those switch stat ports and many retransmissions in tcp maybe occurs. And that may cause application slow problem eventually.

I think
for my way,
I'd like to define the traffic path (what application flow is) first.
Then find out no problem at data-link layer along that path (switches, routers) probably look at their stats.
Find out what causes those retransmission: in case no contention along the path (low utilization)
so check it retransmits 'coz of packet loss or delay.
in case packets come between client and server across WAN links, probably its delay, congestion, or link stability.

i think maybe check and prove each assumption case by case.

just to share you my opion.

bar

 

[dfusion]

I'm seeing the same sequence number about eight times for a single packet. Ugh!

Both servers are 100TX, for duplex, connected to a Foundry Big Iron 4000 on the same ethernet blade, and the ports are set for 100 FULL. There are no CRC errors on the switch ports.

Sr. Network Engineer
ArcLight Systems, LLC

http://www.markholloway.com

 

[rgroulx]

Mark,

Hmmmm...doesn't sound like a Layer 2 problem.

Believe it or not, sometimes legacy apps are happier running at slower speeds. Try reducing the port down to 10/half and see what happens.

Ron

"Hurry up and wait!"

 

[dfusion]

I am performing a capture and I do not see a three way handshake. I do know the data is a continuous stream coming from one of our partners over the WAN. They are sending us data 24/7. The pattern I am seeing so far is when the AS400 is put under heavy load with other functions, this problem arises. It only happens during the day and tapers off around 4:00pm pacific time.

The MTU size of the AS400 is 1460 and the MTU size of the Stratus is 1500. However, the packets are just 500 - 600 bytes, so I don't think that is the problem.

This is frustrating. *sigh*

Sr. Network Engineer
ArcLight Systems, LLC

http://www.markholloway.com

 

[tdelliott]

You may want to check the application to see if you can adjust the time between retransmissions. Some apps do not realize that the propagation delay with WAN connections will be greater than local LANs thus the problem is with the applications retransmitting because it does not believe it is getting a response within the proper timeframe.

 

[AlexTovar]

I've seen this problem very often, and I've noticed that AS400 keeps many TCP ports open, there's no "keep alive" messages but sessions don't close. I really donpt think the AS400 box is without resources because ot the TCP ports open. As Ron said, retransmissions had diminished when the link is changed to 10/100 half. Some times AS400 guys setup more than one NIC and it makes the problem arise. It has been solved with only one NIC card active.

This is what I've seen MANY MANY times.

Alex.

 

[AlexTovar]

I've seen this problem very often, and I've noticed that AS400 keeps many TCP ports open, there's no "keep alive" messages but sessions don't close. I really don't think the AS400 box is without resources because ot the TCP ports open. As Ron said, retransmissions had diminished when the link is changed to 10 or 100 half. Some times AS400 guys setup more than one NIC and it makes the problem arise. It has been solved with only one NIC card active.

This is what I've seen MANY MANY times.

Alex.

 

[RADIOMEN]

Does any one know what happened to this problem? If it was resolved?
I have the same problem using the same devices. Any suggestions?

Thanks,