FTP Issues

[linedancer888]

We have an applications analyst who has a script that ftp's from server A to server B, extracting data from server B, daily starting at 6am. Sometimes the ftp takes 15 minutes, other times it takes 3 hours. Other than at month-end, there is only a minimal difference in file sizes. We checked to make sure that both the aix side and network side are set to 100_full_duplex, which they are. We've had the network people put a sniffer on Server A and have found no issues. Is there anything else I need to check?

 

[khalidaaa]

hmm, I don't know about your environment but it could be something other than the network! It might be a busy server disk or RAM! It could be a CPU issue as well! it depends on how it is configured.

Did you check the performance of the box while it is ftping? If the disk is busy with something else then this might effect the data transfer!

 

[linedancer888]

We see no performance issues, I also run an entstat and check for any errors, which there are none.

 

[khalidaaa]

I doubt that you will see any errors in the entstat output because it is not a network problem as i mentioned! how about monitoring the disks while you run the ftp? i guess iostat (on both machines) will do the buz!

 

[pulsar1]

You might take a look at the setting for default gateway and check the route (traceroute) between the two systems. If a system has mutiple lan cards, I have sometime seen where a second default route is incorrectly added, and them sometimes used. If in doubt you could define a specific route between the two.

 

[linedancer888]

I did a traceroute from both systems and don't see any issues:

From egate1:
traceroute hueyii
trying to get source for hueyii
source should be 10.130.70.62
traceroute to hueyii (10.130.70.142) from 10.130.70.62 (10.130.70.62), 30 hops m
ax
outgoing MTU = 1500
1 hueyii (10.130.70.142) 12 ms 9 ms 1 ms

from egate1_admin:
traceroute hueyii
trying to get source for hueyii
source should be 10.130.70.65
traceroute to hueyii (10.130.70.142) from 10.130.70.65 (10.130.70.65), 30 hops m
ax
outgoing MTU = 1500
1 hueyii (10.130.70.142) 2 ms 1 ms 1 ms

from hueyii:
traceroute egate1
trying to get source for egate1
source should be 10.130.70.142
traceroute to egate1 (10.130.70.62) from 10.130.70.142 (10.130.70.142), 30 hops
max
outgoing MTU = 1500
1 egate1 (10.130.70.62) 2 ms 1 ms 1 ms

from hueyii:
traceroute egate1_admin
trying to get source for egate1_admin
source should be 10.130.70.142
traceroute to egate1_admin (10.130.70.65) from 10.130.70.142 (10.130.70.142), 30
hops max
outgoing MTU = 1500
1 egate1_admin (10.130.70.65) 2 ms 1 ms 1 ms

 

[rzs0502]

Are you perhaps using Jumbo Frames?


"If you always do what you've always done, you will always be where you've always been."

 

[linedancer888]

no, we're not using Jumbo Frames.

 

[4star]

Check if there are some scheduled tasks that might be running at that particular time.Or check availability of source and destination when ftp is launched.

4star

 

[linedancer888]

We changed this job to run when there is very little activity.

A few things I did notice were dns related. The first was on the hueyii machine, when we did nslookup it returned heweyii instead of hueyii and 2nd was that when we did nslookup on egate1_admin, nothing returned.

 

[4star]

Hi,Try to put the ip address and hostname in the etc\host file on both Servers and see.

 

[khalidaaa]

do you have enough mbufs when ftping?

check mbufs denied in netstat -m

 

[linedancer888]

In both hosts file I have the IP addresses, the short name and the FQN.

On the system that's initiating the ftp, when I issue the netstat -m command I don't see any mbufs, everything is showed by size. On the system that we're ftp'ing to, when issuing the netstat -m, mbufs failed are 0.

 

[khalidaaa]

What is the os on both?

then i guess the other machine which doesn't have mbufs denied is an SMP system and it is disabled! Just a note for you, if you want to set this, you need to change the extendednetstats to 1!

no -o extendednetstats=1

what about the failed column on that machine!?! is there any nonzero value for that?

what is the output of netstat -v?

 

[linedancer888]

The os on egate1 is 5.2ml 05, hueyii is 5.2ML01.

The failed are all 0 except for CPU2 size 512 it's 4


netstat -v from egate1:
-------------------------------------------------------------
ETHERNET STATISTICS (ent1) :
Device Type: IBM 10/100 Mbps Ethernet PCI Adapter (23100020)
Hardware Address: 00:06:29:dc:1c:76
Elapsed Time: 16 days 21 hours 42 minutes 48 seconds

Transmit Statistics: Receive Statistics:
-------------------- -------------------
Packets: 2259625 Packets: 8456717
Bytes: 281453437 Bytes: 761051022
Interrupts: 69 Interrupts: 8406740
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 1 Packets Dropped: 193
Bad Packets: 0
Max Packets on S/W Transmit Queue: 86
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 1

Broadcast Packets: 1957 Broadcast Packets: 6207465
Multicast Packets: 2 Multicast Packets: 10
No Carrier Sense: 0 CRC Errors: 0
DMA Underrun: 0 DMA Overrun: 0
Lost CTS Errors: 0 Alignment Errors: 0
Max Collision Errors: 0 No Resource Errors: 0
Late Collision Errors: 0 Receive Collision Errors: 0
Deferred: 0 Packet Too Short Errors: 193
SQE Test: 0 Packet Too Long Errors: 0
Timeout Errors: 0 Packets Discarded by Adapter: 0
Single Collision Count: 0 Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 1

General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 200
Driver Flags: Up Broadcast Running
Simplex AlternateAddress 64BitSupport
PrivateSegment DataRateSet

IBM 10/100 Mbps Ethernet PCI Adapter Specific Statistics:
------------------------------------------------
Chip Version: 26
RJ45 Port Link Status : up
Media Speed Selected: 100 Mbps Full Duplex
Media Speed Running: 100 Mbps Full Duplex
Receive Pool Buffer Size: 384
Free Receive Pool Buffers: 128
No Receive Pool Buffer Errors: 0
Inter Packet Gap: 96
Adapter Restarts due to IOCTL commands: 1
Packets with Transmit collisions:
1 collisions: 0 6 collisions: 0 11 collisions: 0
2 collisions: 0 7 collisions: 0 12 collisions: 0
3 collisions: 0 8 collisions: 0 13 collisions: 0
4 collisions: 0 9 collisions: 0 14 collisions: 0
5 collisions: 0 10 collisions: 0 15 collisions: 0
Excessive deferral errors: 0x0
-------------------------------------------------------------
ETHERNET STATISTICS (ent2) :
Device Type: IBM 10/100 Mbps Ethernet PCI Adapter (23100020)
Hardware Address: 00:09:6b:7e:33:dc
Elapsed Time: 16 days 21 hours 42 minutes 44 seconds

Transmit Statistics: Receive Statistics:
-------------------- -------------------
Packets: 47005097 Packets: 9006521
Bytes: 11490519174 Bytes: 1062172104
Interrupts: 209706 Interrupts: 8966584
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 1 Packets Dropped: 28383
Bad Packets: 0
Max Packets on S/W Transmit Queue: 66
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 1

Broadcast Packets: 2925 Broadcast Packets: 6205994
Multicast Packets: 2 Multicast Packets: 10
No Carrier Sense: 0 CRC Errors: 28190
DMA Underrun: 0 DMA Overrun: 0
Lost CTS Errors: 0 Alignment Errors: 13743
Max Collision Errors: 0 No Resource Errors: 0
Late Collision Errors: 0 Receive Collision Errors: 0
Deferred: 0 Packet Too Short Errors: 193
SQE Test: 0 Packet Too Long Errors: 0
Timeout Errors: 0 Packets Discarded by Adapter: 0
Single Collision Count: 0 Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 1

General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 200
Driver Flags: Up Broadcast Running
Simplex AlternateAddress 64BitSupport
PrivateSegment DataRateSet

IBM 10/100 Mbps Ethernet PCI Adapter Specific Statistics:
------------------------------------------------
Chip Version: 26
RJ45 Port Link Status : up
Media Speed Selected: 100 Mbps Full Duplex
Media Speed Running: 100 Mbps Full Duplex
Receive Pool Buffer Size: 384
Free Receive Pool Buffers: 128
No Receive Pool Buffer Errors: 0
Inter Packet Gap: 96
Adapter Restarts due to IOCTL commands: 1
Packets with Transmit collisions:
1 collisions: 0 6 collisions: 0 11 collisions: 0
2 collisions: 0 7 collisions: 0 12 collisions: 0
3 collisions: 0 8 collisions: 0 13 collisions: 0
4 collisions: 0 9 collisions: 0 14 collisions: 0
5 collisions: 0 10 collisions: 0 15 collisions: 0
Excessive deferral errors: 0x0
-------------------------------------------------------------
ETHERNET STATISTICS (ent0) :
Device Type: IBM 10/100 Mbps Ethernet PCI Adapter (23100020)
Hardware Address: 00:02:55:89:d4:47
Elapsed Time: 13 days 1 hours 18 minutes 40 seconds

Transmit Statistics: Receive Statistics:
-------------------- -------------------
Packets: 3398426 Packets: 48338242
Bytes: 1097666688 Bytes: 8975479371
Interrupts: 9049 Interrupts: 47664477
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 0 Packets Dropped: 193
Bad Packets: 0
Max Packets on S/W Transmit Queue: 26
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 1

Broadcast Packets: 1190 Broadcast Packets: 4692430
Multicast Packets: 0 Multicast Packets: 24609
No Carrier Sense: 0 CRC Errors: 0
DMA Underrun: 0 DMA Overrun: 0
Lost CTS Errors: 0 Alignment Errors: 0
Max Collision Errors: 0 No Resource Errors: 0
Late Collision Errors: 0 Receive Collision Errors: 0
Deferred: 0 Packet Too Short Errors: 193
SQE Test: 0 Packet Too Long Errors: 0
Timeout Errors: 0 Packets Discarded by Adapter: 0
Single Collision Count: 0 Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 1

General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 200
Driver Flags: Up Broadcast Running
Simplex AlternateAddress 64BitSupport
PrivateSegment DataRateSet

IBM 10/100 Mbps Ethernet PCI Adapter Specific Statistics:
------------------------------------------------
Chip Version: 26
RJ45 Port Link Status : up
Media Speed Selected: 100 Mbps Full Duplex
Media Speed Running: 100 Mbps Full Duplex
Receive Pool Buffer Size: 384
Free Receive Pool Buffers: 126
No Receive Pool Buffer Errors: 0
Inter Packet Gap: 96
Adapter Restarts due to IOCTL commands: 0
Packets with Transmit collisions:
1 collisions: 0 6 collisions: 0 11 collisions: 0
2 collisions: 0 7 collisions: 0 12 collisions: 0
3 collisions: 0 8 collisions: 0 13 collisions: 0
4 collisions: 0 9 collisions: 0 14 collisions: 0
5 collisions: 0 10 collisions: 0 15 collisions: 0
Excessive deferral errors: 0x0

 

[khalidaaa]

I don't see any problems here on all adapters for this machine except the Broadcast Packets!

i can see that this is too high! As a rule of thumb this should be at less than 20 percent of the total packets received! Are you using Broadcast so often?

what about the vmstat -v on hueyii machine?

 

[linedancer888]

I don't know all that much about networks (as you can tell), so I don't understand about the Broadcase Packets.

vmstat -v
1441792 memory pages
1359151 lruable pages
120 free pages
1 memory pools
95292 pinned pages
80.1 maxpin percentage
20.0 minperm percentage
80.0 maxperm percentage
80.3 numperm percentage
1091774 file pages
0.0 compressed percentage
0 compressed pages
0.0 numclient percentage
80.0 maxclient percentage
0 client pages
0 remote pageouts scheduled
6854 pending disk I/Os blocked with no pbuf
258952 paging space I/Os blocked with no psbuf
11179553 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf

 

[khalidaaa]

I think this is an indication of a name resolution problem! because the DNS is not used or something!

I read above that you had a problem in the nslookup or so!?!

did you check the /etc/environment file for resolution sequence?

 

[khalidaaa]

oh sorry i meant netstat -v not vmstat -v! my mistake!

could you please show the netstat -v on hueyii ?

 

[linedancer888]

yes, there were a few issues with the name resolution, but I had them fixed, the last one I found was fixed yesterday. I did check the resolv.conf and netsvc.conf and both look ok, I had to add the netsvc.conf to the hueyii machine. hosts=local,bind4