transferspeed from memory to device to low

[RMGBELGIUM]

Hi all,

I've got a weird problem with one of our machines:
Since we did a migration installation from AIX 4.3.3 to AIX5.2 we saw how the speed from our backups decreased from about 40 Mb /sec ( over gigabit ) to about 5 Mb /sec.After some testing , we discovered that if we do a copy of a file from one filesystem to another we also get 5 Mb /sec.But the strange thing is : reading a file , we get 20 Mb /sec !
So every transfer from memory to either disk or tapedevice, we only get 5 Mb /sec.Anyone seen this kind of behaviour before?

output from svmon -G :

size inuse free pin virtual
memory 1572864 1572724 140 131589 1971778
pg space 1826816 1427037

work pers clnt lpage
pin 131585 0 0 0
in use 1216999 354376 1349 0


Thx in advance,

greetz

RMGBelgium

 

[sweetdaddyd]

I think we need to isolate the different scenarios. It sounds like you're doing backups over the network. Are the filesystem moves on the same system or does it involve nfs? What does iostat and paging look like? Did the network settings change after the upgrade? Is it local SCSI disk or a SAN? Is the filesystem transfer on the same disk, or disk1 to disk2? You need to isololate network from local, and disk to disk from same disk.

 

[plamb]

what ML of 5.2 ML-07 has a problem with backups there is a APAR for it IY62852

 

[RMGBELGIUM]

The network troughput gives a speed of 45 Mb /sec (ftp in combination with dd), so the problem doesn't lie there.The problem lies with every disk ( internal as well as SSA disks).There are no nfs mounts or exports on this server.
We have a Tape library direct attached to this server, and even on this server , the backups go slow.
We did tests with the dd command, and we saw that the speed increased when the blocksize was raised from 32k blocks to 320k blocks.

This is the output of vmstat 1

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
3 2 1928610 123 0 2 2 1402 624 0 1098 39872 2614 39 34 12 16
1 2 1928617 253 0 1 0 1777 4665 0 851 33085 1876 5 27 29 39
1 2 1928617 256 0 1 0 1615 4455 0 859 31515 1938 9 26 27 38
1 2 1928693 242 0 3 0 1640 4541 0 919 36741 2351 22 28 18 33
1 2 1928693 230 0 2 0 1774 4586 0 855 32722 2783 4 28 32 35
1 2 1928693 158 0 2 0 1614 5433 0 884 30654 1705 5 27 27 41
1 2 1928693 267 0 1 0 1776 6610 0 847 32383 2675 4 28 29 39
1 2 1928693 258 0 2 0 1618 10630 0 862 31964 2528 5 26 26 42


and iostat :

hdisk6 0.0 0.0 0.0 0 0
hdisk5 2.0 12.0 3.0 12 0
hdskB112 0.0 0.0 0.0 0 0
hdskB109 86.8 4393.0 80.8 4404 0
hdskB114 0.0 0.0 0.0 0 0
hdskB110 0.0 0.0 0.0 0 0
hdskB107 0.0 0.0 0.0 0 0
hdskB106 0.0 0.0 0.0 0 0
hdskB104 0.0 0.0 0.0 0 0
hdskB111 0.0 0.0 0.0 0 0
hdskB115 0.0 0.0 0.0 0 0
hdskB113 0.0 0.0 0.0 0 0
hdskB103 0.0 0.0 0.0 0 0
hdskG204 89.8 373.1 103.7 132 242
hdskB116 0.0 0.0 0.0 0 0
hdskG206 3.0 31.9 4.0 0 32
hdskB101 0.0 0.0 0.0 0 0
hdskH204 60.8 241.4 70.8 0 242
hdskH208 0.0 0.0 0.0 0 0
hdskH206 3.0 31.9 4.0 0 32
hdskG208 0.0 0.0 0.0 0 0
hdskH201 4.0 39.9 5.0 0 40
hdskG201 5.0 47.9 7.0 8 40
hdskG202 0.0 0.0 0.0 0 0
hdskH203 1.0 8.0 1.0 0 8
hdskG203 1.0 12.0 2.0 4 8
hdskH202 0.0 0.0 0.0 0 0
hdskH103 0.0 0.0 0.0 0 0
hdskC412 0.0 0.0 0.0 0 0
hdskH101 17.0 4002.0 69.8 4012 0
hdskG205 0.0 0.0 0.0 0 0
hdskH205 0.0 0.0 0.0 0 0
hdskB312 0.0 0.0 0.0 0 0
hdskC409 0.0 0.0 0.0 0 0
hdskB309 0.0 0.0 0.0 0 0
cd0 0.0 0.0 0.0 0 0

This looks all quite normal...

We cannot install a ML for the moment, because /usr is filled for 96 % and the disks of rootvg are plenty full, so we cannot expand ...
Does 5.2 have ML07? I tought the last one was 05?

greetz,

R.

 

[AIX5L]

Hi

What are your Performance Tunning Settings before ?
Its possible that they where lost after migration ?
In AIX 4.3.3 the settings will be made over vmtune and must be made manually set after each boot by script.
In AIX 5.2 your settings will be in /etc/tunables/nextboot

what shows the putput of: vmstat -v, entstat -d entx

 

[RMGBELGIUM]

The machine has been tuned right,we set our parameters in the startupscripts we use. Mirror write consistency has been disabled, there arent to many hotspots, and there are enough buffers for LVM

here's the output from vmo -a

memory_frames = 1572864
pinnable_frames = 1441262
maxfree = 200
minfree = 120
minperm% = 10
minperm = 148819
maxperm% = 20
maxperm = 297639
strict_maxperm = 0
maxpin% = 80
maxpin = 1258292
maxclient% = 20
lrubucket = 131072
defps = 1
nokilluid = 0
numpsblks = 1826816
npskill = 14272
npswarn = 57088
v_pinshm = 0
pta_balance_threshold = 50
pagecoloring = 0
framesets = 2
mempools = 1
lgpg_size = 0
lgpg_regions = 0
num_spec_dataseg = n/a
spec_dataseg_int = n/a
memory_affinity = n/a
htabscale = -1
force_relalias_lite = 0
relalias_percentage = 0


ioo -a :

minpgahead = 2
maxpgahead = 16
pd_npages = 65536
maxrandwrt = 0
numclust = 4
numfsbufs = 256
sync_release_ilock = 1
lvm_bufcnt = 9
j2_minPageReadAhead = 2
j2_maxPageReadAhead = 128
j2_nBufferPerPagerDevice = 512
j2_nPagesPerWriteBehindCluster = 32
j2_maxRandomWrite = 0
j2_nRandomCluster = 0
jfs_clread_enabled = 0
jfs_use_read_lock = 1
hd_pvs_opn = 35
hd_pbuf_cnt = 2496


iostat -s gives :

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk6 4.3 46.0 9.0 25003188 62490287
hdisk5 6.5 62.7 11.7 56785712 62376127
hdskB112 4.4 142.5 5.8 255616531 15280347
hdskB109 10.6 197.1 13.2 266102650 108444442
hdskB114 9.4 196.6 11.7 246690735 127032735
hdskB110 2.7 54.2 3.3 93409299 9687215
hdskB107 7.3 132.6 9.5 131803199 120280455
hdskB106 4.0 79.6 4.3 41871883 109437479
hdskB104 10.0 147.6 12.1 65375987 215250631
hdskB111 12.4 208.9 15.6 283573355 113525631
hdskB115 6.3 174.9 7.9 322695027 9694219
hdskB113 6.9 204.0 8.7 371952915 15723555
hdskB103 13.0 242.7 16.2 406034599 55359543
hdskG204 33.0 199.1 42.6 153593371 224780704
hdskB116 0.1 6.6 0.1 5870651 6699111
hdskG206 11.7 263.2 21.0 361040291 139249580
hdskB101 1.6 19.6 2.2 28515023 8672403
hdskH204 26.9 151.3 32.6 62820147 224780704
hdskH208 6.7 98.1 15.6 146693043 39705895
hdskH206 8.9 177.4 11.4 197889492 139249580
hdskG208 2.2 38.8 4.1 34130195 39705895
hdskH201 5.0 121.4 6.0 181351808 49450424
hdskG201 7.5 214.1 15.0 357437500 49450424
hdskG202 20.0 1229.1 50.5 1841943496 494221416
hdskH203 5.3 208.1 8.0 349039016 46599721
hdskG203 8.1 315.4 16.2 552817964 46599721
hdskH202 13.6 860.7 24.8 1141696464 494221416
hdskH103 12.5 551.4 22.7 840264827 207724531
hdskC412 0.0 1.8 0.0 163448 3344107
hdskH101 26.1 680.5 42.2 598897659 694650050
hdskG205 0.1 5.4 0.4 9660236 590608
hdskH205 0.2 2.2 0.3 1356028 2760124
hdskB312 0.0 1.9 0.1 263080 3344107
hdskC409 0.1 3.9 0.1 1448268 6007063
hdskB309 0.1 4.4 0.1 2348732 6007063
cd0 0.0 0.0 0.0 0 0



vmstat -v:

1572864 memory pages
1488196 lruable pages
172 free pages
1 memory pools
131558 pinned pages
80.1 maxpin percentage
10.0 minperm percentage
20.0 maxperm percentage
21.2 numperm percentage
315889 file pages
0.0 compressed percentage
0 compressed pages
0.0 numclient percentage
20.0 maxclient percentage
1242 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
877618 paging space I/Os blocked with no psbuf
126641 filesystem I/Os blocked with no fsbuf
7953 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf



greetz,

R.