heavy "wait" after upgrade from 4.3 to 5.2

[noober]

Hi all. I upgraded AIX from 4.3 to 5.2 over the weekend and I am seeing a performance slowdown all of a sudden. I also noticed in topas that my WAIT% appears to jump up to 90% a whole lot and i never noticed this before. Not sure where to go...help!?!

 

[bi]

Have you applied the latest maintenance level filesets? To check, run this command: instfix -i|grep AIX_ML

The latest level is 02. The output from the command would be 5200-02_AIX_ML if you have that level installed.

Did you have any vmtune or schedtune changes in the 4.3 version (run at boot time to make the changes)? Both schedtune and vmtune are being phased out and are replaced in 5.2 with schedo and vmo. Do a man on schedtune and vmtune for more info.

 

[noober]

Hi bi, I did load 5200-01 becuase thats what IBM shipped with my upgrade media kit. I have the 02 on order. I will have to dig into cron I guess and see if some vmtune and schedtune stuff is set up...I inherited this system in October and I dont know it as well as I would like too.

Thanks for the info and if you have any more feel free to treat me like a know-nothing ding-bat and pass it on! -thanks

 

[bi]

If there were any changes that were made, they would be in either /etc/inittab or in a script that is run when the system boots. They wouldn't be in cron because for some of the changes to take effect, the system has to be booted.

 

[bi]

If you have the bandwidth, you could download the ML from

http://www-912.ibm.com/eserver/support/fixes/fcgui.jsp.

The file is 216 MB.

One thing I noticed on a couple of systems that were upgraded from 4.3 to 5.2: rootvg was mirrored on the systems when they were at 4.3, and quorum for the volume group was turned off (as recommeded by IBM). After the upgrade, quorum for rootvg was turned back on, making the mirroring basically useless (because if you lose one of the two disks, the system would stop and that's not why you mirror). It was easy enough to change quorum from two to one disks through smitty.

 

[noober]

Funny that you mention inittab, because there is something new in there called /usr/sbin/tunrestore. Its a script that looks like its doing quite a bit and all of it beyond me. I am guessing this is new and got installed with 5.2 and it does use a lot of default values that I wish I had records of before my upgrade becuase I am not sure if the defaults are what I need! Am I up a creek or what :}

 

[bi]

Do you have a backup of your old /etc/inittab?

 

[noober]

Man your good! I happen to have a copy and the following lines are in the old 4.3 inittab:

schedtune:2nce:/usr/samples/kernel/schedtune -r 5 -m 6 >/dev/console 2>&1
vmtune:2nce:/usr/samples/kernel/vmtune -R 64 -F 184 >/dev/console 2>&1

Since you said the schedtune and vmtune are obsolete, I need to figure out what these two lines were doing in 4.3 and then figure out how to do the same in 5.2?

 

[bi]

Check out this Redbook:

http://publib-b.boulder.ibm.com/Red...56b96005a67ef?OpenDocument&Highlight=0,5L,AIX

It explains what the vmo, ioo, and schedo. vmo and ioo are replacing vmtune. schedo replaces schedtune. You can make the changes through smitty tuning fast path.

You also can do a man on vmtune and schedtune to see what the replacement parms are. You could also do a vmo -a, ioo -a and schedo -a to see what the values are set at now and verify that really is your problem.

 

[noober]

I'm gonna have to put you on the payroll! It looks like (according to vmo -a and schedo -a) that the settings that were getting done in the 4.3 inittab were picked up and still set in 5.2

That means that my problem is somewhere else. I am going to check out the link to the book you gave me (thanks!) and read up in the performace tips guide tonight.


-thanks

 

[bi]

You might also go to the RedBooks home page and search on AIX AND performance to see if the performance might offer some clues.

 

[aixmurderer]

Paging space OK?

IBM Certified Confused - MQSeries
IBM Certified Flabbergasted - AIX 5 pSeries System Administration
MS Certified Windblows Rebooter

 

[sbix]

Are you using ESS? There was some problems on 5.1 exactly like this

 

[noober]

Hey aixmurderer, take a look at this, two [vmstat -s] taken ten minutes apart. Only a couple of users on the system, an instance of an informix database runnings and a light to medium busy batch process going:

----------------------10:52
22202873 total address trans. faults
1408089 page ins
2109040 page outs
324569 paging space page ins
368594 paging space page outs
0 total reclaims
8289933 zero filled pages faults
38062 executable filled pages faults
9789070 pages examined by clock
40 revolutions of the clock hand
1722275 pages freed by the clock
62912 backtracks
0 lock misses
234961 free frame waits
0 extend XPT waits
527684 pending I/O waits
1452266 start I/Os
1452266 iodones
31147191 cpu context switches
25520206 device interrupts
0 software interrupts
0 traps
149079679 syscalls

----------------------11:02
25378649 total address trans. faults
1409812 page ins
2132239 page outs
326256 paging space page ins
368732 paging space page outs
0 total reclaims
9266080 zero filled pages faults
38068 executable filled pages faults
9789070 pages examined by clock
40 revolutions of the clock hand
1722275 pages freed by the clock
63594 backtracks
0 lock misses
234961 free frame waits
0 extend XPT waits
529409 pending I/O waits
1475053 start I/Os
1475053 iodones
32028705 cpu context switches
25711500 device interrupts
0 software interrupts
0 traps
164329363 syscalls

 

[sbix]

Well doing some maths comes out you have 40 page in-outs per second
What says the lsps -a command?

 

[noober]

lsps -a
-------
Page Space
hd6

Physical Volume
hdisk0

Volume Group
rootvg

Size
3072MB

%Used
24

Active
yes

Auto
yes

Type
lv

 

[sbix]

Well ... give a look to the syscalls, if you subtract the last value from the first one and divide the result by 600 (number of seconds elapsed) you will have a 25416 average no of syscalls per second ... looks like too high

 

[sbix]

I did same calculations on a system 6H1 with 2 Gb RAM running AIX 4.3.3, Oracle 7.3 without paging activity, 50 Oracle users working + 50 batch running and I have only 23075 syscall per second

 

[noober]

You guys have any ideas about that or since its 'syscalls' would it be something I should contact IBM about?

 

[bi]

noober, they may ask you to put the ML2 on. There is a fix in there that is for "Filesystem read performance degrades after update from AIX 4.3". (APAR IY45430). And there is a fix for a problem with asynchronous IO, which your database may be using.