M80 latency problem

[jseva]

Just migrated from an h50 to an m80 this weekend, and system response time is slower than before (sporadically).
At times the keyboard will lag for 2 characters, and at times two-words worth. I don't think it's a network issue since my tty0 also experiences the same sporadic response issue. Any ideas? Please help.

 

[MikeLacey]

the output of 'vmstat 1', does it show the system as short of memory or very busy? Mike
michael.j.lacey@ntlworld.com
Email welcome if you're in a hurry or something -- but post in tek-tips as well please, and I will post my reply here as well.

 

[PSD]

jseva,

A strange problem have you go ML-07 installed? check via:

instfix -i |grep ML

Also, post up the output of netstat -m from the box.

The M80 should be about 3 X times quicker than your old H50 - is there something different about this server i.e. less memory therefore introducing a memory bottleneck - as Mike says check out vmstat as the first port of call.

Hope this helps

PSD
HACMP Specialist

 

[jseva]

the m80 has 3X more memory than the h50. here's the netstat output...

******* CPU 0 *******
By size inuse calls failed delayed free hiwat freed
32 91 56905 0 0 165 2320 0
64 35 3723 0 0 29 1160 0
128 35 4670 0 0 157 580 0
256 57 273738 0 0 183 1392 0
512 38 31248 0 0 42 145 0
1024 5 27922 0 0 35 362 0
2048 0 238 0 0 2 362 0
4096 1 17 0 0 2 435 0
8192 1 2 0 0 0 36 0
16384 0 0 0 0 65 87 0
65536 1 1 0 0 0 4096 0
******* CPU 1 *******
By size inuse calls failed delayed free hiwat freed
32 28 44523 0 0 100 2320 0
64 2 105 0 0 62 1160 0
128 14 542 0 0 18 580 0
256 40 211475 0 0 72 1392 0
512 18 22193 0 0 30 145 0
1024 11 22055 0 0 25 362 0
2048 0 259 0 0 2 362 0
4096 0 80 0 0 17 435 0
16384 0 0 0 0 65 87 0
******* CPU 2 *******
By size inuse calls failed delayed free hiwat freed
32 19 22763 0 0 109 2320 0
64 26 348 0 0 38 1160 0
128 23 879 0 0 41 580 0
256 34 135059 0 0 46 1392 0
512 9 11530 0 0 31 145 0
1024 10 11209 0 0 34 362 0
2048 0 133 0 0 2 362 0
4096 1 113 0 0 34 435 0
8192 0 1 0 0 1 36 0
16384 0 0 0 0 65 87 0

The box has 5 more CPUs with no glaring difference from the above listings...

 

[jseva]

ML 07 is installed.

 

[teddy74]

Hi Jseva ,

How was the upgrade done ,was it using a mksysb restore or otherwise and did it go thro smooth , because i faced a problem previously when i upgraded from a H70 to H80 using a mksysb restore and while restoring from the mksysb ,it did not find the requisite drivers for the H80 and then we put in the New os cd and it picked up some files from that CD and then went ahead restoring from the Mksysb ,but the system was misbehaving and then we finally fixed it by reapplying the ML or fix onceagain on the new system.Hope it helps,
regards,
rajesh

rajeshrcmc@hotmail.com
"Better to ask a question once and be thought a fool once, than never to ask and remain a fool forever"

 

[PSD]

From the netstat -m are you getting any MBUF denied errors?

iI also have to agree with rajesh I always like to do new installs when migrating to new hardware, but I guess the you booted from AIX 4.3.3 installation media and chose the restore from system backup option, this would then pick off all the missing drivers on the msysb for the differing hardware.

Cheers

PSD
HACMP Specialist

 

[jseva]

No errors whatsover...

Streams mblk statistic failures:
0 high priority mblk failures
0 medium priority mblk failures
0 low priority mblk failures

Just to clarify, we did not upgrade our H50. We bought a new box with AIX 4.3.3., and copied files over (not including AIX).

Thanks guys for your assistance...

 

[MikeLacey]

jseva emailed me with the vmstat 1 output
[tt]
Here's the output of vmstat 1: (Thanks for your help!)

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 0 418345 11433 0 0 0 14 26 0 127 428 211 2 5 91 2
0 2 418345 11433 0 0 0 0 0 0 1105 3317 489 1 1 98 0
0 2 418155 11636 0 0 0 0 0 0 1113 4268 605 3 1 96 0
0 2 418588 11157 0 0 0 0 0 0 1097 29501 680 1 2 97 0
0 2 417998 11782 0 0 0 0 0 0 1054 6743 649 1 2 96 0
2 2 418610 11063 0 0 0 0 0 0 1164 15919 867 4 3 92 1
0 2 418736 10926 0 0 0 0 0 0 1073 16274 683 5 2 92 0
1 2 418641 11036 0 0 0 0 0 0 1127 187094 781 7 6 86 1
2 2 418705 10980 0 0 0 0 0 0 1057 1250809 532 8 17 75 0
2 2 418606 11114 0 0 0 0 0 0 1051 1107743 561 7 15 77 0
1 2 418925 10793 0 0 0 0 0 0 1032 44361 311 2 12 86 0
2 2 419477 10155 0 0 0 0 0 0 1153 461599 1319 6 7 86 1
0 2 419591 10018 0 0 0 0 0 0 1080 19118 626 4 5 90 1
2 2 419786 9804 0 0 0 0 0 0 1107 606381 625 4 11 85 1
3 3 419807 9775 0 0 0 0 0 0 1152 694789 851 7 12 79 2
[/tt]
Not much help there I'm afraid, it's not busy or running out of memory during this run of vmstat.

The obvious question is: Were you getting your reponse time problems when this run was made?

And another, though I suspect the answer will be satifactory: How much swap in proportion to your physical memory do you have? Mike
michael.j.lacey@ntlworld.com
Email welcome if you're in a hurry or something -- but post in tek-tips as well please, and I will post my reply here as well.

 

[jseva]

AIX tech support recommends decreasing physical memory. We currently have 16GB, and they want us to bring it down to around 6GB. Per IBM, since we don't need the 16GB for any of our applications, memory flushing is taking too long than normal.

Do you guys buy that?

 

[MikeLacey]

Sounds reasonable.

On an optimization course a couple of months (HP, but it should apply) I was told that on "large" systems (>1.5gb RAM, which surprised me) setting the buffer cache to its default value (a fairly large percentage of total RAM) is actually counter-productive as the box spends too much of its time managing the large buffer cache.

As an aside (related-ish). After that course I went looking for away to fix the size of the file system cache in AIX (a good idea on a machine that is just a DB server, apparently) and I couldn't find one.... Mike
michael.j.lacey@ntlworld.com
Email welcome if you're in a hurry or something -- but post in tek-tips as well please, and I will post my reply here as well.