performance help

[dwcasey]

This is a little B50 with 128MB ram.

Here's a vmstat snapshot:

# vmstat 10 20
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 0 32803 123 0 1 2 13 96 0 258 157 56 7 3 81 8
0 0 32825 125 0 0 2 17 85 0 1271 643 739 18 6 69 6
1 0 32727 193 0 0 5 16 79 0 457 852 958 18 8 40 35
0 2 32742 121 0 1 4 15 78 0 600 951 1040 14 11 0 76
3 1 32769 127 0 1 11 42 186 0 605 1174 1272 22 9 46 23
0 0 32672 262 0 0 1 11 56 0 429 1075 815 8 6 81 5
0 0 32794 122 0 0 5 19 191 0 554 1316 1282 21 8 61 9
10 0 32825 124 0 0 2 32 189 0 700 1457 1259 35 12 44 9
0 0 32486 357 0 1 1 11 66 0 627 729 638 12 4 76 7
0 0 32750 120 0 0 2 20 85 0 664 811 759 14 7 72 7
5 2 32762 122 0 0 1 9 86 0 705 502 751 11 7 2 80
11 1 32784 126 0 0 2 26 167 0 791 1143 1165 31 12 30 28
3 0 32835 126 0 1 1 12 44 0 490 656 611 14 4 80 2
0 0 32855 122 0 0 0 4 19 0 487 514 498 10 3 86 1
0 0 32872 124 0 2 1 12 69 0 752 681 706 17 6 73 4
7 0 32828 122 0 1 1 19 95 0 920 1027 1076 25 8 57 9
13 0 32994 260 0 5 9 73 504 0 907 2593 986 31 22 32 15
0 1 32848 207 0 1 0 0 0 0 552 1214 775 18 12 8 62
6 0 32605 218 0 1 1 9 67 0 571 1008 916 26 9 43 21
2 0 32815 120 0 0 4 31 240 0 673 1358 1090 24 8 65 3


Hope that formats o.k.

 

[dwcasey]

Also, lsps shows:

: lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
hd6 hdisk0 rootvg 256MB 24 yes yes lv

 

[aixat]

1) add more memory.
2) update AIX code to latest level.
3) install bos.adt.sample then use vmtune to tune memory.

 

[dwcasey]

Roger that! Thanks.

I did a little more digging on the application(s) this machine is running and it can launch up to 80 instances of the program (proxy server) to handle the load. Each instance will take anywhere from 200kb to 300kb of memory.

Thanks again.

 

[Morsing]

Is it too slow??
It's not paging but has a lot of context switches. I don't think more memory will help a lot.

Read the Performance tools and techniques Redbook or the Performance Tuning Redbook and go from there.

Cheers Henrik Morsing
IBM Certified AIX 4.3 Systems Administration

 

[bandez]

Hi ,

No paging, lot of context switch means that the scheduler time slices not correct or you have too much processes on the system.

Try to adjust the time slice with schedtune and read the performance guide.
It will interesting ; )

 

[dwcasey]

Lots of context switching, eh? I'll check out the book(s) and see what I can do.

Thanks guys!

 

[dwcasey]

So, by running schedtune by itself I get:

: ./schedtune

THRASH SUSP FORK SCHED
-h -p -m -w -e -f -d -r -t -s
SYS PROC MULTI WAIT GRACE TICKS SCHED_D SCHED_R TIMESLICE MAXSPIN
0 4 2 1 2 10 16 16 1 1

CLOCK
-c
%usDELTA
100


With the -h over the SYS column, does that mean that the 0 is the value for 'h'?

Reading some of the PerfGuide and they have a formula to see if memory is overcommitted:

po/fr > 1/h or po*h > fr

So is the 'h' in this formula the same 'h' in the schedtune command? So, if h=0, then 1/0 is, well, not valid, correct?

 

[aixat]

http://WWWWWWWWWRRRRRRRRRROOOOOOOOOOOONNNNNNNNNNGGGGGGGGG.

every one is wrong about schedtune.

If you call IBM evey they will tell you not to touch schedtune.

System is getting low on memory and that cause context switch.

if you reduce % of memory for file pages your system will run f

faster.

read redbooks on performance and read the

the memory section first.

do not touch schedtune.

z

 

[Morsing]

aixat, I don't agree. Sorry, just had to say that.

Cheers Henrik Morsing
IBM Certified AIX 4.3 Systems Administration

 

[dwcasey]

I've heard the same thing about schedtune. Stay away! You can really dork up a system with this command.

I'll see what I can find on % memory for file pages in the books.

Thanks again.

 

[Morsing]

My "don't agree" was about the memory. Please explain to me how low memory can cause context switches. I'm very curious to know?!?!?!?

BTW: In IBMs own Redbook "Performance tools" they say that if you have lots of CS you can increase performance slightly by increasing the time slice with schedtune -t. If it's in a redbook then that's what IBM says. This does not mean that you should not use it with caution.

Cheers Henrik Morsing
IBM Certified AIX 4.3 Systems Administration

 

[bandez]

Hi there,

Solve the problem if you would buy more memry if the system would make a lot of paging and the free memory pages is close to zero. But the situation is NOT that.
You have a lot of free pages and a little paging but lot of context switch. I cannot explain it better !!!!

You have too much processes or the scheduler time slice is not correct.
This symphtom can be coused by blocked processes (you dont have any because the b column on the vmstat's output almost 0) and you could check: #crash
>dlock

 

[mrn]

How long since last reboot? --
| Mike Nixon
| Unix Admin
|

http://www.instantflashshop.com

----------------------------

 

[Morsing]

I'm still a bit curious about how this affects the system?? You didn't say if it was slow or what happened.

If there's no paging and no blocked processes how does the CS get so high if the processes doesn't voluntarily give up the processor?? I mean, if the processes all want full CPU attention, the CS shouldn't be more than 100 because of the 10ms time slice.

If must be a lot of processes that only want i little processing time.

Cheers Henrik Morsing
IBM Certified AIX 4.3 Systems Administration

 

[MoshiachNow]

WHo is saying there is any problem with the performance here? the wait is average around 10%,not too bad.

No paging means there is enough memory for whtever the machine is running in the posted snapshot.

Context switching is in average 1000 per second,also not too bad.

So where is a problem ?
"Long live king Moshiach !"
h

http://www.noahide.com/movement.htm

 

[Morsing]

Exactly. Maybe I wasn't too clear but the original question wasn't too clear either, "performance help" was all the question I got...

A thousand context switches means ten switches per time slice. This means that at least five processes very very often asks for a very little CPU time. Strange but not impossible.

Cheers Henrik Morsing
IBM Certified AIX 4.3 Systems Administration

 

[Morsing]

Hadn't noticed this question:

> Reading some of the PerfGuide and they have a formula to see if memory is overcommitted:

> po/fr > 1/h or po*h > fr

> So is the 'h' in this formula the same 'h' in the schedtune command? So, if h=0, then 1/0 is, well, not valid, correct?

h=0 means that the load control mechanism is disabled. The value is by default 0 if you have more than 128Mb memory and 6 if you have less.

Cheers Henrik Morsing
IBM Certified AIX 4.3 Systems Administration

 

[dwcasey]

Machine has been up 147 days.

I changed h to 6 and m to 4. I used h=6 because I'm on the edge of memory, this machine has 128mb and m=4 because of this in the Performance Guide:

"While m=2 is appropriate for a desktop, single-user configuration, it is frequently too small for larger, multiuser, or server configurations with large amounts of RAM.
If the system you are installing is larger than 32 MB, but less than 128 MB, and is expected to support more than five active users at one time, consider raising the minimum level of multiprogramming of the VMM memory-load-control mechanism."

Here's the latest vmstat snapshot:

: vmstat 15 15
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 0 32487 126 0 1 2 13 94 0 258 165 65 7 3 81 8
2 0 32480 120 0 2 7 30 155 0 406 913 827 26 7 55 12
2 0 32209 262 0 12 3 18 96 0 471 932 700 14 5 74 7
3 1 32311 121 0 2 4 24 176 0 577 793 876 28 7 28 37
0 3 32361 124 0 0 3 15 88 0 544 555 573 17 7 51 25
7 0 32356 123 0 0 5 24 112 0 547 969 886 27 7 56 10
4 0 32260 125 0 8 5 25 123 0 527 1031 1030 27 8 56 10
5 0 32429 340 0 1 6 40 203 0 639 1406 873 22 11 61 6
5 1 32242 343 0 6 1 11 47 0 812 1716 1039 27 18 0 54
1 0 32079 396 0 1 1 5 28 0 703 881 940 20 7 61 12
1 0 32191 252 0 1 8 38 241 0 755 1073 1218 29 10 49 12
8 0 32345 122 0 0 2 16 122 0 652 716 902 24 5 66 5
5 0 32401 124 0 1 2 19 151 0 455 675 724 20 5 69 6
7 4 32133 287 0 1 5 26 151 0 507 835 1008 26 9 3 62
3 5 32218 120 0 0 3 13 90 0 369 750 721 19 6 55 21

: ./vmtune
vmtune: current values:
-p -P -r -R -f -F -N -W
minperm maxperm minpgahead maxpgahead minfree maxfree pd_npages maxrandwrt
6346 25384 2 8 120 128 524288 0

-M -w -k -c -b -B -u -l -d
maxpin npswarn npskill numclust numfsbufs hd_pbuf_cnt lvm_bufcnt lrubucket defps
26207 3328 832 1 93 80 9 131072 1

-s -n -S -h
sync_release_ilock nokillroot v_pinshm strict_maxperm
0 0 0 0

number of valid memory pages = 32758 maxperm=77.5% of real memory
maximum pinable=80.0% of real memory minperm=19.4% of real memory
number of file memory pages = 9012 numperm=27.5% of real memory

Since this is a proxy server, I ran a ps v on all the proxy processes and noticed that one had a significantly larger amount PGIN

PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM COMMAND
26106 - A 4:02 185345 1240 608 32768 984 596 0.1 0.0 ./ns-proxy

If it doesn't format correctly, the PGIN is the 185345 number.

Most of these ns-proxy processes are single threaded.

Performance is a little slow. System response time is slow and there appears to be a "lag" when surfing and an occasional timeout.

 

[MoshiachNow]

Since the system was not rebooted for long time,

I would also check for zombies , since if there are too many - they also slow down the system:

ps –auw | grep defunct "Long live king Moshiach !"
h

http://www.noahide.com/movement.htm