performance help

[dwcasey]

This is a little B50 with 128MB ram.

Here's a vmstat snapshot:

# vmstat 10 20
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 0 32803 123 0 1 2 13 96 0 258 157 56 7 3 81 8
0 0 32825 125 0 0 2 17 85 0 1271 643 739 18 6 69 6
1 0 32727 193 0 0 5 16 79 0 457 852 958 18 8 40 35
0 2 32742 121 0 1 4 15 78 0 600 951 1040 14 11 0 76
3 1 32769 127 0 1 11 42 186 0 605 1174 1272 22 9 46 23
0 0 32672 262 0 0 1 11 56 0 429 1075 815 8 6 81 5
0 0 32794 122 0 0 5 19 191 0 554 1316 1282 21 8 61 9
10 0 32825 124 0 0 2 32 189 0 700 1457 1259 35 12 44 9
0 0 32486 357 0 1 1 11 66 0 627 729 638 12 4 76 7
0 0 32750 120 0 0 2 20 85 0 664 811 759 14 7 72 7
5 2 32762 122 0 0 1 9 86 0 705 502 751 11 7 2 80
11 1 32784 126 0 0 2 26 167 0 791 1143 1165 31 12 30 28
3 0 32835 126 0 1 1 12 44 0 490 656 611 14 4 80 2
0 0 32855 122 0 0 0 4 19 0 487 514 498 10 3 86 1
0 0 32872 124 0 2 1 12 69 0 752 681 706 17 6 73 4
7 0 32828 122 0 1 1 19 95 0 920 1027 1076 25 8 57 9
13 0 32994 260 0 5 9 73 504 0 907 2593 986 31 22 32 15
0 1 32848 207 0 1 0 0 0 0 552 1214 775 18 12 8 62
6 0 32605 218 0 1 1 9 67 0 571 1008 916 26 9 43 21
2 0 32815 120 0 0 4 31 240 0 673 1358 1090 24 8 65 3


Hope that formats o.k.

 

[soma1958]

I have seen the vmstat output the only I noticed sometimes the wa column reaches greater than 40 which means
extensive disk in/out need.

Check your uptime output when performance is good and check it when performance slows and investigate.

please investigate when is the peak performance time, what process is causing and what application by using topas tool or "ps aux | grep -v kproc | head -10". That might guide you to the real bottleneck.

Rebooting the system will defrag the memory and kill any run away process.

Thanks for all
Latif

 

[dwcasey]

No defunct procs.

 

[Morsing]

It could be a network problem. Can you post the 'no -a' output.

Cheers Henrik Morsing
IBM Certified AIX 4.3 Systems Administration

 

[dwcasey]

no -a (so many parms!)

extendednetstats = 0
thewall = 65516
sockthresh = 85
sb_max = 1048576
somaxconn = 1024
clean_partial_conns = 0
net_malloc_police = 0
rto_low = 1
rto_high = 64
rto_limit = 7
rto_length = 13
inet_stack_size = 16
arptab_bsiz = 7
arptab_nb = 25
tcp_ndebug = 100
ifsize = 8
arpqsize = 1
ndpqsize = 50
route_expire = 1
send_file_duration = 300
fasttimo = 200
routerevalidate = 0
nbc_limit = 0
nbc_max_cache = 131072
nbc_min_cache = 1
nbc_pseg = 0
nbc_pseg_limit = 65516
strmsgsz = 0
strctlsz = 1024
nstrpush = 8
strthresh = 85
psetimers = 20
psebufcalls = 20
strturncnt = 15
pseintrstack = 12288
lowthresh = 90
medthresh = 95
psecache = 1
subnetsarelocal = 1
maxttl = 255
ipfragttl = 60
ipsendredirects = 1
ipforwarding = 0
udp_ttl = 30
tcp_ttl = 60
arpt_killc = 20
tcp_sendspace = 16384
tcp_recvspace = 16384
udp_sendspace = 9216
udp_recvspace = 41920
rfc1122addrchk = 0
nonlocsrcroute = 0
tcp_keepintvl = 150
tcp_keepidle = 14400
bcastping = 0
udpcksum = 1
tcp_mssdflt = 512
icmpaddressmask = 0
tcp_keepinit = 150
ie5_old_multicast_mapping = 0
rfc1323 = 0
pmtu_default_age = 10
pmtu_rediscover_interval = 30
udp_pmtu_discover = 0
tcp_pmtu_discover = 0
ipqmaxlen = 100
directed_broadcast = 1
ipignoreredirects = 0
ipsrcroutesend = 1
ipsrcrouterecv = 0
ipsrcrouteforward = 1
ip6srcrouteforward = 1
ip6_defttl = 64
ndpt_keep = 120
ndpt_reachable = 30
ndpt_retrans = 1
ndpt_probe = 5
ndpt_down = 3
ndp_umaxtries = 3
ndp_mmaxtries = 3
ip6_prune = 2
ip6forwarding = 0
multi_homed = 1
main_if6 = 0
main_site6 = 0
site6_index = 0
maxnip6q = 20
llsleep_timeout = 3
tcp_timewait = 1
tcp_ephemeral_low = 32768
tcp_ephemeral_high = 65535
udp_ephemeral_low = 32768
udp_ephemeral_high = 65535
delayack = 0
delayackports = {}
sack = 0
use_isno = 1
tcp_newreno = 0
tcp_nagle_limit = 65535

 

[soma1958]

Do the following:
no -o extendednetstats=1
netstat -m

then check the following values for example:
16 mbufs in use:
0 mbuf cluster pages inuse
4 kbuts allocated to mbufs
0 requests for mbufs denied
0 calls to protocl darain routine1

if the requests for mbufs denied > 0 the you have to increase thewall size.

1- If you have 433 then the value of the wall should be set to 131072(128 megabytes) or have of the memory, which ever is smaller (no -o thewall=131072).

I hope ythis will help
Latif

 

[dwcasey]

Sorry it took me so long to get back. Looks like we're doing good on mbufs:

14 mbufs in use:
15 mbuf cluster pages in use
63 Kbytes allocated to mbufs
0 requests for mbufs denied
0 calls to protocol drain routines
0 sockets not created because sockthresh was reached

 

[gileb]

Hi, if its a proxy, you dont need so much mem for io, so try the vmtune -P 30 and vmtune -p 15 for example. and wait for a few hours to see what happens

 

[dwcasey]

Thanks for the info. I will try it and see what vmstat looks like before/after.

 

[dwcasey]

Actually, I did some poking around in the Proxy Admin console, and I believe there is a lot of I/O. The proxy reads/writes A LOT of files to/from the disk for cache purposes...I believe it's setup to use approximately 6gb of space on it's own FS and there are rules for how long it retains these files.

I investigated tuning the Proxy server itself, but the docs say DONT CHANGE the default settings unless told to do so.

So, maybe I'm going about this the wrong way?

 

[mouse123]

In the top utility i could see Virtual memory is 0.0M free
Server H50
OS 4.3.3
Memory is 2GB

askprod:/#lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
paging01 hdisk1 rootvg 1792MB 12 yes yes lv
paging00 hdisk0 rootvg 2048MB 11 yes yes lv
hd6 hdisk0 rootvg 512MB 42 yes yes lv
Here is my top output

askprod#top

askprod load averages: 2.64, 1.82, 1.69 Wed Dec 18 14:58:36 2002
Cpu states: 45.8% user, 16.6% system, 17.5% wait, 20.1% idle
Real memory: 1960.9M free 1463.0M procs 416.1M files 3840.0M total
Virtual memory: 0.0M free 409.6M used 409.6M total

PID USER PRI NICE SIZE RES STAT TIME CPU% COMMAND
108472 oracle 120 0 28969K 12952K run 6:51 24.8% oracle
131758 oracle 109 0 30189K 14164K run 0:01 18.1% oracle
516 root 127 21 12K 7396K run 122542:43 12.5% Kernel (wait)
1290 root 127 21 12K 7396K run 120009:28 11.8% Kernel (wait)
1032 root 127 21 12K 7396K run 119401:20 7.2% Kernel (wait)
774 root 127 21 12K 7396K run 118996:52 6.5% Kernel (wait)
189180 oracle 73 0 28413K 11444K sleep 1:46 5.5% oracle
220202 oracle 75 0 28345K 11628K sleep 0:03 4.6% oracle
82170 oracle 65 0 28701K 12452K sleep 0:30 2.5% oracle
140536 oracle 65 0 30185K 14164K sleep 0:00 2.3% oracle
153856 root 63 0 1032K 1112K run 0:00 1.6% monitor
180388 oracle 62 0 29821K 13804K sleep 0:19 0.9% oracle
1548 root 16 21 12K 7392K sleep 962:24 0.4% Kernel (lrud)
2064 root 37 21 64K 7440K sleep 325:02 0.4% Kernel (gil)
146552 oracle 60 0 28769K 12316K sleep 9:43 0.2% oracle
4164 root 60 0 165K 36K sleep 769:25 0.0% syncd

Also if watch through monitor, the IO is sometime 96% on hdisk2.

AIX monitor v1.14: askprod Wed Dec 18 15:06:15 2002
Sys 8.2% Wait 36.2% User 23.0% Idle 32.5% Refresh: 1.00 s
0% 25% 50% 75% 100%
=====

http://WWWWWWWWWWWWWWWWWWWWWWWWWW>>>>>>>>>>>>>>>>.......................

Runnable processes 1.00 load average: 1.23, 1.40, 1.55

Memory Real Virtual Paging (4kB) Process events File/TTY-IO
free 1962.9 MB 0.0 MB 587.6 pgfaults 1158 pswitch 0 iget
procs 1461.0 MB 409.6 MB 378.7 pgin 6124 syscall 105 namei
files 416.1 MB 12.0 pgout 549 read 0 dirblk
total 3840.0 MB 409.6 MB 0.0 pgsin 324 write 1112209 readch
0.0 pgsout 2 fork 156600 writech
DiskIO Total Summary 3 exec 0 ttyrawch
read 1523.0 kByte/s 0 rcvint 0 ttycanch
write 48.0 kByte/s 0 xmtint 0 ttyoutch
transfers 201.9 tps
active 1/4 disks Netw read write
en0 44.4 119.5 kB/s
TOPdisk read write busy lo0 0.0 0.0 kB/s
hdisk2 1523 48 kB/s 96%
hdisk1 0 0 kB/s 0%
hdisk0 0 0 kB/s 0%
cd0 0 0 kB/s 0%


Should i add another pagging space in hdisk2? or should i add more memory

I need help from expert?

 

[Hars]

Hi,

What filesystems are in hdisk2 ?

It looks few processes are operating on big files leading to swapping of processes. Since consistent paging util and high context switching is seen, it would improve by adding physical memory.

One more thing to look at is what filesystems the top cpu procs are using. Check out the jfslog's, increase the size if possible.

Check out svmon in redbooks. it will help in analyzing.

All the best
Cheers
Hars

 

[Morsing]

Hmmm.... At first you say you have 2Gb of memory but top reports 4Gb. And can you have more than that in a B50??

Cheers Henrik Morsing
IBM Certified AIX 4.3 Systems Administration

 

[Yegolev]

different guy, different machine, this is a h50. all i know is the zero virtual looks weird. IBM Certified -- AIX 4.3 Obfuscation

 

[mouse123]

Hi Hars

What filesystems are in hdisk2 ?

It is jfs and running oracle 8i. hdisk2 is configured RAID5 (connected SSA D20)and there is no pagging created.

If I create pagging space in hdisk2, will the problem solve?
and also in top utility will it show virtual memory? (now it shows zero virtual memory free )

Thanks

 

[mouse123]

Hi Morsing

I have different machine, which is H50 with 2GB memory

 

[AIXSPadmin]

Reading all of these postings to the original question gave me a good laugh for the night. All I can say is that I am glad I don't need any help with my systems.

 

[mrn]

Thanks for the helpful comment AIXSPadmin.... --
| Mike Nixon
| Unix Admin
|

http://www.instantflashshop.com

----------------------------