Performance help

[mouse123]

I have asked this question before also, still I am waiting for some aix expert reply

In the top utility i could see Virtual memory is 0.0M free
Server H50
OS 4.3.3
Memory is 2GB

askprod:/#lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
paging01 hdisk1 rootvg 1792MB 12 yes yes lv
paging00 hdisk0 rootvg 2048MB 11 yes yes lv
hd6 hdisk0 rootvg 512MB 42 yes yes lv
Here is my top output

askprod#top

askprod load averages: 2.64, 1.82, 1.69 Wed Dec 18 14:58:36 2002
Cpu states: 45.8% user, 16.6% system, 17.5% wait, 20.1% idle
Real memory: 1960.9M free 1463.0M procs 416.1M files 3840.0M total
Virtual memory: 0.0M free 409.6M used 409.6M total

PID USER PRI NICE SIZE RES STAT TIME CPU% COMMAND
108472 oracle 120 0 28969K 12952K run 6:51 24.8% oracle
131758 oracle 109 0 30189K 14164K run 0:01 18.1% oracle
516 root 127 21 12K 7396K run 122542:43 12.5% Kernel (wait)
1290 root 127 21 12K 7396K run 120009:28 11.8% Kernel (wait)
1032 root 127 21 12K 7396K run 119401:20 7.2% Kernel (wait)
774 root 127 21 12K 7396K run 118996:52 6.5% Kernel (wait)
189180 oracle 73 0 28413K 11444K sleep 1:46 5.5% oracle
220202 oracle 75 0 28345K 11628K sleep 0:03 4.6% oracle
82170 oracle 65 0 28701K 12452K sleep 0:30 2.5% oracle
140536 oracle 65 0 30185K 14164K sleep 0:00 2.3% oracle
153856 root 63 0 1032K 1112K run 0:00 1.6% monitor
180388 oracle 62 0 29821K 13804K sleep 0:19 0.9% oracle
1548 root 16 21 12K 7392K sleep 962:24 0.4% Kernel (lrud)
2064 root 37 21 64K 7440K sleep 325:02 0.4% Kernel (gil)
146552 oracle 60 0 28769K 12316K sleep 9:43 0.2% oracle
4164 root 60 0 165K 36K sleep 769:25 0.0% syncd

Also if watch through monitor, the IO is sometime 96% on hdisk2.

AIX monitor v1.14: askprod Wed Dec 18 15:06:15 2002
Sys 8.2% Wait 36.2% User 23.0% Idle 32.5% Refresh: 1.00 s
0% 25% 50% 75% 100%
=====

http://WWWWWWWWWWWWWWWWWWWWWWWWWW>>>>>>>>>>>>>>>>.......................

Runnable processes 1.00 load average: 1.23, 1.40, 1.55

Memory Real Virtual Paging (4kB) Process events File/TTY-IO
free 1962.9 MB 0.0 MB 587.6 pgfaults 1158 pswitch 0 iget
procs 1461.0 MB 409.6 MB 378.7 pgin 6124 syscall 105 namei
files 416.1 MB 12.0 pgout 549 read 0 dirblk
total 3840.0 MB 409.6 MB 0.0 pgsin 324 write 1112209 readch
0.0 pgsout 2 fork 156600 writech
DiskIO Total Summary 3 exec 0 ttyrawch
read 1523.0 kByte/s 0 rcvint 0 ttycanch
write 48.0 kByte/s 0 xmtint 0 ttyoutch
transfers 201.9 tps
active 1/4 disks Netw read write
en0 44.4 119.5 kB/s
TOPdisk read write busy lo0 0.0 0.0 kB/s
hdisk2 1523 48 kB/s 96%
hdisk1 0 0 kB/s 0%
hdisk0 0 0 kB/s 0%
cd0 0 0 kB/s 0%


Should i add another pagging space in hdisk2? or should i add more memory

I need help from expert?

 

[AIXSPadmin]

Some points about paging space:

Do not put more than one paging space LV on a physical volume.

Avoid putting a paging space LV on a heavily active LV.

Make each paging space LV roughly equal in size.

Do not extend paging space LVs across multiple PVs.

Put paging space LVs on PVs that are attached to different disk controllers.

 

[mouse123]

askprod:/#lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
paging01 hdisk1 rootvg 1792MB 7 yes yes lv
paging00 hdisk0 rootvg 2048MB 6 yes yes lv
hd6 hdisk0 rootvg 512MB 23 yes yes lv
askprod:/#lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 2 closed/syncd N/A
hd6 paging 32 64 2 open/syncd N/A
hd8 jfslog 1 2 2 open/syncd N/A
hd4 jfs 2 4 2 open/syncd /
hd2 jfs 54 108 2 open/syncd /usr
hd9var jfs 10 20 2 open/syncd /var
hd3 jfs 10 20 2 open/syncd /tmp
hd1 jfs 37 74 2 open/syncd /home
paging00 paging 128 128 1 open/syncd N/A
paging01 paging 112 112 1 open/syncd N/A
app-lynx jfs 1 1 1 open/syncd /sprd/lynx
askprod:/#lsvg -p rootvg
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk0 active 542 267 108..18..00..32..109
hdisk1 active 542 282 108..17..00..48..109

All pagings are different lv
should i remove one paging (paging00) on hdisk0
and create another paging on hdisk2

these will help?

 

[Morsing]

Delete paging00 and increase hd6 and paging01 to 2048MB.
Do not put paging spaces on other VGs. You cannot remove VGs with active paging spaces on them. (Or has that been fixed in AIX 5??)

Cheers Henrik Morsing
IBM Certified AIX 4.3 Systems Administration

 

[mouse123]

Thanks Henrik

you can see my IO on hdisk2 is 96% in monitor tool, I belive that makes more slow system

 

[Morsing]

Hmmm... Your hdisk2 is doing a lot of seeks. What's on it??

Cheers Henrik Morsing
IBM Certified AIX 4.3 Systems Administration

 

[mouse123]

Hi Henrik

Oracle is running and it is jfs with RAID5 (more than 300 users are accessing ) and if see virtual memory it shows zero in monitoring tool

 

[AIXSPadmin]

You don't mention the volume group that hdisk2 belongs to, however, if hdisk2 is a different volume group and you have more than one PV in that VG then I would move the jfslog for that VG to another disk. A jfslog should not be on a heavily accessed disk, much the same as paging.

I don't see that you need more paging space, just make them equal in size. Paging works in a round-robbin manner in 4k blocks. That is why the paging space LVs should be of equal sizes, if they are not as in your original case with hd6 being much smaller then it may be used 100% before the others and then your paging isn't evenly distributed.

vmstat would give more information as well as iostat, about what is causing your performance problems.

 

[mouse123]

Hi AIXSPadmin

askprod:/#lsvg -p nsprvg
nsprvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk2 active 950 49 00..00..00..00..49
askprod:/#

hdisk2 is different vg and it is ssa disks with RAID 5. oracle installed (jfs) in this there is no paging is created

askprod:/#lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
paging01 hdisk1 rootvg 1792MB 7 yes yes lv
paging00 hdisk0 rootvg 2048MB 7 yes yes lv
hd6 hdisk0 rootvg 512MB 25 yes yes lv
askprod:/#
I am going to increase paging on hd6 as paging as 2048 and i will remove paging00 since it is in hdisk0. IS it ok

 

[AIXSPadmin]

Yes, you should remove paging00 since it is on the same pv as hd6, however, if you make hd6 2048 then make paging01 2048 too. If you have other pv's in the nsprvg besides hdisk2 and they are not as heavily used as this disk then move the jfslog off hdisk2 if that is where it is and move it to a less used disk in the nsprvg.

 

[mouse123]

Hi AIXSPadmin

As I mension this hdisk2 is SSA disks which has 16 9.1 pdisk's (SSA RAID Array RAID 5). When I do

askprod:/#lspv
hdisk0 000095884b0e7990 rootvg
hdisk1 000095886ea7149a rootvg
hdisk2 000095886ea342fa nsprvg

If you could see my io load on hdisk2 in monitor. it is almost 96%
do you thing if i add another pagging on this volume, will it help to reduce this io load

AIX monitor v1.14: askprod Wed Dec 18 15:06:15 2002
Sys 8.2% Wait 36.2% User 23.0% Idle 32.5% Refresh: 1.00 s
0% 25% 50% 75% 100%
=====

http://WWWWWWWWWWWWWWWWWWWWWWWWWW>>>>>>>>>>>>>>>>.......................

Runnable processes 1.00 load average: 1.23, 1.40, 1.55

Memory Real Virtual Paging (4kB) Process events File/TTY-IO
free 1962.9 MB 0.0 MB 587.6 pgfaults 1158 pswitch 0 iget
procs 1461.0 MB 409.6 MB 378.7 pgin 6124 syscall 105 namei
files 416.1 MB 12.0 pgout 549 read 0 dirblk
total 3840.0 MB 409.6 MB 0.0 pgsin 324 write 1112209 readch
0.0 pgsout 2 fork 156600 writech
DiskIO Total Summary 3 exec 0 ttyrawch
read 1523.0 kByte/s 0 rcvint 0 ttycanch
write 48.0 kByte/s 0 xmtint 0 ttyoutch
transfers 201.9 tps
active 1/4 disks Netw read write
en0 44.4 119.5 kB/s
TOPdisk read write busy lo0 0.0 0.0 kB/s
hdisk2 1523 48 kB/s 96%
hdisk1 0 0 kB/s 0%
hdisk0 0 0 kB/s 0%
cd0 0 0 kB/s 0%


Thanks for your help in advance

 

[sectorseveng]

mouse123 - I think you are being bitten by the RAID5 parity write problem. Fixing the paging as suggested by AIXSPadmin is a good idea to improve the overall performance of your system (and should be done anyway, just on general principle) but it won't help much with the load on hdisk2.

Please see my recent posts on the "performance problem" thread raised by tangcov, I think it has relevance to your situation. Basically the solution is to get as many independant disk heads involved in doing disk/io as you can get.

 

[mouse123]

Hi Sectorseveng
I did changed my paging setting now it looks like
askprod:/#lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
paging01 hdisk1 rootvg 1792MB 20 yes yes lv
hd6 hdisk0 rootvg 2880MB 13 yes yes lv

AS YOU SAID IT DIDN'T HELP LOAD ON hdisk2 still my monitor utility shows (100% i/o on hdisk2 and virtual memory is zero)

IX monitor v1.14: askprod Mon Jan 13 11:20:37 2003
Sys 21.0% Wait 31.8% User 45.2% Idle 2.0% Refresh: 1.00 s
0% 25% 50% 75% 100%
===============

http://WWWWWWWWWWWWWWWWWWWWWW>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>.

Runnable processes 4.98 load average: 2.92, 3.35, 3.17

Memory Real Virtual Paging (4kB) Process events File/TTY-IO
free 2249.8 MB 0.0 MB 1060.9 pgfaults 3542 pswitch 0 iget
procs 854.1 MB 409.6 MB 1292.0 pgin 5312 syscall 120 namei
files 416.1 MB 249.0 pgout 1092 read 0 dirblk
total 3520.0 MB 409.6 MB 25.9 pgsin 694 write 9129128 readch
1.0 pgsout 2 fork 598266 writech
DiskIO Total Summary 3 exec 0 ttyrawch
read 5168.1 kByte/s 0 rcvint 0 ttycanch
write 988.2 kByte/s 0 xmtint 0 ttyoutch
transfers 737.2 tps
active 3/4 disks Netw read write
en0 56.7 292.8 kB/s
TOPdisk read write busy lo0 0.0 0.0 kB/s
hdisk2 5065 984 kB/s 100%
hdisk0 60 0 kB/s 13%
hdisk1 44 4 kB/s 6%
cd0 0 0 kB/s 0%

Also I have gone through your posting in (TANGCOV performance problem)

In my server vmtune is working and it looks like

askprod:/#/usr/samples/kernel/vmtune
vmtune: current values:
-p -P -r -R -f -F -N -W
minperm maxperm minpgahead maxpgahead minfree maxfree pd_npages maxrandwrt
26213 78641 2 8 120 128 524288 0

-M -w -k -c -b -B -u -l -d
maxpin npswarn npskill numclust numfsbufs hd_pbuf_cnt lvm_bufcnt lrubucket defps
419422 37376 9344 1 93 96 9 131072 1

-s -n -S -L -g -h
sync_release_ilock nokilluid v_pinshm lgpg_regions lgpg_size strict_maxperm
0 0 0 0 0 0

number of valid memory pages = 524277 maxperm=15.0% of real memory
maximum pinable=80.0% of real memory minperm=5.0% of real memory
number of file memory pages = 83261 numperm=15.9% of real memory

also filemon working

hdisk2 ( RAID5 ) in that oracle is runing as jfs.

askprod:/#lsattr -El hdisk2
pvid 000095886ea342fa0000000000000000 Physical volume identifier False
queue_depth 45 Queue depth True
write_queue_mod 1 Write queue depth modifier True
adapter_a ssa0 Adapter connection False
adapter_b none Adapter connection False
primary_adapter adapter_a Primary adapter True
reserve_lock yes RESERVE device on open True
connwhere_shad 95883BEE186E4CE SSA Connection Location False
max_coalesce 0xe0000 Maximum coalesced operation True
size_in_mb 127555 Size in Megabytes False
location Location Label True

AIX setting for aio looks good as a rule of thump


Please help me out in this. DBA also did reindexing the database

 

[bi]

If you have 300 people accessing this database, it is very busy. I'm surprised you only have 3 hdisks. Oracle recommends more mount points than what you are using.

Has the DBA increased the size of the database SGA to take advantage of the 2 GB of memory? There are other tricks the DBA can do to make the database performance better (through changes to the init.ora file), but it appears your big problem is that you have too much on hdisk2. If the DBA can identify the busiest tables and make sure they are separated from each other and are on different disks, that might help too.

Where are the files for the rollback segments and log files? They should not be on RAID5 disks because the parity cost is going to slow you way down. And the files for rollback and logs should be on different disks from each other. Also, no other filesystems that contain logging directories should be on the RAID5 disk.

 

[mouse123]

Hi Bi

I have 18 pdisks in hdisk2 with RAID 5

Here is the oracle mount points
/dev/ora-arch 7864320 986408 88% 458 1% /oraarch
/dev/ora-dump 9830400 373740 97% 102 1% /oradump
/dev/ora-data 13107200 599580 96% 88 1% /oradata1
/dev/ora-datb 13238272 1280432 91% 63 1% /oradata2
/dev/ora-datc 13238272 76164 100% 66 1% /oradata3
/dev/ora-datd 13107200 561908 96% 67 1% /oradata4
/dev/wpc-data 19660800 271268 99% 745 1% /wpc
/dev/wrk-temp 3145728 2989468 5% 7535 1% /temp
/dev/ins-tech 131072 124224 6% 204 1% /tech
/dev/app-all 4194304 641032 85% 54503 6% /sprd
/dev/ins-rest 131072 121372 8% 39 1% /restore
/dev/ora-date 8912896 407056 96% 36 1% /oradata5


Rollbak segments and log files also kept in hdisk5

 

[bi]

You need to better distributed those 18 disks and the files on them -- basically make several different mount points on a combination of RAID5 and RAID1 (mirrored). This is going to require down time because you are going to have to back up the filesystems, remove the LVs, rebuild the arrays, create new LVs and then restore. Are your indexes on this disk, too?

What kind of LV is hdisk5?

Although you have a large number of mount points, which is what Oracle recommends for performance, in effect by having all these files on one hdisk, you are defeating the purpose of having different mount points.

 

[mouse123]

Bi

Sorry it is hdisk2 only not hdisk5

askprod#lsvg -l datavg

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
log-dflt jfslog 2 2 1 open/syncd N/A
log-arch jfslog 2 2 1 open/syncd N/A
log-orcl jfslog 2 2 1 open/syncd N/A
log-wpc jfslog 2 2 1 open/syncd N/A
ora-arch jfs 60 60 1 open/syncd /oraarch
ora-dump jfs 75 75 1 open/syncd /oradump
ora-data jfs 100 100 1 open/syncd /oradata1
ora-datb jfs 101 101 1 open/syncd /oradata2
ora-datc jfs 101 101 1 open/syncd /oradata3
ora-datd jfs 100 100 1 open/syncd /oradata4
wpc-data jfs 150 150 1 open/syncd /wpc
wrk-temp jfs 24 24 1 open/syncd /temp
ins-tech jfs 1 1 1 open/syncd /tech
app-all jfs 32 32 1 open/syncd /sprd
ins-rest jfs 1 1 1 open/syncd /restore
ora-date jfs 68 68 1 open/syncd /oradata5
ora-tmmd jfs 48 48 1 open/syncd /tmmdump
tmp-ptch jfs 32 32 1 closed/syncd /patch

 

[bi]

Again, hdisk2 is going to continue to be close to 100% busy because you've got your entire database (it appears) on one disk. (I"m assuming your indexes are somewhere on this disk, too.)

And with your redos and rollbacks on a RAID5, they are not going to write as fast as the database may require for good performance.

Also, you have 4 jfslogs on this disk. Why? I thought AIX only requires one jfslog per volume group. jfslogs write a lot, too.

I highly recommend that you redesign this disk subsystem. If you will post your email address I will be happy to help you -- and we can post what was done.

 

[stantmann]

CPU Performance looks good in the examples you gave. If the Oracle process take a lot of disk i/o's it might suit you to create another volume group (ie, ORACLEVG) and add additional filesystems. Put all your disk intensive DB's onto that drive/filesystem. This would ensure that the OS is not slowed heavily by the disk writes.

 

[bi]

mouse123,

Go to IBM's Redbooks web site (

http://www.redbooks.ibm.com/),

do a search on "performance tuning" and take a look at Database Performance Tuning on AIX. Chapter 9 talks about disk subsystems, including RAID considerations for the various files in a database. Chapters 5 and 15 talk about Oracle database tuning, too.

Creating another volume group really isn't necessary and putting all your disk intensive files onto that drive is going to put you right back where you are now -- only the hdisk that is 100% busy is going to change! All those pdisks used on hdisk2 can be in the same volume group, you just have to break all the pdisks into smaller pieces (hdisks) and spread the database files around.