system performance - ouput from nmon tool

[shoux]

Hi all !

can someone for the said report ? especially in system performance issue. at present, my system is in bad condition




??nmon-v10p???a=disk-Adapters????Host=SERVER_1???Refresh=2 secs???17:23.53??
??Memory-Use?????????????????????Paging????????????????????????Stats????????????
? Physical PagingSpace pages/sec In Out FileSystemCache ?
?% Used 93.7% 6.6% to Paging Space 0.5 0.0 (numperm) 76.2%?
?% Free 6.3% 93.4% to File System 1505.5 43.3 Process 12.9%?
?MB Used 7679.7MB 169.3MB Page Scans 1755.4 System 4.6%?
?MB Free 512.3MB 2390.7MB Page Cycles 0.0 Free 6.3%?
?Total(MB) 8192.0MB 2560.0MB Page Steals 1477.1 ------?
? Page Faults 467.0 Total 100.0%?
?Min/Maxperm 1548MB( 19%) 6191MB( 76%) note: % of memory ?
?Min/Maxfree 120 128 Total Virtual 10.5GB User 85.3%?
?Min/Maxpgahead 2 8 Accessed Virtual 1.5GB 14.4% Pinned 10.2%?
????????????????????????????????????????????????????????????????????????????????

Thanks


shoux

 

[costiles]

Capture the results of vmstat 2 10 - and display it here. That is a good start for performance monitoring - that is vmstat at 2 second intervals for 10 iterations. That will tell us more than the jumbled report that you have displayed.

 

[shoux]

Thanks for the response. herewith the vmstat report.

[#] >vmstat 2 10
System Configuration: lcpu=8 mem=8192MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 7 547074 529 0 0 1 394 887 0 1431 15887 1080 2 4 73 21
2 11 547108 164 0 0 0 287 335 0 1863 48806 2803 5 18 18 59
1 14 547123 170 0 1 0 174 203 0 1436 27642 1814 2 15 16 67
1 13 547296 134 0 1 0 624 712 0 1591 36810 2246 5 15 17 63
2 7 548443 174 0 1 0 1578 2376 0 2129 39656 3529 7 15 26 53
2 8 549935 168 0 5 0 1556 2242 0 1820 56528 3101 10 20 23 47
2 8 548956 823 0 2 1 700 968 0 1860 47377 2835 12 15 26 47
2 7 544823 3375 0 7 0 0 0 0 1810 46477 2956 6 19 24 50
1 7 544136 2601 0 15 0 46 64 0 1793 51438 2833 7 17 24 51
1 10 544455 1861 0 29 0 0 0 0 1815 54275 1826 7 16 15 63


shoux

 

[hirschaj]

You I/O wait is really high. iostat and filemon can help you determine what is causing such high wait percentages.


Jim Hirschauer

http://www.aixexpert.com

 

[shoux]

here is ouput from iostat 2 5

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.5 2.0 0.5 4 0
hdisk1 0.0 0.0 0.0 0 0
dac0 0.0 0.0 0.0 0 0
dac0-utm 0.0 0.0 0.0 0 0
dac1 0.0 1070.0 262.5 1396 744
dac1-utm 0.0 0.0 0.0 0 0
hdisk2 98.5 1070.0 262.5 1396 744
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle % iowait
77.0 27035.0 12.2 3.5 56.8 27.4

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.0 0.0 0.0 0 0
hdisk1 0.0 0.0 0.0 0 0
dac0 0.0 0.0 0.0 0 0
dac0-utm 0.0 0.0 0.0 0 0
dac1 0.0 1432.0 356.5 56 2808
dac1-utm 0.0 0.0 0.0 0 0
hdisk2 99.5 1432.0 356.5 56 2808
cd0 0.0 0.0 0.0 0 0

 

[hirschaj]

Looks like hdisk2 is your culprit. What is on that one?


Jim Hirschauer

http://www.aixexpert.com

 

[shoux]

Hi Jim,

Thanks for the response.
For your info, my machine consists of ten harddisks with total space storage of 364 GB. The hardisks are grouped into three main hard disk, hdisk0, hdisk1 and hdisk2. Hdisk0 and hdisk1 (rootvg) stores all the operating system whereas hdisk2 (datavg) contains the database files.
Recently we detected that the hdisk2 has reached maximum performance whereby the percentage of time is active reads (% tm_act) 100% in every second. This would mean, the hdisk2 is extremely busy and will cause the I/O process to become slow. Therefore, I plan to reorganize/reformatted the hardisk2 into multiple harddisk i.e. create additional volume group such as follow:-

hdisk0 - rootvg
hdisk1 - rootvg
hdisk2 - datavg1
hdisk3 - datavg1
hdisk4 - datavg1
hdisk5 - datavg1
hdisk6 - datavg2
hdisk7 - datavg2
hdisk8 - datavg2
hdisk9 - datavg3

Machine specification
Regatta/p670 – FustT,8cpu,8GB memory.

Please advise me whether the exercise will increase the system performance.


Thank you in advance.



shoux

 

[hirschaj]

Spreading the load over multiple spindles should really help a lot. There is no real performance gain by using multiple volume groups. The method you use to spread out the LV's will be the key to improving performance. I would use filemon to determine what my most heavily accessed LV's are before I decided on my final layout. What database are you using? If it is Oracle there are a bunch of ways to improve performance already documented on these forums.


Jim Hirschauer

http://www.aixexpert.com

 

[p5wizard]

But as the disks are FastT (raid5?) volumes, I don't see where the extra spindles would come from...

What is your filesystem layout on datavg - hdisk2? JFS or JFS2? One or more FS? Do you have enough file system buffers?

Please post an output of "lsvg -l datavg" and I'll have a look.


HTH,

p5wizard

 

[shoux]

Thank you.

Jim,
I'm using Universe database

p5wizard,

here is the ouput

datavg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
lvapp jfs2 229 229 1 open/syncd /app
lv02 jfs2 504 504 1 open/syncd /uv2
lv01 jfs2 841 841 1 open/syncd /uv1
lv03 jfs2 459 459 1 open/syncd /uv3
lvuvspool jfs2 100 100 1 open/syncd /uvspool






shoux

 

[p5wizard]

OK, jfs2 - what is your setting for number of FS buffers

ioo -L|grep j2_nBufferPerPagerDevice

do a "vmstat -sv" every 10 seconds and look at the values for "client filesystem I/Os blocked with no fsbuf" (last line outputted normally)

If this counter keeps increasing, you probably need to up that ioo parameter, but you need to unmount/mount your jfs2 filesystems for a change to have effect. The default is 512 but you can go "as high as you want" (2G-1) - it does use more memory though, so it needs to be tuned to a reasonably high value - I'd start by doubling to 1024.



HTH,

p5wizard

 

[hirschaj]

One other really important thing to remember when dealing with database performance issues is that bad SQL can be the cause of the problem in the first place. Whenever I see a database with I/O issues I ask the programmers to take a look at their SQL to make sure they are not doing anything that would cause a hugh amount of I/O unintentionally. So far they have always found bad SQL and my I/O problems go away once they correct the code.


Jim Hirschauer

http://www.aixexpert.com

 

[p5wizard]

That is very true Jim, but in the mean time it won't hurt to check if most of the IO-wait isn't just a wait for an empty FSBUF structure or its JFS2 counterpart...

Not a lot of effort involved.

HTH,

p5wizard

 

[hirschaj]

Hi p5wizard,

I was not implying that shoux should not tune his filesystem buffers, just that it is wise to see if you hican fix the root of the problem. Hopefully tuning can get them by until the root cause is dicovered.


Jim Hirschauer

http://www.aixexpert.com

 

[p5wizard]

Just wanted to make sure shoux gets my point...

HTH,

p5wizard

 

[shoux]

Hi guy ! thanks for the respon

p5wizard

here is the ouput. can you guide me how to increase the ioo parameter to be 1024?

#ioo -L|grep j2_nBufferPerPagerDevice
j2_nBufferPerPagerDevice 512 512 512 0 2G-1 M


#vmstat -sv 10


0 remote pageouts scheduled
10913440 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
25554 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
1050457 external pager filesystem I/Os blocked with no fsbuf


Thank you guy !


shoux

 

[p5wizard]

You gotta check if that last value keeps increasing while your database is active...

smitty ioo I think, but as stated earlier, you have to stop the DB, unmount FSs, mount FSs again for the new value to have effect.


HTH,

p5wizard

 

[p5wizard]

That's smitty TunIoo

use 2nd AND 3rd option to change value
2nd is for current running system
3rd is for next reboot

THEN

unmount & mount your DB filesystems


HTH,

p5wizard

 

[shoux]

Hi !

An error occured while running this command "Smitty TunIoo"


1800-007 There are currently no SMIT
screen entries available for this FastPath.
This FastPath may require installation of
additional software before it can be accessed.


Thank you



shoux

 

[p5wizard]

oslevel -r ?

HTH,

p5wizard