ps -ef¦ grep takes forever

[alexhu]

RS/6000 S70 8 x 125Mhz procs
Running Oracle 8.0.5
Load factor about 4
Total processes on the system ~2000

When I check the databases are up ( ps -ef| grep ora_ )
the result takes at least 1 minute to come back.

This has slowed down over the last month or so

example of sar is

09:00:10 24 30 42 4
09:20:11 25 33 33 8
09:40:12 26 33 32 9
10:00:16 23 30 34 13
10:20:16 26 31 29 14

so the system isn't really slow in itself

other commands (df -k, errpt, uptime ) come back in 'normal' speed

Any ideas ?

Alex

 

[slt589]

How long does ps -fe take (ie. without the grep)

 

[alexhu]

ps -ef on its own takes 6 seconds to start listing

Alex

 

[unixfreak]

I have the same problem on my servers. It seems to depend on the length of the command line that started the process.

We haven't got any other problems so I have accepted this as "normal" behavior.

Cheers

 

[slt589]

6 seconds to print 1st line seems a long time.

Once it starts printing, does it stall?

Does aix have vmstat, or top? (or something else)

Start vmstat or top before you run ps.

Observe behaviour when ps starts.

Look for excessive paging, context switches, run queue

How long does it take for full listing (ie print to screen)?

How long for ps -fe > /dev/null 2>&1 (ie no listing)

 

[slt589]

BTW, how are you eliminating your grep process from the output ?

 

[alexhu]

ps -ef| grep ora_ | grep -v grep

Alex

 

[slt589]

ps -fe | grep [o]ra_

avoids the 2nd pipe and associated process

 

[bi]

Could it be that there are a lot of ora processes starting and ending and that's why it's taking so long? To check to see if an Oracle database is up, I usually just run ps -ef | grep pmon. pmon is Oracle's process monitor.

 

[arvibm]

Hi,

Try to trace out the process consuming maximum amount of memory.

ps aux | more
svmon -Pt 5

Post the output of following commands.

sar -u 5 10
vmstat 5 7
go to directory-/usr/samples/kernel
and run ./vmtune
post the outputs.

Regards

arvibm

 

[alexhu]

processes using max memory vary considerably

/home/oracle > sar -u 5 10

AIX rs6000 3 4 004003234C00 11/03/03

09:20:09 %usr %sys %wio %idle
09:20:14 28 46 25 1
09:20:19 22 40 37 1
09:20:24 38 45 16 0
09:20:29 29 26 41 4
09:20:34 27 26 44 4
09:20:39 30 53 16 2
09:20:44 26 51 21 2
09:20:49 15 19 62 4
09:20:54 18 19 58 5
09:20:59 16 47 35 2

Average 25 37 35 2
/home/oracle > vmstat 5 7
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 4 629869 77 0 34 31 64 131 0 167 247 244 99 99 85 95
4 25 629604 597 0 256 171 916 2416 0 1996 10542 7035 13 29 4 55
4 23 630631 151 0 299 161 1082 3083 0 2030 19414 8423 18 24 3 56
3 23 631126 71 0 281 159 1053 2684 0 2103 13204 4748 15 17 4 63
4 19 630912 471 0 186 218 1185 3099 0 2183 15024 7078 18 27 4 52
3 17 631359 99 0 256 167 967 2401 0 2132 13945 9266 18 47 3 32
7 23 631436 105 0 297 102 835 1929 0 1991 16624 8627 27 42 2 29
/home/oracle > cd /usr/samples/kernel
/usr/samples/kernel > ./vmtune
vmtune: current values:
-p -P -r -R -f -F -N -W
minperm maxperm minpgahead maxpgahead minfree maxfree pd_npages maxrandwrt
104650 418600 2 8 120 128 524288 0

-M -w -k -c -b -B -u -l -d
maxpin npswarn npskill numclust numfsbufs hd_pbuf_cnt lvm_bufcnt lrubucket defps
419422 20992 5248 1 93 593 9 131072 1

-s -n -S -L -g -h
sync_release_ilock nokilluid v_pinshm lgpg_regions lgpg_size strict_maxperm
0 0 0 0 0 0

number of valid memory pages = 524277 maxperm=79.8% of real memory
maximum pinable=80.0% of real memory minperm=20.0% of real memory
number of file memory pages = 307936 numperm=58.7% of real memory


Alex

 

[arvibm]

Hi Alexhu,

System is CPU Bound if in sar -u 60 30 output %usr+%sys > 80%.but ur sar output seems to be quite normal.Problem with ur vmstat output.In ur vmstat output paging space I/O is taking place approaching the systems memory limit.wa value of vmstat indicates I/O wait percentage of CPU AND IF THE WA VALUE IS NON-ZERO , SIGNIFICANT TIME SPENT ON WAITING FILE I/O.Check ur iostat output also if %io wait > 25% and %tm_acct > 70% the system is I/O Bound.
From the output which u have given above i can say lot of paging is happening which might be a cause of problem.Pls make sure that paging space is not mirrored.how many cpu's u have on ur system.what is the total amount of memory.
I recommend u to change ur vmtune parameters.

Pls set the default value of some parameter listed below.This will help in performance improvement

This can be set by the command /usr/kernel/sample/vmtune
-f minfree 120xN(default) if 4 cpu it is 480
-F maxfree 128xN (default) 512
-R = maxfree -minfree =32
-P min perm 5
-p maxperm 10
-s 1
-b 200
-B 800

N->No. of CPU'S

Take mksysb backup of server before changing vmtune parameters of server.This entry u have to put in the /etc/rc before the mount all command...

Take reboot of server after changing vmtune parameters.

I hope this will resolve ur problem.Sorry for late reply as i was not in office from last 3-4 days.


Regards

arvibm