What is healthy paging space usage?

[Einstein47]

I have a number of 4.3.3 servers that run WebSphere 4.0 among other apps. It seems that the paging space used tends to be quite high on the development servers. I would like to know what would be considered "normal" paging space usage?

One of our team members feels that any paging space used is bad and wants to reboot all our servers daily to minimize using paging space. I have tried to tell her that some usage is good, but I didn't have solid numbers to back up my argument. I know that the WebSphere java applications take longer to load after a reboot, and I have tried to explain this as well without success.

Also, can I (or should I) try to control the paging space usage? It seems that often times a server will use up all available paging space and then a hard reboot is necessary. I would like to eliminate this by being proactive; however, rebooting once a day seems like overkill.

If anyone can shed some light on paging space usage, I will be eternally grateful. Einstein47
("For every expert, there is an equal and opposite expert." - Arthur C. Clarke)

 

[AIXSPadmin]

If you could send an lsps -s; lsps -a; vmstat; svmon -G; bootinfo -r; uptime; and a monitor or topas then it would be more beneficial in describing where the problem may be. We are running WebSphere on about a dozen nodes and don't have paging problems. The info would help tremendously.

 

[Einstein47]

Here you go - but since it is a snap-shot of just one server at one point in time, I don't know how useful it will be:

Topas Monitor for host:    sandbox              EVENTS/QUEUES    FILE/TTY
Wed Aug  7 13:02:50 2002   Interval:  2         Cswitch    2446  Readch    29666
                                                Syscall    4919  Writech   20690
Kernel   28.5   |########                    |  Reads       114  Rawin         0
User     71.4   |####################        |  Writes       35  Ttyout     1157
Wait      0.0   |                            |  Forks         2  Igets         0
Idle      0.0   |                            |  Execs         3  Namei       438
                                                Runqueue    8.0  Dirblk        5
Interf   KBPS   I-Pack  O-Pack   KB-In  KB-Out  Waitqueue   0.0
en0        2.5    12.0    10.5     0.7     1.8
lo0        0.0     0.0     0.0     0.0     0.0  PAGING           MEMORY
                                                Faults      321  Real,MB    1023
Disk    Busy%     KBPS     TPS KB-Read KB-Writ  Steals        0  % Comp     78.9
hdisk0    1.0      4.0     1.0     0.0     4.0  PgspIn        0  % Noncomp   9.6
hdisk1    0.0      0.0     0.0     0.0     0.0  PgspOut       0  % Client    0.5
                                                PageIn        0
amgr     (30444)  4.6% PgSp:13.8mb notes        PageOut       1  PAGING SPACE
Xvnc     (58702)  4.0% PgSp:23.3mb root         Sios          1  Size,MB     832
Xvnc     (35866)  2.0% PgSp:11.4mb root                          % Used     50.0
dtterm   (61334)  2.0% PgSp: 1.1mb root                          % Free     49.9
java     (32174)  1.3% PgSp:232.5m: root
java     (48594)  0.7% PgSp:72.7mb root
dsmscoutd(11610)  0.7% PgSp: 1.1mb root            Press "h" for help screen.
gil      (1032)   0.7% PgSp: 0.0mb root            Press "q" to quit program.
dtterm   (41282)  0.7% PgSp: 1.0mb root
event    (31738)  0.7% PgSp: 1.4mb notes
topas    (55890)  0.7% PgSp: 0.5mb root
server   (18864)  0.7% PgSp: 5.0mb notes
topas    (62854)  0.7% PgSp: 0.4mb root
endpoint (4646)   0.7% PgSp: 0.4mb root
oninit   (24028)  0.7% PgSp: 1.4mb root
maps     (32516)  0.0% PgSp: 1.1mb notes

lsps -s
Total Paging Space   Percent Used
      832MB              50%

lsps -a
Page Space  Physical Volume   Volume Group    Size   %Used  Active  Auto  Type
paging00    hdisk0            rootvg         384MB      54     yes   yes    lv
hd6         hdisk1            rootvg         448MB      47     yes   yes    lv

vmstat
kthr     memory             page              faults        cpu
----- ----------- ------------------------ ------------ -----------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
 1  0 267255 32768   0   1   2   5   31   0 1798 3575 2639 23  9 67  1

svmon -G
               size      inuse       free        pin    virtual
memory       262134     229365      32769      11860     267256
pg space     212992     105542

               work       pers       clnt
pin           11860          0          0
in use       205503      23862          0

bootinfo -r
1048576

uptime
  01:04PM   up 1 day,   1:48,  22 users,  load average: 1.53, 1.06, 1.16

Einstein47
("For every expert, there is an equal and opposite expert." - Arthur C. Clarke)

 

[Yegolev]

Our paging is designed mainly as a buffer against spikes. Right this minute on our biggest box:

lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
paging03 hdisk6 pagingvg 5120MB 1 yes yes lv
paging02 hdisk5 pagingvg 5120MB 1 yes yes lv
paging01 hdisk2 pagingvg 5120MB 1 yes yes lv
hd6 hdisk1 rootvg 1024MB 2 yes yes lv

As you can see, we have extra for "emergencies", but this machine runs SAP and Oracle.

We are installing WebSphere somewhere but I have not had a chance to play with it yet. I suggest getting some data over time to see what your utilization is like, mainly to determine what is causing your lockups. I guess you should probably have either more paging or more RAM; paging is cheaper. =) IBM Certified -- AIX 4.3 Obfuscation

 

[AIXSPadmin]

If possible add some extra paging space off of rootvg and onto another vg. Also, increase your paging to 1024MB per paging space.

Our production WebSphere node has 4GB memory and has 2 paging spaces each of 1024MB. Our paging is at 22% but vmstat doesn't report any page-ins or page-outs. WebSphere does use resources.

I don't know how many JVMs you are running, but I saw 2 in your topas output. Also, how much memory is each JVM configured for? This would chew up memory.

 

[Yegolev]

Just checked one of the development WebSphere boxes, it has 2GB of RAM and a single 2GB paging space. IBM Certified -- AIX 4.3 Obfuscation