finding bottlenecks

[MrTom]

one of my servers is driving me mad! it's only purpose in life is to run weblogic over a java application but it seems to be able to consume all system resources.

during bad periods the load averages hit between 2 and 7. if i monitor cpu usage during this period in top then it only seems to be using 20-50% cpu time and not even constantly. i've watched vmstat for swapping and there is little if any (i have 1.5GB free). iowait is always around 0% in top and disk access is very low - since the java app doesn't store any data on the local drives.

i don't believe that the network can be saturated, if there's a good way to log traffic that would help me.

is there anything else i've missed that could cause the load to rocket? how reliable is top? (i only have the 1 cpu)

 

[crowe]

based on what you have said, with the system not really showing signs of being taxed (cpu, mem, and IO) I would initially suspect a LOT of small requests. I have seen systems that show the exact signs that you are discribing, and it ended up being tons of small processes bombarding the cpu, which caused the run-sz queue (load average) to display high values. Basically if you take a small, non cpu intensive process, it will not cause the processor to work hard. When you stack up a couple of thousand of these small non cpu intensive requests, you get exactly what your discribing. However, this is also not the end all answer. It just happened to be what I have experienced on a couple of systems.
The solution ... one way is to add a cpu, kindive like having 2 spindles in a RAID set. When one is busy, the other could service a request.
Hope this helps...

crowe

 

[MrTom]

yes, the java process uses many threads. i found a sun patch which could be the cause of my problems -

http://access1.sun.com/cgi-bin/rpatch2html?README.106980-17

i've also read about an alternative libthread library in /lib/lwp which i might try to use as well as some extra jvm start options.