CPU Utilization issues

[dsm666]

Good morning,
I need some help figuring out what's eating up my cpu. My application can't get enough cpu to do its job. this is a sunfire V440 2CPU's at 1/593 GHZ with 8GB of memory. In the morning hours the box is at less than 3%. I can't figure out what else is using the CPU. We use foglight and it shows that my user is not using much at all.

load averages: 3.24, 3.38, 3.43 us352 08:37:51
193 processes: 186 sleeping, 4 running, 1 zombie, 2 on cpu
CPU states: 29.3% idle, 3.8% user, 66.9% kernel, 0.0% iowait, 0.0% swap
Memory: 8.0G real, 5.3G free, 562M swap in use, 13.1G swap free

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
4080 ceunix 1 10 0 4048K 2504K run 0:01 2.67% cmumboxd
27764 ceunix 1 59 0 5312K 2168K sleep 18:46 2.12% cmuctld
3814 ceunix 1 59 0 4192K 3320K sleep 0:01 1.93% ftpd
4084 ceunix 1 59 0 4064K 2584K sleep 0:00 0.94% cmumboxd
4091 ceunix 1 0 0 4048K 2344K run 0:00 0.85% cmumboxd
4083 ceunix 1 50 0 4048K 2344K sleep 0:00 0.72% cmumboxd
3890 ceunix 1 59 0 4064K 2584K sleep 0:00 0.70% cmumboxd
4090 ceunix 1 59 0 4032K 1464K sleep 0:00 0.67% cmumboxd
4064 ceunix 1 59 0 4048K 2504K sleep 0:00 0.32% cmumboxd
4087 ceunix 1 60 0 4048K 2488K sleep 0:00 0.28% cmumboxd
4072 ceunix 1 59 0 4080K 2544K sleep 0:00 0.21% cmumboxd
27785 ceunix 1 43 0 4040K 2504K sleep 1:38 0.19% cmumboxd
27765 ceunix 1 49 0 5328K 2384K sleep 1:58 0.19% cmuauthd
3628 ceunix 1 59 0 4080K 2600K sleep 0:00 0.19% cmumboxd
6133 daemon 1 59 0 12.0M 10.9M sleep 0:07 0.16% vasd

 

[dsm666]

look at it now:

load averages: 5.62, 3.96, 3.60 us352 08:46:48
164 processes: 159 sleeping, 3 running, 2 on cpu
CPU states: 0.5% idle, 12.9% user, 86.6% kernel, 0.0% iowait, 0.0% swap
Memory: 8.0G real, 5.3G free, 511M swap in use, 13.1G swap free

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
9196 ceunix 1 10 0 4048K 2496K sleep 0:36 30.19% cmumboxd
5917 ceunix 1 39 0 6336K 4448K run 0:03 2.54% cmusvwd
27764 ceunix 1 49 0 5312K 2168K sleep 19:13 2.19% cmuctld
9784 ceunix 1 50 0 4048K 2504K sleep 0:00 1.50% cmumboxd
9800 ceunix 1 59 0 4080K 2656K sleep 0:00 0.84% cmumboxd
9797 ceunix 1 50 0 4048K 2344K sleep 0:00 0.78% cmumboxd
9804 ceunix 1 0 0 4032K 1464K sleep 0:00 0.60% cmumboxd
8600 ceunix 1 59 0 7568K 3216K run 0:01 0.56% sshd
27787 ceunix 1 59 0 7528K 6128K sleep 0:43 0.34% cmuacd
27765 ceunix 1 59 0 5328K 2384K sleep 2:01 0.26% cmuauthd
9781 ceunix 1 59 0 4072K 2536K sleep 0:00 0.26% cmumboxd
9303 ceunix 1 59 0 4080K 2600K sleep 0:00 0.19% cmumboxd
27785 ceunix 1 23 0 4040K 2504K sleep 1:40 0.18% cmumboxd
8617 ceunix 1 59 0 4080K 2560K sleep 0:00 0.14% cmumboxd
15234 ceunix 1 49 0 6816K 4000K cpu/0 0:06 0.13% top

 

[dsm666]

here is what vmstat shows
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m3 m1 in sy cs us sy id
1 0 0 14402544 6435424 1249 2043 3006 1 1 0 1 13 0 0 8 1351 17468 1095 16 28 56
us352:ceunix $