most cpu activity in iowait... 1

[Chapter11]

I'm watching my new 5.1 system running Oracle, and I'm noting some strange behavior.

Virtually all cpu time is being categorized under i/o wait (as opposed to idle, system, or user. The machine has 4 cpus, and all of them are exhibiting this.

Paging activity is negligible.

Processes on cpu is negligible (0 or 1 at 1 sec intervals from vmstat).

Disk activity is almost nonexistent (watching topas).

Network activity is the same.

This machine is just sitting here, apparently idle in all respects except that it thinks it's choking on i/o activity. I'm also watching iostat at 1-sec intervals, and there's activity, but I don't think there's enough activity to justify throwing 4 cpus into 95%+ iowait.

The utilization of this machine is practically the same from before its upgrade from 4.3.3 to 5.1.

In userland, the issue is presenting as slow/no response to users.

Right now the system is relatively fine except the iowait times for the cpus are still much higher than I think they should be. I'm debating blaming Oracle unless someone has any idea what this is about.

 

[metzlej]

can you post the output of vmstat (vmstat 1 40) and have you performed any tuning with vmtune, etc?

Also, have you ran filemon to see which logical volumes are running the hottest?

Jarrett
IBM Certified Systems Expert pSeries HACMP for AIX
IBM Certified Advanced Technical Expert for AIX 5L and pSeries
- AIX 5L Systems Administration
- AIX 5L Performance and System Tuning
- p690 Technical Support
- pSeries HACMP for AIX

 

[Chapter11]

problem appears to be resolved.

I did do some vmstat checks while I was troubleshooting: most interesting info from it was the size of the "b" column - unusual number of processes waiting on i/o, but wasn't anything I already knew.

there is some vmtune work -p 5 -P 15 which is what we've found works well on our Oracle machines.

there was virtually no disk activity of any sort when the problem was going on - I could sit for several seconds watching the disk section of topas and have *NO* activity on any disk.

solution: WLM was in active mode, with almost nothing changed besides defaults. only changes were additions of classes and subclasses with assignment rules - no restrictions of any type (cpu, mem, diskio) were configured. Changed WLM to passive mode and the system is behaving again.