Tablespace create slow (udb) on aix - high i/o waits

[johnofg]

On running udb create tablespace (only process on the system), it takes excessive long to run. On monitoring, vmstat shows high io waits and no page scans, iostat shows low data throughput with disk begin 100$ busy. During periods where there are no page scans, system response is minimal (telnet). Once pages scan start, response returns.

vmstat
-------
2 4 530987 923166 0 0 0 0 0 0 898 1555 393 2 25 33 39
2 4 530987 923160 0 0 0 0 0 0 843 1555 392 2 25 12 60
2 4 530987 923155 0 0 0 0 0 0 803 1634 395 2 25 23 49
2 10 530987 923152 0 0 0 0 0 0 1013 1740 988 2 26 20 52
2 10 531081 923048 0 0 0 0 0 0 819 2188 517 2 26 2 70
3 3 530987 922631 0 0 0 7129 186652 0 1174 2469 1900 2 45 17 36
3 9 531917 922030 0 0 0 9150 245239 0 1263 3871 1772 3 55 10 32
3 5 531748 921172 0 0 0 15984 183671 0 2187 12052 6665 5 67 2 26
2 1 531275 922477 0 0 0 7178 87435 0 1524 5978 2192 4 47 16 33
1 2 530914 922831 0 0 0 0 0 0 817 5518 449 4 28 15 53
2 2 530855 922889 0 0 0 0 0 0 804 5071 440 3 28 24 46
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 2 530855 922889 0 0 0 0 0 0 739 4337 430 2 28 23 47
2 2 530955 922788 0 0 0 0 0 0 725 4482 433 3 28 24 45
2 2 530949 922793 0 0 0 0 0 0 729 5435 439 4 28 24 44
2 2 530917 922825 0 0 0 0 0 0 725 4539 430 3 27 24 46
2 2 530855 922886 0 0 0 0 0 0 703 4947 434 3 28 21 48
2 2 530855 922885 0 0 0 0 0 0 704 5025 435 3 28 21 48

IOstat
-------

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 77.4 6.0 1.5 12 0
hdisk35 99.9 113.9 24.0 0 228
hdisk38 99.9 373.5 21.5 0 748
hdisk42 99.9 257.7 23.5 0 516
hdisk11 99.9 149.8 23.0 0 300
hdisk15 99.9 321.6 22.0 0 644
hdisk19 99.9 271.7 21.0 0 544


System will burst into high io rates once in a while during which time, page scans are taking place.

NMON shows one cpu 100% busy during slow downs, all other cpu's are waiting on IO.

System has AIX 5.1, 5 cpus, and 6 GB ram.

Changing vmtune to reduce file cache does not affect slowdowns.

Does anyone have any suggestions?

thanks.

 

[bimux]

This is a wild guess .. did you setup asyncio ? ..
Are you setup in a disk array ? .. What Raid level ? ...

Another wild guess ... Did you try to setup release behind read write on the filesystem mounted ?

It seems more a UDB problem ... can ou get to the latest service pack ?

 

[czins]

Hello,

I had some similar effects with another application running under AIX5.1-ML04. After migrating to AIX5.1-ML05 the I/O wait has disappeared.

Regards
Stefan

 

[sbix]

Some simple questions:
This tablespace is spread across all the disks?
The tablespace is built on a filesystem or on a raw device?
The hosting LV is striped across all the disks?
If so, these disks are under the same controller?
If so is it a SCSI controller?

 

[Asp7479]

This is a known bug in Aix 5L. You need to upgrade the device drivers for FC card 6228 and 6239. Also, Flash the cards with the latest Microcode. I had this problem and now it does not happen after doing the above.

 

[johnofg]

First, thanks for all the responses.

For czins and asp7479, We will try SL5 as that was planned anyways. I will check to see if our FC are up to date.

Bimux, we do have async IO on. We are using an IBM ESS storage system -- the underlying disks are confirgured as raid 5 arrays.

Sbix, the tablespace is allocated with 3 containers (using files, not raw devices) on 3 different file systems. The file systems are hosted on an ESS storage system. There are two FC interfaces to the ESS.
The IO waits seem to occur when I create the table space with more than 1 container.

Thanks for all you help.

 

[johnofg]

We have upgraded AIX (5.1) to ML05 and it appears that the excessive wait IO's have gone away (at least when formatting UDB tablespaces.)

We had a diffent FC card but the drivers and microcode was already at the latest available.

Thanks to czins and asp7479 for your info.