High IO Wait!!! 2

khalidaaa · Nov 28, 2006

Hi Guros,

I'm having a problem lately with one of our servers that shows a constant high IO wait as shown below:

Code:

# vmstat 1
System Configuration: lcpu=1 mem=2048MB
kthr     memory             page              faults        cpu     
----- ----------- ------------------------ ------------ -----------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa 
 1  2 409176 123834   0   0   0 341 1157   0 1203 85628 3738 21 20 52  7
 0  1 409180 123830   0   0   0 2613 45774   0 615 4536 3885 19  8  0 73
 0  6 409180 123831   0   0   0 2692 53878   0 602 4175 3941 12  5  0 83
 1  0 408936 124073   0   0   0 1505 36609   0 557 3655 2345 13  7  0 80
 1  0 408936 124078   0   0   0 238 5456   0 421 14732 2843 48  3  0 49
 1  0 408936 124071   0   0   0 131 1344   0 451 6569 3320 36  3  0 61
 0  1 409180 123834   0   0   2  88 2376   0 500 8147 5644 32  9  0 59
 0  1 409180 123834   0   0   0   0    0   0 426 4212 3315 25  7  0 68
 0  1 409180 123834   0   0   0   0    0   0 450 5914 6163 15  7  0 78
 3  0 409191 123819   0   0   0  72 1633   0 500 6695 2504 27  5  0 68
 1  1 409191 123824   0   0   0   9  286   0 382 16289 4880 62  7  0 31
 0  1 409191 123824   0   0   0   0    0   0 398 5731 5404 46  3  0 51
 2  1 409191 123824   0   0   0   0    0   0 440 4135 3389 15  4  0 81
 0  1 409191 123823   0   0   0   0    0   0 480 6328 6123 27  4  0 69
 0  2 409216 123799   0   0   0 185 2276   0 771 3938 2184 11  2  0 87
 1  1 409216 123799   0   0   0   0    0   0 668 18927 5324 78 14  1  7
 0  1 409219 123796   0   0   0   0    0   0 521 4202 3293 19  3  0 78
 0  1 408975 124041   0   0   0   9   17   0 435 4956 4699 21  4  0 75
 0  1 408975 124035   0   0   0  64  535   0 641 5219 4591 20  9  0 71
 0  5 408975 124035   0   0   0   9   32   0 533 6426 5376 19  6  0 75
kthr     memory             page              faults        cpu     
----- ----------- ------------------------ ------------ -----------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa 
 1  0 408975 124039   0   0   0  10   40   0 319 18123 3911 69 10  0 21
 0  2 408975 124041   0   0   0   9  334   0 497 5316 4539 28  4  0 68
 0  1 408975 124039   0   0   0   0    0   0 496 6276 5050 38  4  0 58
 0  1 408996 124014   0   0   0  60 1700   0 635 5517 4588 14  5  0 81
 0  1 409021 123995   0   0   0  55 1199   0 620 6931 4886 27  7  0 66
 1  1 409043 123967   0   0   0  53  527   0 579 19974 4732 76  7  0 17
 3  0 409066 123948   0   0   0  90 2472   0 621 6163 4192 40  6  0 54
 0  1 409106 123904   0   0   0 204 5459   0 716 5472 3855 23  4  0 73
 0  1 408866 124145   0   0   0 144 2158   0 723 7554 6014 33  9  0 58
 0  1 408871 124141   0   0   0  71  873   0 1533 7859 4898  9 11  0 80
 0  1 408876 124135   0   0   0 138 3002   0 1529 22091 6251 54  8  0 38
 0  2 408876 124134   0   0   0 603 13920   0 745 12588 4285 54  6  0 40
 0  2 408881 124127   0   0   0  18  141   0 627 7805 5727 37  8  0 56
 1  0 408881 124133   0   0   0   9   46   0 744 5075 3707 14  8  0 78
 2  1 408902 124106   0   0   0  28   65   0 915 7763 6668 29  6  0 65
 1  1 408912 124096   0   0   0   0    0   0 408 9884 3023 37  6  0 57
 0  1 408912 124096   0   0   0   0    0   0 365 12311 4133 61  5  0 34

Code:

# iostat hdisk0 1

System configuration: lcpu=1 disk=12

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0         17.1              21.0     19.6       52.3       7.1     
                " Disk history since boot not available. "


tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        247.5              28.7      5.9        0.0      65.3     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          99.0     1239.6     145.5          0      1252

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        1194.9              20.2     12.1        0.0      67.7     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0         100.0     1454.5     136.4          0      1440

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        292.0              33.0      6.0        0.0      61.0  

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          90.0     1673.0     143.0          0      1673

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        1014.0              13.0      3.0        0.0      84.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          99.0     1416.0     139.0          0      1416

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        292.0               9.0      4.0        0.0      87.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          98.0     1041.0     151.0          0      1041

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        892.0              15.0      3.0        0.0      82.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          97.0     1444.0     137.0          0      1444

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        288.1              11.9      7.9        1.0      79.2     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          98.0     1378.2     145.5          0      1392

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        733.0               8.0      3.0        0.0      89.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0         100.0     1377.0     134.0          4      1373

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        291.0              16.0      4.0        0.0      80.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          99.0     1493.0     135.0          0      1493

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        792.0              12.0     28.0        0.0      60.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          98.0     1369.0     130.0          0      1369

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        291.0              13.0      4.0        0.0      83.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0         100.0     1556.0     130.0          0      1556

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        809.0               6.0      3.0        0.0      91.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          99.0     1460.0     126.0          0      1460

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        291.0              43.0      5.0        0.0      52.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          87.0     1201.0     144.0         12      1189

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        865.0              39.0      5.0        0.0      56.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          96.0     982.0     117.0          4       978

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        290.0              10.0      4.0        0.0      86.0     

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          99.0     1481.0     132.0          0      1481

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        768.0              20.0      2.0        0.0      78.0

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          94.0     1282.0     130.0          0      1282

I beleive that one of the lv's on that disk was causing the problem! i shifted this lv (with migratepv) to another disk and now i'm having the same problem with that disk! The lv contains a database data file.

one more thing that scares me is this process:

Code:

# ps aux | more
USER       PID %CPU %MEM   SZ  RSS    TTY STAT    STIME  TIME COMMAND
root       516 60.1  0.0   16   16      - A      Oct 27 27449:34 wait

Any help is appreciated

Thanks

Regards,
Khalid

khalidaaa · Dec 2, 2006

Again Thank you for the support

mbourne2006, you are right, but unfortunately this is a production system and i'll have to arrange for the downtime in order to do this!

kHz, can i do that while the database is up and running?

Regards,
Khalid

kHz · Dec 3, 2006

You should be able to do it while the database is up because it is copying the lp and when the last partition is copied then it is gone from the old region. It would be best for the database to be down, but if that isn't possible, then do it when there is the least amount of activity.

khalidaaa · Dec 3, 2006

Thanks kHz.

I will try to do this tomorrow as i'm stuck today with something else and i will let you know

Regards,
Khalid

Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

High IO Wait!!! 2

khalidaaa

Technical User

khalidaaa

Technical User

kHz

MIS

khalidaaa

Technical User

Similar threads

Part and Inventory Search

Sponsor