top shows iowait is in high 90's %

[Scunningham99]

Hi we have a solaris v880 runing version 8
# uname -a
SunOS eurfinunx001 5.8 Generic_108528-29 sun4u sparc SUNW,Sun-Fire-880

Top shows iowait in high 90's. This server is connectet to eva5000 hp san and also has internal disks.

I have looked at iostat -xtn, to see which mount's are producing wait. Can anyone help me further as to finding the root of the problem.

# top
last pid: 24983; load averages: 0.13, 0.09, 0.12 01:47:50
224 processes: 223 sleeping, 1 on cpu
CPU states: 0.0% idle, 0.8% user, 1.0% kernel, 98.2% iowait, 0.0% swap
Memory: 8192M real, 3180M free, 8174M swap free

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
24456 oraprod 14 0 0 1129M 1067M sleep 9:53 2.22% oracle
15467 oraprod 1 58 0 1127M 1067M sleep 0:51 0.36% oracle
13799 oraprod 11 58 0 1129M 1075M sleep 7:46 0.23% oracle
24960 root 1 58 0 2048K 1592K cpu0 0:00 0.10% top
24975 oraprod 1 58 0 1126M 1057M sleep 0:00 0.06% oracle
19162 oraprod 1 48 0 1128M 1070M sleep 10:27 0.05% oracle
2296 applprd5 18 59 0 188M 83M sleep 47:16 0.05% java
24481 applprod 8 58 0 11M 7800K sleep 0:48 0.03% jre
19323 oraprod 1 58 0 1128M 1069M sleep 0:34 0.03% oracle
18422 oraprod 11 58 0 1148M 1078M sleep 8:56 0.02% oracle
18463 oraprod 1 58 0 1126M 1050M sleep 1:06 0.02% oracle
7032 oraprod 1 58 0 1127M 1069M sleep 4:16 0.02% oracle
18995 applprod 18 59 0 65M 31M sleep 1:28 0.02% java
23775 orademo 1 48 0 366M 348M sleep 0:03 0.01% oracle
19021 applprod 21 59 0 64M 29M sleep 1:28 0.01% java


# iostat -xtn 2 2
tty
tin tout
0 54
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
43.9 6.4 1971.7 159.3 0.0 0.3 0.0 5.1 0 11 hsx0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx12
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx72
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx84
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t6d0
0.3 0.0 14.4 0.1 0.0 0.0 0.0 3.6 0 0 c4t8d0
0.3 0.2 14.4 1.5 0.0 0.0 0.0 6.0 0 0 c4t9d0
0.3 0.0 14.4 0.1 0.0 0.0 0.0 3.7 0 0 c4t10d0
0.3 0.2 14.4 1.5 0.0 0.0 0.0 6.1 0 0 c4t11d0
0.4 0.0 21.5 0.3 0.0 0.0 0.0 3.9 0 0 c4t12d0
0.4 0.0 21.5 0.1 0.0 0.0 0.0 4.0 0 0 c4t13d0
0.1 0.3 2.3 2.7 0.0 0.0 11.2 8.4 0 0 c2t1d0
0.1 0.3 2.3 2.7 0.0 0.0 9.7 8.0 0 0 c2t2d0
0.1 0.3 2.2 2.7 0.0 0.0 11.6 8.0 0 0 c2t3d0
0.1 0.3 2.2 2.7 0.0 0.0 11.2 7.7 0 0 c2t4d0
0.0 0.3 0.0 2.7 0.0 0.0 0.0 8.8 0 0 c2t5d0
0.0 0.1 0.0 2.1 0.0 0.0 0.0 4.8 0 0 c2t6d0
0.4 0.0 21.8 0.3 0.0 0.0 0.0 3.9 0 0 c3t8d0
0.4 0.0 21.7 0.1 0.0 0.0 0.0 4.0 0 0 c3t9d0
1.0 0.7 40.7 6.8 0.0 0.0 14.2 6.9 0 1 c3t10d0
1.0 0.7 40.8 6.8 0.0 0.0 14.2 6.8 0 1 c3t11d0
1.1 0.7 43.2 6.8 0.0 0.0 17.4 7.2 0 1 c3t12d0
1.1 0.7 43.2 6.8 0.0 0.0 18.7 7.3 0 1 c3t13d0
0.2 0.3 10.0 2.4 0.0 0.0 0.0 10.1 0 0 c1t3d0
0.2 0.5 9.6 3.7 0.0 0.0 0.0 8.4 0 0 c1t4d0
0.4 2.0 27.3 17.1 0.1 0.2 39.1 69.7 0 2 c1t1d0
0.2 0.3 9.6 2.4 0.0 0.0 0.0 10.4 0 0 c1t5d0
0.5 2.0 27.4 17.1 0.1 0.1 33.6 60.2 0 1 c1t0d0
0.2 0.5 10.0 3.7 0.0 0.0 0.0 8.2 0 0 c1t2d0
43.9 6.4 1971.7 159.3 2.6 0.3 51.8 5.1 0 11 c6t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0 0 eurfinunx001:vold(pid690)
tty
tin tout
0 1158
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 1.5 0.0 23.9 0.0 0.0 0.0 0.6 0 0 hsx0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx12
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx72
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx84
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t6d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t8d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t9d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t10d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t11d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t12d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t13d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t2d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t3d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t4d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t5d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t6d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t8d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t9d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t10d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t11d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t12d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t13d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t3d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t4d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t5d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t2d0
0.0 1.5 0.0 23.9 0.0 0.0 0.0 0.7 0 0 c6t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 eurfinunx001:vold(pid690)
# iostat -xtn
tty
tin tout
0 54
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
43.9 6.4 1971.7 159.3 0.0 0.3 0.0 5.1 0 11 hsx0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx12
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx72
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx84
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t6d0
0.3 0.0 14.4 0.1 0.0 0.0 0.0 3.6 0 0 c4t8d0
0.3 0.2 14.4 1.5 0.0 0.0 0.0 6.0 0 0 c4t9d0
0.3 0.0 14.4 0.1 0.0 0.0 0.0 3.7 0 0 c4t10d0
0.3 0.2 14.3 1.5 0.0 0.0 0.0 6.1 0 0 c4t11d0
0.4 0.0 21.5 0.3 0.0 0.0 0.0 3.9 0 0 c4t12d0
0.4 0.0 21.5 0.1 0.0 0.0 0.0 4.0 0 0 c4t13d0
0.1 0.3 2.3 2.7 0.0 0.0 11.2 8.4 0 0 c2t1d0
0.1 0.3 2.3 2.7 0.0 0.0 9.7 8.0 0 0 c2t2d0
0.1 0.3 2.2 2.7 0.0 0.0 11.6 8.0 0 0 c2t3d0
0.1 0.3 2.2 2.7 0.0 0.0 11.2 7.7 0 0 c2t4d0
0.0 0.3 0.0 2.7 0.0 0.0 0.0 8.8 0 0 c2t5d0
0.0 0.1 0.0 2.1 0.0 0.0 0.0 4.8 0 0 c2t6d0
0.4 0.0 21.8 0.3 0.0 0.0 0.0 3.9 0 0 c3t8d0
0.4 0.0 21.7 0.1 0.0 0.0 0.0 4.0 0 0 c3t9d0
1.0 0.7 40.7 6.8 0.0 0.0 14.2 6.9 0 1 c3t10d0
1.0 0.7 40.8 6.8 0.0 0.0 14.2 6.8 0 1 c3t11d0
1.1 0.7 43.2 6.8 0.0 0.0 17.4 7.2 0 1 c3t12d0
1.1 0.7 43.2 6.8 0.0 0.0 18.7 7.3 0 1 c3t13d0
0.2 0.3 10.0 2.4 0.0 0.0 0.0 10.1 0 0 c1t3d0
0.2 0.5 9.6 3.7 0.0 0.0 0.0 8.4 0 0 c1t4d0
0.4 2.0 27.3 17.1 0.1 0.2 39.1 69.7 0 2 c1t1d0
0.2 0.3 9.6 2.4 0.0 0.0 0.0 10.4 0 0 c1t5d0
0.5 2.0 27.4 17.1 0.1 0.1 33.6 60.2 0 1 c1t0d0
0.2 0.5 10.0 3.7 0.0 0.0 0.0 8.2 0 0 c1t2d0
43.9 6.4 1971.7 159.3 2.6 0.3 51.8 5.1 0 11 c6t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0 0 eurfinunx001:vold(pid690)



# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfaf1588,0
1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfaf165c,0
2. c1t2d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfaaf3bb,0
3. c1t3d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfaaf3f3,0
4. c1t4d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfaf1254,0
5. c1t5d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfaf1768,0
6. c2t1d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@4/sd@1,0
7. c2t2d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@4/sd@2,0
8. c2t3d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@4/sd@3,0
9. c2t4d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@4/sd@4,0
10. c2t5d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@4/sd@5,0
11. c2t6d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@4/sd@6,0
12. c3t8d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@5/sd@8,0
13. c3t9d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@5/sd@9,0
14. c3t10d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@5/sd@a,0
15. c3t11d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@5/sd@b,0
16. c3t12d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@5/sd@c,0
17. c3t13d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,700000/pci@2/scsi@5/sd@d,0
18. c4t8d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,600000/pci@2/scsi@5/sd@8,0
19. c4t9d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,600000/pci@2/scsi@5/sd@9,0
20. c4t10d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,600000/pci@2/scsi@5/sd@a,0
21. c4t11d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,600000/pci@2/scsi@5/sd@b,0
22. c4t12d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,600000/pci@2/scsi@5/sd@c,0
23. c4t13d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>
/pci@9,600000/pci@2/scsi@5/sd@d,0
24. c6t0d0 <COMPAQ-HSV110(C)COMPAQ-3025 cyl 25598 alt 2 hd 128 sec 128>
/swsp@0,1/ssd@0,0
Specify disk (enter its number): df -k^C
Specify disk (enter its number):

# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/rootvol 62329116 30701664 31004161 50% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 11226864 16 11226848 1% /var/run
swap 11228448 1600 11226848 1% /tmp
/dev/vx/dsk/sandg/prod59
209674240 143056017 62454622 70% /u01
/dev/vx/dsk/fin/u01 284734464 198178750 81287585 71% /u02
/dev/null 0 0 0 0% /dev/odm
# vxprint -hrt
Disk group: rootdg

DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 0 1056654087.1025.eurfinunx001

dm rootdisk c1t0d0s2 sliced 10175 143349311 -
dm rootmirror c1t1d0s2 sliced 20095 143318784 -

sd rootdiskPriv - rootdisk 126599616 10175 PRIVATE c1t0d0 ENA

v rootvol - ENABLED ACTIVE 126569088 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 126569088 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 126599615 1 0 c1t0d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 126569087 1 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 126569088 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 126569088 0 c1t1d0 ENA

v swapvol - ENABLED ACTIVE 16739520 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 16739520 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 126609791 16739520 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 16739520 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 126569088 16739520 0 c1t1d0 ENA

Disk group: fin

DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg fin default default 96000 1056654530.1169.eurfinunx001

dm fin01 c1t3d0s2 sliced 20095 143318784 -
dm fin02 c1t5d0s2 sliced 20095 143318784 -
dm fin03 c3t8d0s2 sliced 9167 35354136 -
dm fin04 c3t12d0s2 sliced 9167 35354136 -
dm fin05 c3t13d0s2 sliced 9167 35354136 -
dm fin06 c3t9d0s2 sliced 9167 35354136 -
dm fin07 c3t10d0s2 sliced 9167 35354136 -
dm fin08 c3t11d0s2 sliced 9167 35354136 -
dm fin09 c2t2d0s2 sliced 9167 35354136 -
dm fin10 c2t4d0s2 sliced 9167 35354136 -
dm fin11 c2t6d0s2 sliced 9167 35354136 -
dm fin101 c1t2d0s2 sliced 20095 143318784 -
dm fin102 c1t4d0s2 sliced 20095 143318784 -
dm fin103 c4t8d0s2 sliced 9167 35354136 -
dm fin104 c4t9d0s2 sliced 9167 35354136 -
dm fin105 c4t10d0s2 sliced 9167 35354136 -
dm fin106 c4t11d0s2 sliced 9167 35354136 -
dm fin107 c4t12d0s2 sliced 9167 35354136 -
dm fin108 c4t13d0s2 sliced 9167 35354136 -
dm fin109 c2t1d0s2 sliced 9167 35354136 -
dm fin110 c2t3d0s2 sliced 9167 35354136 -
dm fin111 c2t5d0s2 sliced 9167 35354136 -

v u01 - ENABLED ACTIVE 569468928 SELECT u01-04 fsgen
pl u01-04 u01 ENABLED ACTIVE 569468928 STRIPE 2/128 RW
sv u01-S01 u01-04 u01-L01 1 143318784 0/0 4/4 ENA
v2 u01-L01 - ENABLED ACTIVE 143318784 SELECT - fsgen
p2 u01-P01 u01-L01 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin111-02 u01-P01 fin111 0 1155 LOG c2t5d0 ENA
p2 u01-P02 u01-L01 ENABLED ACTIVE 143318784 CONCAT - RW
s2 fin101-02 u01-P02 fin101 0 143318784 0 c1t2d0 ENA
p2 u01-P03 u01-L01 ENABLED ACTIVE 143318784 CONCAT - RW
s2 fin102-02 u01-P03 fin102 0 143318784 0 c1t4d0 ENA
p2 u01-L01-01 u01-L01 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin11-01 u01-L01-01 fin11 0 1056 LOG c2t6d0 ENA
sv u01-S02 u01-04 u01-L02 1 35354136 0/143318784 3/3 ENA
v2 u01-L02 - ENABLED ACTIVE 35354136 SELECT - fsgen
p2 u01-P04 u01-L02 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin111-03 u01-P04 fin111 1155 297 LOG c2t5d0 ENA
p2 u01-P05 u01-L02 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin103-02 u01-P05 fin103 0 35354136 0 c4t8d0 ENA
p2 u01-P06 u01-L02 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin105-02 u01-P06 fin105 0 35354136 0 c4t10d0 ENA
sv u01-S03 u01-04 u01-L03 1 35354136 0/178672920 3/3 ENA
v2 u01-L03 - ENABLED ACTIVE 35354136 SELECT - fsgen
p2 u01-P07 u01-L03 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin111-04 u01-P07 fin111 1452 297 LOG c2t5d0 ENA
p2 u01-P08 u01-L03 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin107-02 u01-P08 fin107 0 35354136 0 c4t12d0 ENA
p2 u01-P09 u01-L03 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin03-02 u01-P09 fin03 0 35354136 0 c3t8d0 ENA
sv u01-S04 u01-04 u01-L04 1 35354136 0/214027056 3/3 ENA
v2 u01-L04 - ENABLED ACTIVE 35354136 SELECT - fsgen
p2 u01-P10 u01-L04 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin111-05 u01-P10 fin111 1749 297 LOG c2t5d0 ENA
p2 u01-P11 u01-L04 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin07-02 u01-P11 fin07 0 35354136 0 c3t10d0 ENA
p2 u01-P12 u01-L04 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin04-02 u01-P12 fin04 0 35354136 0 c3t12d0 ENA
sv u01-S05 u01-04 u01-L05 1 35353272 0/249381192 3/3 ENA
v2 u01-L05 - ENABLED ACTIVE 35353272 SELECT - fsgen
p2 u01-P13 u01-L05 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin111-06 u01-P13 fin111 2046 297 LOG c2t5d0 ENA
p2 u01-P14 u01-L05 ENABLED ACTIVE 35353272 CONCAT - RW
s2 fin109-02 u01-P14 fin109 0 35353272 0 c2t1d0 ENA
p2 u01-P15 u01-L05 ENABLED ACTIVE 35353272 CONCAT - RW
s2 fin110-02 u01-P15 fin110 0 35353272 0 c2t3d0 ENA
sv u01-S06 u01-04 u01-L06 1 143318784 1/0 4/4 ENA
v2 u01-L06 - ENABLED ACTIVE 143318784 SELECT - fsgen
p2 u01-P16 u01-L06 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin111-07 u01-P16 fin111 2343 1155 LOG c2t5d0 ENA
p2 u01-P17 u01-L06 ENABLED ACTIVE 143318784 CONCAT - RW
s2 fin01-02 u01-P17 fin01 0 143318784 0 c1t3d0 ENA
p2 u01-P18 u01-L06 ENABLED ACTIVE 143318784 CONCAT - RW
s2 fin02-02 u01-P18 fin02 0 143318784 0 c1t5d0 ENA
p2 u01-L06-01 u01-L06 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin11-02 u01-L06-01 fin11 1056 1056 LOG c2t6d0 ENA
sv u01-S07 u01-04 u01-L07 1 35354136 1/143318784 3/3 ENA
v2 u01-L07 - ENABLED ACTIVE 35354136 SELECT - fsgen
p2 u01-P19 u01-L07 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin111-08 u01-P19 fin111 3498 297 LOG c2t5d0 ENA
p2 u01-P20 u01-L07 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin104-02 u01-P20 fin104 0 35354136 0 c4t9d0 ENA
p2 u01-P21 u01-L07 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin106-02 u01-P21 fin106 0 35354136 0 c4t11d0 ENA
sv u01-S08 u01-04 u01-L08 1 35354136 1/178672920 3/3 ENA
v2 u01-L08 - ENABLED ACTIVE 35354136 SELECT - fsgen
p2 u01-P22 u01-L08 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin111-09 u01-P22 fin111 3795 297 LOG c2t5d0 ENA
p2 u01-P23 u01-L08 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin108-02 u01-P23 fin108 0 35354136 0 c4t13d0 ENA
p2 u01-P24 u01-L08 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin06-02 u01-P24 fin06 0 35354136 0 c3t9d0 ENA
sv u01-S09 u01-04 u01-L09 1 35354136 1/214027056 3/3 ENA
v2 u01-L09 - ENABLED ACTIVE 35354136 SELECT - fsgen
p2 u01-P25 u01-L09 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin111-10 u01-P25 fin111 4092 297 LOG c2t5d0 ENA
p2 u01-P26 u01-L09 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin08-02 u01-P26 fin08 0 35354136 0 c3t11d0 ENA
p2 u01-P27 u01-L09 ENABLED ACTIVE 35354136 CONCAT - RW
s2 fin05-02 u01-P27 fin05 0 35354136 0 c3t13d0 ENA
sv u01-S10 u01-04 u01-L10 1 35353272 1/249381192 3/3 ENA
v2 u01-L10 - ENABLED ACTIVE 35353272 SELECT - fsgen
p2 u01-P28 u01-L10 ENABLED ACTIVE LOGONLY CONCAT - RW
s2 fin111-11 u01-P28 fin111 4389 297 LOG c2t5d0 ENA
p2 u01-P29 u01-L10 ENABLED ACTIVE 35353272 CONCAT - RW
s2 fin09-02 u01-P29 fin09 0 35353272 0 c2t2d0 ENA
p2 u01-P30 u01-L10 ENABLED ACTIVE 35353272 CONCAT - RW
s2 fin10-02 u01-P30 fin10 0 35353272 0 c2t4d0 ENA

Disk group: sandg

DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg sandg default default 98000 1128415452.2206.eurfinunx001

dm san01 c6t0d0s2 sliced 32511 419348480 -

v prod59 - ENABLED ACTIVE 419348480 SELECT - fsgen
pl prod59-01 prod59 ENABLED ACTIVE 419348480 CONCAT - RW
sd san01-01 prod59-01 san01 0 419348480 0 c6t0d0 ENA
#

Looks like the swap vol and the san mount have highest waits.. can anyone help or point me in the right direction ... or even provide some commands to try..


thanks in advance




Sy UK

 

[Scunningham99]

heres some more output also there are no faulty disks in the messages file...

# vmstat 2 5
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr hx hx hx hx in sy cs us sy id
0 877 0 11075448 3136728 93 129 46 0 0 0 0 50 0 0 0 690 611 345 4 2 94
0 356062523 0 11179104 3216024 115 949 0 4 0 0 0 152 0 0 0 1165 16153 3478 69 6 25
0 356062711 0 11185328 3220704 155 1249 0 0 0 0 0 122 0 0 0 1250 16310 3759 57 9 35
0 356062844 0 11191336 3225264 3 34 0 0 0 0 0 91 0 0 0 527 8713 1874 55 10 35
0 356062947 0 11199232 3231912 1 0 0 0 0 0 0 61 0 0 0 584 8085 2671 34 3 63
#




Sy UK

 

[Annihilannic]

How about the output of vxstat -vdg fin -i 5 -c 3 at a time when the system is busy?

The iostat output doesn't show a hell of a lot of I/O, does it? I think network traffic can also drive up %iowait... are you seeing a lot of that? Try running netstat -i 5 for a while, or monitoring the netstat -k <interface_name> counters periodically, specifically rbytes64 and wbytes64 (if I recall their names correctly).

Annihilannic.

 

[Scunningham99]

# vxstat -vdg fin -i 5 -c 3
OPERATIONS BLOCKS AVG TIME(ms)
TYP NAME READ WRITE READ WRITE READ WRITE

Wed 02 Jul 2008 09:55:48 BST
dm fin01 640859 900791 59260978 14290970 6.0 11.2
dm fin02 621618 900791 56948944 14290970 6.1 11.5
dm fin03 1315927 97363 128455410 1566868 3.9 5.7
dm fin04 3121232 2063218 255399919 40260115 37.2 5.6
dm fin05 3204424 2044042 255689796 40192822 39.0 5.6
dm fin06 1231072 23076 128261738 435088 4.0 8.7
dm fin07 2854737 2063218 241432677 40260115 32.5 5.5
dm fin08 2931984 2044042 241744174 40192822 32.2 5.5
dm fin09 266771 1008787 13496144 16136692 66.0 4.9
dm fin10 272213 1008787 13136062 16136692 71.1 4.9
dm fin11 0 353653 0 12379827 0.0 3.6
dm fin101 648959 1478046 59096682 22063071 5.7 6.9
dm fin102 631921 1478046 56799670 22063071 5.9 7.2
dm fin103 828868 24700 85290072 395200 3.6 7.8
dm fin104 785935 556331 85276772 8901268 3.8 6.1
dm fin105 838240 24700 84724854 395200 3.7 7.9
dm fin106 794526 556331 84715992 8901268 3.9 6.2
dm fin107 1272730 97363 126880794 1566868 3.9 5.6
dm fin108 1186980 23076 126672306 435088 4.0 8.7
dm fin109 265196 989247 13488714 15822938 74.1 5.1
dm fin110 265118 989247 12995642 15822938 74.2 5.1
dm fin111 0 769301 0 16122883 0.0 7.9
vol u01 9603237 9021678 2129767340 160065032 42.9 8.5
vol u01-L01 1280880 1478046 115896352 22063071 5.8 10.1
vol u01-L02 1667108 24700 170014926 395200 3.7 12.4
vol u01-L03 2588657 97363 255336204 1566868 3.9 10.5
vol u01-L04 5975969 2063218 496832596 40260115 35.0 7.1
vol u01-L05 530314 989247 26484356 15822938 74.2 6.0
vol u01-L06 1262477 900791 116209922 14290970 6.1 16.0
vol u01-L07 1580461 556331 169992764 8901268 3.9 8.7
vol u01-L08 2418052 23076 254934044 435088 4.0 14.0
vol u01-L09 6136408 2044042 497433970 40192822 35.8 7.1
vol u01-L10 538984 1008787 26632206 16136692 68.5 6.0

# netstat -i 5
input ge0 output input (Total) output
packets errs packets errs colls packets errs packets errs colls
0 0 0 0 0 1440872003 0 -787018621 177776 337967
0 0 0 0 0 20726 0 40955 0 0
0 0 0 0 0 20769 0 41018 0 0
0 0 0 0 0 20575 0 40718 0 0
0 0 0 0 0 20705 0 40823 0 0
0 0 0 0 0 20609 0 40780 0 0
0 0 0 0 0 20643 0 41001 0 0
0 0 0 0 0 20582 0 40931 0 0
0 0 0 0 0 20490 0 40769 0 0
0 0 0 0 0 20762 0 41085 0 0
0 0 0 0 0 21214 0 41553 0 0
0 0 0 0 0 22096 0 42555 0 0
0 0 0 0 0 20879 0 41267 0 0
0 0 0 0 0 20617 0 40993 0 0
0 0 0 0 0 20733 0 41066 0 0
0 0 0 0 0 20691 0 41059 0 0
0 0 0 0 0 20594 0 40944 0 0
0 0 0 0 0 20752 0 41078 0 0
0 0 0 0 0 20701 0 41051 0 0
0 0 0 0 0 20704 0 41069 0 0
0 0 0 0 0 20843 0 41232 0 0
0 0 0 0 0 20793 0 41147 0 0
0 0 0 0 0 20798 0 41222 0 0
0 0 0 0 0 20654 0 41033 0 0
0 0 0 0 0 20913 0 41252 0 0
0 0 0 0 0 20747 0 41128 0 0
0 0 0 0 0 20776 0 41071 0 0
0 0 0 0 0 20801 0 41209 0 0
0 0 0 0 0 20897 0 41168 0 0
0 0 0 0 0 21558 0 41940 0 0
0 0 0 0 0 20642 0 41009 0 0
0 0 0 0 0 20582 0 40984 0 0
0 0 0 0 0 20566 0 40963 0 0
0 0 0 0 0 20603 0 41008 0 0
0 0 0 0 0 20628 0 41013 0 0
0 0 0 0 0 20705 0 41059 0 0
0 0 0 0 0 20609 0 41001 0 0
0 0 0 0 0 20579 0 40949 0 0
0 0 0 0 0 20584 0 40941 0 0
0 0 0 0 0 20664 0 41054 0 0
0 0 0 0 0 20649 0 41008 0 0
0 0 0 0 0 20625 0 41003 0 0
0 0 0 0 0 20597 0 40973 0 0
0 0 0 0 0 20596 0 40966 0 0
0 0 0 0 0 20648 0 41012 0 0
0 0 0 0 0 20653 0 41005 0 0
0 0 0 0 0 20584 0 40954 0 0
0 0 0 0 0 20772 0 41126 0 0
0 0 0 0 0 20519 0 41165 0 0
0 0 0 0 0 20560 0 40928 0 0
0 0 0 0 0 20547 0 40909 0 0
0 0 0 0 0 20586 0 40975 0 0
0 0 0 0 0 20600 0 40988 0 0
0 0 0 0 0 20595 0 40991 0 0
0 0 0 0 0 20519 0 40921 0 0
0 0 0 0 0 20531 0 40930 0 0
0 0 0 0 0 20517 0 40916 0 0
0 0 0 0 0 20578 0 40964 0 0
0 0 0 0 0 20511 0 40908 0 0
0 0 0 0 0 20663 0 41039 0 0
0 0 0 0 0 20574 0 40954 0 0
0 0 0 0 0 20600 0 40965 0 0
0 0 0 0 0 20553 0 40909 0 0
0 0 0 0 0 20640 0 40992 0 0
0 0 0 0 0 20603 0 40972 0 0
0 0 0 0 0 20609 0 41005 0 0
0 0 0 0 0 20591 0 40975 0 0
0 0 0 0 0 20581 0 40972 0 0
0 0 0 0 0 20555 0 40943 0 0
0 0 0 0 0 20615 0 40999 0 0
0 0 0 0 0 20692 0 41047 0 0
0 0 0 0 0 20730 0 41129 0 0
0 0 0 0 0 23137 0 43429 0 0
0 0 0 0 0 20696 0 41082 0 0
0 0 0 0 0 20386 0 40586 0 0
0 0 0 0 0 20674 0 41053 0 0
0 0 0 0 0 20531 0 40922 0 0
0 0 0 0 0 20703 0 41077 0 0
0 0 0 0 0 20605 0 40974 0 0
0 0 0 0 0 20584 0 40944 0 0
0 0 0 0 0 20557 0 40941 0 0
0 0 0 0 0 20678 0 41031 0 0
0 0 0 0 0 20575 0 40941 0 0
0 0 0 0 0 20618 0 40982 0 0
0 0 0 0 0 20718 0 41085 0 0
0 0 0 0 0 20549 0 40949 0 0
0 0 0 0 0 20728 0 41074 0 0
0 0 0 0 0 20715 0 41074 0 0
0 0 0 0 0 20639 0 40986 0 0
0 0 0 0 0 21098 0 41473 0 0
0 0 0 0 0 21308 0 41683 0 0
0 0 0 0 0 20574 0 40976 0 0
0 0 0 0 0 20784 0 41131 0 0
0 0 0 0 0 20548 0 40866 0 0
0 0 0 0 0 20604 0 40979 0 0
0 0 0 0 0 20692 0 41035 0 0
0 0 0 0 0 20774 0 40997 0 0
0 0 0 0 0 20625 0 40891 0 0
0 0 0 0 0 20695 0 40960 0 0
0 0 0 0 0 21101 0 41526 0 0
0 0 0 0 0 20840 0 41185 0 0
0 0 0 0 0 20618 0 40978 0 0
0 0 0 0 0 20642 0 40999 0 0
0 0 0 0 0 20969 0 41307 0 0
0 0 0 0 0 20715 0 41060 0 0
0 0 0 0 0 20760 0 41154 0 0
0 0 0 0 0 20735 0 41108 0 0
0 0 0 0 0 20812 0 41182 0 0
0 0 0 0 0 20688 0 41057 0 0
0 0 0 0 0 20707 0 41379 0 0
0 0 0 0 0 20744 0 41100 0 0
0 0 0 0 0 20771 0 41144 0 0
0 0 0 0 0 20518 0 40684 0 0
0 0 0 0 0 20840 0 41195 0 0
0 0 0 0 0 20647 0 41012 0 0
0 0 0 0 0 20619 0 41015 0 0
0 0 0 0 0 20597 0 40962 0 0
0 0 0 0 0 20664 0 41075 0 0
0 0 0 0 0 20563 0 40978 0 0
0 0 0 0 0 20741 0 41118 0 0
0 0 0 0 0 20622 0 40985 0 0
0 0 0 0 0 20730 0 41088 0 0
0 0 0 0 0 20545 0 40921 0 0
0 0 0 0 0 20753 0 41115 0 0
0 0 0 0 0 20650 0 41012 0 0
0 0 0 0 0 20726 0 41088 0 0
0 0 0 0 0 20537 0 40917 0 0
0 0 0 0 0 20800 0 41136 0 0
0 0 0 0 0 20493 0 40790 0 0
0 0 0 0 0 20645 0 41023 0 0
0 0 0 0 0 20554 0 40917 0 0
0 0 0 0 0 20644 0 41005 0 0
0 0 0 0 0 20558 0 40948 0 0
0 0 0 0 0 20556 0 40949 0 0
0 0 0 0 0 20656 0 41025 0 0
0 0 0 0 0 20725 0 41139 0 0
0 0 0 0 0 20597 0 40967 0 0
0 0 0 0 0 20628 0 41027 0 0
0 0 0 0 0 20587 0 40974 0 0
0 0 0 0 0 20555 0 40970 0 0
0 0 0 0 0 20497 0 40896 0 0
0 0 0 0 0 20589 0 40997 0 0
0 0 0 0 0 20703 0 41061 0 0
0 0 0 0 0 20720 0 41071 0 0
0 0 0 0 0 21001 0 41224 0 0
0 0 0 0 0 20975 0 41232 0 0
0 0 0 0 0 20793 0 41109 0 0
0 0 0 0 0 20947 0 41268 0 0
0 0 0 0 0 20731 0 41042 0 0
0 0 0 0 0 20868 0 41215 0 0
0 0 0 0 0 21841 0 42210 0 0
0 0 0 0 0 21634 0 42033 0 0
0 0 0 0 0 20654 0 40903 0 0
0 0 0 0 0 20951 0 41309 0 0
0 0 0 0 0 20770 0 41144 0 0
0 0 0 0 0 20599 0 40796 0 0
0 0 0 0 0 20490 0 40704 0 0
0 0 0 0 0 20413 0 40583 0 0
0 0 0 0 0 20368 0 40551 0 0
0 0 0 0 0 20600 0 40986 0 0
0 0 0 0 0 20586 0 40950 0 0
0 0 0 0 0 20680 0 41045 0 0
0 0 0 0 0 20547 0 40943 0 0
0 0 0 0 0 20748 0 41116 0 0
0 0 0 0 0 20546 0 40940 0 0
0 0 0 0 0 21253 0 41557 0 0
0 0 0 0 0 21000 0 41187 0 0
0 0 0 0 0 20980 0 41277 0 0
0 0 0 0 0 20914 0 41180 0 0
0 0 0 0 0 21501 0 41801 0 0
0 0 0 0 0 20716 0 41067 0 0
0 0 0 0 0 20524 0 41227 0 0
0 0 0 0 0 20687 0 41088 0 0
0 0 0 0 0 20656 0 41033 0 0
0 0 0 0 0 20543 0 40923 0 0
0 0 0 0 0 20619 0 40997 0 0
0 0 0 0 0 20662 0 41012 0 0
0 0 0 0 0 20825 0 41223 0 0
0 0 0 0 0 21930 0 42341 0 0
^C#

Any ideas ???

Sy UK

 

[Annihilannic]

4000 packets/sec coming in and 8000 packets per second going out consistently... that's a fair bit of network traffic! Any idea what it is?

Also they're not on ge0 judging by those 0 counters, do you know which NIC they're going through? Try doing a netstat -k <nicname> a couple of times with 5 seconds or so in between and see how much the byte counters are rising.

I actually need the second sample of data from the vxstat output (the first set which you posted are the counters since they were previously reset, the subsequent ones are for the 5 second sample intervals).



Annihilannic.

 

[Scunningham99]

# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.128.9.1 netmask ffffff00 broadcast 10.128.9.255
ether 0:3:ba:1d:67:5
qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 10.128.2.47 netmask ffff0000 broadcast 10.128.255.255
ether 0:3:ba:1d:67:5
you have mail in /var/mail/root
# netstat -k qfe0
qfe0:
ipackets 1435275540 ierrors 0 opackets 3595757778 oerrors 177776 collisions 337967
defer 0 framing 0 crc 0 sqe 0 code_violations 0 len_errors 0
ifspeed 100000000 buff 0 oflo 0 uflo 0 missed 0 tx_late_collisions 174463
retry_error 0 first_collisions 512 nocarrier 0 inits 2 nocanput 34
allocbfail 0 runt 0 jabber 0 babble 3313 tmd_error 0 tx_late_error 0
rx_late_error 0 slv_parity_error 0 tx_parity_error 0 rx_parity_error 0
slv_error_ack 0 tx_error_ack 0 rx_error_ack 0 tx_tag_error 0
rx_tag_error 0 eop_error 0 no_tmds 0 no_tbufs 0 no_rbufs 0
rx_late_collisions 0 rbytes 4157063585 obytes 1495263708 multircv 23282 multixmt 0
brdcstrcv 48923421 brdcstxmt 118464 norcvbuf 34 noxmtbuf 0 phy_inits 5
tx_inits 6 rx_inits 4 ipackets64 1435275540 opackets64 3595757778 rbytes64 98646344097
obytes64 5327254710748 align_errors 0 fcs_errors 0 sqe_errors 0
defer_xmts 0 ex_collisions 0 macxmt_errors 0 carrier_errors 0
toolong_errors 0 macrcv_errors 0 link_up 1 link_duplex 2

# netstat -k qfe0
qfe0:
ipackets 1435276281 ierrors 0 opackets 3595758335 oerrors 177776 collisions 337967
defer 0 framing 0 crc 0 sqe 0 code_violations 0 len_errors 0
ifspeed 100000000 buff 0 oflo 0 uflo 0 missed 0 tx_late_collisions 174463
retry_error 0 first_collisions 512 nocarrier 0 inits 2 nocanput 34
allocbfail 0 runt 0 jabber 0 babble 3313 tmd_error 0 tx_late_error 0
rx_late_error 0 slv_parity_error 0 tx_parity_error 0 rx_parity_error 0
slv_error_ack 0 tx_error_ack 0 rx_error_ack 0 tx_tag_error 0
rx_tag_error 0 eop_error 0 no_tmds 0 no_tbufs 0 no_rbufs 0
rx_late_collisions 0 rbytes 4157172787 obytes 1495384821 multircv 23282 multixmt 0
brdcstrcv 48923614 brdcstxmt 118464 norcvbuf 34 noxmtbuf 0 phy_inits 5
tx_inits 6 rx_inits 4 ipackets64 1435276281 opackets64 3595758335 rbytes64 98646453299
obytes64 5327254831861 align_errors 0 fcs_errors 0 sqe_errors 0
defer_xmts 0 ex_collisions 0 macxmt_errors 0 carrier_errors 0
toolong_errors 0 macrcv_errors 0 link_up 1 link_duplex 2

# netstat -k qfe0 5
qfe0:
ipackets 1435276943 ierrors 0 opackets 3595759418 oerrors 177776 collisions 337967
defer 0 framing 0 crc 0 sqe 0 code_violations 0 len_errors 0
ifspeed 100000000 buff 0 oflo 0 uflo 0 missed 0 tx_late_collisions 174463
retry_error 0 first_collisions 512 nocarrier 0 inits 2 nocanput 34
allocbfail 0 runt 0 jabber 0 babble 3313 tmd_error 0 tx_late_error 0
rx_late_error 0 slv_parity_error 0 tx_parity_error 0 rx_parity_error 0
slv_error_ack 0 tx_error_ack 0 rx_error_ack 0 tx_tag_error 0
rx_tag_error 0 eop_error 0 no_tmds 0 no_tbufs 0 no_rbufs 0
rx_late_collisions 0 rbytes 4157217347 obytes 1496455790 multircv 23282 multixmt 0
brdcstrcv 48923693 brdcstxmt 118464 norcvbuf 34 noxmtbuf 0 phy_inits 5
tx_inits 6 rx_inits 4 ipackets64 1435276943 opackets64 3595759418 rbytes64 98646497859
obytes64 5327255902830 align_errors 0 fcs_errors 0 sqe_errors 0
defer_xmts 0 ex_collisions 0 macxmt_errors 0 carrier_errors 0
toolong_errors 0 macrcv_errors 0 link_up 1 link_duplex 2

# netstat -k qfe0
qfe0:
ipackets 1435278761 ierrors 0 opackets 3595761132 oerrors 177776 collisions 337967
defer 0 framing 0 crc 0 sqe 0 code_violations 0 len_errors 0
ifspeed 100000000 buff 0 oflo 0 uflo 0 missed 0 tx_late_collisions 174463
retry_error 0 first_collisions 512 nocarrier 0 inits 2 nocanput 34
allocbfail 0 runt 0 jabber 0 babble 3313 tmd_error 0 tx_late_error 0
rx_late_error 0 slv_parity_error 0 tx_parity_error 0 rx_parity_error 0
slv_error_ack 0 tx_error_ack 0 rx_error_ack 0 tx_tag_error 0
rx_tag_error 0 eop_error 0 no_tmds 0 no_tbufs 0 no_rbufs 0
rx_late_collisions 0 rbytes 4157504837 obytes 1497675565 multircv 23282 multixmt 0
brdcstrcv 48924283 brdcstxmt 118464 norcvbuf 34 noxmtbuf 0 phy_inits 5
tx_inits 6 rx_inits 4 ipackets64 1435278761 opackets64 3595761132 rbytes64 98646785349
obytes64 5327257122605 align_errors 0 fcs_errors 0 sqe_errors 0
defer_xmts 0 ex_collisions 0 macxmt_errors 0 carrier_errors 0
toolong_errors 0 macrcv_errors 0 link_up 1 link_duplex 2

# netstat -k qfe0
qfe0:
ipackets 1435279555 ierrors 0 opackets 3595761910 oerrors 177776 collisions 337967
defer 0 framing 0 crc 0 sqe 0 code_violations 0 len_errors 0
ifspeed 100000000 buff 0 oflo 0 uflo 0 missed 0 tx_late_collisions 174463
retry_error 0 first_collisions 512 nocarrier 0 inits 2 nocanput 34
allocbfail 0 runt 0 jabber 0 babble 3313 tmd_error 0 tx_late_error 0
rx_late_error 0 slv_parity_error 0 tx_parity_error 0 rx_parity_error 0
slv_error_ack 0 tx_error_ack 0 rx_error_ack 0 tx_tag_error 0
rx_tag_error 0 eop_error 0 no_tmds 0 no_tbufs 0 no_rbufs 0
rx_late_collisions 0 rbytes 4157609316 obytes 1498233131 multircv 23282 multixmt 0
brdcstrcv 48924519 brdcstxmt 118464 norcvbuf 34 noxmtbuf 0 phy_inits 5
tx_inits 6 rx_inits 4 ipackets64 1435279555 opackets64 3595761910 rbytes64 98646889828
obytes64 5327257680171 align_errors 0 fcs_errors 0 sqe_errors 0
defer_xmts 0 ex_collisions 0 macxmt_errors 0 carrier_errors 0
toolong_errors 0 macrcv_errors 0 link_up 1 link_duplex 2

# netstat -k qfe0
qfe0:
ipackets 1435280578 ierrors 0 opackets 3595763007 oerrors 177776 collisions 337967
defer 0 framing 0 crc 0 sqe 0 code_violations 0 len_errors 0
ifspeed 100000000 buff 0 oflo 0 uflo 0 missed 0 tx_late_collisions 174463
retry_error 0 first_collisions 512 nocarrier 0 inits 2 nocanput 34
allocbfail 0 runt 0 jabber 0 babble 3313 tmd_error 0 tx_late_error 0
rx_late_error 0 slv_parity_error 0 tx_parity_error 0 rx_parity_error 0
slv_error_ack 0 tx_error_ack 0 rx_error_ack 0 tx_tag_error 0
rx_tag_error 0 eop_error 0 no_tmds 0 no_tbufs 0 no_rbufs 0
rx_late_collisions 0 rbytes 4157729181 obytes 1498884568 multircv 23282 multixmt 0
brdcstrcv 48924690 brdcstxmt 118464 norcvbuf 34 noxmtbuf 0 phy_inits 5
tx_inits 6 rx_inits 4 ipackets64 1435280578 opackets64 3595763007 rbytes64 98647009693
obytes64 5327258331608 align_errors 0 fcs_errors 0 sqe_errors 0
defer_xmts 0 ex_collisions 0 macxmt_errors 0 carrier_errors 0
toolong_errors 0 macrcv_errors 0 link_up 1 link_duplex 2

# netstat -k ge0
ge0:
ipackets 0 ierrors 0 opackets 0 oerrors 0 collisions 0
ifspeed 0 rbytes 0 obytes 0 multircv 0 multixmt 0 brdcstrcv 0
brdcstxmt 10058 norcvbuf 0 noxmtbuf 0 inits 0 mac_mode 0 xmit_dma_mode 0
rcv_dma_mode 0 nocarrier 0 nocanput 0 allocbfail 0 pause_rcv_cnt 0
pause_on_cnt 0 pause_off_cnt 0 pause_time_cnt 0 txmac_urun 0
txmac_maxpkt_err 0 excessive_coll 0 late_coll 0 first_coll 0
defer_timer_exp 0 peak_attempt_cnt 0 jabber 0 no_tmds 0
txinits 0 drop 0 rxinits 0 no_free_rx_desc 0 rx_overflow 0
rx_hang 0 rxtag_error 0 rx_align_err 0 rx_crc_err 0 rx_length_err 0
rx_code_viol_err 0 pci_badack 0 pci_dtrto 0 pci_data_parity_err 0
pci_signal_target_abort 0 pci_rcvd_target_abort 0 pci_rcvd_master_abort 0
pci_signal_system_err 0 pci_det_parity_err 0 pci_bus_speed 0
pci_bus_width 0 tx_late_error 0 rx_late_error 0 slv_parity_error 0
tx_parity_error 0 rx_parity_error 0 slv_error_ack 0 tx_error_ack 0
rx_error_ack 0 ipackets64 0 opackets64 0 rbytes64 0 obytes64 0
align_errors 0 fcs_errors 0 sqe_errors 0 defer_xmts 0
ex_collisions 0 macxmt_errors 0 carrier_errors 0 toolong_errors 0
macrcv_errors 0 ge_csumerr 0 ge_queue_cnt 0 ge_queue_full_cnt 0
link_up 0

i have run netstat -i again this time values are alot smaller but iowait was still approx 70+


# netstat -i 5
input ge0 output input (Total) output
packets errs packets errs colls packets errs packets errs colls
0 0 0 0 0 1541436828 0 -593048755 177776 337967
0 0 0 0 0 378 0 299 0 0
0 0 0 0 0 222 0 192 0 0
0 0 0 0 0 595 0 528 0 0
0 0 0 0 0 588 0 541 0 0
0 0 0 0 0 464 0 402 0 0
0 0 0 0 0 713 0 741 0 0
0 0 0 0 0 454 0 614 0 0
0 0 0 0 0 514 0 519 0 0
0 0 0 0 0 504 0 460 0 0
0 0 0 0 0 604 0 540 0 0
0 0 0 0 0 754 0 1201 0 0
0 0 0 0 0 793 0 693 0 0
0 0 0 0 0 1652 0 1840 0 0
0 0 0 0 0 763 0 953 0 0
0 0 0 0 0 265 0 223 0 0
0 0 0 0 0 421 0 342 0 0
0 0 0 0 0 555 0 641 0 0
0 0 0 0 0 492 0 550 0 0
0 0 0 0 0 904 0 837 0 0
0 0 0 0 0 1107 0 1017 0 0
0 0 0 0 0 1501 0 1750 0 0
0 0 0 0 0 388 0 495 0 0
0 0 0 0 0 455 0 410 0 0
0 0 0 0 0 288 0 214 0 0
0 0 0 0 0 255 0 184 0 0
0 0 0 0 0 495 0 562 0 0
0 0 0 0 0 280 0 301 0 0
0 0 0 0 0 160 0 106 0 0
0 0 0 0 0 333 0 436 0 0
0 0 0 0 0 255 0 185 0 0
0 0 0 0 0 274 0 207 0 0
0 0 0 0 0 390 0 544 0 0
0 0 0 0 0 891 0 823 0 0
0 0 0 0 0 4499 0 4659 0 0
0 0 0 0 0 615 0 564 0 0
0 0 0 0 0 308 0 254 0 0
0 0 0 0 0 322 0 265 0 0
0 0 0 0 0 370 0 317 0 0
0 0 0 0 0 257 0 173 0 0
0 0 0 0 0 451 0 373 0 0
0 0 0 0 0 654 0 592 0 0
0 0 0 0 0 547 0 496 0 0
0 0 0 0 0 1162 0 1080 0 0
0 0 0 0 0 324 0 249 0 0
0 0 0 0 0 521 0 552 0 0
0 0 0 0 0 259 0 237 0 0
0 0 0 0 0 323 0 326 0 0
^Cyou have mail in /var/mail/root
# top
last pid: 1167; load averages: 1.27, 1.45, 1.32 09:50:13
301 processes: 299 sleeping, 2 on cpu
CPU states: 0.0% idle, 26.3% user, 1.8% kernel, 71.9% iowait, 0.0% swap
Memory: 8192M real, 2660M free, 8174M swap free

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
29801 oraprod 1 10 0 1131M 1075M cpu0 20:39 24.82% oracle
15226 oraprod 15 0 0 1129M 1066M sleep 25:54 0.38% oracle
1161 root 1 50 0 2112K 1576K cpu3 0:00 0.22% top
1076 oraprod 1 45 0 1131M 1071M sleep 0:01 0.19% oracle
665 oraprod 1 58 0 1131M 1072M sleep 0:01 0.16% oracle
28911 oraprod 11 58 0 1131M 1075M sleep 0:11 0.13% oracle
566 oraprod 1 58 0 1132M 1074M sleep 0:02 0.11% oracle
7032 oraprod 1 58 0 1127M 1070M sleep 13:16 0.11% oracle
27677 oraprod 1 58 0 1127M 1062M sleep 0:17 0.10% oracle
19162 oraprod 1 48 0 1128M 1070M sleep 15:38 0.09% oracle
15257 oraprod 1 58 0 1129M 1072M sleep 27:57 0.06% oracle
28387 oraprod 1 58 0 1139M 1083M sleep 0:19 0.06% oracle
6785 orademo 1 48 0 366M 348M sleep 0:40 0.05% oracle
29241 oraprod 1 58 0 1131M 1073M sleep 0:38 0.05% oracle
18380 oraprod 11 58 0 1140M 1053M sleep 7:24 0.05% oracle

# vxstat -vdg fin -i 5 -c 5
OPERATIONS BLOCKS AVG TIME(ms)
TYP NAME READ WRITE READ WRITE READ WRITE

Thu 03 Jul 2008 09:51:33 BST
dm fin01 664350 925638 61721810 14663906 6.0 11.2
dm fin02 644431 925638 59385622 14663906 6.2 11.5
dm fin03 1371737 97363 134544722 1566868 3.9 5.7
dm fin04 3254442 2074828 267932096 40545871 36.3 5.6
dm fin05 3341327 2055466 268228934 40474498 38.0 5.6
dm fin06 1282195 23076 134348586 435088 4.0 8.7
dm fin07 3016323 2074828 255467448 40545871 32.6 5.5
dm fin08 3099670 2055466 255793470 40474498 32.2 5.5
dm fin09 266771 1008787 13496144 16136692 66.0 4.9
dm fin10 272213 1008787 13136062 16136692 71.1 4.9
dm fin11 0 362721 0 12697207 0.0 3.6
dm fin101 672872 1523382 61546424 22719313 5.7 6.9
dm fin102 655397 1523382 59229464 22719313 5.9 7.2
dm fin103 868022 24700 89596576 395200 3.6 7.8
dm fin104 822563 556331 89581972 8901268 3.8 6.1
dm fin105 877436 24700 89034982 395200 3.7 7.9
dm fin106 831205 556331 89024856 8901268 3.9 6.2
dm fin107 1328554 97363 132972058 1566868 3.9 5.6
dm fin108 1238113 23076 132760354 435088 4.0 8.7
dm fin109 265196 989247 13488714 15822938 74.1 5.1
dm fin110 265118 989247 12995642 15822938 74.2 5.1
dm fin111 0 782066 0 16473536 0.0 7.9
vol u01 9877859 9111957 2234285936 161661642 42.9 8.5
vol u01-L01 1328269 1523382 120775888 22719313 5.8 10.1
vol u01-L02 1745458 24700 178631558 395200 3.7 12.4
vol u01-L03 2700291 97363 267516780 1566868 3.9 10.5
vol u01-L04 6270765 2074828 523399544 40545871 34.5 7.1
vol u01-L05 530314 989247 26484356 15822938 74.2 6.0
vol u01-L06 1308781 925638 121107432 14663906 6.1 16.0
vol u01-L07 1653768 556331 178606828 8901268 3.9 8.7
vol u01-L08 2520308 23076 267108940 435088 4.0 14.0
vol u01-L09 6440997 2055466 524022404 40474498 35.2 7.1
vol u01-L10 538984 1008787 26632206 16136692 68.5 6.0

Thu 03 Jul 2008 09:51:38 BST
dm fin01 0 1 0 1 0.0 10.0
dm fin02 0 1 0 1 0.0 10.0
dm fin03 0 0 0 0 0.0 0.0
dm fin04 0 0 0 0 0.0 0.0
dm fin05 0 0 0 0 0.0 0.0
dm fin06 0 0 0 0 0.0 0.0
dm fin07 0 0 0 0 0.0 0.0
dm fin08 0 0 0 0 0.0 0.0
dm fin09 0 0 0 0 0.0 0.0
dm fin10 0 0 0 0 0.0 0.0
dm fin11 0 0 0 0 0.0 0.0
dm fin101 0 2 0 32 0.0 5.0
dm fin102 0 2 0 32 0.0 10.0
dm fin103 0 0 0 0 0.0 0.0
dm fin104 0 0 0 0 0.0 0.0
dm fin105 0 0 0 0 0.0 0.0
dm fin106 0 0 0 0 0.0 0.0
dm fin107 0 0 0 0 0.0 0.0
dm fin108 0 0 0 0 0.0 0.0
dm fin109 0 0 0 0 0.0 0.0
dm fin110 0 0 0 0 0.0 0.0
dm fin111 0 0 0 0 0.0 0.0
vol u01 0 3 0 33 0.0 10.0
vol u01-L01 0 2 0 32 0.0 10.0
vol u01-L02 0 0 0 0 0.0 0.0
vol u01-L03 0 0 0 0 0.0 0.0
vol u01-L04 0 0 0 0 0.0 0.0
vol u01-L05 0 0 0 0 0.0 0.0
vol u01-L06 0 1 0 1 0.0 10.0
vol u01-L07 0 0 0 0 0.0 0.0
vol u01-L08 0 0 0 0 0.0 0.0
vol u01-L09 0 0 0 0 0.0 0.0
vol u01-L10 0 0 0 0 0.0 0.0

Thu 03 Jul 2008 09:51:43 BST
dm fin01 0 1 0 16 0.0 0.0
dm fin02 0 1 0 16 0.0 10.0
dm fin03 0 0 0 0 0.0 0.0
dm fin04 0 0 0 0 0.0 0.0
dm fin05 0 0 0 0 0.0 0.0
dm fin06 0 0 0 0 0.0 0.0
dm fin07 0 0 0 0 0.0 0.0
dm fin08 0 0 0 0 0.0 0.0
dm fin09 0 0 0 0 0.0 0.0
dm fin10 0 0 0 0 0.0 0.0
dm fin11 0 0 0 0 0.0 0.0
dm fin101 0 2 0 32 0.0 5.0
dm fin102 0 2 0 32 0.0 5.0
dm fin103 0 0 0 0 0.0 0.0
dm fin104 0 0 0 0 0.0 0.0
dm fin105 0 0 0 0 0.0 0.0
dm fin106 0 0 0 0 0.0 0.0
dm fin107 0 0 0 0 0.0 0.0
dm fin108 0 0 0 0 0.0 0.0
dm fin109 0 0 0 0 0.0 0.0
dm fin110 0 0 0 0 0.0 0.0
dm fin111 0 0 0 0 0.0 0.0
vol u01 0 3 0 48 0.0 10.0
vol u01-L01 0 2 0 32 0.0 10.0
vol u01-L02 0 0 0 0 0.0 0.0
vol u01-L03 0 0 0 0 0.0 0.0
vol u01-L04 0 0 0 0 0.0 0.0
vol u01-L05 0 0 0 0 0.0 0.0
vol u01-L06 0 1 0 16 0.0 10.0
vol u01-L07 0 0 0 0 0.0 0.0
vol u01-L08 0 0 0 0 0.0 0.0
vol u01-L09 0 0 0 0 0.0 0.0
vol u01-L10 0 0 0 0 0.0 0.0

Thu 03 Jul 2008 09:51:48 BST
dm fin01 0 1 0 16 0.0 10.0
dm fin02 0 1 0 16 0.0 10.0
dm fin03 0 0 0 0 0.0 0.0
dm fin04 1 6 16 208 0.0 5.0
dm fin05 0 4 0 64 0.0 10.0
dm fin06 0 0 0 0 0.0 0.0
dm fin07 2 6 32 208 10.0 5.0
dm fin08 0 4 0 64 0.0 5.0
dm fin09 0 0 0 0 0.0 0.0
dm fin10 0 0 0 0 0.0 0.0
dm fin11 0 0 0 0 0.0 0.0
dm fin101 0 2 0 32 0.0 10.0
dm fin102 0 2 0 32 0.0 10.0
dm fin103 0 0 0 0 0.0 0.0
dm fin104 0 0 0 0 0.0 0.0
dm fin105 0 0 0 0 0.0 0.0
dm fin106 0 0 0 0 0.0 0.0
dm fin107 0 0 0 0 0.0 0.0
dm fin108 0 0 0 0 0.0 0.0
dm fin109 0 0 0 0 0.0 0.0
dm fin110 0 0 0 0 0.0 0.0
dm fin111 0 1 0 9 0.0 20.0
vol u01 3 12 48 320 6.7 9.2
vol u01-L01 0 2 0 32 0.0 10.0
vol u01-L02 0 0 0 0 0.0 0.0
vol u01-L03 0 0 0 0 0.0 0.0
vol u01-L04 3 6 48 208 6.7 8.3
vol u01-L05 0 0 0 0 0.0 0.0
vol u01-L06 0 1 0 16 0.0 10.0
vol u01-L07 0 0 0 0 0.0 0.0
vol u01-L08 0 0 0 0 0.0 0.0
vol u01-L09 0 4 0 64 0.0 10.0
vol u01-L10 0 0 0 0 0.0 0.0

Thu 03 Jul 2008 09:51:53 BST
dm fin01 0 1 0 12 0.0 0.0
dm fin02 0 1 0 12 0.0 0.0
dm fin03 0 0 0 0 0.0 0.0
dm fin04 0 0 0 0 0.0 0.0
dm fin05 0 0 0 0 0.0 0.0
dm fin06 0 0 0 0 0.0 0.0
dm fin07 0 0 0 0 0.0 0.0
dm fin08 0 0 0 0 0.0 0.0
dm fin09 0 0 0 0 0.0 0.0
dm fin10 0 0 0 0 0.0 0.0
dm fin11 0 0 0 0 0.0 0.0
dm fin101 0 3 0 34 0.0 10.0
dm fin102 0 3 0 34 0.0 6.7
dm fin103 0 0 0 0 0.0 0.0
dm fin104 0 0 0 0 0.0 0.0
dm fin105 0 0 0 0 0.0 0.0
dm fin106 0 0 0 0 0.0 0.0
dm fin107 0 0 0 0 0.0 0.0
dm fin108 0 0 0 0 0.0 0.0
dm fin109 0 0 0 0 0.0 0.0
dm fin110 0 0 0 0 0.0 0.0
dm fin111 0 0 0 0 0.0 0.0
vol u01 0 4 0 46 0.0 7.5
vol u01-L01 0 3 0 34 0.0 10.0
vol u01-L02 0 0 0 0 0.0 0.0
vol u01-L03 0 0 0 0 0.0 0.0
vol u01-L04 0 0 0 0 0.0 0.0
vol u01-L05 0 0 0 0 0.0 0.0
vol u01-L06 0 1 0 12 0.0 0.0
vol u01-L07 0 0 0 0 0.0 0.0
vol u01-L08 0 0 0 0 0.0 0.0
vol u01-L09 0 0 0 0 0.0 0.0
vol u01-L10 0 0 0 0 0.0 0.0
#


thanks for your help!!!
#


#




Sy UK

 

[Annihilannic]

Well, network is virtually idle now, disks are definitely idle... no idea! What do your DBAs say, is the DB performing okay? If not, what are its primary waits?

Annihilannic.