iostat how to identify slow disk? 1

[Scunningham99]

Hi all

How can i identify slow disk from iostat -xtn. The one in this case is "ssd8"

extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
20.2 167.1 310.3 1627.9 0.0 0.2 0.0 1.0 0 15 hsx1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx202
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx1207
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx1408
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t6d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 ssd0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 ssd1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.7 0 0 ssd2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 ssd3
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 ssd4
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0 0 ssd5
0.2 0.9 4.3 12.0 0.0 0.0 2.2 27.4 0 1 ssd6
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0 ssd7
0.2 0.3 5.2 2.4 0.0 0.0 18.8 27.4 0 0 ssd8
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 ssd9
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.6 0 0 ssd10
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 0 0 ssd11
20.2 167.2 310.4 1628.3 0.0 0.2 0.0 1.0 0 16 c2t0d2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 euratlsunx06:vold()

AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf8ee305,0
1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfc27da5,0
2. c1t2d0 <SEAGATE-ST373405FSUN72G-0438 cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf9482d8,0
3. c1t3d0 <HITACHI-DK32EJ72FSUN72G-2Q09 cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w210000008715306a,0
4. c1t4d0 <SEAGATE-ST373405FSUN72G-0438 cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf8eab4c,0
5. c1t5d0 <SEAGATE-ST373405FSUN72G-0438 cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf948473,0
6. c1t8d0 <SEAGATE-ST373405FSUN72G-0638 cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfc2ab2c,0
7. c1t9d0 <SEAGATE-ST373405FSUN72G-0638 cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfbfc39c,0
8. c1t10d0 <SEAGATE-ST373405FSUN72G-0638 cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfb6a9a5,0
9. c1t11d0 <SEAGATE-ST373405FSUN72G-0438 cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf6b3672,0
10. c1t12d0 <FUJITSU-MAN3735FSUN72G-0704 cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0102a3b01,0
11. c1t13d0 <SEAGATE-ST373405FSUN72G-0438 cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf94720b,0
12. c2t0d2 <COMPAQ-HSV110(C)COMPAQ-3025 cyl 51198 alt 2 hd 128 sec 128>
/swsp@0,1/ssd@0,2

euratlsunx06# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c1t0d0s0 39247075 2248258 36606347 6% /
/dev/dsk/c1t0d0s6 8263373 587466 7593274 8% /usr
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
/dev/dsk/c1t0d0s1 4133838 73933 4018567 2% /var
swap 11806768 16 11806752 1% /var/run
dmpfs 11806752 0 11806752 0% /dev/vx/dmp
dmpfs 11806752 0 11806752 0% /dev/vx/rdmp
swap 11806840 88 11806752 1% /tmp
/dev/dsk/c1t1d0s2 70592505 15194536 54692044 22% /u01
/dev/dsk/c1t0d0s5 8263373 924489 7256251 12% /opt
/dev/vx/dsk/san/volume01
419411968 398292197 19799833 96% /san01
/dev/dsk/c1t0d0s7 2407311 1824602 534563 78% /export/home
/dev/null 0 0 0 0% /dev/odm

THANKS IN ADVANCE?

Sy UK

 

[Annihilannic]

You can work back from /etc/path_to_inst. Here is an example looking for disk 'sd0'. Replace the 'sd' with 'ssd' and the 0 with 8:

[tt]# awk '$2 == 0 && $3 == "\"sd\""' /etc/path_to_inst
"/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0" 0 "sd"
# ls -l /dev/dsk | grep "/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0"
lrwxrwxrwx 1 root root 51 Sep 11 2004 c0t0d0s0 -> ../../devices/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0:a
lrwxrwxrwx 1 root root 51 Sep 11 2004 c0t0d0s1 -> ../../devices/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0:b
lrwxrwxrwx 1 root root 51 Sep 11 2004 c0t0d0s2 -> ../../devices/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0:c
lrwxrwxrwx 1 root root 51 Sep 11 2004 c0t0d0s3 -> ../../devices/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0:d
lrwxrwxrwx 1 root root 51 Sep 11 2004 c0t0d0s4 -> ../../devices/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0:e
lrwxrwxrwx 1 root root 51 Sep 11 2004 c0t0d0s5 -> ../../devices/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0:f
lrwxrwxrwx 1 root root 51 Sep 11 2004 c0t0d0s6 -> ../../devices/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0:g
lrwxrwxrwx 1 root root 51 Sep 11 2004 c0t0d0s7 -> ../../devices/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0:h
#[/tt]

I had to do this so often (usually diagnosing disk errors in /var/adm/messages) that I wrote a small script to do it.


Annihilannic.

 

[tuka]

1. run iostat -E (look for ssd8 and get the Serial No:



2. run iostat -En (Match Serial No. from above to
the correct cxtxdx)

 

[Scunningham99]

Hi When I issue the above 2 commands it still displays sdd tag for most of the drives. Please advise?

euratlsunx06# iostat -E
sd6 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: TOSHIBA Product: DVD-ROM SD-M1401 Revision: 1009 Serial No: 12/20/00
Size: 18446744073.71GB <-1 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0216K1JWTB
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0201K0ZFYD
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd2 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0638 Serial No: 0244K2M7EK
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd3 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0638 Serial No: 0241K2HAPK
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd4 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: FUJITSU Product: MAN3735F SUN72G Revision: 0704 Serial No: 0303V84098
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd5 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: HITACHI Product: DK32EJ72FSUN72G Revision: 2Q09 Serial No: 0340W1F8G6
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd6 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0638 Serial No: 0244K2E3BP
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd7 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0217K1K0ZK
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd8 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0216K1HSHY
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd9 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0638 Serial No: 0236K2CX8F
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd10 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0215K1H5YX
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd11 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0217K1K7NW
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd13 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0
Vendor: COMPAQ Product: HSV110 (C)COMPAQ Revision: 3025 Serial No: 5849ARA04G
Size: 429.49GB <429488340992 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0




euratlsunx06# iostat -En
c0t6d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: TOSHIBA Product: DVD-ROM SD-M1401 Revision: 1009 Serial No: 12/20/00
Size: 18446744073.71GB <-1 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0216K1JWTB
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0201K0ZFYD
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd2 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0638 Serial No: 0244K2M7EK
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd3 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0638 Serial No: 0241K2HAPK
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd4 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: FUJITSU Product: MAN3735F SUN72G Revision: 0704 Serial No: 0303V84098
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd5 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: HITACHI Product: DK32EJ72FSUN72G Revision: 2Q09 Serial No: 0340W1F8G6
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd6 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0638 Serial No: 0244K2E3BP
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd7 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0217K1K0ZK
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd8 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0216K1HSHY
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd9 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0638 Serial No: 0236K2CX8F
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd10 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0215K1H5YX
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
ssd11 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373405FSUN72G Revision: 0438 Serial No: 0217K1K7NW
Size: 73.40GB <73400057856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c2t0d2 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0
Vendor: COMPAQ Product: HSV110 (C)COMPAQ Revision: 3025 Serial No: 5849ARA04G
Size: 429.49GB <429488340992 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0

THANKS IN ADVANCE TO THE BOTH OF YOU!

Sy UK

 

[cndcadams]

try iostat -xctn will give you cxtxdx

thanks

CA

 

[Scunningham99]

Thanks for your reply.. but this doesnt seem to yield desired results either:-
euratlsunx06# iostat -xctn
tty cpu
tin tout us sy wt id
0 17 55 5 41 0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
16.1 105.9 261.7 1098.9 0.0 0.1 0.0 1.1 0 11 hsx1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx202
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx1207
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 hsx1408
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t6d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 ssd0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 ssd1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.7 0 0 ssd2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 ssd3
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 ssd4
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0 0 ssd5
0.1 0.8 1.8 10.7 0.0 0.0 3.1 30.8 0 1 ssd6
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0 ssd7
0.1 0.3 2.2 2.1 0.0 0.0 13.3 28.7 0 0 ssd8
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 ssd9
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.6 0 0 ssd10
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 0 0 ssd11
16.1 105.9 261.7 1099.0 0.0 0.1 0.0 1.2 0 11 c2t0d2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 euratlsunx06:vold()

any ideas??



Sy UK

 

[Annihilannic]

No luck with my idea?

Annihilannic.

 

[Scunningham99]

Hi Annihlannic

Doesent seem to work, however not sure if i am doing it correctly:-

euratlsunx06# awk '$2 == 0 && $3 == "\"sd\""' /etc/path_to_inst "/sbus@1f,0/SUNW
,fas@e,8800000/sd@0,0" 8 "sdd"
"/pci@8,700000/scsi@1/sd@0,0" 0 "sd"
awk: can't open /sbus@1f,0/SUNW,fas@e,8800000/sd@0,0
record number 2738
euratlsunx06# ls -l /dev/dsk | grep "/pci@8,700000/scsi@1/sd@0,0" 0 "sd"
grep: can't open 0
grep: can't open sd
euratlsunx06# ls -l /dev/dsk | grep "/pci@8,700000/scsi@1/sd@0,0"

please advise!

Sy UK

 

[cndcadams]

what os is this?

I found this script on sun that gives controller and serial#, we can then check it against your iostat -E output.

SENA disk serial numbers
Script to display disk serial numbers for fibre channel devices and devices in a Sun Enterprise Network Array (SENA):

#!/bin/sh

for disk in `format < /dev/null | egrep "[0-9]+\." | awk '{print $2}'`
do
serial=`luxadm display /dev/rdsk/${disk}s2 2>/dev/null | grep Serial | awk '{print $3}'`
echo "$disk\t$serial"
done


hope this helps.

CA

 

[Annihilannic]

Sorry, that was just an example using sd0, I should have clarified. To search for ssd8 the first command is:

[tt]awk '$2 == 8 && $3 == "\"ssd\""' /etc/path_to_inst[/tt]

Then copy and paste the output of that into the next command:

[tt]ls -l /dev/dsk | grep <paste here>[/tt]

I hope that's clearer.

Annihilannic.

 

[Annihilannic]

D'oh, not *all* of the output, just the first part, e.g.

[tt]"/sbus@1f,0/SUNW,fas@e,8800000/sd@0,0"[/tt]

Annihilannic.

 

[Scunningham99]

Hey annihilannic

that worked a treak.

the script before was good but did not show internal disks. But thanks for info anyway as im sure it will come usefull in the future.

thabns again to all of you for all your help!!

Sy UK