IBM EXP300 defunct disks

[davemak]

Recently installed IBM x330 with an EXP300 external disk unit, this contains 14 73gb drives. Problem I have is that i keep getting 'defunct' disks, first failure there were seven drives defunct, they were brought back online ok but it took the raidmanager approx 24 hours to re-sync. This week another three drives have gone defunct, do not think it is a drive problem but maybe the raid controller (serverRaid-4Mx, anybody any suggestions? Have attached output from the config and event logs.

ServeRAID defunct drive event log for controller 1 (37 entries):

Date and time Serial number Correlation ID Failure codeescription
December 4, 2002 7:17:52 PM GMT+00:00 3EK2EY4W 80001f24 40030001efunct drive (FRU Part # 06P5319) - Physical drive added to server on controller 1, channel 2, SCSI ID 0.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2K408 80001f25 40030001efunct drive (FRU Part # 06P5319) - Physical drive added to server on controller 1, channel 2, SCSI ID 1.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2K7R3 80001f25 40030001efunct drive (FRU Part # Not available) - Physical drive added to server on controller 1, channel 2, SCSI ID 2.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2DMXR 80001f26 40030001efunct drive (FRU Part # Not available) - Physical drive added to server on controller 1, channel 2, SCSI ID 3.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2AR2Y 80001f27 40030001efunct drive (FRU Part # 06P5319) - Physical drive added to server on controller 1, channel 2, SCSI ID 4.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2K8VA 80001f27 40030001efunct drive (FRU Part # 06P5319) - Physical drive added to server on controller 1, channel 2, SCSI ID 5.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2ET2Z 80001f28 40030001efunct drive (FRU Part # Not available) - Physical drive added to server on controller 1, channel 2, SCSI ID 6.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2JMZG 80001f28 40030001efunct drive (FRU Part # Not available) - Physical drive added to server on controller 1, channel 2, SCSI ID 8.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2JMRT 80001f29 40030001efunct drive (FRU Part # Not available) - Physical drive added to server on controller 1, channel 2, SCSI ID 9.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2K5EV 80001f29 40030001efunct drive (FRU Part # Not available) - Physical drive added to server on controller 1, channel 2, SCSI ID 10.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2F539 80001f2a 40030001efunct drive (FRU Part # Not available) - Physical drive added to server on controller 1, channel 2, SCSI ID 11.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2JXYM 80001f2a 40030001efunct drive (FRU Part # Not available) - Physical drive added to server on controller 1, channel 2, SCSI ID 12.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2EZQK 80001f2b 40030001efunct drive (FRU Part # Not available) - Physical drive added to server on controller 1, channel 2, SCSI ID 13.
December 4, 2002 7:17:52 PM GMT+00:00 3EK2JSZP 80001f2b 40030001efunct drive (FRU Part # Not available) - Physical drive added to server on controller 1, channel 2, SCSI ID 14.
December 4, 2002 11:26:27 PM GMT+00:00 3EK2K7R3 8007b0e0 4002670cefunct drive (FRU Part # Not available) - I/O subsystem error on controller 1, channel 2, SCSI ID 2.
December 4, 2002 11:26:33 PM GMT+00:00 3EK2ET2Z 8007b33c 4002760cefunct drive (FRU Part # Not available) - I/O subsystem error on controller 1, channel 2, SCSI ID 6.
December 4, 2002 11:26:35 PM GMT+00:00 3EK2DMXR 8007b47d 4002740fefunct drive (FRU Part # Not available) - I/O subsystem error on controller 1, channel 2, SCSI ID 3.
December 4, 2002 11:26:35 PM GMT+00:00 3EK2JMRT 8007b47e 4002760cefunct drive (FRU Part # Not available) - I/O subsystem error on controller 1, channel 2, SCSI ID 9.
December 4, 2002 11:26:35 PM GMT+00:00 3EK2F539 8007b480 4002760cefunct drive (FRU Part # Not available) - I/O subsystem error on controller 1, channel 2, SCSI ID 11.
December 5, 2002 8:51:45 AM GMT+00:00 3EK2K7R3 8041c3c3 40020005efunct drive (FRU Part # Not available) - User marked failed on controller 1, channel 2, SCSI ID 2.
December 9, 2002 11:25:56 AM GMT+00:00 3EK2PJ0F 825d8e85 40020005efunct drive (FRU Part # Not available) - User marked failed on controller 1, channel 2, SCSI ID 11.
December 9, 2002 11:35:33 AM GMT+00:00 3FP00TB3 825e8b83 10f52100efunct drive (FRU Part # Not available) - SCSI error on controller 1, channel 2, SCSI ID 9.
December 9, 2002 2:23:04 PM GMT+00:00 3EK2PXVJ 80000de3 40020004efunct drive (FRU Part # Not available) - User accepted on controller 1, channel 2, SCSI ID 9.
December 9, 2002 2:23:04 PM GMT+00:00 3EK2EZQK 80000de3 40020004efunct drive (FRU Part # Not available) - User accepted on controller 1, channel 2, SCSI ID 13.
December 9, 2002 2:23:04 PM GMT+00:00 3EK2JSZP 80000de4 40020004efunct drive (FRU Part # Not available) - User accepted on controller 1, channel 2, SCSI ID 14.
December 23, 2002 3:03:21 PM GMT+00:00 3EK2EY4W 800014cb 4002740fefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 0.
December 23, 2002 3:03:21 PM GMT+00:00 3EK2K408 800014cb 4002760cefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 1.
December 23, 2002 3:03:21 PM GMT+00:00 3EK2PYHB 800014cc 4002740fefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 2.
December 23, 2002 3:03:21 PM GMT+00:00 3EK2PX3V 800014cd 4002760cefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 3.
December 23, 2002 3:03:21 PM GMT+00:00 3EK2AR2Y 800014ce 4002760cefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 4.
December 23, 2002 3:03:21 PM GMT+00:00 3EK2K8VA 800014cf 4002760cefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 5.
December 23, 2002 3:03:21 PM GMT+00:00 3EK2JMZG 800014d0 4002740fefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 9.
December 23, 2002 3:03:21 PM GMT+00:00 3EK2K5EV 800014d1 4002760cefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 11.
December 23, 2002 3:03:21 PM GMT+00:00 3EK2JXYM 800014d3 4002760cefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 13.
December 23, 2002 3:03:21 PM GMT+00:00 3EK2EZQK 800014d4 4002760cefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 14.
December 29, 2002 10:03:49 PM GMT+00:00 3EK2K408 8003e0f4 4002700cefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 1.
December 29, 2002 10:03:49 PM GMT+00:00 3EK2Q2MB 8003e0f5 4002700cefunct drive (FRU Part # 06P5319) - I/O subsystem error on controller 1, channel 2, SCSI ID 6.
------------------------------------------------------------------------
ServeRAID hard event log for controller 1 (0 entries):

----------------------------------------------
null
Server name : columbus


December 30, 2002 2:30:45 PM GMT+00:00

Configuration summary
---------------------------

Server name.....................columbus
ServeRAID Manager agent.........5.11.05
ServeRAID Manager console.......5.11.05
Number of controllers...........1
Operating system................Windows 2000

Information for controller 1
-------------------------------------------------------
Controller type.................ServeRAID-4Mx
BIOS version....................5.11.05
Firmware version................5.11.05
Device driver version...........5.11.05
Controller FRU..................06P5737
Battery FRU.....................37L6903
Serial number...................00213495
Part number.....................24P2589
Physical slot...................1
Battery-backup cache............Installed
Read-ahead cache mode...........Adaptive
Stripe-unit size................8K
Rebuild rate....................High
Hot-swap rebuild................Enabled
Data scrubbing..................Enabled
Auto-synchronization............Enabled
Clustering......................Disabled
Unattended mode.................Disabled
BIOS-compatibility mapping......Extended
Number of arrays................2
Number of logical drives........2
Number of hot-spare drives......0
Number of ready drives..........0

Array A
--------------------
Array identifier................A
Array size in MB................69430
Free space in MB................0
Number of logical drives........1
Stripe order (channel/device)...1/0 1/1
Number of physical drives.......2

Logical drives in array A
--------------------------------
Logical drive...................1
Array letter....................A
State...........................Okay
RAID level......................1
Data space in MB................34715
Parity space in MB..............34715
Date created....................12/04/2002
Write-cache mode................Write through
Merge-group number..............207
Merge-group state...............Non-shared


Physical drives in array A
--------------------------------
Type............................Hard disk drive
Channel.........................1
SCSI ID.........................0
Vendor..........................IBM-ESXS
Product or model number.........ST336752
FRU part number.................06P5352
Serial number...................3ET0MESG
Firmware level..................B843
Size in MB......................34715
State...........................Online
Array letter....................A
PFA error.......................No

Type............................Hard disk drive
Channel.........................1
SCSI ID.........................1
Vendor..........................IBM-ESXS
Product or model number.........ST336752
FRU part number.................06P5352
Serial number...................3ET0PB8Q
Firmware level..................B843
Size in MB......................34715
State...........................Online
Array letter....................A
PFA error.......................No

Array B
--------------------
Array identifier................B
Array size in MB................980084
Free space in MB................0
Spare space in MB...............75413
Number of logical drives........1
Stripe order (channel/device)...2/0 2/1 2/2 2/3 2/4 2/5 2/6 2/8 2/9 2/10 2/11 2/12 2/13 2/14
Number of physical drives.......14

Logical drives in array B
--------------------------------
Logical drive...................2
Array letter....................B
State...........................Offline
RAID level......................5E
Data space in MB................840060
Parity space in MB..............64620
Date created....................12/09/2002
Write-cache mode................Write through
Merge-group number..............207
Merge-group state...............Non-shared


Physical drives in array B
--------------------------------
Type............................Hard disk drive
Channel.........................2
SCSI ID.........................0
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2EY4W
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................1
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2K408
Firmware level..................B245
Size in MB......................70006
State...........................Defunct
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................2
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2PYHB
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................3
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2PX3V
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................4
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2AR2Y
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................5
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2K8VA
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................6
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2Q2MB
Firmware level..................B245
Size in MB......................70006
State...........................Defunct
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................8
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2PXVJ
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................9
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2JMZG
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................10
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2PJ0F
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................11
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2K5EV
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................12
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2JSZP
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................13
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2JXYM
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................14
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2EZQK
Firmware level..................B245
Size in MB......................70006
State...........................Defunct
Array letter....................B
PFA error.......................No


SCSI channel 1
-------------------
Type............................Hard disk drive
Channel.........................1
SCSI ID.........................0
Vendor..........................IBM-ESXS
Product or model number.........ST336752
FRU part number.................06P5352
Serial number...................3ET0MESG
Firmware level..................B843
Size in MB......................34715
State...........................Online
Array letter....................A
PFA error.......................No

Type............................Hard disk drive
Channel.........................1
SCSI ID.........................1
Vendor..........................IBM-ESXS
Product or model number.........ST336752
FRU part number.................06P5352
Serial number...................3ET0PB8Q
Firmware level..................B843
Size in MB......................34715
State...........................Online
Array letter....................A
PFA error.......................No

Type............................Enclosure
Channel.........................1
SCSI ID.........................8
Vendor..........................IBM
Product or model number.........FTlV1 S2
Serial number...................000
Firmware level..................0

SCSI channel 2
-------------------
Type............................Hard disk drive
Channel.........................2
SCSI ID.........................0
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2EY4W
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................1
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2K408
Firmware level..................B245
Size in MB......................70006
State...........................Defunct
Array letter....................B
PFA error.......................No
Write-cache mode................Not available

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................2
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2PYHB
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................3
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2PX3V
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................4
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2AR2Y
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................5
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2K8VA
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................6
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2Q2MB
Firmware level..................B245
Size in MB......................70006
State...........................Defunct
Array letter....................B
PFA error.......................No
Write-cache mode................Not available

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................8
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2PXVJ
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................9
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2JMZG
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................10
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2PJ0F
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................11
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2K5EV
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................12
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2JSZP
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................13
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2JXYM
Firmware level..................B245
Size in MB......................70006
State...........................Online
Array letter....................B
PFA error.......................No
Write-cache mode................Write through

Type............................Hard disk drive
Channel.........................2
SCSI ID.........................14
Vendor..........................IBM-ESXS
Product or model number.........ST373405
FRU part number.................06P5319
Serial number...................3EK2EZQK
Firmware level..................B245
Size in MB......................70006
State...........................Defunct
Array letter....................B
PFA error.......................No
Write-cache mode................Not available

Type............................Enclosure
Channel.........................2
SCSI ID.........................15
Enclosure status................Okay
Fan 1 status....................Okay
Fan 2 status....................Okay
Power supply 1 status...........Okay
Power supply 2 status...........Okay
Temperature status..............Normal
Enclosure ID....................0
Vendor..........................IBM
Product or model number.........EXP300
Serial number...................6836009
Firmware level..................D014
...............................
FRU type........................MIDPLANE
FRU vendor......................IBM
FRU date of manufacture.........10/2002
FRU part number.................19K1239
FRU serial number...............6836009
...............................
FRU type........................CARD
FRU vendor......................IBM
FRU date of manufacture.........10/2002
FRU part number.................19K1171
FRU serial number...............1R1232AS818
...............................
FRU type........................CARD
FRU vendor......................IBM
FRU date of manufacture.........10/2002
FRU part number.................19K1171
FRU serial number...............1R1232AS817
...............................
FRU type........................Power
FRU vendor......................IBM
FRU date of manufacture.........10/2002
FRU part number.................07K5985
FRU serial number...............1R0KH2A3169
...............................
FRU type........................Power
FRU vendor......................IBM
FRU date of manufacture.........10/2002
FRU part number.................07K5985
FRU serial number...............1R0KH2A3163
...............................
FRU type........................CARD
FRU vendor......................IBM
FRU date of manufacture.........10/2002
FRU part number.................19K1173
FRU serial number...............1R0222AS130
...............................
FRU type........................CARD
FRU vendor......................IBM
FRU date of manufacture.........10/2002
FRU part number.................19K1172
FRU serial number...............1R0242AT473


End of the configuration information for controller 1

 

[neilcresswell]

Ok, the first thing you want to do is get online and download IBM Update Express v2.02(ftp://ftp.pc.ibm.com/pub/pccbbs/bp_server/01r0530.iso) and verify the server is at the latest levels of F/W, NVRam etc...Then get the IBM HDD Update program v1.13 (ftp://ftp.pc.ibm.com/pub/pccbbs/pc_servers/73p6458.iso) and update the FW of all HDD's installed in the EXP.
From the look of the error logs it appears to be a communication error between the controller and each disk. This could be caused by a number of things, Scsi Cable, Termination, Faulty ServeRaid Card, Faulty BackPlane in EXP etc etc.

After Performing all code updates, if it still fails check the obvious cables etc then get it looked at by IBM.

N

 

[borrim]

davemak, I found a really good tid on ibm's site regarding multiple drive failures and what to do. you might want to look at it. Under support, do a search for "MIGR-39144" or try this url:

http://www-1.ibm.com/support/docview.wss?uid=psg1MIGR-39144

 

[jeremymc]

We have had the same problem, it was the EXP300 back plan. IBM replaced the backplane and fixed it.