Hello,
I have a very odd situation, which I have been able to reproduce many times in different machines.
If I config a RAID 1 mirror, regardless of stripe size or cache, and run a simple hdparm -t /dev/sda on that mirror (after it's done synching, I know), it's 16-18MB/sec. If I break the mirror by doing this
# ipssend setstate 1 1 1 DDD
and run hdparm -t /dev/sda again, I get 70MB/sec
The drives I'm using are ST3146707LC's which are extremely fast Cheetah 10K.7 146GB drives. Both are brand new. Now in case anyone says that this is happening as they're not IBM FRU's, I also did the exact same test with a couple of IBM Flash'ed Atlas 10K IV's 73GB, same thing. Mirror them, and it's slow as mollases, break the mirror, it flies.
This was reproduced with ServeRAID 4LX, 4L, 4M, and also with 4H in two different servers weeks apart. I am starting to realize that I simply cannot run RAID 1 as it's ridiculously slow. Here's the catch: This happens ONLY in Linux. In Windows '2003 it's performing normally, 70-80MB/sec in Atto using a 32MB test block. Linux however is a totally different story.
So, this is an odd scenario, I have tried changing stripe size, changing kernel, changing write cache mode, everything makes no difference. What was really amazing is that when I pulled the 4Lx and replaced with a 4H there was no difference.
Testing is being done on IBM xSeries 350's and 360's.
IF ANYONE has any clue I would appreciate a post. We're running BIOS 7.12 and also Driver 7.10, with Firmware 7.12 as well.
Controller type : ServeRAID-4Lx
Controller FRU : 06P5737
Serial number : 03181976
Part number : 24P2531
Actual BIOS version : 7.12.02
Firmware version : 7.12.02
Boot block version : 7.12.02
Device driver version : 7.10.18
Controller slot information : 3
05:03.0 RAID bus controller: IBM ServeRAID Controller
Subsystem: IBM ServeRAID-4Lx
Flags: bus master, 66Mhz, slow devsel, latency 96, IRQ 201
Memory at ecffe000 (32-bit, prefetchable) [size=8K]
Capabilities: [80] Power Management version 2
[root@santorini ~]# hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 50 MB in 3.06 seconds = 16.35 MB/sec
[root@santorini ~]# hdparm -T /dev/sda
/dev/sda:
Timing cached reads: 756 MB in 2.01 seconds = 376.74 MB/sec
[root@santorini ~]#
Watch this. I'll force-fail one of the drives:
[root@santorini ~]# ipssend setstate 1 1 1 DDD
Found 1 IBM ServeRAID controller(s).
Set device state has been initiated for controller 1...
Command completed successfully.
[root@santorini ~]# hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 174 MB in 3.02 seconds = 57.61 MB/sec
[root@santorini ~]#
I'll do it again
[root@santorini ~]# hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 176 MB in 3.03 seconds = 58.11 MB/sec
[root@santorini ~]# hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 176 MB in 3.01 seconds = 58.50 MB/sec
[root@santorini ~]#
Here are the drives
Device is a Hard disk
SCSI ID : 0
PFA (Yes/No) : No
State : Online (ONL)
Size (in MB)/(in sectors): 140014/286749208
Device ID : SEAGATE ST314670D7013KS2GB2H
FRU part number : 4 Seagate Al
Target on SCSI ID 1
In any event this is a serious problem, and I'm sure it's affecting many others as well, if anyone has any clues other than simply adding a 3rd drive and running RAID 5, I'm all ears.
Thanks
Vlad
I have a very odd situation, which I have been able to reproduce many times in different machines.
If I config a RAID 1 mirror, regardless of stripe size or cache, and run a simple hdparm -t /dev/sda on that mirror (after it's done synching, I know), it's 16-18MB/sec. If I break the mirror by doing this
# ipssend setstate 1 1 1 DDD
and run hdparm -t /dev/sda again, I get 70MB/sec
The drives I'm using are ST3146707LC's which are extremely fast Cheetah 10K.7 146GB drives. Both are brand new. Now in case anyone says that this is happening as they're not IBM FRU's, I also did the exact same test with a couple of IBM Flash'ed Atlas 10K IV's 73GB, same thing. Mirror them, and it's slow as mollases, break the mirror, it flies.
This was reproduced with ServeRAID 4LX, 4L, 4M, and also with 4H in two different servers weeks apart. I am starting to realize that I simply cannot run RAID 1 as it's ridiculously slow. Here's the catch: This happens ONLY in Linux. In Windows '2003 it's performing normally, 70-80MB/sec in Atto using a 32MB test block. Linux however is a totally different story.
So, this is an odd scenario, I have tried changing stripe size, changing kernel, changing write cache mode, everything makes no difference. What was really amazing is that when I pulled the 4Lx and replaced with a 4H there was no difference.
Testing is being done on IBM xSeries 350's and 360's.
IF ANYONE has any clue I would appreciate a post. We're running BIOS 7.12 and also Driver 7.10, with Firmware 7.12 as well.
Controller type : ServeRAID-4Lx
Controller FRU : 06P5737
Serial number : 03181976
Part number : 24P2531
Actual BIOS version : 7.12.02
Firmware version : 7.12.02
Boot block version : 7.12.02
Device driver version : 7.10.18
Controller slot information : 3
05:03.0 RAID bus controller: IBM ServeRAID Controller
Subsystem: IBM ServeRAID-4Lx
Flags: bus master, 66Mhz, slow devsel, latency 96, IRQ 201
Memory at ecffe000 (32-bit, prefetchable) [size=8K]
Capabilities: [80] Power Management version 2
[root@santorini ~]# hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 50 MB in 3.06 seconds = 16.35 MB/sec
[root@santorini ~]# hdparm -T /dev/sda
/dev/sda:
Timing cached reads: 756 MB in 2.01 seconds = 376.74 MB/sec
[root@santorini ~]#
Watch this. I'll force-fail one of the drives:
[root@santorini ~]# ipssend setstate 1 1 1 DDD
Found 1 IBM ServeRAID controller(s).
Set device state has been initiated for controller 1...
Command completed successfully.
[root@santorini ~]# hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 174 MB in 3.02 seconds = 57.61 MB/sec
[root@santorini ~]#
I'll do it again
[root@santorini ~]# hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 176 MB in 3.03 seconds = 58.11 MB/sec
[root@santorini ~]# hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 176 MB in 3.01 seconds = 58.50 MB/sec
[root@santorini ~]#
Here are the drives
Device is a Hard disk
SCSI ID : 0
PFA (Yes/No) : No
State : Online (ONL)
Size (in MB)/(in sectors): 140014/286749208
Device ID : SEAGATE ST314670D7013KS2GB2H
FRU part number : 4 Seagate Al
Target on SCSI ID 1
In any event this is a serious problem, and I'm sure it's affecting many others as well, if anyone has any clues other than simply adding a 3rd drive and running RAID 5, I'm all ears.
Thanks
Vlad