IBM DS 300 SAN / Performance Problems on Virtual Server 2005

[marcelhaas]

hello community..

we've set up a iscsi san storage infrastructure based on ibm ds 300 and qlogic iscsi hba's.

last days we detected a performance bottleneck on our virtual machines when writing data to san attached logical drives.

some information about our infrastructure:

ibm ds300 (as mentioned)
redundant controllers
1 raid 5 array with 6 x 300gb disks
san clients attached over qlogic iscsi hba's 4010
gigabit ethernet

i made some performance tests and monitored the performance counters on the selected virtual servers. with this tests i detected that the AVG. Disk Queue Lenth is over recommended value of 2.

based on this i think that the bottleneck is the disk. for example if i write batch files to the disk for testing the write performance i can see the queue up arround 100% but cpu activity is low. as i understand the worker for the queue don't allocates enough cpu memory.

i'am pretty sure, that it is a issue that is also virtual server related, because we have that problem just on our virtual machines. when i did the performance messurements for exemple on the host plattform which has also san attaches disks, performance is ok.

Some results of my disk benchmark tests(used the tool diskbench which writes batchfiles to the disk and analyses the time in which the file was written and iterates the size of the file):

Results of writing data on a san attached disk / physical server:

Starting Batch Create File Bench...

240 MB; c:\test; 251658240 bytes; 1156 ms; 207.612 MB/s
260 MB; c:\test; 272629760 bytes; 922 ms; 281.996 MB/s
280 MB; c:\test; 293601280 bytes; 1016 ms; 275.591 MB/s
300 MB; c:\test; 314572800 bytes; 1469 ms; 204.221 MB/s
320 MB; c:\test; 335544320 bytes; 1141 ms; 280.456 MB/s

Create Batch File Bench ended

Results of writing data on a san attached disk / virtual server:


Starting Batch Create File Bench...

240 MB; c:\test; 251658240 bytes; 42188 ms; 5.689 MB/s
260 MB; c:\test; 272629760 bytes; 2023 ms; 128.522 MB/s
280 MB; c:\test; 293601280 bytes; 466699 ms; 0.600 MB/s
300 MB; c:\test; 314572800 bytes; 4066 ms; 73.783 MB/s
320 MB; c:\test; 335544320 bytes; 3194 ms; 100.188 MB/s

Create Batch File Bench ended


here you ca see, that the performance is inconsistent and pretty slow.


does anyone here have exerience with san performance problems? i have a lot of screenshots and results of my analysis, but can't post it here. if someone is interested, i would send this docs over email.

i talked to ibm about this problem and gave them all information and config logs. they gave me a feedback that the config is ok and all hardware is working fine.

thats nice, but it didn't help..

 

[s36e175]

Hi, DS300 is a very slow system. IBM has now withdrawn this model (and the DS400) after only 12 months in the field.
They have been replaced with the *much* faster DS3000 but the iSCSI model (DS3300 won't be available until June).

I have been involved in a a couple of DS300 situations and, trust me, there is no way to make a DS300 run fast.

Regards.

 

[marcelhaas]

Hey, this is bad news;-)

Thanks for your feedback anyway. the most anoiying thing is that the write cache is disabled if you use two controllers. i plugged out the second controller, but still then i can't enable the write caching.

so, you recommend to replace the lovely DS 300, or what do you plan i your case?

Regards, Marcel

 

[marcelhaas]

update:

ibm has withdrawn the DS 300 because it does not meet some EU specifications. (RoSH Standard)

http://en.wikipedia.org/wiki/RoHS

pretty bad!