Symmetrix 3930 sustained bandwidth for HDTV editing?

[AxelMertes]

Hi!

I am currently investigating the alternatives for installing a new 8+ TB RAID secured storage solution for our new video workstations. The system will work in uncompressed HDTV video modes, eating ~250 MB/s per system at peak (dual stream realtime), 125 MB/s on average. A total of 3 such systems plus access from a rendering farm of 40 CPUs will be there. We expect an minimum needed bandwidth of 500 MB/s at least to work fluently.

Current options are e.g. striping two dual-active-active SATA-to-FC RAIDs such as from Infortrend, utilizing 32 SATA drives such as the 250 GB Maxtor, and up to 4 2GBit FC ports. Each enclosure is just 4 HE in size. The expected bandwith is about 150 MB/s per active controller, meaning up to 600 MB/s with 4 controllers. Price for such installation is about 25,000 Euros for 8 TB storage, 6 TB net RAID5 disk space. We have already Gadzoox 2 GBit switches. This stuff is available now.

Another option would be the upcoming Medea StreamRAID48, that will utilize 48 SATA drives in incredible 4 HE rack space, including a 16 port 2 GBit switch, dual-active-active controllers and 8 RAID sets of 5+1 disks. That will give us incredible 15 TB/12 TB net in 4 HE! But they are still working on the release and its to be expected to market in 2. quarter 2004. For now just the smaller 3 HE 12 drive model is available for OEMs.

Another option now opens on the horizon and thats why I am asking in this forum:

We could buy one or two used but almost new Symmetrix 3930 with 9.2 TB and 12+ GB cache each. I knew these monsters just from exhibitions and main frames, but never actually used them. I read lot about its famous features, but when you look close you find its usually off-the-shelf hardware components put together to build these devices PLUS advanced software I don´t really need or can not utilize in my application. But, given the idea of having a comparison of 256 times 32GB disk vs. 32 times 250GB disks, the Symmetrix should be really fast.

If you consider my application as just wanting ONE large single Windows NTFS partition with maximum disk space (6+ TB net) how complex is the setup of the Symmetrix 3930 then? We may do software striping of smaller than 2 TB sized slices if necessary - some controllers have addressing problems beyond 2 TB LUNs. I don´t need all these advanced mainframe software / caching stuff. I just need pure disk space and pure non-cached sustained disk performance of large block sequential data - usually the opposite of data base users.

All info I found so far is that it has two cache buses with 360 MB/s each. But is that per channel director, or disk director? Or is that total bandwidth at max peak?

If I consider that the whole system will be used as ONE large disk array (shared storage for HDTV content) I expect some 200 disks working at once when reading / writing data. Given the idea that at least 10 MB/s come in/out each disk as a very minimum expectation, this should yield into 2 GByte/s transfer speeds easily. But where is the real bottleneck in these systems? Feeding the speed out using only 1 GBit connections could be a bottleneck.

Has anyone ever tested the total sustained bandwidth of such systems as the Symmetrix 3930? Think of four host computers with dual 2 GBit HBAs each, reading sequencial data and maximum disk speed, how much megabytes per second will I see? 300, 600, 800?

Is the disk director principle in the 3930 comparable to what DataDirect does with their 8000 and 8500 models? They can feed up to 800+ MB/s sustained from 56 disks minimum.

Does the 3930 use FC or SCSI disks? What brand is the RAID controller? The ED-1032 within is a brocade as far as I know these devices, right?

You see, I have a lot of questions, but I guess not many customers in my business ever worked or thought about such systems as the Symmetrix 3930. So I just ask here, as I can see some people do know the 3930 for real.

I would be very pleased if someone around could give me some hints and answers.

Btw, I have an offer for the 3930 that is close to the SATA RAID stuff - now you may understand why I think about this. I know that power consumption and space is much less for the SATA stuff, but its definetly slow drives compared to the 3930. I just want to make a right decision and don´t want to buy the wrong thing.

Best regards,


Axel Mertes

 

[comtec17]

I dont have but a minute to answer this question and can get back to it later but for now here are some of my thoughts.

A 3930 cannot use 2GB director ports so you are only limited to 1GB per port.

Currently, a windows system can only see up to 2TB of disk. This might have been increased with Windows 2003 but 2k and NT are limited to 2TB.

You shouldn't need a huge Symmetrix box to do this. I would recommend using a CX600 or a StorageTek D280. Here is the site for STK with specs

http://www.storagetek.com/products/disk_storage/disk_systems.html

With STK SANtricity you can set to use no cache and get great performance. The STK also uses FC drives.

Another solution would be the Hitachi Thunder 9500 series.

http://www.hds.com/products/systems/9500v/

The ED-1032 is a McData switch not brocade.

This is all on personal preferance, cost, and tons of homework. Every vendor will tell you that their stuff is better so make sure you ask for benchmark results or to see a proof of concept in their labs. Sorry, but I have to take off but can answer more later if you want.

 

[AxelMertes]

Hi Comtec17,

your infos are welcome. I would appreciated a more detailed answer on the concept behind the 3930. As I wrote I have a good offer regarding price, so I need to understand if the unit will fit or not.

Btw, I have info from two vendors that have successfully created NTFS volumes far beyond 2 TB size using Windows XP Professional. The 2 TB limitation seems to be hardware inherent, not software inherent. NTFS is told to handle 64 TB volume size and 2 TB file size maximum. Many SCSI/FC devices may limit LUNs at 2 TB either, but some already overcome the barrier through extra addressing bits. You simply need to make 2 TB slices and stripe them in Windows XP Professional. This is known to work. We don´t need any "Server OS", except for a gateway machine interfacing to Gigabit Ethernet for those machines not connected to the SAN itself.

Looking forward to your more detailed answers,

thanks in advance,

Axel

 

[Chacalinc]

Hi Axel,

You can get the info about 3930 from here:

http://www.emc.com/products/systems/symmetrix_5000_3000.jsp

My recommendation is to use RAID 3. RAID 5 (with all disk arrays) is not a good choice for stream video. R5 has a poor performance in writes. ATA drives are huge, but slower than FC. Or you can use a CLARiiON with RAID 10, but it could be expensive.

"All info I found so far is that it has two cache buses with 360 MB/s each. But is that per channel director, or disk director? Or is that total bandwidth at max peak?":

+----+  +----+
| CD |  | CD |
+----+  +----+
   |      |
--------------------cache   (bus X)
--------------------cache   (bus Y)
   |      |
+----+  +----+
| DD |  | DD |
+----+  +----+

CD: Channel Director
DD: Disk Director

bus X and Y are 360 MB/Seg (together: 720 MB/s). Odd cards (CD1-CD3-DD5-DD7 i.e.) are connected by bus X and even card by bus Y.

Hope this help you.

 

[AxelMertes]

Hi Chacalinc,

is it correct then to assume that the maximum total performance possible is 720 MB/s. Given the system as you described this is a physical boarder not breakable with this design, right?

I guess then I´d better go for the Infortrend solutions or e.g. the new 5884 controller from LSI Logic. They can at least deliver 780 MB/s sustained per controller.

Thanks for this info.


Axel

 

[Chacalinc]

"this is a physical boarder not breakable with this design, right?" physically, yes, but:

that's not quite correct... you can think that 720 MB/s is the maximum performance, but that is the hardware. EMC has a very good microcode and a big cache. the cache is used very efficiently so you can get better performance than using the machine as hardware only. (sorry, I don't know how to say it in english).

Another thing to think is: the new 5884 controller from LSI Logic deliver 780 MB/s, but the backend is the same? can this array deliver at least 780 MB/s in the backend? (because you have at least 2 controllers, will it be a bottleneck?). be careful with that.

Anyway, I don't say that 3930 is the right solution for you, I'm showing you the technical things only... I didn't test the Symm boxes on stream video solutions. I strongly recommend you RAID 3. This is the best solutions for you.

And how we say in spanish: the cheap things costs a lot!

 

[AxelMertes]

Hi Chacalink,

the Infortrend EonStore16 (SATA) does constant 125-140 MB/s RAID5 read/write even with fragmented disks and multiple streams at the same time, with ONE controller. The dual-active-active variant will be probably twice as fast, as it comes with up to four 2 GBit ports and two physical controllers. So scalabilty of the EonStore16 will be something like 250 MB/s per 4 HE chassis.

The 5884 in its SATA variant does maximize throughput at 780 MB/s. I guess you need at least two drive chassis to get there, such as 28 disks. But to nice thing is it scales easily beyond that disks size by adding further chassis (up to 224 disk total), but the limit will always be 780 MB/s, no more, but apparently then no less either.

I have no idea about the price tag for this 5884 SATA device. If anyone is around and could give me an average for that I´d appreciate this strongly. I guess it´ll be more expensive than the Infortrends, but at a certain amount of disks this might change, as I do not buy always new controllers as I will do with Infortrend solutions. On the other hand do the many controllers on the Infortrend then add up an incredible bandwidth, but you need many many ports on switches for it, which is then the pricey bottle-neck.

Thanks for this comment.

Anyone else who has some ideas/alternatives I should look at?

Axel