Legato Speed Issues

[jddotman]

My issue is I have around 40 million small tiff files on our production NAS server. It totals to around 4.4 terabytes. The backup using Networker 7.2 and an IBM 3583 library with an IBM 3580 LTO3 tape drive takes around 40 hours for a full backup.
I'm pretty sure this is due to the millions of tiny files and the fact that when the LTO3 drive slows below 40mb a sec it stops and you get the whole head repositioning issue.
The Legato server is Win2k3 HP-DL380 G2. The link speed is 1 gbps and it is %27 utilized. I've checked the disk and memory usage on the client box and those stats are fine. The CPU on the client (dual proc compaq B2000 NAS G2) are getting hammered and staying around %90 with 8 save streams running.
I divided up the nas and did some tests. When I run a backup on 60 files that total 600 gig I get 40mb + per sec. If I run a backup on one of the directories with 5 million small tif files that total 218 gig I get 6-7 mb per sec.
Would a file device to stage pool backup actually improve performance enough? I'm trying to get them to purchase more tape drives as that would help. Can you think of any other ideas.

 

[ofladung]

I would suggest a VTL (virtual tape library). Backup to disk is a nice feature if you have slow clients and want to collect the data bevor you stream it to tapes, but it works without compression and is limitted to 2TB per device (only some of a few "problems"). A VTL act like a real jukebox with compression and is very fast.

regards,
Oliver

 

[605]

Where do you get the info about the 2TB limitation from? This is new to me and i do not think it is true. It might be a bug but i am sure the limitation was never intended.

 

[ofladung]

oops - that's what my trainer said (7.x advanced admin course), if you use a file type device.

 

[ofladung]

I found this article on the web:

http://www.storagenetworking.org/Discussion/forum_posts.asp?TID=19&PN=1&get=last

I think it depends on the OS you have.

 

[jddotman]

Thanks for the link.

 

[AcclaroVeritas]

Did anyone have a solution for this? I'm also backing up several different volumes with millions of tiny files. It seems to keep the backups (traditional ethernet network) backups.

 

[Rexxy]

have you considered investing in Legato's snapimage module, I have seen this in a customers site and it worked amazingly well, below is an exerpt from a website regardiung snapimage......


NetWorker SnapImage Module - Two Unique Solutions; One Powerful Module

Solution One: Live, Block-Level Backup for Protecting Many Small Files

Solution Two: Enables Remote Backup of NDMP-Compliant NAS File Servers Natively vs. NFS

The NetWorker SnapImage Module enables an enterprise with a limited backup window to fully protect active file systems containing many small files. Managed by Legato NetWorker, SnapImage delivers high performance, low impact data protection through live, block-level backup. SnapImage is also a standards-based NDMP agent that enables remote backup of NDMP-compliant NAS file servers natively vs. NFS in heterogeneous environments. SnapImage provides superior leverage, scalability, and manageability of your


cheers,

Rexxy

 

[605]

Of course the backup to a disk will improve the whole scenario in this case. Disks do not need to reposition (well, actually pausing the stream is nothing else but 'a kind of logical repositioning' but this time is only a fraction of seconds while on tape devices it will at least take 2 seconds or more - depending on the device type.

There are a lot of other solutions available which all depend on the scenario. NAS is a general term that needs to be specified - is it a NDMP type of filer or is it a Windows based filer? Compaq points to Windows. In this case, you can install NetWorker Client based solutions (extensions for the NetWorker client software) because you must install the NetWorker client on the filer.

One solution already comes with the product: rawasm .
You may backup the whole partition or the physical disk by running an image backup. This works fine. And it is for free ;-) However, you must of course stop the access to the disk during the backup. And of course you can only run a raw recovery.

Snapimage is the better, more flexible solution. It is creating a snapshot of your filesystem and but runs a fast image backup. Compared to rawasm the major advantage in my eyes is that you can restore single files from the backup.

 

[1018DADA]

All good views, how about making this Huge file system a storage node, that should help some, also snapimage is good alternative, other factors to consider, spit the huge server into two server solution.
Twick the NIC parameters to make it more efficient network throughtput, looks like you are also reaching Max performance on your CPU on the client.
You should look into getting faster raid controller, on the server optimized for read.
Get the fastest drives on the client if you can ( this will require rebuild) and stagger the backups over two days vs starting all save sets at the same time, this can be acomplished by creating partial save sets, different group and scedule with two clients to the same server.
Doing all of the above smaller changes probably wont yeild as much benefit as getting an NDMP complient NAS and using NDMP backups.

 

[jddotman]

I tried the disk backup solution from legato and the results were much the same due to the CPU limitation on the NAS (Compaq B2000 with a DL380 dual proc 1.2 ghz head). I created a 100 GB advanced file system device and the throughput stayed much the same. We're going to migrate to a SAN solution to replace the NAS.

 

[ofladung]

keep in mind that you need a Legato powersnap module to backup SAN snapshots. We have an EVA 3000, and, because there's no module available, we use "Business Copy" with scripts.

Oliver

 

[disldrautoldmc]

jddotman,
keep in mind that the problem stay on milion of small files and not in the performance of Networker.
If your NAS support NDMP backup protocol, is possible that you can increase backup speed.

Bye.