Extremely slow NDMP restores (NetWorker 7.2 + NetApp)

[angelshark]

Hello,

We were backing up our NetApp filers by UNC about 6 months ago but in order to optimize performance we decided to implement NDMP backups using NetWorker 7.2.

We have had a very important performance increase for the backups but the restores were much slower than before. Most restores would take about 1 to 3 hours even for small single files which is slow but were still acceptable for most of our operations.

Nonetheless we've been experiencing very slow NDMP restores since about 1 month. A simple 200MB restore can take up to 26 hours. This data belongs to a NetApp volume of 420GB and the size of this volume has barely changed during the last month.

The configuration of our NetWorker server/NetApp servers apparently hasn't changed either.

Has any of you seen this problem? Any help is appreciated.

Thank you in advance.

 

[mybackupfailed]

I've seen the same thing using Commvault. They claim it is a Netapp bug (#132492-not public viewable) that I pasted below. I've tried turning DAR off and it doesn't seem to help much. NDMP restores have a hard time picking and choosing single files out of a large NDMP backup. I've had better luck restoring the entire directory to the server that hosts the media locally. Or splitting the contents up for backup so the restore doesn't have to parse through the entire archive.

%%% TITLE: NDMP Data server requests data from offsets that have already passed.
%%% DESCRIPTION:
In some specific interactions with the Attempo backup application during a DAR
recover, the Netapp NDMP Data Server requests byte offsets that have already passed.
This results in generating error messages of the form:
"RESTORE: Could not find <file-name> on tape at specified location : no header."
in NDMP Log messages sent to the backup application by the Netapp NDMP Data Server.
Also, in such cases the file mentioned in the log message will not get recovered.
%%% WORKAROUND:
The only workaround is to not use DAR for recovering files.

 

[angelshark]

I asked NetApp and they told me we don't have that bug with our backups.

Sometimes when I start an NDMP restore, NetWorker loads the required tape but then it stays on "Ready for reading, idle" for a long time before starting to actually read the tape. I wondering why does it do that.

Do you have any ideas?

Thank you.

 

[mybackupfailed]

Are you backing up to tape? Does your Netapp have a direct connection to the tape library? I am running 3-way backups with the tape library connected to a different media server. Commvault recommended using a tape library directly attached to the filer for faster restores which is pretty worthless advice now since they sold use the 3-way backup solution in this configuration. But if you have this setup this may also be slowing down your restores (filer sends tape offset read commands to the media server and the entire process bogs down).

 

[angelshark]

I am backing up to SDLT320 tapes. Our NetWorker serveur is SCSI attached to a StorageTek L700 library. Our NetApp servers are gigabit ethernet attached to the same switch on which the networker server is attached.

What can we do to significantly improve our NDMP restore performance?

Thanks.

 

[mybackupfailed]

Restore data to the NetWorker server attached to the StorageTek (if NetWorker supports it) and manually move the restored files to the NetApp.

 

[angelshark]

You can't restore NetApp data to a different platform.

 

[mybackupfailed]

Guess this is a Commvault-only feature. They license it separately and sell it probably for this scenario.

 

[angelshark]

I have disabled the DAR option by removing the DIRECT=y application information and now the restores are faster even though all the backup volume is read from tape. NetApp tech support says that DAR wasn't designed for tape backups but for disk backups.