One server is extremely slow in backup

[rjecne]

Hello,

I have a CommVault GalaxyExpress 6.1 installation that has been working without problems except for a single server. I have a Windows Domain Controller that backs up at a transfer rate of 0.08GB per hour. The other four servers (no DC) back up at between 18 and 60 GB per hour. I have verified connectivity and updates. The DC is also a WSUS server. All are connected to the exact same Cisco switch and all show full connectivity. Not sure where else to look, and the manuals don't offer any troubleshoot or solutions for this. Any assistance would be greatly appreciated.

Thanks,
Robert

 

[baddos]

Is the backup job going to the same destination as the other servers (i.e. magnetic disk, tape library, etc)?

Have you verified there are no ethernet errors on the Cisco switch or the NIC driver?

 

[rjecne]

I have verified that there are no errors anywhere as far as ethernet communications goes. The servers are all set to 100MBps Full Duplex and all are going to the exact same tape drive for backup.

 

[rjecne]

I should also add that I was doing individual server backups using the EZBackup option and selecting the correct retention policy. Four servers backed up without any issues over a weekend, the DC ran for 36 hours and had 2.4GB transferred.

 

[cabraun]

I would recommended makeing sure you have the latest NIC driver on all servers. I know it might seems strange that all of a sudden a NIC will just start acting up, but I have seen it before.

About a year ago, all of a sudden 2 CommVault Oracle Agent backups just out of the blue started running really slow. Backups that just days earlier took 3 hours were suddenly taking a full 24 hours. No changes had been made anywhere on these servers or on the network, it was just totally random.

I rebooted, etc but still very slow. Then I updated the NIC drivers on the clients and the Media Agents and that night, and ever since, we have been back to normal speeds.

 

[rjecne]

Cabraun, thanks for your idea, but I have verified that the driver is the most current. Any other ideas? It is really frustrating that the DC, which is the smallest (data wise) of the group is taking the most time.

 

[birky]

Hi

If you try and manually copy some files from the slow host to another box what is performance like compared with one of the other hosts?

What nic are you using...is it the same as the others?



Birky
CommVault Certified Engineer

 

[605]

I do not know the Express version and how to control streams here. However, in general for Windows 2003 servers you must ensure that you do not create 2 streams from partitions residing on the same physical disk at the same time. This will slow down the transfer rate to less than 10% of a single stream. And i am pretty sure that 6.1 does not support 2008 yet.

You don't believe me ? - You better run a short test:
- Copy one large file from any disk and calculate the transfer speed. For example, use "copy file > nul".
- Now do the same for another drive residing on the same physical disk.
- Finally, from two CMD windows, initiate each copy process at the same time ... and watch the difference!

BTW - it must be a PHYSICAL disk attached to the system - the effect will not show with a virtual system.

 

[rjecne]

Hello,

I have tried copying a large group of files in a series of folders, some large single files and some smaller ones. They copy around to the different servers at about the same speed, so performance is equivalent. They are all Dell Servers with Intel Pro 1000 ethernet adapters.

605 - Express is a single stream solution. When the jobs are created, it queues them up and runs a single server with a single stream at a time.

Thanks,
Robert

 

[getndz]

May be an antivirus programm, i.e trendmicro.
Stop the services during the backup jobs

 

[kitelife]

We get similar behaviour with port speed mismatches.

check the port speed is the same as all the other ports on your switch.

if so check that the port speed on the server is the same as the other servers