"Shadow copies deleted" during BAB backup job

[KaJuHe]

I have BAB V9 for Windows with the latest patches running on a W2K server, and when i try to backup a remote Windows 2003 server, let's call it XXX in this post, every now
and then (occurs randomly about once a month at least) i experience a problem and a drive (F won't be backed up (Approximately 0 file(s) 0 KB will be processed.) as expected as follows:

This is when everything goes OK ie.in over 90% of the cases:

Information 2005/05/14 19:09:19 Connected with client agent at \\XXX. (User = xx.local\admin)
Information 2005/05/14 19:09:19 BrightStor AB Client Agent for Windows is version 9.0, build 2020...
Information 2005/05/14 19:09:28 Backup C: to 5/14/05 7:02 PM sequence 1 session 4
...
Information 2005/05/14 21:06:59 Source Directory: F:
Information 2005/05/14 21:06:59 Backup Session 6 on Media 5/14/05 7:02 PM Serial # 000046L2
Information 2005/05/14 21:11:10 Approximately 529,233 file(s) 123 261,20 MB will be processed.
Information 2005/05/14 23:41:43 529,233 file(s) 124 952,79 MB sent by agent @ 807,53 MB/min
Information 2005/05/14 23:42:14 ** Summary for \\XXX **


This is when the problem occurs, ie.in about 10% of the cases:
Information 2005/05/16 19:09:35 Connected with client agent at \\XXX. (User = xx.local\admin)
Information 2005/05/16 19:09:35 BrightStor AB Client Agent for Windows is version 9.0, build 2020...
Information 2005/05/16 19:09:50 Backup C: to 5/16/05 7:02 PM sequence 1 session 4
...
Information 2005/05/16 21:37:08 Source Directory: F:
Information 2005/05/16 21:37:08 Backup Session 6 on Media 5/16/05 7:02 PM Serial # 000058L2
Information 2005/05/16 21:37:08 Approximately 0 file(s) 0 KB will be processed.
Information 2005/05/16 21:37:09 0 file(s) 4 KB sent by agent @ 152 KB/min
Information 2005/05/16 21:37:46 ** Summary for \\XXX **

XXX EventViewer log:
Error 2005/05/16 21:17:51 VolSnap
The Shadow copies of volume F: were deleted because the shadow copy storage could not grow in time.Consider reducing the IO load on the system or choose a shadow copy storage volume that is not being shadow copied.

NOTE: another interesting this that there is actually no errors in the log, only INFORMATION about the "0 file(s)" issue so this means i have to browse the logs manually basicly daily to confirm that everything went ok with the backup.I have tried to reschedule the job using different time to backup but have not noticed any difference, backup is also the only job running on this server at that time so i see no real way as how to "reduce the IO load on the system"

I can see what the EventViewer log says but have no real idea about the correct way of solving this,
any ideas as how this problem could be solved in the correct way?

Kind regards KH

 

[davidmichel]

run ck disk
defrag
set virus scan program to scan incoming and not outgoing files

 

[KaJuHe]

Thanks for the quick response.
Virus scan program has always scanned only incoming files on this server.Defragment analysis gives a fragmentation rate of total 8% and 17% for files and the analysis suggests "You do not need to defragment this volume" however i've now run the defragment anyway.Checkdisk can only be run rarely effectively on this 24x7 server since in use; the server+disks are about 1 year old and from a "major/reliable" make so i'm quite confident with the hardware.

I am confident that the solution was found today from the following link: (needs yet to be tested,should work)

http://support.microsoft.com/?id=826936

A supported fix is available from Microsoft both as hotfix and included in the sp1 also:

"833167 A Volume Shadow Copy Service (VSS) update package is available for Windows Server 2003

You may also experience a problem in the Volsnap.sys driver that causes shadow copy deletion when there is a high level of input/output, especially when the disk write cache is disabled (for example, on a domain controller computer).

By default, the shadow copy provider that is included in Microsoft Windows Server 2003 is used to create shadow copies for backup purposes. The shadow copies are implemented by using a copy-on-write implementation. The differences are also copied in the shadow copy storage area. In Volume Shadow Copy service, you can use the shadow copy management tool, or you can use the vssadmin command.

In the original released version of Windows Server 2003, the initial shadow copy storage area that is allocated for shadow copy creation is 100 megabytes (MB). However, the actual space that is used is much smaller. As time passes, the allocated space may increase as more data is modified on the original volume. However, when you have high input/output traffic on the original volume, the shadow copy storage area cannot grow fast enough to hold all the copy-on-write changes. This causes deletion of all the shadow copies on the original volume. This problem is more noticeable in domain controller configurations. By default, the disk write cache is disabled in domain controller configurations.

This hotfix makes some classes of writer (SQL, Active Directory, and others) less likely to experience time-out errors. In part, this hotfix is designed to improve the performance of the default shadow copy driver (Volsnap.sys) when there are high input/output levels. The hotfix also includes a larger initial size for the shadow copy storage area: 300 MB instead of 100 MB. Additionally, if a 300 MB initial size is not sufficient, and if shadow copies are still lost because of high input/output levels, this hotfix includes a way for you to configure the initial size of the shadow copy storage area when the shadow copy storage area is created. You can configure the initial size of the shadow copy storage area by using the configurable registry key setting that this hotfix provides.

 

[davidmichel]

good find