Losing Shadow Copies 1

[gregmuir2]

I'm just getting started with Shadow Copies on our servers. For once, it appears Microsoft might have actually made an improvement. Then again, I'm posting about a problem I'm having with it so maybe they haven't gotten it right just yet.

I have two servers here, call 'em A and B. One is running Windows Server 2003 and the other is Server 2003 Data Center edition. Server A has no problems with the shadow copies. I have twice-daily backups going back to last week. Yay! Server B, the Data Center edition one, loses the backups each night. I can trigger multiple backups during the day for testing, 5 or 6 of them. When I come in the next day, the only backup I see is the default one for 7am. I at first thought this was a Veritas problem since I'm using Veritas BackupExec on these servers. However, Server A does not experience this problem. I spoke with a rep from Veritas and they promised the ONLY time data gets deleted is when you tell it to do an archive and delete job. Your regular full and diff jobs will not delete the data.

So, I'm guessing this is a straight-up Windows problem. I have this running on a very fat array so I have 100gb set aside for Shadow Copy. If every instance of your oldest backup gets overwritten, does it disappear from the backup overview? Is there any other setting that would delete backups? I know that defrag can corrupt your shadow copies but that isn't something you would see from the console. Help! Thanks.

 

[porkchopexpress]

Do you get any errors in the event log?

Take a look at this KB article under the "Shadow copies are lost during backup and during times when there are high levels of input/output" section.

http://support.microsoft.com/default.aspx?scid=kb;en-us;826936

When you are the IT director, it's your job to make sure the IT works. If it does work they know already and if it doesn't, they don't want to hear your pathetic excuses.

 

[porkchopexpress]

I've had this a couple of time but only when the server was very busy.





When you are the IT director, it's your job to make sure the IT works. If it does work they know already and if it doesn't, they don't want to hear your pathetic excuses.

 

[gregmuir2]

Hmm. Reading those links right now. Here's the way we have Server B setup, the one with the problem.

It is an HP DL380 Storage Server with an 838gb SCSI RAID-5 array. This serves as a file repository for several different important directories. The "Public Drive" is the one everybody in the company knows about and is currenty 125gb. In addition, Veritas uses a folder on there for Backup-to-Disk operations late at night. In addition, we have a robotic tape library that gets a full backup of the Public folder once a week with differentials nightly. It also gets a copy of the system backups of all our other servers that are performed b2d.

This system was setup for us by "someone who knows what he's doing." I don't know enough to say otherwise but it seems to me that a good backup-to-disk system would be with SATA drives in a RAID-5. The SCSI drives seem like they're great for high-availability stuff that people are working on over the day but the SATA's should be fine for the long, tedious backup jobs. Cheaper drives, cheaper computer, etc. We've got an old server here we don't know what to do with, too slow to keep in the rack but it should fit the bill nicely. Already has a built-in SATA raid setup, I think I can have four drives spinning on that one. There's also room and a controller for a cheapie standard ATA hard drive to use for the OS.

Anyway, to get back to the point I was going to make, the Veritas backups fragment the disk something fierce. I can have the drive completely defragmented, run a set of backups and every single backup file is fragmented to hell and back. In addition, I see that Shadow Copy likes 16kb clusters and this drive is formatted for 4kb clusters. That sounds like it would certainly be an issue the next time I tried defragging.

I ran the vssadmin list writers as suggested and there were no errors shown. I guess I'll have to try running it tomorrow morning if the backups disappear and see what they show. From what the KB article says, it sounds like shadow copies can be lost during the high read-write times even if it's not trying to run a shadow copy during that period.

 

[gregmuir2]

Aha!!! I found the error in the event logs. The shadow copies definately were deleted because the shadow copy storage could not grow in time. I would imagine the heavy load here was caused by the b2d jobs running in the background. This occurred at 11:46pm, right in the middle of when the backups are running. WTF? Insane that ALL shadow copies get dropped because of this.

 

[porkchopexpress]

Thats the error i was looking for and yes it does indicate that excessive activity is causing it, any possibility you could stop one of those jobs running at the weekend to test or is it in constant use.

We used to get it on our DC's if the virus scan and backup ran into each other.





When you are the IT director, it's your job to make sure the IT works. If it does work they know already and if it doesn't, they don't want to hear your pathetic excuses.