System State Failed to Create VSS Snapshot 1

[mitmont]

Hello,

We've got a new install of CommVault 8.0 about 85 servers to backup with Exchange 2007, SQL, SPS, DC's, normal file and application servers on 2003SP2 and 2008SP2 OS's. We're doing a Disk to Disk to Tape configuration with 10TB disk and an ML6000 Tape drive. On the 2008 servers we get a

Error Code: [17:55] Description: Failed to create a VSS snapshot and it is required in order to run a system state backup. Failing the scan.
At the same time we get an message in the event log "Cannot scan [\].


We have the default subclient with VSS selected and System State Backup checked. This is only occuring on the 2008 servers and for SQL, SPS and Exchange servers. In the EvMgr logs on the client server we get the dropped rmask errors. Anyone solved this?

 

[mitmont]

Additional information. The errors are only applying to the System State. For all of the default subclient I've unchecked the System State and the jobs run on every client server. Some of the forums and research indicate that a possible contributing factor is the drive configuration concerning the drives that are marked active in teh Disk Manager. After verifying this every server has the D:OS drive marked as active however the OS is actually on the C: drive. One server had the small 78MB EISA drive marked as the active drive.
The servers all have an additional agent installed ie... SPS, SQL, Exchange. I have a 2008 x32 with just the file agent; runs fine, I have a 2008 x64 with one agent; runs fine.
So the issue is really in the System State. So has anyone run into the issue and solved it and if so what steps did ou take.

Thanks in advance.
Mitmont

 

[Psy053]

I recommend creating an additonal Subclient, and then separating the System State Backup from the Default Subclient.

That way you can run the System State backup independantly of the default content, which will make the logs files much easier to go through.

If you are able to do this, and then post the logged error details, it should make it easier to diagnose.

 

[mitmont]

Thanks Psy053,

Did that. The default subclient works great without the system state check box BUT... The system state subclient fails every time on every client that I had the original problem. The issue then is no System State, no good recovery plan except complete rebuild. Not a great a solution.

 

[baddos]

The windows hosts that this happens on, are they all the same OS or a mix of 2003/2008 boxes? System State must be working on some of them.

 

[theravager]

Make sure you have enough space availble on the server to use vss for this.

Also you should check that the vss writers are not active and delete any existing shadow copies, this causes vss to fail.

 

[Psy053]

The reason I suggested the separation of the System State from Default Subclient is so that when you run the System State backup, you will be able to view the logs files once it errors out. There will usually be something within the logs that will help you pin point the cause of the problems.

Also, I would recommend that you have a look at the Event Viewer Logs on the problem Servers to see if there are any VSS related errors there.

If you have no joy after those steps, you could probably post up your logs and I can take a look at them.

 

[mitmont]

I'll check the backup logs for the System State. For the OS's they're all 2008 Enterprise X64. We're looking at the original disk configuration as the issue. The servers are all Dell PE with M600 blades and 6850s. The drives came configured with 78MB EISA, D:OS 2GB and the remaining 60+GB to 300GB depending on the drives. The OS is on the C: drive but not marked active. The common similiarity is that the active partition is either the D: 2GB drive or on one server the 78MB partition. Commvault pointed us to a link

http://social.technet.microsoft.com...l/thread/dbab7d2a-0a4e-48a6-850b-da9f31e3e371

for a solution.

We took one of our production/nonproduction servers to attempt to move the Active disk to the OS. !!! Do not do the step unless you want to have more work to move it back!! After the change the server on boot up would not boot due to bootmgr files missing ie, the ones on the last active drive.

Since this is a new install Commvault and production servers stay active for 3+ years we'll rebuild OSs for the HyperV hosts and determine the next course for SQL, Exchange and SPS.

Our work today is to reload the OS in a configuration that marks the C: OS drive as active and then test the commvault. I'll post back the results.

 

[mitmont]

OK, Anyone following this thread. The issue comes to the disk configuration. Here's what I found out. It is the disk configuration. We rebuilt that server with a straight 2008 x64 OS install. 78MB EISA and used the remaining 67GB unallocated space for the OS. Completed the install with no errors. Ran the first backup. No problems. The C: drive is the Page, System, Active Boot and Primary Partition.

My second server test, Figure how to do it without reinstalling the OS. There is a way to do it. I'll post the exact procedures. In essence you have to drop the server and run the 2008 recovery environment. The two configurations I dealt with were with the Active partitions on the 2GB drive and the EISA drive. Both servers are fully functional after the action with the C: drive as the Active partition. I'll monitor for stability for the next few days before I put them into production and I'll post the corrective actions.

Our servers are Dell PE M600 Blades. The Disk Manager shows the active partion. If you're using CommVault you'll see the VSS Snapshot error on File system backups. Narrow the backup and you'll find it's the system state portion. Eliminate the System State and the job runs and completes. The actions I took corrected the system state VSS Snapshot error for 2008 x64, SQL, SPS, Hyper V guest servers. All of our 2003 server have no problems with any backups.

Mitmont

 

[mitmont]

Before you do this have as good a backup as you can, for virtual servers export and copy it off to another server for protection. Disclaimer: This worked for me on 4 servers. I have 8 more to go. This may or may not work for your situation but it points you in the direction for the fix. Alternate solution. Rebuild the server if you want a good System State backup.

For everyone, the drive configuration can be changed and these are the steps I took to make the changes.
The server I was working on was a 2008x64 Hyper-V host with 2 virtuals, Dell M600 Blades originally configured from Dell. I did this on 3 other servers that were hosts without any virtuals populated yet and the same result. I got the VSS Snapshot to backup the system state without a problem and discover the Virtual servers on the initial setup of the default client.

Verify the Active and the Boot partitions (drives)
Perform the 2008 Recovery Environment from the Win2008 disk.
Click Repair on the second screen. Click the operating system and then click on the Command Prompt on the next
screen.
This section is for the drive with a drive letter that you can get to.
Once you're here
1. At the Command Prompt verify the directory contents: (change the drive letters based on your server)
2. Type cd\
3. C:\Dir /a
4. D:
5. D: \Dir /a
6. Verify the Boot directory and Bootmgr file location.
7. On the drive with the Boot Directory type
8. Xcopy /h bootmgr d:
9. Robocopy c:\boot d:\boot /mir
(You want the Bootmgr and Boot folder on the drive with the OS)
10. Restart the server
11. After startup go to Server Manager/Disk Manager
12. Change the Active Drive to C: (where the Operating system is
13. There should be no request to restart the server at this change.
14. Reboot
15. Go to Server Manager/Disk Manager and delete the d:\OS (the drive without the Operating System
16. Reboot.
17. You should receive an error concerning the \boot\BCD missing
18. Reboot into the Recovery Console as before.
19. At the screen for selecting the operating system you may not have an identified partition to select.
20. Press Next
21. Select Command Prompt
22. Type “bootrec /RebuildBcd. This will identify the volume where the OS is C: and ask if you want to add it to the configuration.
23. Select Yes
24. You’ll get a confirmation.
25. Reboot.
26. The server should be restored to a working manner.

For the Active and boot on the EISA drive. One added step I did was to have a copy of the Bootmgr file from another 2008 server.

When you get to the Command prompt as before.
1. At the Command Prompt verify the dir contents:
2. Type cd\
3. C:\Dir /a
4. D:
5. D: \Dir /a
6. Type Diskpart
7. Type Select Disk 0, Enter
8. Type List Partition
9. Find the Partition with the 2008 Operating System
10. Type Select Partition # (corresponding to the result of item 9)
11. Type Active
12. Exit
13. Verify the drive C and D for the 2008 OS
14. Change to the C or D Drive with the 2008 OS
15. Type Bootrec \Fixboot, Enter
16. Type Bootrec \RebuildBcd
17. Connect Floppy Drive and copy the Bootmgr hidden file by typing
18. Change directory to the floppy A: or B:
19. Type xcopy /h bootmgr C: or D: (whichever the 2008 OS is on)
20. Reboot
21. Should be good to go.

This worked well for me. One issue from the Hyper-V virtual server. On restarting the virtuals I got an error
:An error occurred while attempting to change the state of the virtual machine TMember. The Virtual machine could not be started because hypervisor is not running.

The solution was to Log into the server as the local administrator, at the command prompt type
bcdedit /set hypervisorlaunchtype auto

Reboot the server and restart the virtual server.

This entire process took about 20-30 minutes for server #3 and 4 after everything was worked out. Pretty much it. These procedures will probably help in some other aspects of drive configuration and troubleshooting. Hope this helps.