Backing up VMs... lots of them...

[Chapter11]

Good afternoon. I am about at my wit's end to figure out how to do this.

Does anyone have a *good* way of backing up 200+ VMs across 13 HA ESX hosts, backed with 16TB of SAN space?

All of the hosts are managed by a VirtualCenter Server.

---

Now, I have done ESX-level backups on some of our other ESX servers, that are not SAN-attached (all DASD), and don't have VMs sliding around all over the place with VMotion auto-load-leveling. These work fine, and I'm happy with them. The short version on these, is that a snapshot is taken (vcbSnapshot), a backup is taken directly into our backup software (Networker 7.4.4), then the snapshot is released.

This does not work in our main HA environment, on account of a VM being able to float amongst hosts.

If I snapshot a given VM, the vmdk files are still accessible only to the host that is actually running the VM.

---

I've read enough about VCB to not be comfortable with its ability to scale.

For image-style backups, it seems to want to copy the entire vmdk directory over to the VCB proxy, then my backup software backups up that proxy.

I'm not thrilled about that for a couple of reasons:
1. It requires yet more disk space, 1-to-1. As this is several terabytes, this is not so easy. (and before anyone pops off about how cheap disk space is, imagine it's your $40k that you have to spend and can't bill back to anyone).
2. Designing the backup method as a two-stage backup also requires that recovery be two-stage. Both of these processes being two-stage is undesirable complexity, and makes DR that much more difficult.

This methodology does not strike me as scalable to and above my current environment.

The vmdk-mount-into-Windows method, while inappropriate for some of my VMs, also seems to suffer from lack of scaling. I haven't dug into this one in as much detail, but it seems to require a drive letter for every mounted vmdk? There aren't enough letters to go around - I would have to have to have 2-3 VCB servers per ESX.

---

I would like to minimize the number of transits across whatever networks (san or lan) are involved - at this scale, I don't believe I can afford to shuffle data multiple times between its living space and backup media.

---

Yes I am aware of Avamar and other de-dupe technology. Those still do not resolve the "VM can be anywhere in this HA" problem for ESX-level backups.

 

[mrdenny]

I assume that your VMs are spread across many LUNs? Could you clone a LUN, back it up, then destroy the clone, clone the next LUN, back it up, etc until all the LUNs are backed up?

Denny
MVP
MCSA (2003) / MCDBA (SQL 2000)
MCTS (SQL 2005 / SQL 2005 BI / SQL 2008 DBA / SQL 2008 DBD / SQL 2008 BI / MWSS 3.0: Configuration / MOSS 2007: Configuration)
MCITP (SQL 2005 DBA / SQL 2008 DBA / SQL 2005 DBD / SQL 2008 DBD / SQL 2005 BI / SQL 2008 BI)

My Blog

 

[ArizonaGeek]

We use Vizioncore vRanger to snapshot our VMs and store them on a Dell PE1950 with a 6tb MD1000 SAS array. We then use Backup Exec to pick that up to tape for off site storage. I highly recommend the software.

http://www.vizioncore.com/products/vRangerPro/

Cheers
Rob

The answer is always "PEBKAC!

 

[needcoffee]

Chapter11,
You could go traditional putting agents in each guest or use a VCB solution. VCB is usually more cost effective froma licesing standpoint.

You could use LUNS formatted as NTFS and mapped to your VCB proxy to use as mount points for the VCB jobs. There are IO considerations but you only need enough space in a LUN to hold 7-8 snapshots at a time (plus a little padding). You do not require a Windows driver letter for each vmdk. Read the VMware backup guide to see how the mount point works and the differences between file level and full image VCB options.

Know limitations of VCB Proxy 1.5 in a SAN environment using SAN mode backup. Not sure if these apply in a vSphere 4.0 environment.

1. You can only run 1 VCB snapshot against 1 VM guest in a LUN at a time and mount it to the proxy. You have to unmount the VCB snapshot before proceeding to the next guest in the LUN. If you have multiple VM guests' config and VMDK files in the same LUN this is an important limitation. You WILL receive SCSI reservation issues if you ignore this and your jobs will fail intermittently.

2. A Windows VCB proxy can usually handle 4-8 mounted snapshots at a time with consideration to rule 1.

3. You can have multiple Windows VCB Proxies if you need to scale. Not always cost effective.

4. High volume SQL and Exchange guests will have issues even with the VSS driver from VMware.

Regards,
needcoffee

 

[Chapter11]

If I read #1 right, then I can only VCB-snapshot 1 VMDK out of an entire LUN at a time. Correct?

 

[needcoffee]

Let me clarify my point.

LUN 1 & LUN 2 contains the VMDK files for Guest 1 and 2.

LUN 3 & LUN 4 contains the VMDK files for Guest 3 and 4.

You can run VCB snapshots for Guest 1 & Guest 3, or Guest 1 & Guest 4, or Guest 2 & 3, or Guest 2 & Guest 4 at the same time. The trick is not having snapshots for guests using the same LUNS at the same time ie. Unmount Guest 1 before snapping Guest 2 since they use the same LUNS.

needcoffee

 

[NickFerrar]

Good luck with it - we use VCB (via NetBackup v6.5.3.1) to backup up about 4TB across two sites and get a very high failure rate. Unfortunately failures with VCB usually generate a generic snapshot 156 error but can be for a variety of reasons so can take a lot of time sorting out (Monday's I lose about half a day of a team member whilst the weekly full backup failures are investigated and massaged through). Sometimes it works smoothly for a couple of weeks but usually it's a complete nightmare.
Without throwing bags of money at the problem and putting in tiered storage I can't really see an easy solution either, VCB just doesn't scale to enterprise level environments.