Synchronized VM's backup/replication

[ccoope@cmers.com]

We have a very sensitive custom application that needs a set of VM's to snapshot within a relatively tight timeframe. Does anyone have any ideas on how to guarantee that the set of VM's could backup/replicate essentially simultaneously. Each VM runs in it's own datastore on EqualLogic SAN storage. From our observations, backup jobs run every step in serial, and only submit the snapshot requests to VMWare when they actually are working on each individual VM in a Job, and not all VM's at the beginning of the scheduled task.

[tsightler]

As far as I know VMware doesn't really provide a way to do this and Veeam doesn't have a method either. How critical is the timing required for your case? Is it a matter of minutes or seconds?

[ccoope@cmers.com]

It would likely be seconds, as they would like to stay transactionally consistent.

I wonder if there is a way to script the activity.

[Gostev]

No, this is not possible. Hard to achieve even in theory, because even if you issue snapshot creation for multiple VMs within a few seconds (cannot really do it instantly, as this requires SOAP communication to vCenter) - this is still only "request" to create snapshot, and nothing more. Actual snapshots may be created at different times (for some VMs it will take faster, for other VMs slower). More importantly though, if you use VSS integration to achieve transactionally consistent backups (which you really should), this will split VMs in time even more. For some VMs VSS freeze will take seconds, for other - minutes.

What is the application, you need this for? I assume it is some custom application? You should not really need anything like this for typical apps... I only recall seeing similar request once before in 3 years, and that other one was for some home-built application.

[tsightler]

It would likely be seconds, as they would like to stay transactionally consistent.
I wonder if there is a way to script the activity.

I don't know any way you could achieve this with VMware snapshots. That being said, you could use the Equallogic snapshot technology to achieve this and that's probably what you'll need to do.

[Beevoir]

The only possible issue (And a browse of the Equallogic information didn't give many clues) was the consistency aspect of the VSS snaps, and I believe that you would still have the same problems with the VSS snap time issues on an individual machine basis.

When you shut the machine down for routine maintenance, do you have a specified order to do this in to preserve consistency? Database hot suspend etc?

[Gostev]

Good question...

[tsightler]

It's possible, I don't know anything about the users application, however, Equallogic offers multiple ways to achieve consistent snapshots with their ASM/VE tools. As far as I know ASM/VE doesn't use VSS, but it's typically not a requirement since a snapshot will be "crash consistent" anyway. If your applicaiton can recover from a "system power off" then it can recover from a non-VSS snapshot. I've got several cases where frontend and backend services need to be carefully snapped together in reasonably close time, including replicating multiple volumes "in sync", and Equallogic tools provide ways to accomplish this fairly trivially. We've done this for over a decade starting with NetApp based storage and snapshots back in 1999, and that was before technologies like VSS even existed. We created scripts that would mount the snapshots and replicated volumes and run and recovery procedures, disk checks, etc. and then integrity tests on the applications to verify their state so that we'd know our snapshots were good and recoverable, similar to the technology Veeam provides with Surebackup.

[Gostev]

Tom is right. There are some important gotchas to keep in mind though, even with most typically virtualized applications. For example, Active Directory domain controller will never recover properly from non-VSS snapshot. In fact, attempting such recovery will brake DC so bad you would have to demote it (due to USN rollback). Or take Exchange server, its boot up after restore will take about 15-20 minutes longer if no VSS was used during backup (did not test myself on decent Exchange, but got these numbers from VMware preso/testing). All of this on top of regular danger of crash-consistency (that very little chance of something braking due to improper shutdown). So I would definitely recommend performing constant recovery testing similar to what Tom had just explained. Thanks.

[tsightler]

For example, Active Directory domain controller will never recover properly from non-VSS snapshot. In fact, attempting such recovery will brake DC so bad you would have to demote it (due to USN rollback)

This isn't 100% accurate. It's actually fairly trivial to recover AD from a non-VSS snapshot, but yes, you do have to have some basic knowledge of AD administration and recovery, and I agree you can get yourself into quite a mess if you don't understand how AD replication works and the steps to perform a manual AD restore. Also, if you happen to only have a single domain controller, for example some smaller businesses, then there's no possibility of USN rollback so crash consistent isn't much different than a VSS based backup.

As far as Exchange, I've never experienced the issue you refer to of the system taking 15-20 minutes longer without VSS, but to be fair, I'm not sure there would ever be a reason to do this. Once again, Equallogic offers the solution of using ASM/ME (Microsoft Edition) if you require VSS level protection of a Windows VM and I'm not suggesting out-of-hand that VSS is not valuable, only that, based on the application, you might not need it.

[ccoope@cmers.com]

Yeah, we have ASM/VE in place and in the queue to be further tested, which does seem to create fairly consistent near-simultaneous snaps in good timeframes. It doesn't directly use VSS, I don't believe, but it does grab the state information and everything else from the VM's using VMWare's API's (making for a lot of extra data in the snapshots), and is supposed to be crash consistent. This whole issue is for a Websphere/Filenet based environment, with some MS SQL and other goodies all frankensteined together, and it's all very sensitive to the transaction consistency between the 3 machines. EqualLogic can snapshot and replicate around I believe 8 volumes simultaneously, which is what we need for this to work (the various drives are broken out into separate volumes/datastores). The main issue we are having outside of the timing has to do with the shear number of manual steps required to recover from our snapshots, and the extreme space inefficiency of the EqualLogic SmartCopy/SmartReplication snapshots. It's why we were looking at Veeam in the first place, rather than having to buy another array just to have enough space for the snapshots required in the SLA's. We will likely use a combination of the two, as there are quite a few other VM's that Veeam works perfectly fine for outside of this application such as DC's, Terminal Servers, Fileservers, Exchange, vCenter, etc.

[tsightler]

The main issue we are having outside of the timing has to do with the shear number of manual steps required to recover from our snapshots, and the extreme space inefficiency of the EqualLogic SmartCopy/SmartReplication snapshots.

Well, the manual steps could probably be automated with some scripting, assuming enough effort, but I understand the issue with the inefficiency of EQL snapshots as I've been harping on that since my earliest days of EQL ownership. What funny is that just last week I was on a call with a Dell Equallogic engineer and made the comment that we barely use EQL snapshots because they are so inefficient and his response was "I've never heard anyone call EQL snapshots "inefficient" before!!". I asked him if he ever actually used EQL storage, talked to any EQL customers, or participated in any EQL community forums because I think that may very well be the #1 complaint about the arrays.

It would be nice if VMware could implement some type of "consistency group" type of snapshot in the future as that really is a required feature, however, if I could ask for any feature at all, it would be a CDP style recovery of a VM and you could likely leverage this feature to provide exact point-in-time recovery of multiple VM's.

For now we just have to work with what we have.

[Gostev]

Would small service downtime be acceptable for "consistency group" type of snapshot?

[tsightler]

Probably not in our environment, unless it was very, very short, no more than current "stun" time with regular VMware snapshots. Even that can be a pain sometimes.

[edubya]

I need to be able to back up (or at least snap) two VMs at the same time for consistency. We use an application that archives files once placed into our file server. The database VM needs to be snapped at the same time as the file server to avoid the chance to server mismatch on a restore job.

I have tried simply adding both servers to the same job, but that processes one after the other. Is there a way to tie two VMs together so they snap and backup simultaneously?

[dellock6]

No, inside a job VMs are processed sequentially.
I had to do the same thing, and best way is to create two jobs, place only one VM in each job, put in place two proxies, and schedule both jobs to start at the same time. Each proxy will run a job at that time, so snapshots will be really near in time.

Luca.