Transactionally Consistent Backups

[Chihiro]

I have seen various posts on the subject, and have read the manual and release notes, but would like to summarize/clarify in a single thread. I'm just looking for general comments/feedback from the public and Veeam Product Managers.

My understanding is that Veeam Backup offers TWO methods of performing transactionally consistent [hot] backups: VMware Tools Quiescence and VSS Integration.

I understand that VMware Tools Quiescence is enabled by default for newly created Veeam Backup jobs. The User Guide states It is recommended to use it unless you are backing up VMs with highly transactional applications... (Exchange and SQL are given as two examples.) However, since using this option introduces the possibility of corrupting the [running] guest machine (if it has highly transactional applications), and with "highly transactional" being quite relative, wouldn't it better/safer to NOT have this option enabled by default and forcing the backup admin to enable it for specific guest machines which were known to only contain "lowly transactional" applications? I don't know about everyone else, but I'd be half-guessing which of my servers are "highly transactional" or not. I do appreciate that VMware Tools Quiescence is simply a "function" of VMware Tools installed on the guest machine (and is therefore a VMware issue), and that transactionally consistent [hot] backups are a main selling point of Veeam Backup...

As for VSS Integration, Veeam PMs on this forum have previously stated ...we recommend to use VSS option rather than VMware tools quiescence for all Windows machines that support VSS. If that's the case, why is Quiescence the one enabled by default? (I know one can only enable one or the other.) Is there any harm in enabling VSS (say when selecting an entire ESX host) for ALL guests (Windows machines that support and don't support VSS, non-Windows machines) when creating a new job?

One final question I have is about VSS in general: Is the VSS service enabled by default on all Windows machines that support VSS? In other words, does one have to specifically "enable" or setup VSS on supported machines?

The bottom line is, I want to perform transactionally consistent backups wherever possible, but without the risk of damaging running, production VMs. What's the best practice?

[Gostev]

As for VSS Integration, Veeam PMs on this forum have previously stated ...we recommend to use VSS option rather than VMware tools quiescence for all Windows machines that support VSS. If that's the case, why is Quiescence the one enabled by default?

It is enabled by default in the UI, but it is NOT used if you use Veeam VSS (our code disables VMware tools quiescence automatically in this case). With UI settings set this way, if you choose not to enable Veeam VSS, VMware tools quiescence will be automatically used instead.

(I know one can only enable one or the other.) Is there any harm in enabling VSS (say when selecting an entire ESX host) for ALL guests (Windows machines that support and don't support VSS, non-Windows machines) when creating a new job?

Yes, you do not want to do this. Your Veeam Backup jobs will throw warning, or error out on Linux hosts (depending on "Continue even if VSS fails" checkbox).

One final question I have is about VSS in general: Is the VSS service enabled by default on all Windows machines that support VSS? In other words, does one have to specifically "enable" or setup VSS on supported machines?

Yes, VSS service works out of box, no need to enable anything on Windows VMs.

The bottom line is, I want to perform transactionally consistent backups wherever possible, but without the risk of damaging running, production VMs. What's the best practice?

For Windows-based VMs, my recommendation is as follows:
1. Using Veeam VSS integration (Backup Consistence step of the Backup Job wizard) is first best choice.
2. Next is to use VMware Tools VSS integration (available in ESX 3.5 U2 or later).
3. Next is to run backup without VMware Tools quiescence. Backup will be crash consistent but often restorable, production VM will not be affected.
4. Worst is to use VMware Tools quiescence. Depending on applications running in your VM, for instance MS SQL, both VM data in backup and production VM may get corrupted.

[tsightler]

2. Next is to use VMware Tolls VSS (available in ESX 3.5 U2 or later).

How do you use this? Does this happen automatically on supported VM's which using ESX 3.5 U2 or later and VMware Tools?

3. Next is to run backup with VMware Tools quiescence (backup will be crash consistent but often restorable, production VM will not be affected).
4. Worst is to use VMware Tools quiescence (depending on applications running in your VM, for instance MS SQL, both VM data in backup and production VM applications may get corrupted).

[/quote]

This is confusing. What's the difference between "run backup with VMware Tools quiescence" and "use VMware Tools quiescence"?

Thanks!

[Chihiro]

Gostev, thanks for the clarification.
Regarding your recommendations for Windows-based VMs, like tsightler, I am slightly confused.

1. Using Veeam VSS integration (Backup Consistence step of the Backup Job wizard) is first best choice.

Are you suggesting that by using Veam VSS implementation (for Windows machines that support VSS), transactionally consistent backups are guaranteed, and there is absolultely no chance of corrupting the running VM and corresponding backup? Even if the VM contains non-VSS-aware transactional applications?

2. Next is to use VMware Tolls VSS integration (available in ESX 3.5 U2 or later).

Are you implying VMware Tools quiescence and VSS should both be disabled in Veeam, and then VMware Tools VSS (available in ESX 3.5 U2 or later) should somehow be individually setup/configured in each VM (running Windows machines that support VSS) to achieve transactionally consistent backups? Are these guaranteed to be transactionally consistent and corruption-free?

3. Next is to run backup with VMware Tools quiescence. Backup will be crash consistent but often restorable, production VM will not be affected.

I thought VMware Tools quiescence was one of the methods for transactionally consistent backups (not crash consistent).

4. Worst is to use VMware Tools quiescence. Depending on applications running in your VM, for instance MS SQL, both VM data in backup and production VM may get corrupted.

Is #4 in reference to #3? (You mention VMware Tools quiescence in both.) If so, the last part of #4 contradicts the last part of #3 ...

[Chihiro]

Oh, and I need to be have some kind of backups until I can run transactionally consistent backups with confidence.

If I perform cold backups (VMs in shutdown state; various Windows and Linux OS's), can I assume all options/risks related to transactionally consistent backups (through VMware Tools quiescence, VSS integration, etc.) are irrelevant? Can I leave either enabled on my jobs and rest assured they won't have any effect on the cold backups?

[Gostev]

How do you use this? Does this happen automatically on supported VM's which using ESX 3.5 U2 or later and VMware Tools?

That is correct. This will happen automatically if you have VSS integration component of VMware Tools installed, and not using Veeam VSS.

This is confusing. What's the difference between "run backup with VMware Tools quiescence" and "use VMware Tools quiescence"?

Just a missing word on my end, I've corrected the original recommendation.

[Gostev]

Are you suggesting that by using Veam VSS implementation (for Windows machines that support VSS), transactionally consistent backups are guaranteed, and there is absolultely no chance of corrupting the running VM and corresponding backup? Even if the VM contains non-VSS-aware transactional applications?

"Absolutely no chance of corrupting the running VM and corresponding backup" - NO. There is always a chance of corruption due to a bug in VMware snapshot code, in Veeam VSS implementation, in Microsoft VSS, or simply cosmic ray hitting your production or backup storage and changing single but critical bit

"Best reliability comparing to all other image-level backup applications" - YES, having tested other VSS integration implementations I can guarantee that.

Of course, VSS quiesce requires VSS-aware applications, and will not work with application that are not VSS-aware. But I am not aware of any Microsoft transactional applications which are not VSS-aware.

Are you implying VMware Tools quiescence and VSS should both be disabled in Veeam, and then VMware Tools VSS (available in ESX 3.5 U2 or later) should somehow be individually setup/configured in each VM (running Windows machines that support VSS) to achieve transactionally consistent backups? Are these guaranteed to be transactionally consistent and corruption-free?

To make use of VMware Tools VSS, you should:
a) Individually setup VMware VSS integration components in VMware Tools in each VM.
b) Do not use Veeam VSS.
c) Make sure VMware Tools quiescence is enabled in Veeam Backup job settings (default).

Since VMware VSS is not developed by Veeam, I cannot guarantee anything about its performance. I can only say that their implementation is not as complete and does not fully follow Microsoft guideliness for some applications (DC, Exchange) as it comes to performing restores.

I thought VMware Tools quiescence was one of the methods for transactionally consistent backups (not crash consistent).

No, regular VMware Tools quiescence (non VSS integrated) is completely unaware of any applications (simple file system flush and freeze).

[Gostev]

If I perform cold backups (VMs in shutdown state; various Windows and Linux OS's), can I assume all options/risks related to transactionally consistent backups (through VMware Tools quiescence, VSS integration, etc.) are irrelevant? Can I leave either enabled on my jobs and rest assured they won't have any effect on the cold backups?

Everything discussed relates to "hot" backups (running VM) only.

You have to keep in mind however, that cold image-level backup is not a panacea either. Thing is, with some applications (virtualized domain controller, for instance) "cold" backup is actually worse than no backup at all (meaning, it will do more bad than good if restored at a later time). This happens because offline backup can only run without VSS, thus the VM is not getting prepared for correct restore by application's VSS writer. If you attempt to restore such DC backup, you will get USN rollback issue. You can google for "USN rollback" to learn more about the issue and consequences, and understand why does it happen.

On the other hand, Veeam VSS guarantees that your hot DC backup will be consistent, and correctly recoverable.

[Chihiro]

Thanks again Gostov. No anti-cosmic ray protection?

I still find it strange that your product defaults to your "worst" recommendation of using VMware Tools Quiescence. When I first tested your software, I left everything at defaults and backed-up all VMs (some w/ SQL) on my test ESX server. Days later, everything seems fine, but I guess something could be broken and I just haven't noticed yet.

To make use of VMware Tools VSS, you should:
a) Individually setup VMware VSS integration components in VMware Tools in each VM.
b) Do not use Veeam VSS.
c) Make sure VMware Tools quiescence is enabled in Veeam Backup job settings (default).

If VMware Tools Quiescence is enabled and used in your recommendation #2 and #4, what exactly is the difference between these two? For #2, in order to use VMware Tools' native VSS implementation, shouldn't VMware Tools Quiescence be disabled in Veeam instead?

No, regular VMware Tools quiescence (non VSS integrated) is completely unaware of any applications (simple file system flush and freeze).

But your product's User Guide states that VMware Tools Quiescence is a method for transactionally consistent backups. Is it or is it not?

Everything discussed relates to "hot" backups (running VM) only.

So... what are the recommended settings for cold (shutdown) backups? Which options should be enabled/disabled and/or does it not matter?

You have to keep in mind however, that cold image-level backup is not a panacea either. Thing is, with some applications (virtualized domain controller, for instance) "cold" backup is actually worse than no backup at all (meaning, it will do more bad than good if restored at a later time). This happens because offline backup can only run without VSS, thus the VM is not getting prepared for correct restore by application's VSS writer. If you attempt to restore such DC backup, you will get USN rollback issue. You can google for "USN rollback" to learn more about the issue and consequences, and understand why does it happen.

Point taken. I will be virtualizing a DC and will be using your product to hot backup via Veeam VSS Integration. However, I also have several other servers for which I do not know if VMware Tools Quiescence and/or VSS Integration would be harmful or less-effective. E.g. Non-VSS-aware transactional non-Microsoft application [even] running on a VSS-supported OS would be better suited for a cold backup if a transactionally consistent backup was desired--INSTEAD of using an alternative transactionally consistent backup like VMware Tools Quiescence that could potentially damage the source. No?

[Gostev]

I still find it strange that your product defaults to your "worst" recommendation of using VMware Tools Quiescence. When I first tested your software, I left everything at defaults and backed-up all VMs (some w/ SQL) on my test ESX server. Days later, everything seems fine, but I guess something could be broken and I just haven't noticed yet.

The reason is that your test servers were lightly loaded. We have been discussing some specifics about bad behaviour of regular VMware Tools quiescence in real-world environments in the following thread: http://www.veeam.com/forums/viewtopic.php?f=2&t=1489 (see last few posts in the thread).

If VMware Tools Quiescence is enabled and used in your recommendation #2 and #4, what exactly is the difference between these two? For #2, in order to use VMware Tools' native VSS implementation, shouldn't VMware Tools Quiescence be disabled in Veeam instead?

These are just 2 modes. When enabled, VMware Tools quiescence uses either VMware-VSS-integrated quiesce (if available/applicable), or regular quiesce (often referred as "SYNC driver") otherwise.

But your product's User Guide states that VMware Tools Quiescence is a method for transactionally consistent backups. Is it or is it not?

If UG talks about VMware-VSS-integrated quiesce mode there, than it is correct. If not, oh well, then you just found a bug.

So... what are the recommended settings for cold (shutdown) backups? Which options should be enabled/disabled and/or does it not matter?

Yes, it does not matter completely.

Point taken. I will be virtualizing a DC and will be using your product to hot backup via Veeam VSS Integration. However, I also have several other servers for which I do not know if VMware Tools Quiescence and/or VSS Integration would be harmful or less-effective. E.g. Non-VSS-aware transactional non-Microsoft application [even] running on a VSS-supported OS would be better suited for a cold backup if a transactionally consistent backup was desired--INSTEAD of using an alternative transactionally consistent backup like VMware Tools Quiescence that could potentially damage the source. No?

Cold backup is usually best/safest backup choice for most applications. For instance, it is considered to be the best way to backup Linux databases or mail servers (no VSS there), or Oracle running on Windows (VSS implementation not so good). You may want to research on specific application in the corresponding community just to make sure it treats such backup fine. Again, most applications love cold backups, but soon as we are talking about application with multi-master replication (AD DC, Exchange PF, DNS and possibly other apps I've never seen before in my life) - it should be big red sign for cold backup, because you are likely to brake a lot of stuff by restoring such cold backup back into the system at a later time.

For hot backups, I recommend using VSS-integrated quiesce in all cases, no matter of which applications you have running in VM. VSS-integrated quiesce is tons better than no quiesce, because whole OS is being propertly quiesced (file system, registry, WMI, COM and many other components and subsystems are covered by it). So ongoing transactions in OS data "pipes" are getting completed and these "pipes" remain "frozen" until snapshot is created, file system and registry caches are flushed to storage, thus the possibility of data loss is reduced greatly comparing to regular crash-consistent backup. Because in latter case you simply loose all cache and RAM content, and besides actual data loss, this may also affect some poorly written 3rd party apps which will not be able to startup correctly once backup is recovered (due to some essential data missing).

Try running the following command in your VM, this lists all VSS writers registered on the system - just gives idea on how comprehensive VSS quiesce is, and how many OS components it properly flushes and freezes:

Code: Select all

vssadmin list writers

Clean Windows 7 install by itself has 10 out of box VSS writers! And after you install things like SQL Express etc. these products will bring and register own VSS writers on the system too...

[clix]

Hi Anton,

Could you please provide a recommendation for Linux-based VMs?

We are backing up our Red Hat VMs using "Enable VMware Tools Quiescence ". Backups have been successful, but after reading this it looks like it's not a good idea.

Please advise.

[Chihiro]

Based on the above, I think I have the "perfect" backup job definition:

1. DISABLE VMware Tools Quiescence (since it's a potentially destructive method of taking transactionally consistent backups).
2. ENABLE VSS Integration (since it's the best method of taking transactionally consistent backups--but only for Windows 2000/XP/2003).
3. Yes, Perform backup even if VSS fails, so that
a) running, VSS-unaware guests will continue to be backed-up crash consistently sans VSS.
b) Cold (shutdown) guests will have a perfectly consistent backup sans VSS.

This job would be used on any/all types of guests, ensuring all VSS-unaware systems (such as Linux and Windows 2000) that require perfectly consistent backups (such as those with SQL or Oracle) are shutdown beforehand. VMware Tools Quiescence would never be used.

What do you think?

[Gostev]

Could you please provide a recommendation for Linux-based VMs? We are backing up our Red Hat VMs using "Enable VMware Tools Quiescence ". Backups have been successful, but after reading this it looks like it's not a good idea.

The general consensus for hot backups of Linux VM is the following:
- If you are NOT running databases, mail servers or other transactional applications in VM - hot backups using "Enable VMware Tools Quiescence " is fine.
- If you are running transactional applications in VM - you should use pre-freeze and post-thaw scripts to stop/suspend and start/resume services before/after snapshot creation, as described here for instance.

[Gostev]

Chihiro, yes I would do this. One thing I would change is this: I would have separate job for applications with multi-master replication, with "Perform backup even if VSS fails" checkbox cleared. I would not want to have a chance of non-VSS backup of these applications created.

Step 1 is not really necessary, remember that if you are using Veeam VSS, this gets disabled automatically.

Another thing to keep in mind is that Veeam VSS will not work if VM is located in DMZ behind firewall. In that case, best is to use VMware VSS integration (to do that, you install VMware Tools with this option on VM, and make sure "VMware Tools Quiescence" is enabled in Veeam Backup job settings).