VSS and / or VMware Tools Quiescence before backup

[PeterCox]

Do Veeam have any recommendations regarding the use of VSS and / or VMware Tools Quiescence in the guest before Veeam backup?

There have been significant issues with both functions in the past depending on specific version in use.

Are there any Veeam recommendations on the setup of VSS within various versions of MS Windows Server when using Veeam backups of VMs?

Are the calls made by Veeam to VSS both before and after backup?

Is the VSS call to the Windows guest for all Windows drives for all VSS writers?

Taking an Exchange Server as an example - do Veeam call VSS before the backup starts and request and Exchange freeze, then call VSS at the end of the backup triggering an Exchange log truncation and un-freeze (thaw are MS calls it)?

Thanks
Peter

[Vitaliy S.]

Hello Peter,

There are a number of existing threads on this forum which discusses VMware VSS vs. Veeam VSS vs. VMware quiescence, you may want to read them for more information on how these compare to each other. But definitely we recommend to use Veeam VSS whenever possible.

Please have a look at the links below for more information:

http://www.veeam.com/forums/viewtopic.p ... ols#p14018

http://www.veeam.com/forums/viewtopic.p ... 24&start=0

http://www.veeam.com/go/vmware-vss-backup-wp

Also could you please clarify what issues you've met while using VSS integration option.

Hope it helps!

[PeterCox]

Thanks for the references, I will research these articles. As a topic I find VSS is generally very poorly understoond with a lot of people thinking it is just Previous Versions.

The issues we have seen relate to the use of VSS or VMware Tools with systems other than Veeam. To date we have not tested the VSS integration option in Veeam backup and I was looking to see what others have found when using the function with Veeam backup.

Various versions of MS VSS have had a number of issues over the various itterations of Windows Server. There are still issues with VSS on Server 2008 R2 with and without HyperV. There have been a number of issues with the VSS writer for SQL Server. We seen VSS service crashs intermittantly on Server 2003 when using Advanced File options on various backup products.

I believe the problem has been addressed in ESX4 U1, but Quiesce by Tools commonly caused issues with snapshot creation and/or delete on earlier versions of ESX and this crashed a backup tool we used before we switched to Veeam.

[Gostev]

Peter, you are correct that VSS is poorly inderstood as it applies to virtualization. I recommend researching VSS section in the stickied FAQ thread first, before doing further research on links provided by Vitaly. There are 2 great articles referenced that really do great job putting VSS in context of VMware backup challenges, and also highlighting what puts Veeam VSS integration aside of other implementations.

[PeterCox]

Some progress and some more questions:

The Veeam backup wizard mentions VSS quiescence is two places –

Backup Destination – Advanced – Enable VMware Tools quiescence
And
Backup Consistency – Enable Veeam VSS intergration

Since two overlapping VSS freezes is not likely to be something you want one would conclude that only ONE of these two options should be enabled on any given backup job?

When a job with Veeam VSS intergration enabled, and VMware Tools quiescence disabled, finds a VM Guest with out of date tools installed it returns the following in the job log –

Creating snapshot
Failed to perform Veeam VSS freeze: VMware tools status is "Old"

Which leads one to believe that Veeam VSS integration might be nothing more than VMware Tools quiescence by another name? And if so, then why are user credentials required in the - Backup Consistency – Enable Veeam VSS intergration – dialogue of the wizard?

Research continues.

[Vitaliy S.]

Hi Peter,

Let me start from your last question, if you please:

Which leads one to believe that Veeam VSS integration might be nothing more than VMware Tools quiescence by another name? And if so, then why are user credentials required in the - Backup Consistency – Enable Veeam VSS intergration – dialogue of the wizard?

Actually, no. Veeam Backup & Replication 4.1 provides two different technics for creating transactionally consistent backup images — the Enable VMware tools quiescence and Enable Veeam VSS integration options.

Please review the links below for the difference between those two technics:

http://www.veeam.com/forums/viewtopic.p ... SYNC#p3003
http://www.veeam.com/forums/viewtopic.p ... SYNC#p1834

When a job with Veeam VSS intergration enabled, and VMware Tools quiescence disabled, finds a VM Guest with out of date tools installed it returns the following in the job log –
Creating snapshot
Failed to perform Veeam VSS freeze: VMware tools status is "Old"

Yes, that's correct. You should update the VMware tools to get Veeam VSS work. There is nothing to do with VMware Tools quiescence in this case. Because we need VMware Tools updated (installed) to determine what kind of Operating System is installed in your VM. This will give instructions to our VSS driver of how to properly freeze your VM so that every application inside this VM is in consistent state

Since two overlapping VSS freezes is not likely to be something you want one would conclude that only ONE of these two options should be enabled on any given backup job?

Please note that when you select both VSS integration and VMware tools quiescence options for a job at the same time, the VSS module will only be used for processing backed up and replicated VMs. However, if you use both VSS and VMware tools quiescence options and select the Continue backup even if Veeam VSS quiescence fails option for backup jobs or the Continue replication even if Veeam VSS quiescence fails option for replication jobs, all your VMs will be processed with VSS first, and in case of VSS failure (e.g., Linux VMs), VMs will be processed with the VMware tools quiescence option enabled.
This can be very useful when you have both Windows- and Linux-based VMs in one job.

Hope it helps!

[PeterCox]

Yes all of that makes sense.

After reading a number of posts and bolgs and the MS TechNet discussion of how VSS works I have another two questions in relation to Veeam Backup components –

1. Veeam Agent in Guest – a couple of the posts mention the installation of a small agent supplied by Veeam into the Windows guest to coordinate the Veeam VSS Integration activity. Is this still relevant at Veeam Backup 4.1 and above? And if so, where can I find the Agent and the installation documentation?
2. VMware Tools – we use the Typical Install option when installing the VMware tools into the Windows Gust OS. Does this include all functionality required by Veeam Backup or do we need to select another option?

[Vitaliy S.]

Peter,

There is no need in installing Guest agent manually. All of the setup procedures are handled by Veeam automatically, you should only provide proper credentials (should have local admin privileges on the Guest) at advanced job settings for VSS integration option.

Yes, Typical install option would be fine.

Thank you!

[PeterCox]

OK, how can I check that the agent is installed and working on the guest?

If my guest is not contactable by the Veeam Backup server - the guest is public facing in a DMZ no IP path for MS Admin$ or RPC$ etc ports exists how can I get the agent installed?

If I get the agent installed does Veeam Backup contact the agent via ESX Tools or does it still need an open IP path?

[Vitaliy S.]

Actually, Veeam dynamically installs/uninstalls its VSS provider on-the-fly. That's why your Veeam Backup server needs access SMB to IPC$ on the target VM, so the network connection will still be required.

There is an open discussion of how to back up DMZ VMs, might be helpful:

http://www.veeam.com/forums/viewtopic.p ... +agent+vss

[PeterCox]

We have done additional testing on VSS Integration and found the following:

Veeam VSS Integration in a Windows SBS 2003 guest running AD, Exchange and SQL and we get almost what we expected in the Application Event log and the Exchange Database logs are truncated correctly.

The one exception is the timing of events such as the Exchange Event ID 9616 – VSS service snapshot has truncated the logs successfully. This event occurred at the beginning of the Veeam backup job just after the ESX SnapShot was created. We had expected to see this event after the backup completed successfully on the basis that if the backup fails the application, in this case Exchange, believes it has been backed up and has removed committed log entries.

By comparison another well known Image backup utility (I didn't say ShadowProtect) signals the application that it is now backed up after the actual backup completes and only if successful.

We also tested the VMware tools quiescence option without Veeam VSS Integration and the age old problem returned – The ESX Create SnapShot task failed – Operation timed out. The test Guests in this run were Version 4 hardware, we are preparing a retest against Version 7 hardware, but I am not hopeful. This issue with VMware tools has been around since ESX3 at least.

Naturally Veeam backup cannot solve the various issues that exist with the default installation of various MS Application VSS Writers (may be other applications as well), you need to get the VSS installation working first. The major traps we see are early versions of the SQL / MSDE Writer and the writers such as Exchange which are disabled by default.

Be careful NOT to have a Veeam VSS Enabled backup run at the same time as the Previous Versions scheduled task. The default PV schedule is 7AM and 12noon. Better idea to turn off Previous Versions and use Veeam file level restore.

[PeterCox]

A little bit more research and testing and the results:

I read in a blog on another forum that Veeam VSS Integration invokes the VMware Pre and Port freeze scripts. There are two versions of these scripts pre and post ESX3.5U2 and the configuration is documented in the VMware KB 1006671.

Very useful functionality for those (many) applications without VSS Writers.

In our testing Veeam VSS Integration did not invoke either version of the VMWare scripts during a backup on Windows 2003 SP2 servers. We have VMWare Tools quiescense turned off as it is unreliable and in our installation not required as all guest systems are Windows Server with current VMware tools installed and the guest OS is directly reachable by the Veeam backup server.

All of the Windows VSS Writers get invoked as expected by Veeam Backup.

[Vitaliy S.]

Hi Peter,

Actually Veeam VSS integration doesn't invoke Pre and Post freeze scripts, that's why no scripts were invoked during your testings. You should use VMware Tools quiescense in order to use those scripts in the jobs.

[PeterCox]

What is the chance we can have Veeam Backup Version 5 invoke a Pre and a Post backup script inside the Windows Guest VM?

This functionality would be most useful in dealing with applications which do not have a VSS Writer to support VSS Integration.

The user would need to understand that any such script must have a very short execution time, it should not be used to run processes that take more than a few seconds. VSS sets a limit of 60 seconds on a VSS Writer and that value is probably a good benchmark.

[PeterCox]

Our retest with version 7 hardware indicates that the problem with VMtools quiesce on V4 hardware is not an issue with V7 hardware.

We have run a number of backups with V7 hardware and VMtools quiesce only, no Veeam VSS integration, and all have been successful.

[Vitaliy S.]

Peter, thank you for sharing your feedback and research here! Much appreciated.

[Bennyz]

[merged with existing discussion]

I have the following servers running on one ESXi server:
Domain Controller, Windows 2008
Exchange Server, Windows 2008
SQL 2008, Windows 2008 Server
SQL 2005, Windows 2000 Server
Machine-1, Windows 2000 Pro
Machine-2, Windows XP Pro
Machine-3, CentOS

How should I set up backup on the Veeam and on the guest OS, assuming I only want to do full backup while the guests are running? I read some posts http://www.veeam.com/forums/viewtopic.php?p=6097#p6097 talking about if not set up correctly, Veeam backup would damage the OS/database.

[Bennyz]

So what's the conclusion here? Use Veeam VSS but disable VM Quiescence? That probably works for VSS enabled OS, but what about Windows 2000, SQL 2005 and Linux?

[Vitaliy S.]

Benny,

Veeam Backup and Replication cannot damage any VM.

Basically, you need to make sure you've enabled Application-Aware Image Processing for Windows-based VMs. For other VMs (Linux-based) which do not support VSS, you need to use VMware Tools Quiescence. Futhermore, you may want to use pre-freeze and post-thaw scripts with VMware Quiescence (eg. for Windows 2000 VMs).

Should more details be required, please have a look at this topic [FAQ] v5 : Frequently Asked Questions > Answers

Thank you!

[Bennyz]

How do I use "pre-freeze and post-thaw scripts"? Where can I find them?

[Vitaliy S.]

Please search through these forums for usage examples and more info > http://www.veeam.com/forums/search.php? ... e+postthaw