About VMWare Tools Quiescence for Linux guests

[antipolis]

Greetings,

I am quite confused about the "Enable VMWare Tools Quiescence" option.

We are backing up NFS Datastores stored on NetApp appliances, these datastores contain mixed Windows and Linux guests. Now on my initial implementation of Veeam I checked the "Enable VMWare Tools Quiescence" option, because according to the documentation https://helpcenter.veeam.com/docs/backu ... tml?ver=95

To create transactionally consistent backups and replicas for VMs that do not support Microsoft VSS (for example, Linux VMs), you must enable VMware Tools quiescence for the job. In this case, Veeam Backup & Replication will use the VMware Tools to freeze the file system and application data on the VM before backup or replication.

Furthermore the text under the options says "only used for machines with application aware processing disabled". Most of our virtual machines DO have AA processing enabled with pre and post scripts for both windows and linux hosts, but I have a few linux hosts with cannot be processed by AA processing for all kinds of reasons ; so I figured ok let's enable quiescence so that "machines with application aware processing disabled" are processed by it

Now the Veeam Documentation (https://helpcenter.veeam.com/docs/backu ... tml?ver=95) refers to this VMware article : https://pubs.vmware.com/vsphere-55/inde ... p.9.6.html

Which clearly states that "other guest operating system" (in the context of the article other = which is not windows) are "Not applicable" and does "Crash-consistent quiescing". I think back then VMWare had an experimental feature called the sync driver for linux operating systems, but it does not seem to be the case anymore, and as far as I'm aware sync is not included in the open-vm-tools package. Yes, we use OVT and not the official tools,this is the vmware recommendation : https://kb.vmware.com/selfservice/micro ... Id=2073803 :

VMware recommends using OVT redistributed by operating system vendors.

In the light of this, I don't see any upsides of enabling "VMWare Tools Quiescence" for our few non-AA linux guests, "crash consistent quiescing" does not appear very different to my than just backing it up straight away.

Furthermore, there is a lot of downsides for us for activating VMWare Tools Quiescence, from Veeam documentation :

https://helpcenter.veeam.com/docs/backu ... tml?ver=95

[For storage systems working over NFS] If you enable the Enable VMware tools quiescence option in the job settings, Veeam Backup & Replication will not use Backup from Storage Snapshots to process running Microsoft Windows VMs that have VMware Tools installed.

https://helpcenter.veeam.com/docs/backu ... tml?ver=95

If you enable the Enable VMware tools quiescence option in the job settings, Veeam Backup & Replication will not use the Direct NFS transport mode to process running Microsoft Windows VMs that have VMware Tools installed.

And yes, after some testing it appears that both backup from storage snapshot and directnfs are indeed not working for our windows VMs (which is counter intuitive because they have AA processing enabled)

To sum it up, my understanding of it (please correct me if I'm wrong, which was the motivation behind this post) is that in our case (just a few linux VMs without AA processing) enabling VMWare Tools Quiescence has no clear benefits and make us unable to use some veeam features (namely backup from storage snapshots and directnfs)

Sorry for the long post and thank you for your insight

[PTide]

Hi,

Your understanding is correct. You should use Veeam AAIP settings for Microsoft VMs where possible. For all other VMs that are not suitable for Veeam default AAIP options (Linux VMs. MS VMs with custom software) you should specify pre-freeze and post-thaw scripts in the "Scripts" tab of the "Guest processing" settings.

Thanks.

[antipolis]

Then I think the documentation should be updated, this page is misleading : https://helpcenter.veeam.com/docs/backu ... tml?ver=95

You can enable both options for VM quiescence. Such scenario is recommended if you add Microsoft Windows and Linux VMs to the same job. In this case, all VMs will be processed in a transactionally consistent manner — either with application-aware processing or VMware Tools quiescence.

But this is not true for Linux VM because as mentionned in the VMWare article referred above VMWare Tools Quiescence will only be crash consistent for OS other than Windows (which makes this option kind of useless for Linux guests)

[PTide]

Then I think the documentation should be updated

We'll look into that, thanks!

But this is not true for Linux VM because as mentionned in the VMWare article referred above VMWare Tools Quiescence will only be crash consistent for OS other than Windows (which makes this option kind of useless for Linux guests)

As stated in the most recent VMware KB:

Quiesce: If the <quiesce> flag is 1 or true, and the virtual machine is powered on when the snapshot is taken, VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk, or other higher-level application-specific tasks.

On the other hand there is no mention what exactly does VMware quiescence do during snapshot regarding app-specific tasks. Previously it was possible to add custom pre-freeze and post-thaw sripts into specific directory (/usr/sbin/) so VMware Tools could trigger the scripts before and after a snaphost, however I haven't managed to find any info about that in the current VMware documentation. For now you should disable VMware native quiescence and use Veeam's pre-freeze/post-thaw scripts to make a consistent backup. I'll update the thread once I find out more details.

Thank you!

[antipolis]

Thank you for your feedback

Previously it was possible to add custom pre-freeze and post-thaw sripts into specific directory (/usr/sbin/) so VMware Tools could trigger the scripts before and after a snaphost, however I haven't managed to find any info about that in the current VMware documentation.

From my own testing I was never able to make "native" /usr/sbin pre-freeze and post-thaw scripts work with either OVT (open-vm-tools) or official VMWare Tools so I am using the corresponding veeam options

[DeWi]

Are there any news about this?

[PTide]

Hi,

The following steps should make it work:

1. Make sure that you have VMware tools installed
2. Create script /usr/sbin/pre-freeze-script
3. Make sure it has 0700 permissions and is owned by root:

Code: Select all

chmod 0700 /usr/sbin/pre-freeze-script
chown root /usr/sbin/pre-freeze-script

For post-thaw actions use a script named post-thaw-script.

Thanks

[rhys.hammond]

Hello Everyone!

Just hoping to run something by the Veeam team as I am after some clarification regarding application-consistent backups of Linux VMs.
So I thought transitionally consistent backups and 'proper backup and restore of VMs running highly transactional applications' are one the same.

https://helpcenter.veeam.com/docs/backu ... tml?ver=95
"To create transactionally consistent backups and replicas for VMs that do not support VSS (for example, Linux VMs), you must enable VMware Tools quiescence for the job. In this case, Veeam Backup & Replication will use the VMware Tools to freeze the file system and application data on the VM before backup or replication."

Vs

https://helpcenter.veeam.com/docs/backu ... tml?ver=95
'If you use VMware Tools quiescence, Veeam Backup & Replication will quiesce the VM but will not perform application-specific actions required for proper backup and restore of VMs running highly transactional applications.'

It's been discussed in other threads with mention of looking into the help guide and updating it here, vmware-vsphere-f24/about-vmware-tools-q ... 39069.html.

In my mind, we should always be using the pre-freeze and post-thaw scripts to ensure application consistency and never relying on 'enabling VMware Tools Quiescence'.

[DGrinev]

Hi Rhys,

Yes, you're right as best practice states to use pre-freeze/post-thaw scripts, for the rest information please check discussion above.

Thanks!

[hajek]

Hi DGrinev
When you mention "best practice states to use pre-freeze/post-thaw scripts" are you referring the the Veeam pre-freeze/post-thaw scripts (https://helpcenter.veeam.com/docs/backu ... tml?ver=95) or VMware Tools pre-freeze/post thaw scripts (as mentioned by PTide above)? I'm currently partial to the VMware Tools /usr/sbin/pre-freeze-script and /usr/sbin/post-thaw-script because I already have them written and have been using them for a few years and don't require and additional users or SSH access. I assume that in order for the VMware Tools scripts to be executed by Veeam the job needs to be configured with "Enable Vmware Tools quiescence", is that correct?

Reading through the Veeam pre_post_scripts it seems that I would have to provide some ssh credentials to Veeam in order to manage the scripts on the host, is that correct? The document says "Scripts for Linux VMs are uploaded over SSH." but if the scripts exist already does Veeam still need to SSH to the VM? If I have existing /usr/sbin/pre-freeze-script and /usr/sbin/post-thaw-script on a VM can the Veeam pre_post_scripts be configured to execute them?

thanks,

[PTide]

Hi Tomas,

I assume that in order for the VMware Tools scripts to be executed by Veeam the job needs to be configured with "Enable Vmware Tools quiescence", is that correct?

Correct. Also with VMware Tools quiescence you don't have to enable provide guest credentials and can leave application-aware processing option disabled.

Reading through the Veeam pre_post_scripts it seems that I would have to provide some ssh credentials to Veeam in order to manage the scripts on the host, is that correct?

That's correct.

but if the scripts exist already does Veeam still need to SSH to the VM?

Yes, if you specify pre-freeze/post-thaw script in guest-processsing options.

If I have existing /usr/sbin/pre-freeze-script and /usr/sbin/post-thaw-script on a VM can the Veeam pre_post_scripts be configured to execute them

No, you need to use VMware quiescence option instead.

Please mind the following - quiescence ensures that host can capture file-system-consistent guest snapshots. Transaction consistency depends on whether or not vmware tools are aware of your particular linux application.

Thanks

[jessie0]

Hello,

I am new to veeam and I have the same concerns as the above posts.
Considering the date of the last post, I was wondering if there has been any changes to veeam's
Quiescence of Linux VMs.

Questions:
1. Do I still need to "Enable quiescing" and specify a pre-freeze and post-thaw scripts in VB&R 9.5 U4 when backing up transactional applications in Linux VM?
2. Are transactionally consistent application backups for Linux physical servers also created using pre-freeze and post-thaw scripts?
3. If the answer to question 2 is no, how do I go about creating transactionally consistent backups for Linux physical servers?

Plea:
Sorry to trouble whoever is reading this, but I need help with scripts for backing up highly transactionally consistent applications for Linux VMs.

Thanks.

[Andreas Neufert]

Linux do not has a central consistency protocol like Microsoft VSS. So Veeam has to find out which application is installed that hard way and then speak with the application to bring it in consistent state.

If you enable the Veeam own Guest Processing we perform this today only for Oracle DBs. For any other database you need to check if they support crash consistent backup and if not you need to create pre and post script to bring them in consistent state. You can use the Veeam Script processing as part of the Guest processing for it.

VMware Tools quiescense do not do this at all and if used you need to use the scripting approach as well. We recommend to use the Veeam pre post scripting as if something goes wrong you have everything in the Veeam logs instead of 2 places (VMware + Veeam).

The upcoming Veeam Agent for Linux 4 that will be released with v10 will support MySQL and PostgreSQL (without the script processing).

[jessie0]

Hello Andreas Neufert,

Thanks for taking out time to clarify things.
"The upcoming Veeam Agent for Linux 4 that will be released with v10 will support MySQL and PostgreSQL (without the script processing)".
Please when will it be out? can not wait to not have to apply scripts to MySQL and PostgreSQL for Linux servers.

A few question regarding the pre-freeze and post-thaw scripts for Linux VMs and physical servers.
I understand that both scripts should be moved to the /usr/sbin/ directory and the ownership and permission must be set.
However, on the VB&R console, the only option to specifying a path for the script is the C:\ or D:\ of the VB&R server.
hence the following questions:

1. Haven already saved the scripts in my Linux server, do I also need to save the script on my VB&R server(Windows machine) for it to be recognized and executed?
2. What file name and format should I save it as on the VB&R server?

Thanks.

[jessie0]

Hello

I have been attempting a backup of my Linux VM. I have enabled quiescing and supplied the path to the pre-freeze and post-thaw scripts but
each time I run the backup job, I get the following error:

CreateSnapshot failed, vmRef "vm-xxx", timeout "1800000", snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "True"

Any idea what could be the cause of the problem, and possible work-around?
The only explanation as to the reason for this is for a windows VM, not Linux.
Thanks.

[wishr]

Hi Jessie,

This error could be caused by a variety of reasons so the best way to get it sorted is to let our support engineers analyze the debug logs.

Thanks

[PTide]

@jessie0,

This error is most likely caused by wrong permissions on the scripts. Kindly check my older post - the approach that is described there still works in Ubuntu 19.10.

hence the following questions:

1. Haven already saved the scripts in my Linux server, do I also need to save the script on my VB&R server(Windows machine) for it to be recognized and executed?
2. What file name and format should I save it as on the VB&R server?

1.) No (for Linux)
2.) See 1.)

Thanks!

[jessie0]

Hello PTide,

Thanks for the response!
I have come to understand how script application in veeam works, thanksto your help and that of the so many kind people on here.

Thank you.