Host-based backup of VMware vSphere VMs.
Post Reply
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

About VMWare Tools Quiescence for Linux guests

Post by antipolis »

Greetings,

I am quite confused about the "Enable VMWare Tools Quiescence" option.

We are backing up NFS Datastores stored on NetApp appliances, these datastores contain mixed Windows and Linux guests. Now on my initial implementation of Veeam I checked the "Enable VMWare Tools Quiescence" option, because according to the documentation https://helpcenter.veeam.com/docs/backu ... tml?ver=95
To create transactionally consistent backups and replicas for VMs that do not support Microsoft VSS (for example, Linux VMs), you must enable VMware Tools quiescence for the job. In this case, Veeam Backup & Replication will use the VMware Tools to freeze the file system and application data on the VM before backup or replication.
Furthermore the text under the options says "only used for machines with application aware processing disabled". Most of our virtual machines DO have AA processing enabled with pre and post scripts for both windows and linux hosts, but I have a few linux hosts with cannot be processed by AA processing for all kinds of reasons ; so I figured ok let's enable quiescence so that "machines with application aware processing disabled" are processed by it

Now the Veeam Documentation (https://helpcenter.veeam.com/docs/backu ... tml?ver=95) refers to this VMware article : https://pubs.vmware.com/vsphere-55/inde ... p.9.6.html

Which clearly states that "other guest operating system" (in the context of the article other = which is not windows) are "Not applicable" and does "Crash-consistent quiescing". I think back then VMWare had an experimental feature called the sync driver for linux operating systems, but it does not seem to be the case anymore, and as far as I'm aware sync is not included in the open-vm-tools package. Yes, we use OVT and not the official tools,this is the vmware recommendation : https://kb.vmware.com/selfservice/micro ... Id=2073803 :
VMware recommends using OVT redistributed by operating system vendors.
In the light of this, I don't see any upsides of enabling "VMWare Tools Quiescence" for our few non-AA linux guests, "crash consistent quiescing" does not appear very different to my than just backing it up straight away.

Furthermore, there is a lot of downsides for us for activating VMWare Tools Quiescence, from Veeam documentation :

https://helpcenter.veeam.com/docs/backu ... tml?ver=95
[For storage systems working over NFS] If you enable the Enable VMware tools quiescence option in the job settings, Veeam Backup & Replication will not use Backup from Storage Snapshots to process running Microsoft Windows VMs that have VMware Tools installed.
https://helpcenter.veeam.com/docs/backu ... tml?ver=95
If you enable the Enable VMware tools quiescence option in the job settings, Veeam Backup & Replication will not use the Direct NFS transport mode to process running Microsoft Windows VMs that have VMware Tools installed.
And yes, after some testing it appears that both backup from storage snapshot and directnfs are indeed not working for our windows VMs (which is counter intuitive because they have AA processing enabled)

To sum it up, my understanding of it (please correct me if I'm wrong, which was the motivation behind this post) is that in our case (just a few linux VMs without AA processing) enabling VMWare Tools Quiescence has no clear benefits and make us unable to use some veeam features (namely backup from storage snapshots and directnfs)

Sorry for the long post and thank you for your insight
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by PTide »

Hi,

Your understanding is correct. You should use Veeam AAIP settings for Microsoft VMs where possible. For all other VMs that are not suitable for Veeam default AAIP options (Linux VMs. MS VMs with custom software) you should specify pre-freeze and post-thaw scripts in the "Scripts" tab of the "Guest processing" settings.

Thanks.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by antipolis »

Then I think the documentation should be updated, this page is misleading : https://helpcenter.veeam.com/docs/backu ... tml?ver=95
You can enable both options for VM quiescence. Such scenario is recommended if you add Microsoft Windows and Linux VMs to the same job. In this case, all VMs will be processed in a transactionally consistent manner — either with application-aware processing or VMware Tools quiescence.
But this is not true for Linux VM because as mentionned in the VMWare article referred above VMWare Tools Quiescence will only be crash consistent for OS other than Windows (which makes this option kind of useless for Linux guests)
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by PTide »

Then I think the documentation should be updated
We'll look into that, thanks!
But this is not true for Linux VM because as mentionned in the VMWare article referred above VMWare Tools Quiescence will only be crash consistent for OS other than Windows (which makes this option kind of useless for Linux guests)
As stated in the most recent VMware KB:
Quiesce: If the <quiesce> flag is 1 or true, and the virtual machine is powered on when the snapshot is taken, VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk, or other higher-level application-specific tasks.
On the other hand there is no mention what exactly does VMware quiescence do during snapshot regarding app-specific tasks. Previously it was possible to add custom pre-freeze and post-thaw sripts into specific directory (/usr/sbin/) so VMware Tools could trigger the scripts before and after a snaphost, however I haven't managed to find any info about that in the current VMware documentation. For now you should disable VMware native quiescence and use Veeam's pre-freeze/post-thaw scripts to make a consistent backup. I'll update the thread once I find out more details.

Thank you!
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by antipolis »

Thank you for your feedback
PTide wrote: Previously it was possible to add custom pre-freeze and post-thaw sripts into specific directory (/usr/sbin/) so VMware Tools could trigger the scripts before and after a snaphost, however I haven't managed to find any info about that in the current VMware documentation.
From my own testing I was never able to make "native" /usr/sbin pre-freeze and post-thaw scripts work with either OVT (open-vm-tools) or official VMWare Tools so I am using the corresponding veeam options
DeWi
Lurker
Posts: 1
Liked: never
Joined: Mar 16, 2017 11:33 am
Full Name: Dennis Wiche
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by DeWi »

Are there any news about this?
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by PTide »

Hi,

The following steps should make it work:

1. Make sure that you have VMware tools installed
2. Create script /usr/sbin/pre-freeze-script
3. Make sure it has 0700 permissions and is owned by root:

Code: Select all

chmod 0700 /usr/sbin/pre-freeze-script
chown root /usr/sbin/pre-freeze-script
For post-thaw actions use a script named post-thaw-script.

Thanks
rhys.hammond
Veeam Software
Posts: 72
Liked: 15 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

[MERGED] Veeam & ensuring transactionally consistent workloa

Post by rhys.hammond »

Hello Everyone!

Just hoping to run something by the Veeam team as I am after some clarification regarding application-consistent backups of Linux VMs.
So I thought transitionally consistent backups and 'proper backup and restore of VMs running highly transactional applications' are one the same.

https://helpcenter.veeam.com/docs/backu ... tml?ver=95
"To create transactionally consistent backups and replicas for VMs that do not support VSS (for example, Linux VMs), you must enable VMware Tools quiescence for the job. In this case, Veeam Backup & Replication will use the VMware Tools to freeze the file system and application data on the VM before backup or replication."

Vs

https://helpcenter.veeam.com/docs/backu ... tml?ver=95
'If you use VMware Tools quiescence, Veeam Backup & Replication will quiesce the VM but will not perform application-specific actions required for proper backup and restore of VMs running highly transactional applications.'

It's been discussed in other threads with mention of looking into the help guide and updating it here, vmware-vsphere-f24/about-vmware-tools-q ... 39069.html.

In my mind, we should always be using the pre-freeze and post-thaw scripts to ensure application consistency and never relying on 'enabling VMware Tools Quiescence'.
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by DGrinev » 1 person likes this post

Hi Rhys,

Yes, you're right as best practice states to use pre-freeze/post-thaw scripts, for the rest information please check discussion above.

Thanks!
hajek
Influencer
Posts: 11
Liked: 1 time
Joined: Aug 01, 2016 4:11 pm
Full Name: Tomas Hajek
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by hajek »

Hi DGrinev
When you mention "best practice states to use pre-freeze/post-thaw scripts" are you referring the the Veeam pre-freeze/post-thaw scripts (https://helpcenter.veeam.com/docs/backu ... tml?ver=95) or VMware Tools pre-freeze/post thaw scripts (as mentioned by PTide above)? I'm currently partial to the VMware Tools /usr/sbin/pre-freeze-script and /usr/sbin/post-thaw-script because I already have them written and have been using them for a few years and don't require and additional users or SSH access. I assume that in order for the VMware Tools scripts to be executed by Veeam the job needs to be configured with "Enable Vmware Tools quiescence", is that correct?

Reading through the Veeam pre_post_scripts it seems that I would have to provide some ssh credentials to Veeam in order to manage the scripts on the host, is that correct? The document says "Scripts for Linux VMs are uploaded over SSH." but if the scripts exist already does Veeam still need to SSH to the VM? If I have existing /usr/sbin/pre-freeze-script and /usr/sbin/post-thaw-script on a VM can the Veeam pre_post_scripts be configured to execute them?

thanks,
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by PTide »

Hi Tomas,
I assume that in order for the VMware Tools scripts to be executed by Veeam the job needs to be configured with "Enable Vmware Tools quiescence", is that correct?
Correct. Also with VMware Tools quiescence you don't have to enable provide guest credentials and can leave application-aware processing option disabled.
Reading through the Veeam pre_post_scripts it seems that I would have to provide some ssh credentials to Veeam in order to manage the scripts on the host, is that correct?
That's correct.
but if the scripts exist already does Veeam still need to SSH to the VM?
Yes, if you specify pre-freeze/post-thaw script in guest-processsing options.
If I have existing /usr/sbin/pre-freeze-script and /usr/sbin/post-thaw-script on a VM can the Veeam pre_post_scripts be configured to execute them
No, you need to use VMware quiescence option instead.

Please mind the following - quiescence ensures that host can capture file-system-consistent guest snapshots. Transaction consistency depends on whether or not vmware tools are aware of your particular linux application.

Thanks
jessie0
Enthusiast
Posts: 50
Liked: 5 times
Joined: Oct 16, 2019 5:04 am
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by jessie0 »

Hello,

I am new to veeam and I have the same concerns as the above posts.
Considering the date of the last post, I was wondering if there has been any changes to veeam's
Quiescence of Linux VMs.

Questions:
1. Do I still need to "Enable quiescing" and specify a pre-freeze and post-thaw scripts in VB&R 9.5 U4 when backing up transactional applications in Linux VM?
2. Are transactionally consistent application backups for Linux physical servers also created using pre-freeze and post-thaw scripts?
3. If the answer to question 2 is no, how do I go about creating transactionally consistent backups for Linux physical servers?

Plea:
Sorry to trouble whoever is reading this, but I need help with scripts for backing up highly transactionally consistent applications for Linux VMs.

Thanks.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by Andreas Neufert »

Linux do not has a central consistency protocol like Microsoft VSS. So Veeam has to find out which application is installed that hard way and then speak with the application to bring it in consistent state.

If you enable the Veeam own Guest Processing we perform this today only for Oracle DBs. For any other database you need to check if they support crash consistent backup and if not you need to create pre and post script to bring them in consistent state. You can use the Veeam Script processing as part of the Guest processing for it.

VMware Tools quiescense do not do this at all and if used you need to use the scripting approach as well. We recommend to use the Veeam pre post scripting as if something goes wrong you have everything in the Veeam logs instead of 2 places (VMware + Veeam).

The upcoming Veeam Agent for Linux 4 that will be released with v10 will support MySQL and PostgreSQL (without the script processing).
jessie0
Enthusiast
Posts: 50
Liked: 5 times
Joined: Oct 16, 2019 5:04 am
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by jessie0 »

Hello Andreas Neufert,

Thanks for taking out time to clarify things.
"The upcoming Veeam Agent for Linux 4 that will be released with v10 will support MySQL and PostgreSQL (without the script processing)".
Please when will it be out? can not wait to not have to apply scripts to MySQL and PostgreSQL for Linux servers.

A few question regarding the pre-freeze and post-thaw scripts for Linux VMs and physical servers.
I understand that both scripts should be moved to the /usr/sbin/ directory and the ownership and permission must be set.
However, on the VB&R console, the only option to specifying a path for the script is the C:\ or D:\ of the VB&R server.
hence the following questions:

1. Haven already saved the scripts in my Linux server, do I also need to save the script on my VB&R server(Windows machine) for it to be recognized and executed?
2. What file name and format should I save it as on the VB&R server?

Thanks.
jessie0
Enthusiast
Posts: 50
Liked: 5 times
Joined: Oct 16, 2019 5:04 am
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by jessie0 »

Hello

I have been attempting a backup of my Linux VM. I have enabled quiescing and supplied the path to the pre-freeze and post-thaw scripts but
each time I run the backup job, I get the following error:

CreateSnapshot failed, vmRef "vm-xxx", timeout "1800000", snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "True"

Any idea what could be the cause of the problem, and possible work-around?
The only explanation as to the reason for this is for a windows VM, not Linux.
Thanks.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by wishr »

Hi Jessie,

This error could be caused by a variety of reasons so the best way to get it sorted is to let our support engineers analyze the debug logs.

Thanks
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by PTide »

@jessie0,

This error is most likely caused by wrong permissions on the scripts. Kindly check my older post - the approach that is described there still works in Ubuntu 19.10.
hence the following questions:

1. Haven already saved the scripts in my Linux server, do I also need to save the script on my VB&R server(Windows machine) for it to be recognized and executed?
2. What file name and format should I save it as on the VB&R server?
1.) No (for Linux)
2.) See 1.)

Thanks!
jessie0
Enthusiast
Posts: 50
Liked: 5 times
Joined: Oct 16, 2019 5:04 am
Contact:

Re: About VMWare Tools Quiescence for Linux guests

Post by jessie0 » 1 person likes this post

Hello PTide,

Thanks for the response!
I have come to understand how script application in veeam works, thanksto your help and that of the so many kind people on here.

Thank you.
Post Reply

Who is online

Users browsing this forum: No registered users and 70 guests