VMWare Tools Quiescence Issue Domino Server

[kjc3303]

Hi there, hoping someone may be able to help with an issue I am having.

I have logged a support call but thought I'd post on here in case someone has experenced the same issue.

All my VMs are backing fine using MS VSS, I can intant restore and have setup sure backup which works perfectly:D

I have one VM that is a Domino server, Domino does not support MS VSS so am using VMWare Tools Quiescence.
This VM backs up successfully according to the job log but can not IR or sure backup from the successful backup.

I have checked the enable VMware Tools Quiescence and created a prefreeze script and placed in C:\Program Files\VMware\VMware Tools\backupScripts.d

Script

@echo off
if %1 == freeze goto freeze
if %1 == thaw goto thaw
if %1 == freezeFail goto freezeFail

:freeze
Net Stop "Lotus Domino Server (FProgramFilesIBMLotusDominodata)"
exit
:thaw
net start "Lotus Domino Server (FProgramFilesIBMLotusDominodata)"
exit
:freezeFail
net start "Lotus Domino Server (FProgramFilesIBMLotusDominodata)"
exit

I see this scripts works and the server shuts down and starts up whilst the snapshot is taken.

VMWare error

Reason: Failed to lock the file.
Cannot open the disk '/vmfs/volumes/ef3713aa-7c71c00b/Gradus-VS08-05 _Domino_ IR_7408e544-cbe5-407e-a655-ffcd8b87220d/Gradus-VS08-05 (Domino)_1.vmdk' or one of the snapshot disks it depends on.

Veeam Error said that the disk was locked



When I untick VMWare Tools Quiescence and select MS VSS the back up works and I can IR and surebackup the Domino Server but this leaves me with a 'dirty' backup that is onlycrash consistant.

Does anybody have any Ideas if I am missing something obvious or have come accross this before?

Thanks

Kev

[kjc3303]

Edit forgot to mention the Domino Backup is in it's own seperate job.

[Gostev]

Please include your Veeam support case ID for this issue.

[kjc3303]

Hi Gostev

ID#5160896

Thanks

[kjc3303]

Thought I would update with my findings here is my last email to Veeam support.

Hi Zach,

unfortunately I am going to have to cancel the webex later. The good news is I have found the issue and implemented a work around.

This Issue was only occurring when trying to restore a VM (IR, Sure backup and Full VM Backup) that ....
A) was VMware Tools Quiescence and B) Had vmdk's on different Data stores

I have put both vmdk's on the same datastore and I can IR, Surebackup and perform a restore entire VM.

I also tried all variables to see if there was something wrong with my infrastructure and it does seem to be a bug in the Veeam software.

I built a new VM (VMware Tools Quiescence) with one disk on Datastore A - Success
Added a second disk on Datastore A - Success
Removed second disk and added Disk to Datastore B - Failed
Migrated both disks to Datastore B - Success

An additional test I did was the original VM that was failing I performed a VM restore but selected restore to a new location I then manually selected the original locations and it resulted in Success.
Something else I did notice was that when using restoring entire VM option but leaving the defaults the vmdks where both restored to the same datastore (They were on different) Which I assume is why it couldn't be powered on.

[Vitaliy S.]

Hi Kevin, thank you for posting the workaround. I will ask our QC team to try to reproduce this scenario.

[kjc3303]

That would be good thanks.

[Vitaliy S.]

Kevin, we have created similar VM with multiple disks on different datastores, enabled VMware Tools Quiescence and successfully backed it up. SureBackup and Instant Recovery worked for us too.

The only thing that is not clear so far - based on your log files you already had existing snapshot on the VM when you run the backup job with VMware Tools Quiescence enabled.

Can you please confirm that? Could there be any orphaned snapshot which is not visible within a Snapshot Manager?

[kjc3303]

Hi Vitaliy, I am sure that this is not the case as I have performed all recents tests on a brand new VM with no snapshots. I can send over all the log files for my recent tests on the the new VM if it would help?

I had also created a brand new Domino server VM and had the same issue.

Edit: I have had a look through all my datastores and can not see any orphaned vmsn files.

[Vitaliy S.]

Yes, please update your ticket with new log files, so we could take a look at them. On top of that, could you please create a screenshot of Snapshot Manager while running backup job with quiescence enabled?

[kjc3303]

Will do, although will not be until the new year now I'm afraid. Have a good Christmas

Kevin

[Vitaliy S.]

Ok, let's get back to this issue next year. Merry Christmas!