Snapshots failing on ESXi 7.x

[Marc.H]

After upgrading VCenter to 7.0 and running ESXi 6.7 and 7.0 in a mixed Cluster I do see the following error once in a while:
04.06.2020 01:13:04 :: Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-183086, timeout 1800000, snName VEEAM BACKUP TEMPORARY SNAPSHOT, snDescription Please do not delete this snapshot. It is being used by Veeam Backup., memory False, quiesce False

That happens to more than one VM on the same ESX server at that point of time. Putting that host in maintenance and rebooting solves it for some days. Problem appears randomly on one of the 13 ESX Servers in the Cluster. As mentioned they are both ESXi 6.7 and 7.0 (VCA is on 7.0 latest patch.

Anyone else seeing this problem?

Regards
Marc

[HannesK]

Hello,
for technical issues always contact support and work out a solution with them before posting to the forums... what is your case number please? I also assume that you installed cumulative patch 2?

veeam-backup-replication-f2/rules-of-po ... -t755.html

Thanks,
Hannes

[Marc.H]

Hi Hannes,

yes I have CP2 installed. Sorry no case opened yet. Will do that now.

Thanks
Marc

[Marc.H]

Case #04210758

while looking through the list of serves in more details it seems the error only appears on the ESXi 7.0 Hosts. The ESXi 6.7 Hosts seem not to show this error.

I keep the post updated with any news.

[PetrM]

Hi Marc,

I would also recommend to open a ticket with VMware support because snapshot creation process is managed by the hypervisor, backup application is only the requestor.

Putting that host in maintenance and rebooting solves it for some days

I think that it makes sense to check ESXi logs (f.e. hostd.log) on timestamps corresponding to backup job failures and look for error messages related to snapshot creation.

I believe you should be able to reproduce the same issue manually by creating snapshot from the vSphere client.

Thanks!

[b1gsoup]

We had a similar issue after moving to vSphere 7. Our issue was resolved with a firmware upgrade to our HBAs on our hosts.

[Marc.H]

Hi everyone - to keep you updated - I have indeed opened a case with vmware and they confirmed it is a memory problem bug within version 7.0.0 and 7.0.0a with is assumed to be fixed in a future release. Up to then the only workaround is to reboot the host that shows the problem and it will fix it on that specific host for some time again.

[brlomba1]

Hello all

We recently went to ESXi7 and vcenter 7. We've been having intermittent snapshot issues affecting both Replication & Backups. They generally go away after we vmotion a VM and/or reboot the hosts, but it has gotten worse lately. We have an open case with VMware. Anyone seeing this? We are hitting this in 2 different datacenters running on Nimble & UCS and Nimble & Dell.

[PetrM]

Hello,

I'd recommend to continue the investigation with VMware support team as creation or removing snapshot is managed at the level of hypervisor, backup application only plays the role of requestor.

Thanks!

[Andreas Neufert]

Do you use the Nimble together with our Storage Integration? Then it is important that the access policy of the Nimble volume is set to "Volume only" and not not "Snapshot and volumes".
As well we have many ten thousands of customers running on ESXi7 and nothing poped up in that direction more frequently.

[regi]

Hi backup or replication some VM's started to fail after the very latest V10 patch 10.0.1.4854 (My VMware ESXi is 7.0.0, 16324942)
This is totally random 3 backups were ok & same 3 failed on replication 2 days ago but all are good on todays backup/replication
Today 1 VM replication is failed but I moved this VM to a different host and re ran job all are ok
So it's totally random. I believe it all started after build VBR 4854 update & nothing is been changed on VMW side
Please help/advise

Error
Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-32500, timeout 1800000, snName VEEAM BACKUP TEMPORARY SNAPSHOT, snDescription Please do not delete this snapshot. It is being used by Veeam Backup., memory False, quiesce False
Error: An error occurred while saving the snapshot: Error.

Veeam Support - Case # 04403091

[regi]

Hi I'm having the same issue. Updated to ESXi 7 months ago & it was running fine all this time & I updated to latest VBR patch last week and im having this issue (My setup is HPE G10's with Pure SAN AFA) Never seen any like this.

vmware-vsphere-f24/failed-to-create-vm- ... 69508.html

[regi]

See this vmware-vsphere-f24/snapshot-error-t67340.html

[Natalia Lupacheva]

HI Regi,

I've moved your post to this thread to keep the discussions on this issue in one place.
Also, did the suggested workaround (reboot) work for you?

Thanks!

[regi]

Morning like I said reboot did not help. The only way is to VMotion VM & retry or reboot ESX host or services.sh restart on host
Can I reopen this case Veeam Support - Case # 04403091 or a new case for me for this issue ?

All failing VM's are on the same ESX host (It was a different ESX host last week & I have rebooted) will try & reboot the problem host today
(Maybe an issue with VMW 7 some sort of memory issue after installing/upgrading VMW 7 for 80 days & need a servicers restart or reboot. I upgraded all the hosts to VMW 7 80 days ago)

It got worse & never had a single issue with Veeam BR

I have opened a support case with VMW last night waiting for an update

[regi]

Hi had a zoom meeting with VMW re this issue. They advised that they V 7 very new & still having some heap memory issues which will be fixed in the next patch. They asked me to VMotion VM's & reboot all ESX hosts & monitor for the next few days. Will update.

[regi]

Update: Just finished rebooting all esx hosts & re-ran all failed backups/replications from last night all are good
Hopefully I'm sorted for another 80 days I'm sure VMW will patch this issue very soon

Cheers

[Natalia Lupacheva]

Hi Regi,

So, the host reboot is the only available workaround for now.
thanks a lot for the follow-up and sharing the details! Appreciate it.

Thanks!

[regi]

Morning Natalia yes this is the only workaround currently according to VMW. If you can't do VMotion you can issue services restart using SSH
services.sh restart but VMW said it's better to reboot ESX host. Hope VMW will fix this memory heap issue in the next release. Have a good day

/R

[regi]

Update FYI: The VMW patch will be released to address this issue by Nov according to VMW support

[Gostev]

So this patch is not included in vSphere 7 Update 1?

[regi]

Reply from VMW:

Hello Regi,

The version 7.0 U1 is not yet released, and the issue would be fixed in that.

Regards,
Technical Support Engineer
Global Support Services, VMware Inc.

[chsuscale]

anyone here can confirm the problem is gone with 7.0.1.
I've got a server running 7.0.1, 17551050 which is giving me:

Code: Select all

" 17.02.2021 05:41:46 :: Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-116164, timeout 1800000, snName VEEAM BACKUP TEMPORARY SNAPSHOT, snDescription Please do not delete this snapshot. It is being used by Veeam Backup., memory False, quiesce False "
 "An error occurred while saving the snapshot: A digest operation has failed."