Host-based backup of VMware vSphere VMs.
Post Reply
Marc.H
Enthusiast
Posts: 75
Liked: 15 times
Joined: May 23, 2012 7:51 am
Full Name: Marc Holbe
Contact:

Snapshots failing on ESXi 7.x

Post by Marc.H »

After upgrading VCenter to 7.0 and running ESXi 6.7 and 7.0 in a mixed Cluster I do see the following error once in a while:
04.06.2020 01:13:04 :: Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-183086, timeout 1800000, snName VEEAM BACKUP TEMPORARY SNAPSHOT, snDescription Please do not delete this snapshot. It is being used by Veeam Backup., memory False, quiesce False

That happens to more than one VM on the same ESX server at that point of time. Putting that host in maintenance and rebooting solves it for some days. Problem appears randomly on one of the 13 ESX Servers in the Cluster. As mentioned they are both ESXi 6.7 and 7.0 (VCA is on 7.0 latest patch.

Anyone else seeing this problem?

Regards
Marc
HannesK
Product Manager
Posts: 14252
Liked: 2858 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Snapshot error

Post by HannesK »

Hello,
for technical issues always contact support and work out a solution with them before posting to the forums... what is your case number please? I also assume that you installed cumulative patch 2?

veeam-backup-replication-f2/rules-of-po ... -t755.html

Thanks,
Hannes
Marc.H
Enthusiast
Posts: 75
Liked: 15 times
Joined: May 23, 2012 7:51 am
Full Name: Marc Holbe
Contact:

Re: Snapshot error

Post by Marc.H »

Hi Hannes,

yes I have CP2 installed. Sorry no case opened yet. Will do that now.

Thanks
Marc
Marc.H
Enthusiast
Posts: 75
Liked: 15 times
Joined: May 23, 2012 7:51 am
Full Name: Marc Holbe
Contact:

Re: Snapshot error

Post by Marc.H » 1 person likes this post

Case #04210758

while looking through the list of serves in more details it seems the error only appears on the ESXi 7.0 Hosts. The ESXi 6.7 Hosts seem not to show this error.

I keep the post updated with any news.
PetrM
Veeam Software
Posts: 3224
Liked: 518 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Snapshot error

Post by PetrM »

Hi Marc,

I would also recommend to open a ticket with VMware support because snapshot creation process is managed by the hypervisor, backup application is only the requestor.

Marc.H wrote:Putting that host in maintenance and rebooting solves it for some days
I think that it makes sense to check ESXi logs (f.e. hostd.log) on timestamps corresponding to backup job failures and look for error messages related to snapshot creation.

I believe you should be able to reproduce the same issue manually by creating snapshot from the vSphere client.

Thanks!
b1gsoup
Lurker
Posts: 1
Liked: 2 times
Joined: Jun 11, 2020 2:52 pm
Contact:

Re: Snapshot error

Post by b1gsoup » 2 people like this post

We had a similar issue after moving to vSphere 7. Our issue was resolved with a firmware upgrade to our HBAs on our hosts.
Marc.H
Enthusiast
Posts: 75
Liked: 15 times
Joined: May 23, 2012 7:51 am
Full Name: Marc Holbe
Contact:

Re: Snapshot error

Post by Marc.H » 2 people like this post

Hi everyone - to keep you updated - I have indeed opened a case with vmware and they confirmed it is a memory problem bug within version 7.0.0 and 7.0.0a with is assumed to be fixed in a future release. Up to then the only workaround is to reboot the host that shows the problem and it will fix it on that specific host for some time again.
brlomba1
Influencer
Posts: 15
Liked: 1 time
Joined: Oct 03, 2018 8:07 pm
Contact:

[MERGED] Snapshots failing on ESXi 7.x

Post by brlomba1 »

Hello all

We recently went to ESXi7 and vcenter 7. We've been having intermittent snapshot issues affecting both Replication & Backups. They generally go away after we vmotion a VM and/or reboot the hosts, but it has gotten worse lately. We have an open case with VMware. Anyone seeing this? We are hitting this in 2 different datacenters running on Nimble & UCS and Nimble & Dell.
PetrM
Veeam Software
Posts: 3224
Liked: 518 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

[MERGED] Re: Snapshots failing on ESXi 7.x

Post by PetrM »

Hello,

I'd recommend to continue the investigation with VMware support team as creation or removing snapshot is managed at the level of hypervisor, backup application only plays the role of requestor.

Thanks!
Andreas Neufert
VP, Product Management
Posts: 6699
Liked: 1400 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

[MERGED] Re: Snapshots failing on ESXi 7.x

Post by Andreas Neufert »

Do you use the Nimble together with our Storage Integration? Then it is important that the access policy of the Nimble volume is set to "Volume only" and not not "Snapshot and volumes".
As well we have many ten thousands of customers running on ESXi7 and nothing poped up in that direction more frequently.
regi
Enthusiast
Posts: 53
Liked: 22 times
Joined: May 10, 2010 7:21 am
Full Name: Regi Pradeeswaran
Contact:

[MERGED] Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-32500,

Post by regi »

Hi backup or replication some VM's started to fail after the very latest V10 patch 10.0.1.4854 (My VMware ESXi is 7.0.0, 16324942)
This is totally random 3 backups were ok & same 3 failed on replication 2 days ago but all are good on todays backup/replication
Today 1 VM replication is failed but I moved this VM to a different host and re ran job all are ok
So it's totally random. I believe it all started after build VBR 4854 update & nothing is been changed on VMW side
Please help/advise

Error
Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-32500, timeout 1800000, snName VEEAM BACKUP TEMPORARY SNAPSHOT, snDescription Please do not delete this snapshot. It is being used by Veeam Backup., memory False, quiesce False
Error: An error occurred while saving the snapshot: Error.

Veeam Support - Case # 04403091
regi
Enthusiast
Posts: 53
Liked: 22 times
Joined: May 10, 2010 7:21 am
Full Name: Regi Pradeeswaran
Contact:

[MERGED] Snapshots failing on ESXi 7.x

Post by regi »

Hi I'm having the same issue. Updated to ESXi 7 months ago & it was running fine all this time & I updated to latest VBR patch last week and im having this issue (My setup is HPE G10's with Pure SAN AFA) Never seen any like this.

vmware-vsphere-f24/failed-to-create-vm- ... 69508.html
regi
Enthusiast
Posts: 53
Liked: 22 times
Joined: May 10, 2010 7:21 am
Full Name: Regi Pradeeswaran
Contact:

Re: Snapshots failing on ESXi 7.x

Post by regi »

Natalia Lupacheva
Veteran
Posts: 1143
Liked: 302 times
Joined: Apr 27, 2020 12:46 pm
Full Name: Natalia Lupacheva
Contact:

Re: Snapshot error

Post by Natalia Lupacheva »

HI Regi,

I've moved your post to this thread to keep the discussions on this issue in one place.
Also, did the suggested workaround (reboot) work for you?

Thanks!
regi
Enthusiast
Posts: 53
Liked: 22 times
Joined: May 10, 2010 7:21 am
Full Name: Regi Pradeeswaran
Contact:

Re: Snapshot error

Post by regi »

Morning like I said reboot did not help. The only way is to VMotion VM & retry or reboot ESX host or services.sh restart on host
Can I reopen this case Veeam Support - Case # 04403091 or a new case for me for this issue ?

All failing VM's are on the same ESX host (It was a different ESX host last week & I have rebooted) will try & reboot the problem host today
(Maybe an issue with VMW 7 some sort of memory issue after installing/upgrading VMW 7 for 80 days & need a servicers restart or reboot. I upgraded all the hosts to VMW 7 80 days ago)

It got worse & never had a single issue with Veeam BR

I have opened a support case with VMW last night waiting for an update
regi
Enthusiast
Posts: 53
Liked: 22 times
Joined: May 10, 2010 7:21 am
Full Name: Regi Pradeeswaran
Contact:

Re: Snapshot error

Post by regi » 2 people like this post

Hi had a zoom meeting with VMW re this issue. They advised that they V 7 very new & still having some heap memory issues which will be fixed in the next patch. They asked me to VMotion VM's & reboot all ESX hosts & monitor for the next few days. Will update.
regi
Enthusiast
Posts: 53
Liked: 22 times
Joined: May 10, 2010 7:21 am
Full Name: Regi Pradeeswaran
Contact:

Re: Snapshots failing on ESXi 7.x

Post by regi » 1 person likes this post

Update: Just finished rebooting all esx hosts & re-ran all failed backups/replications from last night all are good :mrgreen:
Hopefully I'm sorted for another 80 days :D I'm sure VMW will patch this issue very soon

Cheers
Natalia Lupacheva
Veteran
Posts: 1143
Liked: 302 times
Joined: Apr 27, 2020 12:46 pm
Full Name: Natalia Lupacheva
Contact:

Re: Snapshots failing on ESXi 7.x

Post by Natalia Lupacheva »

Hi Regi,

So, the host reboot is the only available workaround for now.
thanks a lot for the follow-up and sharing the details! Appreciate it.

Thanks!
regi
Enthusiast
Posts: 53
Liked: 22 times
Joined: May 10, 2010 7:21 am
Full Name: Regi Pradeeswaran
Contact:

Re: Snapshots failing on ESXi 7.x

Post by regi »

Morning Natalia yes this is the only workaround currently according to VMW. If you can't do VMotion you can issue services restart using SSH
services.sh restart but VMW said it's better to reboot ESX host. Hope VMW will fix this memory heap issue in the next release. Have a good day

/R
regi
Enthusiast
Posts: 53
Liked: 22 times
Joined: May 10, 2010 7:21 am
Full Name: Regi Pradeeswaran
Contact:

Re: Snapshots failing on ESXi 7.x

Post by regi » 1 person likes this post

Update FYI: The VMW patch will be released to address this issue by Nov according to VMW support
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Snapshots failing on ESXi 7.x

Post by Gostev »

So this patch is not included in vSphere 7 Update 1?
regi
Enthusiast
Posts: 53
Liked: 22 times
Joined: May 10, 2010 7:21 am
Full Name: Regi Pradeeswaran
Contact:

Re: Snapshots failing on ESXi 7.x

Post by regi » 1 person likes this post

Reply from VMW:

Hello Regi,

The version 7.0 U1 is not yet released, and the issue would be fixed in that.

Regards,
Technical Support Engineer
Global Support Services, VMware Inc.
chsuscale
Influencer
Posts: 11
Liked: never
Joined: Jan 09, 2020 9:21 am
Contact:

Re: Snapshots failing on ESXi 7.x

Post by chsuscale »

anyone here can confirm the problem is gone with 7.0.1.
I've got a server running 7.0.1, 17551050 which is giving me:

Code: Select all

" 17.02.2021 05:41:46 :: Failed to create VM snapshot. Error: CreateSnapshot failed, vmRef vm-116164, timeout 1800000, snName VEEAM BACKUP TEMPORARY SNAPSHOT, snDescription Please do not delete this snapshot. It is being used by Veeam Backup., memory False, quiesce False "
 "An error occurred while saving the snapshot: A digest operation has failed."
Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Google [Bot] and 28 guests