Availability for the Always-On Enterprise
Post Reply
CamCamJelly
Novice
Posts: 6
Liked: 1 time
Joined: Oct 29, 2013 2:44 pm
Contact:

Veeam snapshots causing VMs to shutdown

Post by CamCamJelly » Oct 29, 2013 3:08 pm

I have been running Veeam backup and replication jobs successfully without error for the past year. I upgraded to v7 and had no problems with my jobs until the end of last week. I have a handful of VMs that are getting shut off when Veeam puts a backup snapshot on the file at the beginning of the job. The error I get on veeam is

Code: Select all

10/29/2013 2:55:39 AM :: Error: An error occurred while restarting virtual machine after taking a snapshot. The virtual machine will be powered off.
If you try to power the VM on again you get an error on VCenter saying memory cannot be allocated. The only way to remedy this is do a vmotion migration to another host. After that I have to remove the Veeam snapshot and do a consolidation to get rid of the extra VMDK file.

The error on the Vcenter side states "An error occurred while restarting virtual machine after taking a snapshot. The virtual machine will be powered off.
Reason: 0 (Cannot allocate memory).
Cannot open the disk '/vmfs/volumes/516739fd-f3210fd7-682d-90b11c18974e/MyVM/MyVM-000001.vmdk' or one of the snapshot disks it depends on."

I can create snapshot manually on Vcenter without any trouble. I'm on ESXi 5.0 currently on all my hosts.

Case #00468130

tsightler
Veeam Software
Posts: 5209
Liked: 2088 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by tsightler » Oct 29, 2013 3:20 pm 1 person likes this post

Based on the description this is almost certainly caused by the well known VMFS Heap size limitations. Take a look at this VMware KB article:

http://kb.vmware.com/selfservice/micros ... Id=1004424

You might want to limit the number of parallel process as I'm guess that when mulitple VMs on the same host are being backed up this is triggering the issue. It's also likely why it's happening at the start up of the job. Once a few of the VMs are stopped there's enough VMFS heap for the rest.

The KB article also has several workarounds and possible options assuming you are on recent patch levels.

CamCamJelly
Novice
Posts: 6
Liked: 1 time
Joined: Oct 29, 2013 2:44 pm
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by CamCamJelly » Oct 29, 2013 3:30 pm

So according to that KB article the min heap size for ESXi 5.0 Patch 5 (1024429) or later the min VMFS3 heap size is 256. I've looked on my hosts which are ESXi 5.0 (1311175) the minheapsizeMB for VMFS3 is 255. That one MB difference could be making the difference on the jobs failing?

I'm not worried about the parallel processes. I only run one job at a time.

tsightler
Veeam Software
Posts: 5209
Liked: 2088 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by tsightler » Oct 29, 2013 3:35 pm

So did you not enable parallel processing after the V7 upgrade? This option runs multiple backups in a single job.

I believe the article mentions two changes you need to make to reserve the maximum heap with 5.0.

tsightler
Veeam Software
Posts: 5209
Liked: 2088 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by tsightler » Oct 30, 2013 1:54 am 1 person likes this post

My understanding is that, with the proper patch level, MinHeapSizeMB can be 255, while MaxHeapSizeMB should reflect 640. If your host were upgraded from previous versions then the older, lower limits will be in place. I don't have any 5.0 hosts in my lab any more, but my 5.1 host show MaxHeapSizeMB to be 640, and I believe that 5.0 with recent patches should also reflect this maximum value. Perhaps you can post a screenshot of your advanced parameters. Note that this does not eliminate the possibility of the problem occurring, and I can't guarantee this is the issue, but I think it is likely as this is a fairly common issue especially if your hosts are running VMs with large amounts of storage. Also having snapshots open increases heap use.

ESXi 5.5 has significantly reworked VMFS heap handling that will hopefully address this issue so once Veeam has support for 5.5 hopefully these issues will be largely resolved at least until users starting hitting new limits.

CamCamJelly
Novice
Posts: 6
Liked: 1 time
Joined: Oct 29, 2013 2:44 pm
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by CamCamJelly » Oct 30, 2013 2:14 pm 1 person likes this post

I increased the heap size on my hosts and set up the parallel processing. My jobs got through fine last night. I will keep a eye on it for the next few days to see if the problem reoccurs. Thank you for the help!

tsightler
Veeam Software
Posts: 5209
Liked: 2088 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by tsightler » Oct 30, 2013 2:19 pm

Excellent, please let us know if you continue to experience problems.

CamCamJelly
Novice
Posts: 6
Liked: 1 time
Joined: Oct 29, 2013 2:44 pm
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by CamCamJelly » Nov 01, 2013 2:31 pm

I unfortunately had the same errors as before last night. I seem to be getting the errors when I have application-aware image processing enabled. If I have that off the jobs seem to go through fine. From what I've been told I want that option on when doing SQL server and exchange jobs?

v.Eremin
Veeam Software
Posts: 15397
Liked: 1163 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by v.Eremin » Nov 01, 2013 2:36 pm

From what I've been told I want that option on when doing SQL server and exchange jobs?
Yep, AAIP is required for backing up/replicating VMs that run special MS applications such as, Exchange, AD, SQL, Sharepoint, etc.

This options allows VB&R to create transactionally consistent backup of a VM running VSS-aware applications without powering them off. It also ensures successful VM recovery, as well as proper recovery of all applications installed on the VM without any data loss and also it notifies applications about them being backed up.

However, I doubt that this functionality is related to heap size anyhow.

Thanks.

CamCamJelly
Novice
Posts: 6
Liked: 1 time
Joined: Oct 29, 2013 2:44 pm
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by CamCamJelly » Nov 01, 2013 2:56 pm

I looked on the Veeam log to see what the error was and saw this. It looks a little differnt that the one I posted in the OP

Code: Select all

Unable to release guest. Error: RPC function call failed. Function name: [GetFreezeState]. Target machine: [My Server]. RPC error:The remote procedure call failed. Code: 1726
Error: An error occurred while restarting virtual machine after taking a snapshot. The virtual machine will be powered off. 

tsightler
Veeam Software
Posts: 5209
Liked: 2088 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by tsightler » Nov 01, 2013 3:04 pm

Yeah, I don't see any way AAIP could possibly be involved in this issue, it interacts with the guest OS, not the host. The error you posted above would be expected if VMware forced a shutdown of the machine due to lack of VMFS heap as we would be unable to release the guest freeze.

You should certainly feel free to open a case with VMware or Veeam (or both) so that they can look at the logs, but I'm still about 99.995% sure this is a VMFS heap issue. I'd still be interested in seeing a screenshot of your heap settings just because I don't have ESXi 5.0 anywhere and can't find a screenshot online so I'm curious if recent patches make it look the same as 5.1.

CamCamJelly
Novice
Posts: 6
Liked: 1 time
Joined: Oct 29, 2013 2:44 pm
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by CamCamJelly » Nov 05, 2013 3:14 pm

After talking with Veeam support they suspected that the proxy server I was using was causing the problem. The logs make it look like that it didn't get updated properly when I went from v6 to v7. I disabled the proxy server and have had 2 nights of successful jobs with AAIP on. I'll keep an eye on it for the next few days.

v.Eremin
Veeam Software
Posts: 15397
Liked: 1163 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam snapshots causing VMs to shutdown

Post by v.Eremin » Nov 05, 2013 3:25 pm

Have you also tried to reinstall proxy components by re-adding this proxy server? Thanks

Post Reply

Who is online

Users browsing this forum: No registered users and 37 guests