ESX 5.5: An error occurred while consolidating disks

VMware specific discussions

Re: ESX 5.5: An error occurred while consolidating disks

Veeam Logoby Peejay62 » Mon Sep 22, 2014 11:53 am

We run using Vcenter connection. It will be too much effort I am afraid to connect to ESXi host directly because I don't know where or when on what host it will happen... Probably I can make a new job, picking a specific host and select all the vms running on that one. Then I have to sit and wait to see if the consolidations occurs. That might just be the case. You gave me a good idea to dig in and find the root cause for this, thanks

Peter
Peejay62
Expert
 
Posts: 171
Liked: 21 times
Joined: Tue Aug 06, 2013 10:40 am
Full Name: Peter Jansen

Re: ESX 5.5: An error occurred while consolidating disks

Veeam Logoby Vitaliy S. » Mon Sep 22, 2014 12:02 pm

Yes, I was referring to adding the "problematic" host as a standalone one via IP address and then run a couple of test jobs to see if this issue can be reproduced or not.
Vitaliy S.
Veeam Software
 
Posts: 19539
Liked: 1097 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: ESX 5.5: An error occurred while consolidating disks

Veeam Logoby seadave » Sat Jan 17, 2015 6:58 am

I've also had this error. We've been running Veeam 8.0.0.917 for about 3 months without issue. This week we used a mail archiving tool to export a large number of email messages ~1.5M (150G) to a file share. A few days after this, when Veeam snapshot our mailbox server, it filled up the LUN it was on. The LUN had quite bit of free space ~500G, but the VM is 2.2TB so possible it needed more than that. Awoke to find VM hung wanting to retry a file operation. I first expanded free space and then selected "Retry" in vCenter. Wouldn't work so I choose Continue. VM was still hung so needed reboot. It came back up fine. Realized later in the day the VM still needed snapshot consolidation. I attempted one and I got the error:

An error occurred while consolidating disks: 9 (Bad file descriptor).

Realized later that the consolidation process kept cycling during the day. I've never seen that happen before. It appeared like it knew there was a problem and just kept running sometimes failing and sometimes succeeding until all deltas were processed. It was weird. Finally resolved itself right after I had paid for a $1200 afterhours VMware support credit. I couldn't believe it.

Later that night I manually ran Veeam against that VM again and again I got the error after the backup had completed successfully and vCenter was attempting to consolidate the snapshot. This time it only took four attempts before it was resolved. I think I might be getting hit by the 12s bug. Not sure. Wondering about what kind of risk this involves moving forward. I think it might be time to build a clean mailserver and migrate mailboxes to free up the whitespace in the old one.
seadave
Enthusiast
 
Posts: 35
Liked: 6 times
Joined: Thu Oct 30, 2014 12:43 am

Re: ESX 5.5: An error occurred while consolidating disks

Veeam Logoby foggy » Mon Jan 19, 2015 2:09 pm

Looks like Snapshot Hunter tries to consolidate snapshots. You can ask support for logs review to identify whether everything works as expected.
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: ESX 5.5: An error occurred while consolidating disks

Veeam Logoby dellock6 » Mon Jan 19, 2015 3:58 pm

You can quickly look in history as explained in the blog post linked by Alexander and search for snapshot hunter activities, just to be sure.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: ESX 5.5: An error occurred while consolidating disks

Veeam Logoby ken.wilson » Wed May 27, 2015 1:57 pm

award@kahnlitwin.com wrote:I had this problem and found that the Veeam server still had the affected VM's disk mounted. I removed the disk from the Veeam server and cosolidated without problems.


This was my issue as well with one of my VM's. One of my Proxy's still had the disk added but it's strange the job log didn't show an issue and was successful. Either way once the disk was removed from the proxy I was able to consolidate the disk.
ken.wilson
Service Provider
 
Posts: 27
Liked: 2 times
Joined: Mon Sep 26, 2011 2:28 pm
Location: Toronto, Canda
Full Name: Ken Wilson

Re: ESX 5.5: An error occurred while consolidating disks

Veeam Logoby rafael.gonzalez » Mon Jun 01, 2015 5:53 pm 2 people like this post

Ok, so I was hitting the same issue - a particularly busy test Exchange VM was successfully backing up, but would bomb out on the consolidation step - so I always had the yellow bang on the VM. The only way to remedy that was to power off the VM and delete snap/consolidate, then power back on. Not good.

I spoke to veeam (support case ID 00835299) and vmware support over several days, and the word I got was that in our scenario, the situation just could not be dealt with with vanilla settings - so - they sent me this article:

http://kb.vmware.com/selfservice/micros ... Id=2082886

(Article number is 2082886 in case forum hoses the link.)

The short version is that I had to change a setting for our problem VM that allowed it to remain in an "unresponsive state" for the duration needed to consolidate the disks. (I used the PowerCLI option to target the one VM but didn't alter the time value under "Additional Info").

Well - I'm happy to report that the initial results look good, and the backups appear to be completing.

It's not a perfect solution, but until then - it increased joy.
rafael.gonzalez
Lurker
 
Posts: 1
Liked: 2 times
Joined: Wed Oct 08, 2014 12:51 am
Full Name: Rafael Gonzalez

Re: ESX 5.5: An error occurred while consolidating disks

Veeam Logoby iwik » Tue Sep 29, 2015 8:00 am 1 person likes this post

Hi, I hit same problem. My solution was to shutdown vm and run consolidation. It was successful when vm was not running.
iwik
Influencer
 
Posts: 16
Liked: 2 times
Joined: Wed Apr 02, 2014 11:10 am

Re: ESX 5.5: An error occurred while consolidating disks

Veeam Logoby dharmon » Thu Nov 19, 2015 4:10 pm

My fix: Even though none of my proxy servers, including the backup server, showed that they had a drive mounted for a backup or replication, once I stopped the backup server and proxy servers, the consolidation proceeded without error. I turned the backup server and proxy servers back on afterward.
dharmon
Lurker
 
Posts: 1
Liked: never
Joined: Thu Jun 25, 2015 2:05 pm
Full Name: David Harmon

Previous

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 35 guests