VM disk consolidation failure

[zak2011]

Just got VM disk consolidation failure messages for my exchange vm and sharepoint vm.
The following link explains how to consolidate snapshots.
http://kb.vmware.com/selfservice/micros ... Id=2003638

However after doing this I got the error message "unable to access file since it is locked consolidate".
The following article describes why it happens and how to resolve this.

http://kb.vmware.com/selfservice/micros ... Id=2033537

This is the first time I got this issue in my environment. It was a bit surprising to see the virtual hard drives which Veeam was using was also of exchange and sharepoint. However after removing them manually, consolidating snapshots worked.
Any idea as to why Veeam locks on to other VMs hard disks and does not release it? Is this something unique to v7?

Thanks

[Vitaliy S.]

Hi Arun,

No, it's not. What do you mean by using? Were these disks added to the proxy server? Are you using HotAdd backup mode?

Thanks!

[zak2011]

Hi Vitaliy,
Its been happening now for a couple of other VMs also. Seems like the proxy is going to hot add mode.
The job statistics say
9/9/2013 10:44:38 PM :: Using backup proxy This server for disk Hard disk 3 [hotadd]

and then 9/9/2013 10:46:04 PM :: Hot add is not supported for this disk, failing over to network mode (Hard disk 3)
However when I go to the Veeam virtual drives, I can see the hard drives of the other vms are still mounted and not released.
I have to go to the Veeam vm , edit settings and manually remove the mounted drives of the other vms.

[foggy]

Have you reviewed the possible reasons of why hotadd is not supported? Anyway, I would contact support, since disks should definitely be released after the job.

[zak2011]

@foggy- thanks for the KB. I went through them, but does not apply in my environment though. I have logged a case for this
Case#00441952.
Just for information..never had this in v6.5.

[zak2011]

Initially support asked me to turn off hotadd mode for all proxies, keep 'enable parallel processing' enabled and enable only network mode. When hotadd mode and enable parallel processing are enabled, I am getting this error. Support says there was a problem with one proxy. So they have asked me to disable that proxy and reenable hot add mode.
However I have been able to use the proxy in v6.5 and it used to work.

Thanks.

[veremin]

However I have been able to use the proxy in v6.5 and it used to work.

Parallel processing hasn’t been present in version 6.5. So, this might be the reason. Thanks.

[zak2011]

Tried network mode + parallel processing with all the proxies. There was no problem.
However Hotadd+Parallel processing does not work as it immediately gives VM disk consolidation failure

[veremin]

The point was that previously existed proxy couldn’t be tested before with parallel processing option since it literally hasn’t been there. Anyway, did disabling/re-enabling stuff help to solve the issue with parallel processing not working properly with one Hot Add proxy? Thanks.

[zak2011]

disabling hotadd and enabling parallel processing works.
disabling parallel processing with hotadd enabled works.

Thanks

[veremin]

However Hotadd+Parallel processing does not work as it immediately gives VM disk consolidation failure

Is it the only proxy server that is affected by this behavior or there are many of them? Thanks.

[zak2011]

As per support investigation seems to be that it is only with one proxy server.
However I don't know what is it with just that proxy that is causing this?

Thanks

[veremin]

Providing that there is only one proxy affected, it might be worth re-installing proxy components and seeing whether it helps or not. It can be done by deleting/adding proxy server to a console of a backup server. Thanks.

[zak2011]

Thankyou Vladimir. I will do this immediately and update you.
Thanks for your support as always.

[zak2011]

I tried removing it and added it again. Got a warning while trying to add it

Code: Select all

9/20/2013 10:53:23 AM          Collecting hardware info
9/20/2013 10:53:24 AM          Detecting operating system
9/20/2013 10:53:31 AM Warning    Detecting OS version
                                 Failed to detect OS version: Failed to connect to WMI on host 'netmsrv03'.
9/20/2013 10:53:31 AM          Creating temporary folder
9/20/2013 10:53:32 AM          Uploading package VeeamTransport.msi
9/20/2013 10:53:43 AM          Installing package VeeamTransport.msi
9/20/2013 10:53:43 AM          Deleting temporary folder
9/20/2013 10:53:43 AM          Registering client VEEAMSRV01 for package Transport
9/20/2013 10:53:43 AM          All required packages have been successfully installed
9/20/2013 10:53:43 AM          Discovering installed packages
9/20/2013 10:53:44 AM          Creating database records for server
9/20/2013 10:53:44 AM          Detecting server configuration
9/20/2013 10:53:44 AM          Creating configuration database records for installed packages
9/20/2013 10:53:48 AM          Collecting disks and volumes info
9/20/2013 10:53:48 AM Warning    Microsoft Windows server added with warnings

Thanks

[foggy]

Do you still have the the same issue after adding the proxy anew?

[zak2011]

The issue is still there after adding the proxy anew.

[veremin]

Also, it might be worth understanding whether the machine in question is capable of Hot Add; in other words, whether this issue can be reproduced without Veeam presence or not. The way how SCSI Hot Add functionality can be tested manually is described here.

Thanks.

[zak2011]

Thanks Vladmir for the KB.
When the issue was occuring, if i go to the Veeam server and go to edit settings, I can see all the vmdks of those machines that were giving error message ' VM consolidation needed'.
Then i had to manually remove them from settings of the VM. So would this be related to the proxy or to the Veeam server vm.

Thanks

[zak2011]

The latest update I got from support regarding this was considering that this happened with one of the backup proxies enabled, It might be that proxy would be overloaded by parallel backup tasks and/or internal processes resulting in unexpected vddk transport failures
However as Vladimir already had recommended to remove that proxy and readd it, the issue happened just once after I did this ( however support says reinstalling the backup transport doesnt fix the issue) .After that even with this proxy server enabled, the error did not recurr again. So, I cannot be certain to say if readding the proxy fixed the issue or whether it resolved on its own.

Thanks

[zak2011]

Support has asked me to disabled this proxy as a temporary resolution. However I have requested them to take this up for further investigation with research so that we can have a concrete reason as to why it happened also for the benefit of others also who may face this problem in future.