Veeam 6.5 - hangs while removing snapshot

[maverick964_uk]

We have veeam 6.5 update 3 installed. Veeam server is a VM as well.
Vcenter 5.1

On adhoc days we get a backup that hangs. The reason for the hang is that one VM will be still be in the state of "Removing snapshot" in veeam. The one we have now has sat there for over 6hours. Another we've seen has sat there for 59hours!!

there is no indication or alert anywhere as to why this happens. Its not on a particular VM either.

sometimes you can remove the snapshot in vcenter but even then Veeam does not see that its been removed.
Stopping and starting veeam services doesnt help. You have to restart the veeam server.

Could someone help to point where we can troubleshoot this and how we get round it.

I would expect Veeam to have some intelligence that it would timeout on the snapshot removal and move to the next VM.
We go into system testing next week so would like to resolve or offer a workaround.

From the programdata backup logfile we see "could not release disk" but nothing else. I'm assuming veeam is just waiting on ESXi response??

thanks

[veremin]

Have you tried to manually reproduce this situation? You can take the snapshot of your production VM manually, keep the snapshot open for long enough time before deleting it, similar to time it takes to backup the VM, and see if you experience this issue or not.

If not, kindly contact our support team and let them investigate it directly.

Thanks.

[maverick964_uk]

thanks for the reply.

Unfortunately it hangs on random vm's so hard to reproduce.
My thoughts here is that Veeam needs to deal with this situation better.
I would expect a timeout period....

Can we debug this in any way??

[veremin]

In fact, it’s hard to say without taking a look at corresponding logs. Thus, I’ve suggested contact our support that is always ready to assist you in this.

Thanks.

[maverick964_uk]

issue could be that we are running vsphere 5.1 with NO patches!....

I've advised that the customer gets this installed as vanilla 5.1 has alot of problems/issues...

[maverick964_uk]

After further investigations into the veeam logs I see the following:

TryReleaseDevice DynLinkApi.....error
could not release disk error -536870347

My thoughts is that via hotadd transport mode, veeam is having trouble releasing the disk.

We cant kill the job so I reboot the veeam server.
Once it comes up, it cleans up and removes the snapshot......seen in vcenter....

any ideas??

[Gostev]

Hi Mike, please include the Veeam support case ID for this issue. As explained when you click New Topic, support case ID is required when posting about technical issues. Thanks!

[veremin]

My thoughts is that via hotadd transport mode, veeam is having trouble releasing the disk.

Then, it might be worth testing manually ISCSI Hot Add functionality and seeing whether this issue can be reproduced without Veeam presence. Thanks.