Virtual Appliance Mode - Snapshots Cleanup

[ptmartin]

Hi All,
Is anyone else having problems with Virtual Appliance mode? It seems that the virtual hard disks are not removed on the virtualized Veeam backup server once the backup completes. The VSphere client reports that consolidation helper is trying to do its job but cannot. The only way I've found to correct the problem is to manually delete the additional hard disks from the Veeam virtual server, manually create another snapshot of the VM that was being backed up in the VSphere client, then "remove all" snapshots from Snapshot Manager. This seems to kick in the colsolidation helper again (like it was froze previously), and allows the snapshots to clean up properly.

Don't get me wrong, I really think this is going to be aswesome once this issue is worked out. I saw one server backup go from a couple of hours to complete to about 15 minutes.
I'm really excited about it!

So is anyone else seeing anything similar or are there any known workarounds?

Thanks in advance,
Paul

[Gostev]

Hello Paul, this issue with virtual appliance mode was fixed in 4.1.2 and later... if you have this issue with v5, please contact our support for investigation and let me know your support case number. Thanks!

[ptmartin]

Thanks Anton - It is version 5 (enterprise if that matters) and my ticket number is #540760.

[JohnG]

I had a similar issue with VA mode. My VM vmdks would become 'locked' and 'read only' and backups would fail, in addition to the snapshot issue you mentioned. This seemed to happen especially if the Veeam box had restarted in the middle of a backup. The response I received from Veeam support was to do the exact same thing you did to get yours working. Since we have our servers auto-restart after updates are applied, it would be cumbersome to deal with this after every update. My workaround was to ditch VA mode and stick with San Access Mode. And now Veeam automatically retries to backup the VM it was backing up, before the restart.

[Bunce]

We've also started experiencing similar issues with a few VM's on V5 that we never had with V4.. Snapshot removal fails and disks remain mounted on the Veeam VM for one of the larger VM's being backed up (has 4 large VMDK's)

Manual creation and removal of snapshots throuhg VCenter works fine on those (and all) VM's.

Doing some more testing to see if I can nut out the commonalities.

[Vitaliy S.]

This seemed to happen especially if the Veeam box had restarted in the middle of a backup

Well, that explains... Dismount and remove snapshot API calls are made at the end of the job, so if you happen to interrupt the job in the way described above, it is expected to see such a behaviour due to vStorage API (Hot Add mode) design. If I were you I would probably disable auto-restart functionality for this particular VM.

We've also started experiencing similar issues with a few VM's on V5 that we never had with V4.. Snapshot removal fails and disks remain mounted on the Veeam VM for one of the larger VM's being backed up (has 4 large VMDK's)

Andrew, I would suggest opening a support call with our technical team to help you with troubleshooting.

[Gostev]

We've also started experiencing similar issues with a few VM's on V5 that we never had with V4.. Snapshot removal fails and disks remain mounted on the Veeam VM for one of the larger VM's being backed up (has 4 large VMDK's)

Please also send us the logs for this job. Devs actually confirmed that there were some changes in functionality that handles stuck/locked snapshots due to virtual disks still being mounted, so this time I will not say "this functionality did not change since previous release"

[jftuga]

I also have this problem and am awaiting a call back from Veeam Support. In the mean time, has anyone successfully resolved this problem? We are using Veeam 5 Enterprise.

Thanks,
-John

[Gostev]

All, thanks to all the logs files provided earlier we have found the issue, and it is indeed specific to v5 and is resulted by some changes made in v5 logic around stuck disk dismount and stuck VMware snapshots handling. We will produce the hotfix, if you are affected please open support case to be notified on hotfix availability. Thank you.

[jftuga]

So what are we supposed to do in the mean time? (until the hotfix is available)

How can we manually remove the snapshots? Trying this trhough the vSphere client does not work.

[Gostev]

So what are we supposed to do in the mean time? (until the hotfix is available)

Well, it is not like this will be happening for every VM and everytime, only for some VMs and in certain conditions (such as rebooting backup server while the job runs). We did not even have this logic of handling hotadd issues until version 4.1.2 (released 3 months ago), so the fact that this logic does not work reliably right now is hardly a showstopper. You can just continue to run normally while monitoring your VMs. But if you are really concerned, you can always use any backup mode other than Virtual appliance, until the hotfix is available.

How can we manually remove the snapshots? Trying this trhough the vSphere client does not work.

This is described in the first post of this topic.

[ptmartin]

Well I sent my logs to support 2 days ago and still have not heard anything from them about what they discovered or what I should do. Thanks for that I guess ... Hopefully they do not forget to send the hotfix when it is available too.

I have found a workaround for my environment. The machine that seems to have been causing the issue is a large VM with 2 vhd's - the d-drive is 255GB. The solution for me was to change this machine back to backing up via network mode and leave the rest in appliance mode. It seems ok for the last couple of days but obviously not fixed. I had thought it may have something to do with the VM datastore's block size but other VMs in this datastore have not had this issue. All that I am left thinking is that the size of the machine itself must be the issue.

[Gostev]

Unfortunately, we do not know the actual issue for sure, since it sits in vStorage API (which is not our code). Our logic simply works around this HotAdd (or, rather, HotRemove) issue. Essentially, we remove the disks ourselves in those rare cases when we detected they were not removed by vStorage API. We just need to fix this functionality now actually, the fix is in testing since yesterday.

[ptmartin]

Got the fix today and just installed it. Now when I attempt to start a backup job the software freezes and reports that it is "not responding." Calling support again ...

[Gostev]

Our support distributes cumulative patches only, and the current patch addresses around 20 issues, so freeze might not be related to this specific issue fix. A few fixes in this patch require additional things to be done during fix installation, if you missed this there could be problems. Support should be able to trace the issue down quite easily.

[ptmartin]

As previously stated, I applied the hotfix and my server froze when I attempted to run a backup. I restarted the server and tried again after verifying that services all started properly as is customary whenever a server is restarted. The results were the same and I contacted support. The person I spoke with said someone else was working on the case and he asked the tech I was speaking with to have me send him my logs again, which I did. I waited patiently for someone at support to contact me so we could work on the issue. At 2pm I received an email stating that I would be contacted soon. Finally just before the end of the business day I restored the Veeam directory from my vRanger backup server and picked out the files that were changed by this hotfix and returned my configuration to an operational state so I could have my backups run last night.

Today I get an email asking me to check and see if the Veeam services are running now after applying the hotfix. Are you kidding me?? Do you not think that I would make every attempt to get my backups working yesterday before they were needed? Is this the level of support I am to expect whenever there is an issue? If so I think you should reconsider your pricing model because it isn't worth it. This forum provides much more support of the product than I've received anywhere else.

[Gostev]

Hi, Paul. I can confirm that this is not "expected", typically people are very happy with our support. Right now there are pretty special times, and our support is a quiet busy (we had 20K product downloads in 3 weeks, there is immense interest to this new version). So, they may not be responding immediately. But things are slowly going back to normal already anyway.

I can have our support management call you and discuss this situation, if you'd like to submit your feedback directly? Just let me know.

[ptmartin]

I believe I said what I needed to. I would like support to call me when they are in a position to work on the issue with me. The few emails I've received leave more questions than answers.

[Gostev]

Sure, that goes without saying, I just thought you may want to talk to someone senior from support, since I cannot really take action and do anything with your feedback above (I am not involved with technical support). Thanks!

[rfn]

I'm having major problems with this as well... It can run well for days and then suddenly something happens and a replication fails and snapshots are left on the VM and VMDK's aren't removed from the VBR VM. I'm replicating some VM's each hour and since we only have around 10 VM's on our EqualLogic PS4000VX (15K SAS) then there should be enough resources to handle that. We're replicating to a PS4000E (SATA) over gigabit LAN.

When it's running then all is well and it doesn't seem that it's stressing the EqualLogic, so I don't understand why this happens. Today it was our SQL Server that was hit - it has happened before - and it's running with two vRDM disks. Maybe it's hit harder because of this but our Dynamics NAV clients were hanging all over the place because of this error and it wasn't very popular

I will open a support case soon.

[Bunce]

I also received hotfix today, and now jobs hang on startup - GUI is unresponsive (64 bit version). Have to kill process via task manager.

Updated logs have been sent: #542064

Cheers,
A

[ptmartin]

I just finished working with Veeam support and am now working again with the newest hotfix. The underlying issue according to the tech was that when the hotfix was downloaded and extracted, Windows involked a security measure to 'lock' the individual files that were in the hotfix. The tech individually 'unlocked' each file (right-click - properties - unlock) and reapplied the hotfix successfully. The backup is now running. Upon completion, I will change the backup method to virtual appliance mode and update this forum upon completion. I assume given the time that this will be in the morning. The hotfix is being rewritten to take into consideration the Windows security issue that was discovered today. More to follow ...

[bshubinsky]

Just a heads up I got a file today around noon for this hotfix. I'm also having the freezing issues on x64-bit. I was asked to reboot and the problem still persists, I'll have to look into the file locking to see if that fixes it.

[Bunce]

Unblocking each file individually and then recopying over worked for me. Thanks ptmartin.

A

[ptmartin]

I am happy to report that with all of the machines in Virtual Appliance mode (except Veeam backup VM - this is in Network mode) I had 100% successful backups from all machines and fairly decent throughput. I'm satisfied with this and am quite ready to announce that this issue is finally resolved for me. Thanks to all who helped and mostly thanks to those that finally listened.

Veeam Backup and Replication is an amazing accomplishment!! I won't say that I am sorry for the success of this release of software Anton ("we had 20K product downloads in 3 weeks, there is immense interest in this new version") and I don't think that you are either. Forgive me but I don't feel any sympathy for you. This is realy great news and not an excuse for lack of product support. I hope that there is a lesson learned that can be pulled from this where support is concerned. When it becomes evident that they cannot keep up with the demand, they either need to step it up or get additional help. Those of us that are using the software have an obligation to ensure that the backups are successful. The urgency just was not there and it was painfully obvious. We all needed to step up on this one and once we did, the results are undeniably a success.

Bunce - Thank you very much and you are most welcome! I'm glad I could help.

I am going to try to set up SureBackup today and see what that can do in my environment. I hope to apply all that I've learned and be able to achieve the same successful backup results with little to no trouble.

[Gostev]

Glad it works well for you now, thanks for updating!

lack of product support

Well, I would not say that - I believe, all responses were still under SLA per your support agreement, and the issue is resolved in just under 8 days, which involved not just actual investigation and troubleshooting, but also releasing and testing actual hotfix. Does it show lack of support, or does it prove otherwise? I heard stories about those gorilla size software vendors where same thing can take 9 months.

What I really meant to say above when referring to support load, is that it is definitely higher than during other 11 months of the year, and it is hard to expect instant replies/solutions from support immediately after major release happens. Plus, most new issues for newly released versions are unknown, and must go through development for inverstigation. And this requires extra time.

I honestly think that our support is great, I see very little negative feedback but a lot of positive one... and I sincerely hope that down the road, your experience with our support will only become better and better.

[tessi]

Hello,

I have also received the error:

RemoveSnapshot failed, snapshotRef "snapshot-79938", timeout "3600000"
Unable to access file <unspecified filename> since it is locked

Using version 5.0.0.179 - 64bit.

How can I obtain this hotfix?

-Tessi

[tsightler]

I would suggest you download and upgrade to 5.0.1, which includes the above mentioned fixes.