Comprehensive data protection for all workloads
Locked
ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Virtual Appliance Mode - Snapshots Cleanup

Post by ptmartin »

Hi All,
Is anyone else having problems with Virtual Appliance mode? It seems that the virtual hard disks are not removed on the virtualized Veeam backup server once the backup completes. The VSphere client reports that consolidation helper is trying to do its job but cannot. The only way I've found to correct the problem is to manually delete the additional hard disks from the Veeam virtual server, manually create another snapshot of the VM that was being backed up in the VSphere client, then "remove all" snapshots from Snapshot Manager. This seems to kick in the colsolidation helper again (like it was froze previously), and allows the snapshots to clean up properly.

Don't get me wrong, I really think this is going to be aswesome once this issue is worked out. I saw one server backup go from a couple of hours to complete to about 15 minutes.
I'm really excited about it! :D

So is anyone else seeing anything similar or are there any known workarounds?

Thanks in advance,
Paul
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Gostev »

Hello Paul, this issue with virtual appliance mode was fixed in 4.1.2 and later... if you have this issue with v5, please contact our support for investigation and let me know your support case number. Thanks!
ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by ptmartin »

Thanks Anton - It is version 5 (enterprise if that matters) and my ticket number is #540760.
JohnG
Novice
Posts: 8
Liked: never
Joined: Aug 26, 2010 1:23 pm
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by JohnG »

I had a similar issue with VA mode. My VM vmdks would become 'locked' and 'read only' and backups would fail, in addition to the snapshot issue you mentioned. This seemed to happen especially if the Veeam box had restarted in the middle of a backup. The response I received from Veeam support was to do the exact same thing you did to get yours working. Since we have our servers auto-restart after updates are applied, it would be cumbersome to deal with this after every update. My workaround was to ditch VA mode and stick with San Access Mode. And now Veeam automatically retries to backup the VM it was backing up, before the restart.
Bunce
Veteran
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Bunce »

We've also started experiencing similar issues with a few VM's on V5 that we never had with V4.. Snapshot removal fails and disks remain mounted on the Veeam VM for one of the larger VM's being backed up (has 4 large VMDK's)

Manual creation and removal of snapshots throuhg VCenter works fine on those (and all) VM's.

Doing some more testing to see if I can nut out the commonalities.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Vitaliy S. »

JohnG wrote:This seemed to happen especially if the Veeam box had restarted in the middle of a backup
Well, that explains... Dismount and remove snapshot API calls are made at the end of the job, so if you happen to interrupt the job in the way described above, it is expected to see such a behaviour due to vStorage API (Hot Add mode) design. If I were you I would probably disable auto-restart functionality for this particular VM.
Bunce wrote:We've also started experiencing similar issues with a few VM's on V5 that we never had with V4.. Snapshot removal fails and disks remain mounted on the Veeam VM for one of the larger VM's being backed up (has 4 large VMDK's)
Andrew, I would suggest opening a support call with our technical team to help you with troubleshooting.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Gostev »

Bunce wrote:We've also started experiencing similar issues with a few VM's on V5 that we never had with V4.. Snapshot removal fails and disks remain mounted on the Veeam VM for one of the larger VM's being backed up (has 4 large VMDK's)
Please also send us the logs for this job. Devs actually confirmed that there were some changes in functionality that handles stuck/locked snapshots due to virtual disks still being mounted, so this time I will not say "this functionality did not change since previous release" ;)
jftuga
Enthusiast
Posts: 26
Liked: 2 times
Joined: Oct 22, 2010 7:02 pm
Full Name: John Taylor
Location: Athens, GA
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by jftuga »

I also have this problem and am awaiting a call back from Veeam Support. In the mean time, has anyone successfully resolved this problem? We are using Veeam 5 Enterprise.

Thanks,
-John
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Gostev »

All, thanks to all the logs files provided earlier we have found the issue, and it is indeed specific to v5 and is resulted by some changes made in v5 logic around stuck disk dismount and stuck VMware snapshots handling. We will produce the hotfix, if you are affected please open support case to be notified on hotfix availability. Thank you.
jftuga
Enthusiast
Posts: 26
Liked: 2 times
Joined: Oct 22, 2010 7:02 pm
Full Name: John Taylor
Location: Athens, GA
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by jftuga »

So what are we supposed to do in the mean time? (until the hotfix is available)

How can we manually remove the snapshots? Trying this trhough the vSphere client does not work.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Gostev »

jftuga wrote:So what are we supposed to do in the mean time? (until the hotfix is available)
Well, it is not like this will be happening for every VM and everytime, only for some VMs and in certain conditions (such as rebooting backup server while the job runs). We did not even have this logic of handling hotadd issues until version 4.1.2 (released 3 months ago), so the fact that this logic does not work reliably right now is hardly a showstopper. You can just continue to run normally while monitoring your VMs. But if you are really concerned, you can always use any backup mode other than Virtual appliance, until the hotfix is available.
jftuga wrote:How can we manually remove the snapshots? Trying this trhough the vSphere client does not work.
This is described in the first post of this topic.
ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by ptmartin »

Well I sent my logs to support 2 days ago and still have not heard anything from them about what they discovered or what I should do. Thanks for that I guess ... Hopefully they do not forget to send the hotfix when it is available too.

I have found a workaround for my environment. The machine that seems to have been causing the issue is a large VM with 2 vhd's - the d-drive is 255GB. The solution for me was to change this machine back to backing up via network mode and leave the rest in appliance mode. It seems ok for the last couple of days but obviously not fixed. I had thought it may have something to do with the VM datastore's block size but other VMs in this datastore have not had this issue. All that I am left thinking is that the size of the machine itself must be the issue.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Gostev »

Unfortunately, we do not know the actual issue for sure, since it sits in vStorage API (which is not our code). Our logic simply works around this HotAdd (or, rather, HotRemove) issue. Essentially, we remove the disks ourselves in those rare cases when we detected they were not removed by vStorage API. We just need to fix this functionality now :) actually, the fix is in testing since yesterday.
ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by ptmartin »

Got the fix today and just installed it. Now when I attempt to start a backup job the software freezes and reports that it is "not responding." Calling support again ...
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Gostev »

Our support distributes cumulative patches only, and the current patch addresses around 20 issues, so freeze might not be related to this specific issue fix. A few fixes in this patch require additional things to be done during fix installation, if you missed this there could be problems. Support should be able to trace the issue down quite easily.
ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by ptmartin »

As previously stated, I applied the hotfix and my server froze when I attempted to run a backup. I restarted the server and tried again after verifying that services all started properly as is customary whenever a server is restarted. The results were the same and I contacted support. The person I spoke with said someone else was working on the case and he asked the tech I was speaking with to have me send him my logs again, which I did. I waited patiently for someone at support to contact me so we could work on the issue. At 2pm I received an email stating that I would be contacted soon. Finally just before the end of the business day I restored the Veeam directory from my vRanger backup server and picked out the files that were changed by this hotfix and returned my configuration to an operational state so I could have my backups run last night.

Today I get an email asking me to check and see if the Veeam services are running now after applying the hotfix. Are you kidding me?? Do you not think that I would make every attempt to get my backups working yesterday before they were needed? Is this the level of support I am to expect whenever there is an issue? If so I think you should reconsider your pricing model because it isn't worth it. This forum provides much more support of the product than I've received anywhere else.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Gostev »

Hi, Paul. I can confirm that this is not "expected", typically people are very happy with our support. Right now there are pretty special times, and our support is a quiet busy (we had 20K product downloads in 3 weeks, there is immense interest to this new version). So, they may not be responding immediately. But things are slowly going back to normal already anyway.

I can have our support management call you and discuss this situation, if you'd like to submit your feedback directly? Just let me know.
ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by ptmartin »

I believe I said what I needed to. I would like support to call me when they are in a position to work on the issue with me. The few emails I've received leave more questions than answers.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Gostev »

Sure, that goes without saying, I just thought you may want to talk to someone senior from support, since I cannot really take action and do anything with your feedback above (I am not involved with technical support). Thanks!
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by rfn »

I'm having major problems with this as well... It can run well for days and then suddenly something happens and a replication fails and snapshots are left on the VM and VMDK's aren't removed from the VBR VM. I'm replicating some VM's each hour and since we only have around 10 VM's on our EqualLogic PS4000VX (15K SAS) then there should be enough resources to handle that. We're replicating to a PS4000E (SATA) over gigabit LAN.

When it's running then all is well and it doesn't seem that it's stressing the EqualLogic, so I don't understand why this happens. Today it was our SQL Server that was hit - it has happened before - and it's running with two vRDM disks. Maybe it's hit harder because of this but our Dynamics NAV clients were hanging all over the place because of this error and it wasn't very popular :cry:

I will open a support case soon.
Bunce
Veteran
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Bunce »

I also received hotfix today, and now jobs hang on startup - GUI is unresponsive (64 bit version). Have to kill process via task manager.

Updated logs have been sent: #542064

Cheers,
A
ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by ptmartin »

I just finished working with Veeam support and am now working again with the newest hotfix. The underlying issue according to the tech was that when the hotfix was downloaded and extracted, Windows involked a security measure to 'lock' the individual files that were in the hotfix. The tech individually 'unlocked' each file (right-click - properties - unlock) and reapplied the hotfix successfully. The backup is now running. Upon completion, I will change the backup method to virtual appliance mode and update this forum upon completion. I assume given the time that this will be in the morning. The hotfix is being rewritten to take into consideration the Windows security issue that was discovered today. More to follow ...
bshubinsky
Novice
Posts: 8
Liked: never
Joined: Oct 13, 2010 1:04 pm
Full Name: Boris Shubinsky
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by bshubinsky »

Just a heads up I got a file today around noon for this hotfix. I'm also having the freezing issues on x64-bit. I was asked to reboot and the problem still persists, I'll have to look into the file locking to see if that fixes it.
Bunce
Veteran
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Bunce »

Unblocking each file individually and then recopying over worked for me. Thanks ptmartin.

A
ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by ptmartin »

I am happy to report that with all of the machines in Virtual Appliance mode (except Veeam backup VM - this is in Network mode) I had 100% successful backups from all machines and fairly decent throughput. I'm satisfied with this and am quite ready to announce that this issue is finally resolved for me. Thanks to all who helped and mostly thanks to those that finally listened.

Veeam Backup and Replication is an amazing accomplishment!! I won't say that I am sorry for the success of this release of software Anton ("we had 20K product downloads in 3 weeks, there is immense interest in this new version") and I don't think that you are either. Forgive me but I don't feel any sympathy for you. This is realy great news and not an excuse for lack of product support. I hope that there is a lesson learned that can be pulled from this where support is concerned. When it becomes evident that they cannot keep up with the demand, they either need to step it up or get additional help. Those of us that are using the software have an obligation to ensure that the backups are successful. The urgency just was not there and it was painfully obvious. We all needed to step up on this one and once we did, the results are undeniably a success.

Bunce - Thank you very much and you are most welcome! I'm glad I could help.

I am going to try to set up SureBackup today and see what that can do in my environment. I hope to apply all that I've learned and be able to achieve the same successful backup results with little to no trouble.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by Gostev »

Glad it works well for you now, thanks for updating!
ptmartin wrote:lack of product support
Well, I would not say that - I believe, all responses were still under SLA per your support agreement, and the issue is resolved in just under 8 days, which involved not just actual investigation and troubleshooting, but also releasing and testing actual hotfix. Does it show lack of support, or does it prove otherwise? I heard stories about those gorilla size software vendors where same thing can take 9 months.

What I really meant to say above when referring to support load, is that it is definitely higher than during other 11 months of the year, and it is hard to expect instant replies/solutions from support immediately after major release happens. Plus, most new issues for newly released versions are unknown, and must go through development for inverstigation. And this requires extra time.

I honestly think that our support is great, I see very little negative feedback but a lot of positive one... and I sincerely hope that down the road, your experience with our support will only become better and better.
tessi
Lurker
Posts: 1
Liked: never
Joined: Dec 20, 2010 11:03 pm
Full Name: t w
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by tessi »

Hello,

I have also received the error:

RemoveSnapshot failed, snapshotRef "snapshot-79938", timeout "3600000"
Unable to access file <unspecified filename> since it is locked

Using version 5.0.0.179 - 64bit.

How can I obtain this hotfix?

-Tessi
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Virtual Appliance Mode - Snapshots Cleanup

Post by tsightler »

I would suggest you download and upgrade to 5.0.1, which includes the above mentioned fixes.
Locked

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], legil.miguel and 234 guests