Host-based backup of Microsoft Hyper-V VMs.
Post Reply
davidh
Lurker
Posts: 1
Liked: never
Joined: Aug 22, 2013 1:17 pm
Full Name: David Hardwick
Contact:

Veeam 7 2012 Hyper-v

Post by davidh »

Hi,

Has anyone experienced issues since upgrading to Veeam 7 from 6.5?

Environment:

Windows 2012 datacentre
2 hosts
Local storage
10gb backup network between servers
Veeam storage on iSCSI SAN

The issue appears to center around 2 Exchange VMs in a dag across both hosts.
After the Veeam 7 server was install in a fresh VM, the hosts integration components were upgraded. A initial backup was run which froze Host 1, after the reboot a subsequent VSS snapshot either by the host or backup would first freeze the Exchange1 VM, then within an hour BSOD the host. During this entire time the host would show the system process maxing out disk IO.

Host 2 would successfully run a backup, although not truncate logs. We would also see VMs freeze while the backups took place.

We removed all Veeam integration components from the hosts, turned off the Veeam 7 server, but the hosts are still exhibiting the same behavior when the VSS writer runs. We have had to disable all Volume shadow services on both hosts to stop them freezing up as they are in production.

We believe our next steps may be to reinstall the server 2012 hosts from scratch and a new Veeam 6.5, unless anyone has any ideas?

Luckily we were trialing 7 on only 2 of our 10 hosts, sadly those 2 hosts supported 150 users :/

support case 00432125, yet to hear back from escalated support.
jrfolkman
Novice
Posts: 7
Liked: never
Joined: Sep 13, 2012 3:58 pm
Full Name: Jason Folkman
Contact:

Re: Veeam 7 2012 Hyper-v

Post by jrfolkman »

Any update? This is very similar to my setup so I am interested in the result. The big issue I have with "testing" Veeam 7 is that regardless of whether you upgrade your existing Veeam server or create a new one the host integration is updated... So it's not a "zero loss" scenario if there are problems. All that being said I have upgraded Veeam (although in a VMWare environment) many times and not had a problem.

I wish we had the luxury of multiple extra Hosts but we don't so if the host integration causes issues we have big problems.

Thanks,
Jason
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam 7 2012 Hyper-v

Post by Vitaliy S. »

Jason, in order to receive quicker assistance with this issue, it's better to contact our technical support team directly. Thanks!
jrfolkman
Novice
Posts: 7
Liked: never
Joined: Sep 13, 2012 3:58 pm
Full Name: Jason Folkman
Contact:

Re: Veeam 7 2012 Hyper-v

Post by jrfolkman »

I don't need assistance I was just seeing if the other guys issue had been resolved and what they found. Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 7 2012 Hyper-v

Post by foggy »

As far as I can see from the OP's case, the issue is still under investigation.
tsuplex
Novice
Posts: 4
Liked: never
Joined: Nov 30, 2012 12:28 am
Contact:

Re: Veeam 7 2012 Hyper-v

Post by tsuplex »

I believe we're currently seeing the same or similar issue.

We were running 6.5 flawlessly on Windows Server 2012 datacenter prior to upgrading to 7.

After the upgrade, during a backup we started experiencing what we'd describe as a host lockup but the host itself remains contactable, but all VMs stop responding and the Hyper-V management console becomes unusable. I have to stop the job, stop all Veeam services on the host and force-quit-restart the Hyper-V Management service to regain control of the guest VMs.
I too am running Exchange 2013 DAG nodes on each of the two hosts that are experiencing the issue.

The issue still occurs even after disabling CBT and application aware image processing.

My case number is 00432695.

Have you made any progress?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 7 2012 Hyper-v

Post by foggy »

As far as I can see from your case, you've already moved much further in investigation than OP did, so please continue working with support. Thanks.
tsuplex
Novice
Posts: 4
Liked: never
Joined: Nov 30, 2012 12:28 am
Contact:

Re: Veeam 7 2012 Hyper-v

Post by tsuplex »

I was asked to run some backups with ODX disabled on the Windows Server 2012 hosts. The symptom hasn't re-occurred with ODX disabled.

Note that the backups that were misbehaving were local storage backups, not SAN backups.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 7 2012 Hyper-v

Post by foggy »

Well, there were some issues with ODX observed previously in v6.5, however they were supposed to be fixed in v7. I believe that R&D will be able to nail this down either, based on the information provided by you in the case.
tsuplex
Novice
Posts: 4
Liked: never
Joined: Nov 30, 2012 12:28 am
Contact:

Re: Veeam 7 2012 Hyper-v

Post by tsuplex »

I asked if there would be a patch to allow Veeam 7 and ODX to co-exist, the response I got from support is that "this issue related to Windows".

I had no issues with 6.5 on the very same set of servers / operating systems, the issue started occurring immediately after doing an in-place upgrade to 7.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 7 2012 Hyper-v

Post by foggy »

tsuplex wrote:I had no issues with 6.5 on the very same set of servers / operating systems, the issue started occurring immediately after doing an in-place upgrade to 7.
Has support explained this fact to you? I would ask them to do it, as without taking this into account it would be logical to contact MS support (since Veeam support was able to reproduce the behavior without Veeam B&R in action).
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 7 2012 Hyper-v

Post by foggy »

foggy wrote: Has support explained this fact to you?
Since v6.5 did not support ODX, it was disabled - that's why you did not observe this behavior previously. Also you are not observing it with v7, if disable ODX manually. Seems the proper way to address this is to contact MS directly...
jrfolkman
Novice
Posts: 7
Liked: never
Joined: Sep 13, 2012 3:58 pm
Full Name: Jason Folkman
Contact:

Re: Veeam 7 2012 Hyper-v

Post by jrfolkman »

Not trying to be snarky but just understand. If we have a problem with Veeam (whatever the issue) and Veeam TS says its a "Microsoft issue" we should contact Microsoft? It seems that Veeam would have more pull with Microsoft (and be able to explain the issue better, test, etc..) than the customer. It's also strange that the issue wasn't even with a SAN backup (which is what would use ODX right?)...

That being said are there instructions on how to disable ODX? I just searched the V7 manual for "ODX" and did not find anything.

Thanks,
Jason
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 7 2012 Hyper-v

Post by foggy »

jrfolkman wrote:That being said are there instructions on how to disable ODX? I just searched the V7 manual for "ODX" and did not find anything.
Please follow these instructions.
stormgoose
Lurker
Posts: 1
Liked: never
Joined: Sep 26, 2013 1:51 am
Contact:

Re: Veeam 7 2012 Hyper-v

Post by stormgoose »

I am very relieved to finally find a forum online detailing others suffering the woes of Veeam 7, versus 6.5. You have my sincere empathy.

Our setup is simple - 2 x Win 2012 Datacenter Hyper-V Hosts, with 9 production VMs (all running Win 2012 Datacenter too) running on Host1, and replicated (using the Hyper-V inbuilt replica system) to Host2. We aren't using any clustering, and all storage is direct attached on each identical server, using LSI SAS-2 RAID cards with RAID 10 on 6 drives per host.

Veeam 7 is so bad, it has proven dangerous to our setup. I started a backup running in the early hours last night (one of many that have gone wrong), and hoped that with some new setting tweaks (such as bandwidth throttling) it might play ball. I watched it backup for several hours, and it was fine.... This morning, I get a call from my work, and everything is completely going to pieces. As another user put it above, the Hosts are fine, but the VMs themselves timeout. Trying to shut down the Veeam backup service is fruitless, as is trying to stop the backup in the Veeam B & R console. So I have to restart the Host, which with all this going on then takes almost an hour to return to useability. In the interim, my colleagues are pretty upset with the whole ordeal, as frankly am I.

Veeam 7 wasn't available at our time of purchase, and ideally I would have waited till it was released to properly test it. Veeam's selling technique was just a bit questionable (the deal was effectively something like a 40% discount to buy rights to a future version before the end of June 2013, getting a 'free' upgrade to the enterprise plus 7 edition when it came out but without being able to test it - and having to pay more for support later down the track too.) 6.5 was available at the time, and I throughly tested it and found it to be an excellent product. It lacked tape and WAN acceleration support (both offered in 7 enterprise plus and strong needs for us), and on the strength of the quality of 6.5, we went for it. I'm now starting to regret that decision.

Could a Veeam forum admin chime in and tell me where I need to go to file a support request? So far I've emailed a contact I had at Veeam, but not heard anything back....

Good luck to all - will share the outcome if I have anything to contribute on the issues :-)
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam 7 2012 Hyper-v

Post by Gostev » 1 person likes this post

stormgoose wrote:Could a Veeam forum admin chime in and tell me where I need to go to file a support request? So far I've emailed a contact I had at Veeam, but not heard anything back....
To open a case, click Support on http://www.veeam.com
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: Veeam 7 2012 Hyper-v

Post by kte »

Any updates on this ??
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 2012 Hyper-v

Post by veremin »

Hi, Koen,

I'm wondering what particular issues you experience with ODX. Issues with ODX functionality have been previously discussed in both the weekly digest and forum topic. In case you have missed them, it might be worth taking a look now.

Thanks.
Post Reply

Who is online

Users browsing this forum: No registered users and 25 guests