Veeam 7 2012 Hyper-v

[davidh]

Hi,

Has anyone experienced issues since upgrading to Veeam 7 from 6.5?

Environment:

Windows 2012 datacentre
2 hosts
Local storage
10gb backup network between servers
Veeam storage on iSCSI SAN

The issue appears to center around 2 Exchange VMs in a dag across both hosts.
After the Veeam 7 server was install in a fresh VM, the hosts integration components were upgraded. A initial backup was run which froze Host 1, after the reboot a subsequent VSS snapshot either by the host or backup would first freeze the Exchange1 VM, then within an hour BSOD the host. During this entire time the host would show the system process maxing out disk IO.

Host 2 would successfully run a backup, although not truncate logs. We would also see VMs freeze while the backups took place.

We removed all Veeam integration components from the hosts, turned off the Veeam 7 server, but the hosts are still exhibiting the same behavior when the VSS writer runs. We have had to disable all Volume shadow services on both hosts to stop them freezing up as they are in production.

We believe our next steps may be to reinstall the server 2012 hosts from scratch and a new Veeam 6.5, unless anyone has any ideas?

Luckily we were trialing 7 on only 2 of our 10 hosts, sadly those 2 hosts supported 150 users :/

support case 00432125, yet to hear back from escalated support.

[jrfolkman]

Any update? This is very similar to my setup so I am interested in the result. The big issue I have with "testing" Veeam 7 is that regardless of whether you upgrade your existing Veeam server or create a new one the host integration is updated... So it's not a "zero loss" scenario if there are problems. All that being said I have upgraded Veeam (although in a VMWare environment) many times and not had a problem.

I wish we had the luxury of multiple extra Hosts but we don't so if the host integration causes issues we have big problems.

Thanks,
Jason

[Vitaliy S.]

Jason, in order to receive quicker assistance with this issue, it's better to contact our technical support team directly. Thanks!

[jrfolkman]

I don't need assistance I was just seeing if the other guys issue had been resolved and what they found. Thanks

[foggy]

As far as I can see from the OP's case, the issue is still under investigation.

[tsuplex]

I believe we're currently seeing the same or similar issue.

We were running 6.5 flawlessly on Windows Server 2012 datacenter prior to upgrading to 7.

After the upgrade, during a backup we started experiencing what we'd describe as a host lockup but the host itself remains contactable, but all VMs stop responding and the Hyper-V management console becomes unusable. I have to stop the job, stop all Veeam services on the host and force-quit-restart the Hyper-V Management service to regain control of the guest VMs.
I too am running Exchange 2013 DAG nodes on each of the two hosts that are experiencing the issue.

The issue still occurs even after disabling CBT and application aware image processing.

My case number is 00432695.

Have you made any progress?

[foggy]

As far as I can see from your case, you've already moved much further in investigation than OP did, so please continue working with support. Thanks.

[tsuplex]

I was asked to run some backups with ODX disabled on the Windows Server 2012 hosts. The symptom hasn't re-occurred with ODX disabled.

Note that the backups that were misbehaving were local storage backups, not SAN backups.

[foggy]

Well, there were some issues with ODX observed previously in v6.5, however they were supposed to be fixed in v7. I believe that R&D will be able to nail this down either, based on the information provided by you in the case.

[tsuplex]

I asked if there would be a patch to allow Veeam 7 and ODX to co-exist, the response I got from support is that "this issue related to Windows".

I had no issues with 6.5 on the very same set of servers / operating systems, the issue started occurring immediately after doing an in-place upgrade to 7.

[foggy]

I had no issues with 6.5 on the very same set of servers / operating systems, the issue started occurring immediately after doing an in-place upgrade to 7.

Has support explained this fact to you? I would ask them to do it, as without taking this into account it would be logical to contact MS support (since Veeam support was able to reproduce the behavior without Veeam B&R in action).

[foggy]

Has support explained this fact to you?

Since v6.5 did not support ODX, it was disabled - that's why you did not observe this behavior previously. Also you are not observing it with v7, if disable ODX manually. Seems the proper way to address this is to contact MS directly...

[jrfolkman]

Not trying to be snarky but just understand. If we have a problem with Veeam (whatever the issue) and Veeam TS says its a "Microsoft issue" we should contact Microsoft? It seems that Veeam would have more pull with Microsoft (and be able to explain the issue better, test, etc..) than the customer. It's also strange that the issue wasn't even with a SAN backup (which is what would use ODX right?)...

That being said are there instructions on how to disable ODX? I just searched the V7 manual for "ODX" and did not find anything.

Thanks,
Jason

[foggy]

That being said are there instructions on how to disable ODX? I just searched the V7 manual for "ODX" and did not find anything.

Please follow these instructions.

[stormgoose]

I am very relieved to finally find a forum online detailing others suffering the woes of Veeam 7, versus 6.5. You have my sincere empathy.

Our setup is simple - 2 x Win 2012 Datacenter Hyper-V Hosts, with 9 production VMs (all running Win 2012 Datacenter too) running on Host1, and replicated (using the Hyper-V inbuilt replica system) to Host2. We aren't using any clustering, and all storage is direct attached on each identical server, using LSI SAS-2 RAID cards with RAID 10 on 6 drives per host.

Veeam 7 is so bad, it has proven dangerous to our setup. I started a backup running in the early hours last night (one of many that have gone wrong), and hoped that with some new setting tweaks (such as bandwidth throttling) it might play ball. I watched it backup for several hours, and it was fine.... This morning, I get a call from my work, and everything is completely going to pieces. As another user put it above, the Hosts are fine, but the VMs themselves timeout. Trying to shut down the Veeam backup service is fruitless, as is trying to stop the backup in the Veeam B & R console. So I have to restart the Host, which with all this going on then takes almost an hour to return to useability. In the interim, my colleagues are pretty upset with the whole ordeal, as frankly am I.

Veeam 7 wasn't available at our time of purchase, and ideally I would have waited till it was released to properly test it. Veeam's selling technique was just a bit questionable (the deal was effectively something like a 40% discount to buy rights to a future version before the end of June 2013, getting a 'free' upgrade to the enterprise plus 7 edition when it came out but without being able to test it - and having to pay more for support later down the track too.) 6.5 was available at the time, and I throughly tested it and found it to be an excellent product. It lacked tape and WAN acceleration support (both offered in 7 enterprise plus and strong needs for us), and on the strength of the quality of 6.5, we went for it. I'm now starting to regret that decision.

Could a Veeam forum admin chime in and tell me where I need to go to file a support request? So far I've emailed a contact I had at Veeam, but not heard anything back....

Good luck to all - will share the outcome if I have anything to contribute on the issues

[Gostev]

Could a Veeam forum admin chime in and tell me where I need to go to file a support request? So far I've emailed a contact I had at Veeam, but not heard anything back....

To open a case, click Support on http://www.veeam.com

[kte]

Any updates on this ??

[veremin]

Hi, Koen,

I'm wondering what particular issues you experience with ODX. Issues with ODX functionality have been previously discussed in both the weekly digest and forum topic. In case you have missed them, it might be worth taking a look now.

Thanks.