9.5 Hyper-V 2016 Known Issues

[buburip]

I have checked that site, downloaded the vmguest.iso The version of the tools in the ISO is 6.3.9600.16384. So it doesn't help.

No, this is my site.

I modified the ISO VMGuest and the vmbus.sys versions are fine
2008R2: 6.3.9600.17903
2012R2: 6.3.9600.17415

Do not look at the setup.exe version

Checks the XML file: support\amd64\Windows6.x-HyperVIntegrationServices-x64.xml : version 6.3.9600.17903

Insert the iso and force the installation with the cmd command: Dism.exe / online / Add-package /packagepath:D:\support\amd64\Windows6.x-HyperVIntegrationServices-x64.cab / quiet / norestart

And check the version of vmbus.sys .....

[mmaster]

Still no update from MS for the backup of a DC2012 on a 2016 Hyper-v host?

[dredd123]

I'm fighting with this on 6 2012R2 DC's after upgrading one of our Hyper-V clusters to 2016

These VM's are running Integration Services version 6.3.9600.18398 - I would like to try and downgrade to 17415 as suggested by buburip but I can't see how to do it?

It really is high time MS got this sorted out now.

[buburip]

I'm fighting with this on 6 2012R2 DC's after upgrading one of our Hyper-V clusters to 2016

These VM's are running Integration Services version 6.3.9600.18398 - I would like to try and downgrade to 17415 as suggested by buburip but I can't see how to do it?

It really is high time MS got this sorted out now.

Hello,

Uninstall the KB3063109 and install Windows6.2-HyperVIntegrationServices-x64.cab in vmguest.iso , see this tutorial : https://www.pc2s.fr/?p=1568

Or here in translation: https://translate.google.fr/translate?s ... t=&act=url

Or install directly the KB3071740, here : https://www.microsoft.com/en-us/downloa ... x?id=48207 - Source : https://support.microsoft.com/en-us/hel ... based-host

[mvalpreda]

Is there a general consensus on this?

Uninstall KB3063109
Get the integration components to
2008R2: 6.3.9600.17903
2012R2: 6.3.9600.17415

Bit suck since I can't seem to get my 2012R2 to 17415.....only 17903.

[mvalpreda]

The one at https://www.microsoft.com/en-us/downloa ... x?id=48207 is 17903. And I don't see any of those build numbers here: https://social.technet.microsoft.com/wi ... mbers.aspx

Rather not turn off AAP.....so for anyone who has a 2016 Hyper-V host with a 2012 R2 DC....what have you done and where did you get whatever package?

[buburip]

The one at https://www.microsoft.com/en-us/downloa ... x?id=48207 is 17903. And I don't see any of those build numbers here: https://social.technet.microsoft.com/wi ... mbers.aspx

Rather not turn off AAP.....so for anyone who has a 2016 Hyper-V host with a 2012 R2 DC....what have you done and where did you get whatever package?

You have not read the previous posts.
Everything is before and HERE : https://www.pc2s.fr/?p=1568

[buburip]

Hello,

see this tutorial : https://www.pc2s.fr/?p=1568

For 2012R2 :

Uninstall the KB3063109 and or KB3071740 and install Windows6.2-HyperVIntegrationServices-x64.cab in vmguest.iso (version 6.3.9600.17415)

For 2008R2 :

Uninstall the KB3063109 and install Windows6.x-HyperVIntegrationServices-x64.cab in vmguest.iso (version : 6.3.9600.17903)

[mvalpreda]

Thanks. Didn't see the KB3071740 for 2012 R2. Got to 17415, but now I get a new error when trying to backup the 2012 DC.

6/6/2017 8:05:57 AM :: Failed to finalize guest processing. Error: Failed to take in-guest VSS snapshot COM error: Code: 0x80042308

And a bunch of VSS writers are failed (time out).

[buburip]

Thanks. Didn't see the KB3071740 for 2012 R2. Got to 17415, but now I get a new error when trying to backup the 2012 DC.

6/6/2017 8:05:57 AM :: Failed to finalize guest processing. Error: Failed to take in-guest VSS snapshot COM error: Code: 0x80042308

And a bunch of VSS writers are failed (time out).

Tries by disabling "Enable application-aware image processing" and "Enable guest file system indexing" !

[dredd123]

Unfortunately I don't have KB3063109 installed - I had to install the 18398 Integration Services manually from a vmguest.iso from one of our other 2012R2 Hyper-V clusters. Not sure if there is another way to uninstall Integration Services.

Hello,

Uninstall the KB3063109 and install Windows6.2-HyperVIntegrationServices-x64.cab in vmguest.iso , see this tutorial : https://www.pc2s.fr/?p=1568

Or here in translation: https://translate.google.fr/translate?s ... t=&act=url

Or install directly the KB3071740, here : https://www.microsoft.com/en-us/downloa ... x?id=48207 - Source : https://support.microsoft.com/en-us/hel ... based-host

[frankive]

Veeam, I think this topic deserves a blog from Veeam how to solve this until Microsoft gets this fixed. May frustrated users on a product which has been in production for almost a year..

[Salm]

Tries by disabling "Enable application-aware image processing" and "Enable guest file system indexing" !

Hello,

Not to be mean, but the goal of what he's trying to do is making application-aware processing (AAP) work.

So disabling it is not going to help much.

[TT-DL]

microsoft-hyper-v-f25/9-5-hyper-v-known ... ml#p243861

In my experience KB3063109 fixes this COM error, but then you have a version of integration tools that is later than what everyone says allows you to complete AAP backups. It's a vicious cycle

Im still stuck with it as well. I've bitten the bullet and have opened a support case with Microsoft about the issue, as I didn't think the work around outlined here are sustainable long term. ie forever checking every update to make sure it is not going to update the integration tools, and subsequently break AAP again. At the end of the day we all still need to be able to patch, and patch regularly.

Ill post an update if Micorosoft can fix. Otherwise plan B is to abandon 2016 Hyper-V host for us for these machines and run up a 2012 host. Very frustrating...

[TT-DL]

OK guys - an update - no resolution yet.

I now have open support cases with both Veeam and Microsoft about this issue. Despite everyone on this thread saying this is Microsoft issue, if I was adjudicating I would rule in Microsoft favor at the moment. I'll reserve judgement until/if this gets resolved though.

If you are having trouble with this issue, do this to prove the integrity of your VSS architecture.

1) From your Hyper-V host can you take a checkpoint of the machine. If so go to 2).
2) From your Hyper-V host can you use Windows Backup to backup the problem machine? If so go to 3).
3) From your guest VM that is unable to complete a Veeam AAP backup, check the state of your VSS Writers, immediately after completing a Successful windows backup, before trying a veeam backup. Are they all ok? if so go to 4).
4) On the guest VM can you create a disk shadow from the command line? Once complete run 'vssadmin list shadows', can you see the shadow you just created? if so go to 5)
5) Check the state of your VSS writers again on the VM guest. Are they all ok, with nothing in an error/failed state? if so go to 6)
6) Your VSS architecture seems fine!

At this point my microsoft engineer has concluded your "VSS architecture is fine" you can create windows, backups and checkpoints, and a guest snapshot. Everything is working. If you are having problems you need to call the 3rd party vendor."

Try running a Veeam AAP backup again. Backup fails with '32768' error. Check VSS writers. A number of writers in failed state (7) Timed out.

Hang on! I have Veeam telling me it's a Microsoft issue. And now I have Microsoft telling me its a Veeam issue! WTF. I don't care who's issue it is, I just need it fixed.

Got Veeam on the phone, Microsoft Engineer showed Veeam engineer his findings. Veeam agreed we could seemingly replicate all the steps required to complete a AAP backup. They then both suggested a number of patches to volsnap and vssvc.exe which i have now applied and they haven't resolved the issue.

The case remains open with both Veeam and MSoft. However microsoft have proved they can complete backups using the VSS architecture so at this point im siding with them. If you cant complete the above steps on your server then I suggest you open a support case with MSoft to resolve this first. Its only a couple of hundred dollars, cheap in the scheme of a business, even a small one.

If I get a fix from either party ill post it. At this point I feel like Microsoft have proved their case about their VSS architecture working. I remain less convinced on the Veeam front. If anyone from Veeam has any details on the nature of this 'bug' that causes this issue or a bug id with Microsoft. Can you please it or PM me so I can raise it with my Microsoft support tech?

[Mike Resseler]

Hi Daniel,

I see your case is closed? Is that correct?

[eainsworth]

Just migrated my Hyper-V cluster to 2016 over the weekend and ran into this issue with backing up Exchange 2010 on the upgraded cluster. We performed this upgrade hoping for better backup performance with the native 2016 RCT feature. Instead, we no longer have a working backup infrastructure. Has anyone upgraded their Veeam backup server to Windows 2016? Any different results?

[TT-DL]

Hi Daniel,

I see your case is closed? Is that correct?

Hi Mike,

First veeam support agent wasn't very helpful, saw the writers in a failed state and closed the case. Case number - 02169813.

Second support agent, was extremely helpful, and opened a new support case for me after he went through the issue with Msoft Tech. Case number - 02182835. This case should be still open.

Thanks.

[Mike Resseler]

Ah, thanks Daniel, I was indeed looking at the old support case. Not too happy that the first support engineer wasn't very helpful. If you feel it is necessary, please escalate this. We take a lot of pride in our support department and our VP's running the different divisions are working always very hard to solve issues where the engineer didn't do what needs to be done.

I hope that you have better success with the new one, please escalate if necessary. We might need to get it to DEV team if it is locked at some point (although I hope not...)

Thanks again,

Mike

[Mike Resseler]

Hi Edward,
First: Welcome to the forums!

In this thread there should be the solution what you need to do to get Exchange running. It requires a few updates (and can't remember which versions were used) but Exchange was shown a few times as a success. It is AD that is the major issue here. The solution involves running some specific updates on your exchange server to make sure that WS2016 virtualization is supported. Those fixes come from MSFT

Cheers
Mike

[Salm]

Just migrated my Hyper-V cluster to 2016 over the weekend and ran into this issue with backing up Exchange 2010 on the upgraded cluster. We performed this upgrade hoping for better backup performance with the native 2016 RCT feature. Instead, we no longer have a working backup infrastructure. Has anyone upgraded their Veeam backup server to Windows 2016? Any different results?

Hi,

See this post for Exchange backups : microsoft-hyper-v-f25/9-5-hyper-v-known ... ml#p230347

[TT-DL]

Ah, thanks Daniel, I was indeed looking at the old support case. Not too happy that the first support engineer wasn't very helpful. If you feel it is necessary, please escalate this. We take a lot of pride in our support department and our VP's running the different divisions are working always very hard to solve issues where the engineer didn't do what needs to be done.

I hope that you have better success with the new one, please escalate if necessary. We might need to get it to DEV team if it is locked at some point (although I hope not...)

Hi Mike and All,

After spending several hours doing log collection and remote sessions with a second level Veeam support tech he says he now understands exactly what the issue is. Sounds like Veeam could do with some internal kb articles on this issue though (Maybe even a public KB on the issue?). He is just not sure if it is resolvable,maybe something for the Veeam Dev team I think.

Basically he said if it is possible to call the older WMI API (as Microsoft do when they run a backup on a 2k8 R2 Exchange 2010 box) then it can be fixed. Apparently the older API's are all present, the question is, is it possible to call them based on guest OS from within Veeam 9.5+? -Something only the Veeam team can answer.... ?????

Meanwhile everyone that has a 2016 host that is running any application that requires an 'application consistent' snapshot is stuck and desperate for a backup that works that isn't Windows.

BTW - If you are stuck with this Windows Backup works perfectly and will let you flush logs etc. Not the best solution, but something at least. You cant mix and match your veeam/Windows backups though FYI. one or the other.

[Mike Resseler]

Daniel,

Just looked and it seems that you are still doing tests together with our support engineer. Please keep pushing . I'm hoping that support and dev are already working together now.

[frankive]

any news on this?

[matt306]

I've had the same issues with a cluster that's been rolling upgraded to 2016. My current workaround is to keep a single node with 2012R2 (paused) in the cluster as it causes Veeam to behave differently and backups succeed (albeit with no CBT). Obviously if you've already upgraded the functional level to 9, this isn't an option for you.

[frankive]

It is my impression that a huge company as Veeam and as a Microsoft Partner you would push Microsoft in this case? It has gone almos a year now without successful backup of active directory.
Not funny to be a service provider and trying to excuse the customers with this.

[TT-DL]

Daniel,

Just looked and it seems that you are still doing tests together with our support engineer. Please keep pushing . I'm hoping that support and dev are already working together now.

Hi Mike, and All,

Ive requested this earlier, but I'll request it again. Everywhere I look everyone says this is a known 2016 bug. Cool got it. Can anyone actually provide a bug ID for this issue or a link to a relevant KB article?

I have MSoft support telling me no such bug exists. I am honestly going around in circles here. However no one can provide anything solid on this. This is the most elusive 'bug' in the history of the internet. Everyone 'knows' about it, but no one seems to have actually ever seen it. Maybe the loch ness monster created it...

Can someone please provide a link to somewhere MSFT or MSFT bug ID on this issue??

Thanks in advance.

[TT-DL]

I've had the same issues with a cluster that's been rolling upgraded to 2016. My current workaround is to keep a single node with 2012R2 (paused) in the cluster as it causes Veeam to behave differently and backups succeed (albeit with no CBT). Obviously if you've already upgraded the functional level to 9, this isn't an option for you.

Hi Matt, Thanks for the feedback. Im guessing because you have a 2012 host in the cluster veeam is seeing the cluster as a 2012 cluster and subsequently falling backup to the 2012 backup architecture rather than using the new 2016 API. This is also how Windows backup successfully completes for these guests on a 2016 Hyper-V host.

I asked my Veeam support engineer to make me a magic 'tick' in a box somewhere in the veeam interface to fallback like you have inadvertently discovered as a work around. The veeam tech tells me this simply isn't possible and they must use the 2016 API for hyper-v 2016 guests. Its very frustrating, but I like your work around.

[Mike Resseler]

Daniel,

I would love to give you the KB or something similar from MSFT but we don't have it. Unfortunately.

I can give you this: https://social.technet.microsoft.com/Fo ... rverhyperv

Here you will see that it is not us alone that is having this issue, our competition is suffering from it also

[TT-DL]

Hi Mike,

Appreciate the update. I'm feeling pretty frustrated about this, as are a lot of people, to me my current biggest frustration is it does not look like Veeam has owned or driven this at all with Microsoft.

I have just hit 'Contact a Manager' from within 'Support' when logged into the veeam site. I have requested that a Manager or someone at Veeam take some ownership on this issue with Microsoft and make sure a bug has been opened and can be tracked, because the current situation of 'its a bug, no one knows if Microsoft have acknowledge it though' is frankly ridiculous.

I would encourage all users of this forum, to 'Contact a Manager' about this issue and let's start escalating it and trying to get some ownership on it from the Veeam side. Please do this. Someone else is not going to do it. If ou are reading this you need to 'contact a manager'.

I'm particularly annoyed that there are work rounds ie using legacy backup processes to resolve this issue, but Veeam won't touch them as 'its a Msoft bug' but we don't know any more than that and we are not driving it for a resolution. Why don't veeam drop a couple of hundred bucks and open this as a support case with Microsoft if nothing else???

In it's current state it basically renders B&R useless on a 2k16 platform, as pretty well everyone has either AD or Exchange in their environment.