9.5 Hyper-V 2016 Known Issues

[frankive]

worked like a charm with most of the servers and exchange (thanks for the post regarding disable service. It was actually enough to just disable topology service).
all servers was 2012.

My two 2012 R2 servers, fully patched, do not work yet, as they are not seeing that specific update yet.

[Mike Resseler]

Which is the latest rollup you have installed on those R2 servers?

[frankive]

had 1Gb left with updates after waiting a day after installing these, backups went just fine. so the issue is now the domain controller, as the original thread is related too.
We will wait for Microsoft here

by the way; storage spaces direct and 4X800 sata ssd + veeam gives almost 1Gbit in read speed during backup, pretty nice

[Mike Resseler]

Frank,

Glad to see that you are getting somewhere . Is the DC now the only one left (or is the TS also still a problem?)

Mike

[frankive]

the DC is the only one lef, the ts (2008 R2) was also much more happy after the KB3037623 install
have creatt a new domain controller (DC02) which runs on a small HP Microserver (200 USD) with free hyper-v 2012 r2 as host. Then we also have a 100% good backup of the dc for this customer as long as Microsoft has not released the update to solve the issue.

[twinstate_ainsley]

I find myself in the same position:

- Hyper-V host running Windows Server 2016
- VM running Windows Server 2012 R2 and Microsoft Exchange 2013 CU15
- Microsoft patch KB3037623 (referenced in prior posts) was already installed via Windows updates
- AAP still fails on backups

I was considering uninstall that patch and re-installing using the guidelines from the post by SALM (microsoft-hyper-v-f25/9-5-hyper-v-known ... ml#p230347 , see below). What do folks think?
=======================================
My own Exchange runs on 2012 R2 and Exchange 2016 RTM and CU3.

I have confirmed that backups with AAP work for Exchange 2013 and 2016 (latest CU) on Windows Server 2012 R2, provided KB3037623 is installed on the guest OS (this KB updates integration services).

If the KB has not yet been installed, the vss will fail with the following error
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32768'. Failed to create VM recovery snapshot, VM ID '064dac1c-9652-4378-a69c-fa3429ca32a9'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)


The KB is unable to be properly installed if Exchange is already installed (OS will hang on reboot)

The following service must be disabled prior to KB installation
AD Topology.
These services should be set to manual :
Microsoft Exchange Diagnostics
Microsoft Exchange Transport
Microsoft Filtering Management Service
Microsoft Exchange Search Host Controller

The services should be set to Automatic after successful KB installation.

I do not have any information on 2008 R2 Guest OS or previous versions of Exchange.
=======================================

[Mike Resseler]

Thanks Frank!

Glad to hear that TS is solved also. Make sure that you are ready to seize the roles in a worst case scenario (I am assuming that DC01 still has the FISMO roles running) and you should be good to go for now. Obviously, just like you, I'm hoping MSFT to fix this as soon as possible!

[Salm]

I find myself in the same position:

- Hyper-V host running Windows Server 2016
- VM running Windows Server 2012 R2 and Microsoft Exchange 2013 CU15
- Microsoft patch KB3037623 (referenced in prior posts) was already installed via Windows updates
- AAP still fails on backups

What's the error you get ?
AAP should work if the guest OS is up to date.

[Mike Resseler]

To add to @salm's questions:

If you indeed have already installed KB3037623 on the VM and you still are getting issues as described in this thread then please log a support call. Let us also know what the exact error is you are getting. It might be something else

Thanks
Mike

[twinstate_ainsley]

I have already started a support call as I figured that was the next step.

[mkaec]

I'm puzzled why KB3037623 would not be installed. The KB articles indicates this was released in March 2015.

[twinstate_ainsley]

I had a good support call with Veeam even though it hasn't resolved the issue yet. You can read the relevant verbiage from the ticket below. I am going to try using hyper-v quiescence and will let you know how that goes.

"Here's the recap for our WebEx today, and what information I'll need and what steps we're going to take to move forward on this case.

1) We confirmed that VSS is working as expected on both the guest and the host by initiating the same workflow on both sides independently of Veeam
2) We confirmed production checkpoints, which trigger VSS on the guest when taking the host level snapshot, were also working as expected
3) We confirmed the issue we're facing is a known Microsoft issue with backing up Windows 2012 R2 guests on Hyper-V 2016.
While the specific error we were facing (error 32770 creating production checkpoint) is known to affect DCs mainly, it appears we're facing the same issue and same behavior on your Exchange 2013 server. This has been acknowledge by Microsoft and will be fixed in a future update."

"As for our steps moving forward, we're going to reconfigure the job to utilize Hyper-V Quiescence on your Exchange guest. The manual production checkpoint executed properly and engaged VSS on the guest, and as far as our needs for Exchange go, this should be sufficient to make sure a transaction-safe backup is taken. This will not impact your ability to perform Exchange item restores as the Veeam Explorer for Exchange utilizes our file-level restore process to mount the Exchange database locally. As long as the Exchange database was properly prepared for hot backup, it'll be in a state that VEX can safely mount and utilize.

Here's the instructions on how to enable this:
https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Make sure that VSS is disabled on the Exchange server under AAIP, as we reenabled it after correcting the DC settings to ignore the SQL server present. You'll still receive the warning on the DC about the SQL VSS writer not being present, but that's harmless. If you really want to eliminate the warning I believe there's a registry key to disable checking for SQL VSS writer presence, I'll do some digging to find it and let you know if I do."

[kimtay]

My position now:

- Hyper-V host running Windows Server 2016
- VM running Windows Server 2012 R2 and Microsoft SQL Server 2014
- Microsoft patch KB3037623 (referenced in earlier posts) has been installed through Windows updates
- Veeam B&R 9.5 Update 1 running without any problems (after upgrate KB3037623)

Thanks all!

[Salm]

It seems the problem is the creation of production checkpoints.

Have you tried manually creating one ?

If it doesn't work, have you tried switching the VM back to pre-2016 checkpoint creation mode ?

[jassonmc]

Since I've been struggling as well to backup our DCs on Windows Server 2012 and Windows Server 2012 R2 on Hyper-V 2016, I thought to share my current workaround.
The mentioned KB3037623 has never been offered on any of the two servers ia Windows Updates, no matter what I did.
As of today, both servers are fully updated, having installed all optional updates as well and we are not using WSUS, so basically there is nothing more that could be installed apart from manual installations.

Initially found here the right comment: https://www.reddit.com/r/Veeam/comments ... ler_issue/

How I did it:
Reboot service EventSystem in the GUI, which will also restart its 3 to 4 dependent services
AFTER restarting the service EventSystem, only then restart also service VSS
It's important to note that you restart VSS as a last step, otherwise it won't work or if you did it the other way around you have to restart VSS once more, which also fixes the problem.
That's it, backups now run smoothly with AAIP enabled on both DCs.

Since this procedure has to be done on every VM reboot, it's rather handy to have a batch at hand that will be run on system boot as an automated task.
So I created a simple batch file with the following entries:
net stop BITS
net stop SENS
net stop NtFrs
net stop DFSR
net stop EventSystem
net start EventSystem
net start DFSR
net start NtFrs
net start SENS
net start BITS
net stop VSS
net start VSS

Afterwards setup a task that runs on every system boot. I added a 2 minutes delay, just in case the above service restarts could be executed to early, but I'm not sure on that and didn't spend the time to find out.
Hopefully that comes in handy to some of you.

Cheers Juri

[Mike Resseler]

Juri,

Thanks for this detailed explanation. Really appreciated. Unfortunately this doesn't take away the fact that we hope MSFT fixes this at a certain point in time.

But for now, it is a good workaround

Thanks
Mike

[jassonmc]

Absolutely Mike!

From a Microsoft perspective it's totally ridiculous that this issue is still not solved and we have to use such silly self made patches...
Anaway, for now that problem climbs down the priority ladder as this is good enough for us now.

Cheers Juri

[twinstate_ainsley]

So using the Hyper-V quiescence didn't work and I got the following feedback from Veeam support:

=============================

Sorry for the delay in updating you. If the Hyper-V Quiescence setting is not working, we can try to troubleshoot this, but I did check and using Hyper-V Quiescence will not allow for Exchange Item Restores for things such as emails and other exchange items. So because application aware is failing due to the current Hyper-V 2016 bug, your main options to utilize application aware processing so you can perform Exchange Item Restores are the following:

1. Upgrade your affected VM to Windows Server 2016 since this OS is not affected by the bug.

2. Wait for a fix for this from Microsoft Support.

3. Don't enable Application aware processing or Hyper-V Quiescence. Use a possible workaround such as Veeam Endpoint (Limited support as this is not a supported use for Veeam Endpoint but it should work and allow for Exchange Item Restores) or Veeam Agent for Windows (Beta).

====================================

It seems like number 2 is my only option unless anyone has experience with Veeam Endpoint? I may also try the batch file that stops and starts services to see if that does anything for me.

[Mike Resseler]

Hi,

Did you have a support call with MSFT also? Any reaction from there?

[twinstate_ainsley]

No call with Microsoft and not sure I am going to; if Veeam is already working on them for updates, I think they'll have more luck than me

[mkaec]

I don't blame you. It's been several years since I had a call with Microsoft, but trying to get through the first level to someone that cares was difficult and frustrating. My issue ended up being a known issue and a patch was delivered, but I had to fight tooth and nail with the level 1 reps that insisted the behavior I was seeing was "by design".

[Simon_]

Hi all,

I have similar issues to twinstate_ainsley

- Hyper-V Host running Windows Server 2016
- VM running Windows Server 2012 R2 and Microsoft Exchange 2013 CU15
- VM running Windows Server 2012 R2 (no Exchange)
- Microsoft patch KB3037623 is not installed and does not appear in Windows Updates on either 2012 R2 server
- VM running Windows Server 2016 (Domain Controller)
- VM running Windows Server 2016 (Member Server)
- Veeam B&R 9.5 Update1 (9.5.0.823)

I see Veeam backup errors, with AAP disabled (first error) or with AAP enabled (second error):
- Failed to create VM recovery checkpoint (mode: Crash consistent) Details: Job failed (''). Error code: '32774'. Failed to create VM recovery snapshot, VM ID '064dac1c-9652-4378-a69c-fa3429ca32a9'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)
- Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32774'. Failed to create VM recovery snapshot, VM ID '064dac1c-9652-4378-a69c-fa3429ca32a9'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)

In Hyper-V I cannot create a Checkpoint, I see error:
An error occurred while attempting to checkpoint the selected virtual machine(s). Taking a checkpoint of 'servername' failed. The operation cannot be performed while the object is in use.

VSS writers are running and stable across all servers (Host and Guests).

Rebooting the Hyper-V host allows the backups to run and checkpoints to be created for up to 24 hours before the errors return.

I have tried running the Veeam backups with and without AAP enabled, I've checked the Hyper-V Host has the latest Windows Updates and checked my Hyper-V configuration.

At another site I have a Hyper-V Host 2016 running Windows Server 2016 with three VMs (2012 R2 with Exchange 2013 CU15, 2016 Domain Controller, 2016 Remote Desktop Server) that shows no errors in Hyper-V (Checkpoints) or in Veeam (backups)

I'll be interested to see if Microsoft release any updates for this. Its driving me mad!

I haven't logged a support case (yet).

[lenart.plausteiner]

Hi all,

I have similar/same problem with backing up Windows 2008 R2 VM with ApplicationAware enabled. On this VM we have installed Exchange 2010 with latest and greatest SP . This VM has been moved(replicated) from HyperV 2008 R2 to HyperV 2016. When we try to establish backup we get this in our veeam LOG:

Code: Select all

[RTS] Recovery checkpoint snapshot started.
[15.03.2017 09:29:24] <18> Info     [VM NAmen]  Recovery checkpoint snapshot started.
[15.03.2017 09:29:24] <18> Info     Performing on-host backup from CSV with software provider. Copying ctp files to volumes owners
[15.03.2017 09:29:34] <18> Error    [RTS] Failed to update snapshot's state
[15.03.2017 09:29:34] <18> Error    Job failed ('Checkpoint operation for 'Apolon' failed. (Virtual machine ID 19AE9956-BDF8-409F-95D1-8F4F7FE8DEA8)'). Error code: '32768'.
[15.03.2017 09:29:34] <18> Error    Failed to create VM recovery snapshot, VM ID '19ae9956-bdf8-409f-95d1-8f4f7fe8dea8'. (Veeam.Backup.ProxyProvider.CHvWmiProxyErrorException)
[15.03.2017 09:29:34] <18> Error       at Veeam.Backup.ProxyProvider.CHvWmiReconnectableRemoteCommand.InvokeInThread(Delegate dlg, Object[] args)
[15.03.2017 09:29:34] <18> Error       at Veeam.Backup.ProxyProvider.CHvWmiReconnectableRemoteCommand.DoInvoke(CHvWmiProxyRequestContextNdw context, Int32 reconnectsCount, Delegate dlg, Object[] args)
[15.03.2017 09:29:34] <18> Error       at Veeam.Backup.ProxyProvider.CHvWmiReconnectableRemoteCommand.Invoke[Ret](CHvWmiProxyRequestContextNdw context, Func`1 dlg)
[15.03.2017 09:29:34] <18> Error       at Veeam.Backup.ProxyProvider.SHvWmiProxyMethodsHelper.CreateRecovery(Guid vmID, ESnapshotConsistencyLevel consistencyLevel, Byte guestBackupType, CHvIntegrationConnectionInfo hvIntegrationConnectionInfo, Int32 timeout)
[15.03.2017 09:29:34] <18> Error       at Veeam.Backup.ProxyProvider.CHvWmiSnapshotRemoteManager2015.CreateRecovery(Guid vmID, ESnapshotConsistencyLevel consistencyLevel)
[15.03.2017 09:29:34] <18> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointHelper.VeeamAwareSnapshotLogic(CHvVmSnapshotData snapData, Exception& exception)
[15.03.2017 09:29:34] <18> Error    [VM Name]  Failed to update snapshot's state

And we get this to log inside VM:

Code: Select all

15.3.2017 9:18:41   7648                          Restarting HVVSS service.
15.3.2017 9:18:41   7648                          Cannot read retry settings from registry. Using default value 4. Cannot read VmicvssRetryNumber value
15.3.2017 9:18:41   7648                          Win32 error:The system cannot find the file specified.
15.3.2017 9:18:41   7648                           Code: 2
15.3.2017 9:18:41   7648                              Restarting Hyper-V VSS requester
15.3.2017 9:18:41   7648                                  Stopping Hyper-V VSS Requester
15.3.2017 9:18:41   7648                                  Cannot read meta file size settings from registry. Using default value 60. Cannot read StopServiceTimeout value
15.3.2017 9:18:41   7648                                  Win32 error:The system cannot find the file specified.
15.3.2017 9:18:41   7648                                   Code: 2
15.3.2017 9:18:41   7648                                  Starting Hyper-V VSS Requester
15.3.2017 9:18:42   7648                              Restarting Hyper-V VSS requester. Ok.
15.3.2017 9:18:42   7648                          Finishing  CHvVssJob::StaticKeepAliveThread
15.3.2017 9:18:42   9192                          Keep alive thread successfully finished.
15.3.2017 9:18:42   9192  INFO                Finishing snapshot: job id: {9f12f53f-0bdd-43c7-a6b5-14588acf18d1}, logs truncation required: no.. Ok.
15.3.2017 9:18:42   9192  INFO        RPC: FinishSnapshotEx. Ok.

Has annybody resolved this issue?

[Mike Resseler]

@simon

Have you April 2014 update rollup on that 2012 R2 server? You need to have that before you can install https://support.microsoft.com/en-us/hel ... view-hosts.

It might be interesting to log a support case and log the case ID here also

[Mike Resseler]

@lenart

This seems to be a different thing but I can be mistaken. What is the VM version? 8 or still lower?

Please log a support case also and post the case ID here (Plus the follow-up after that)

Thanks
Mike

[lenart.plausteiner]

@Mike

thanks for replay, we haven't yet opened support case. I'll do this right now and post support case ID here.

This VM is version 8 and it was replicated with Veeam B&R to this new Hyper-V cluster. Source was HyperV 2008R2 with MS SCVMM and destination was HyperV 2016 cluster w/o SCVMM.

[lenart.plausteiner]

Support case ID is: #02100205

[Salm]

Hey !

Is your guest OS fully up to date, especially, does it have KB3037623 installed ?

Also, if it's on an Exchange server, see microsoft-hyper-v-f25/9-5-hyper-v-known ... ml#p230347 as it may apply to you as well.

[Salm]

@Mike/Veeam

For what it's worth, I just fully updated a 2012 R2 domain controller and the 2016 hyper-v host with the March updates (even the optional ones), and the VSS bug still has not been corrected.

I find it weird that MS takes such a long time to correct a critical issue such as this...

[Mike Resseler]

@Salm,

Thanks for letting us know. I am still in the process of updating my lab with the latest updates. And yes, I can only agree that it takes too much time to fix this bug...