Hyper-V 2016 Requires Configuration Version 8

[mkaec]

I recently upgraded a Hyper-V host from Windows Server 2012 R2 to Windows Server 2016. The intention was not to upgrade the VM configuration version until I was more comfortable, as once this is done the VM cannot be migrated to a host running 2012 R2. I wanted to have an out if something were to go seriously wrong with Server 2016.

After the upgrade, I tried my Veeam jobs and got an error in Veeam:

Code: Select all

Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed ('Checkpoint operation for '<VM>' failed. (Virtual machine ID <GUID>) '<VM>' could not initiate a checkpoint operation: %%2147754767 (0x8004230F). (Virtual machine ID <GUID>)'). Error code: '32768'. Failed to create VM recovery snapshot, VM ID '<GUID>'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)

And the guest logged:

Code: Select all

Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details CommitSnapshots [hr = 0x800705b4, This operation returned because the timeout period expired.]. 
Operation:
Executing Asynchronous Operation
Context:
Current State: DoSnapshotSet

Eventually, I tried the job with application aware processing turned off. The job completed with the warning:

Code: Select all

Changed block tracking will not be used for this VM until you upgrade VM hardware version to 8.0 or later.

So, I upgraded the VM configuration version, enabled AAP and the job completed successfully.

The clues were there, but it wasn't overtly obvious what the problem was. I hope this post will save others from having to engage in troubleshooting this issue. If you want to backup Hyper-V 2016 with Veeam, the VM configuration version should be 8.0.

[nmdange]

Good to know... it would be nice if Veeam could fall back to the Veeam CBT driver!

[mkaec]

Good to know... it would be nice if Veeam could fall back to the Veeam CBT driver!

That's what I originally thought would happen. But it makes sense that they wouldn't want to devote resources to update and test it for Server 2016.

[Mike Resseler]

Mark,

Actually, it isn't always that simple . But you are right, and this was stated in different guides that you need to be at VM config version 8 (and if it is a cluster 2016).

Now you can leave the VM in a previous config version, without AAP and it will backup, changes only, but with no CBT so the backup will take a bit longer. But this should be a temporary solution (no longer than 1 week or so). Microsoft itself will also advise you to keep your servers not too long in that mixed mode.

These changes are because of the rolling-cluster upgrade possibilities and have a big impact. But I believe this is a good thing because it will become much more usable when we go from 2016 to the next version, or at least I hope so...

[nmdange]

What about if you have a Scale-Out File Server cluster running Windows Server 2016 presenting SMB storage to a Hyper-V 2012 R2 cluster? Would CBT be supported in that situation?

[Mike Resseler]

I actually never tested that. But in that case I think the Veeam CBT will be used. But I am not sure to be honest.

Is this a temporary situation that you are running or are you planning to keep this for a long time?

[nmdange]

Depends on what you mean a "long time" It would just be for the duration of transitioning from Hyper-V 2012 R2 to 2016, but since I have a fairly large environment it would be for quite a few months. With 2016, I'm planning on moving to switch embedded teaming so I can converge my SMB RDMA NICs with the virtual switch traffic. However, that requires two SMB networks on the same subnet which only works on 2016. So for the transition I'm thinking to have the SOFS cluster have RDMA NICs on both the old 2-subnet networks for 2012 R2 Hyper-V hosts, and the new 1-subnet network for 2016 Hyper-V hosts.

Also does an off-host proxy for Hyper-V have to match the version of the host being backed up? Talking SMB here, not SAN with Hardware VSS...

[Mike Resseler]

I'm going to have to investigate to be sure how CBT will work in that case and whether this is supported from our side or not. I will try to get your answer as quick as possible

[Mike Resseler]

Hi,

I got confirmation that this scenario will use Veeam CBT and that the off-host proxy needs to match the version of the Hyper-V host, so 2012 R2 in this case.

Which means you can continue your plans with your upgrades

Cheers
Mike

[nmdange]

Much appreciated

[Chrizz]

Hi there,
I am using Veeam since 2015 and now it is the first time I have a behavior which I could not resolve by myself.

I have 2 questions, maybe they belong to each other:

1) "Changed block tracking will not be used for this VM until you upgrade VM hardware version to 8.0 or later."
2) A very increased duration time for backing up VMs since - yeah, since when I had changed something - but not sure WHAT.

for 1) I know that 8 of these VMs are v 5.0 - but I cannot really change this at the moment because in our HyperV replication scenario there are still 2012R2 servers involved so replication will be broken if I do change HW to 8.0 on the backup host. But this was the same BEFORE that message popped up for the first time.

for 2)
Some days ago my jobs looked like this: (ignore the 2 failed VMs in this example!)

Success 6 Start time 22:00:08 Total size 6,4 TB Backup size 31,0 GB
Warning 0 End time 22:53:33 Data read 230,4 GB Dedupe 1,3x
Error 2 Duration 0:53:25 Transferred 30,9 GB Compression 4,2x
Details
Name Status Start time End time Size Read Transferred Duration Details
EDGE Success 22:32:24 22:36:21 127,0 GB 0,0 B 30,5 KB 0:03:57
Lexware Success 22:07:48 22:20:37 913,0 GB 6,5 GB 1,6 GB 0:12:49
Lync2013 Success 22:02:52 22:04:11 254,0 GB 0,0 B 15,3 KB 0:01:19
skypesettings Error 22:00:16 22:00:17 0,0 B 0,0 B 0,0 B 0:00:00 Task failed. Failed to expand object skypesettings. Error: Cannot find VM skypesettings on host Dora.mediaworx.de
Statistics Error 22:00:17 22:00:17 0,0 B 0,0 B 0,0 B 0:00:00 Task failed. Failed to expand object Statistics. Error: Cannot find VM Statistics on host Dora.mediaworx.de
WAC Success 22:03:52 22:08:31 127,0 GB 0,0 B 13,8 KB 0:04:38
XS1 Success 22:00:41 22:41:44 2,5 TB 110,7 GB 14,7 GB 0:41:03
XS2 Success 22:15:55 22:53:19 2,5 TB 113,3 GB 14,6 GB 0:37:24


Later on it started looking like this:

Success 0 Start time 12:02:43 Total size 6,8 TB Backup size 674,8 GB
Warning 9 End time 18:14:31 Data read 1,3 TB Dedupe 5,1x
Error 0 Duration 6:11:47 Transferred 684,7 GB Compression 2,0x
Details
Name Status Start time End time Size Read Transferred Duration Details
EDGE Warning 12:03:20 12:51:45 127,0 GB 28,9 GB 17,0 GB 0:48:25 Changed block tracking will not be used for this VM until you upgrade VM hardware version to 8.0 or later.
Lexware Warning 12:48:41 13:59:48 913,0 GB 74,1 GB 53,5 GB 1:11:06 Changed block tracking will not be used for this VM until you upgrade VM hardware version to 8.0 or later.
Lync2013 Warning 13:27:52 14:35:55 254,0 GB 74,7 GB 32,8 GB 1:08:03 Changed block tracking will not be used for this VM until you upgrade VM hardware version to 8.0 or later.
skypesettings Warning 12:03:19 13:29:43 100,0 GB 47,5 GB 26,9 GB 1:26:24 Changed block tracking will not be used for this VM until you upgrade VM hardware version to 8.0 or later.
Statistics Warning 12:03:19 12:48:35 127,0 GB 21,5 GB 13,1 GB 0:45:15 Changed block tracking will not be used for this VM until you upgrade VM hardware version to 8.0 or later.
VMM2016 Warning 12:03:19 13:29:28 256,0 GB 46,0 GB 27,8 GB 1:26:08 SQL VSS Writer is missing: databases will be backed up in crash-consistent state and transaction log processing will be skipped
WAC Warning 17:19:27 18:06:07 127,0 GB 38,4 GB 28,2 GB 0:46:40 Changed block tracking will not be used for this VM until you upgrade VM hardware version to 8.0 or later.
XS1 Warning 13:29:32 17:22:37 2,5 TB 507,4 GB 242,3 GB 3:53:04 Changed block tracking will not be used for this VM until you upgrade VM hardware version to 8.0 or later.
XS2 Warning 14:34:23 18:14:05 2,5 TB 535,6 GB 243,2 GB 3:39:42 Changed block tracking will not be used for this VM until you upgrade VM hardware version to 8.0 or later.

Look at the increased "backup time" from 0:53 to 6 hours although the same "total size" is reported. This is huge!

Architecture:
Veeam Backup & Replication 9.5.0.1536
Running in a VM on HyperV 2016 server
All VMs to backup are are Windows based
VMs are stored on the same HyperV host on SAN storage (IBM V7000)
Most of the VMs are HyperV-replicas from other HyperV-hosts with 2012r2 server
The backup repository is on the same SAN storage like the VMs itself (IBM V7000)

The hardware was not changed at all and shows no problems in our production environment.

Any ideas on this?

Thanks - Chrizz

[PTide]

Hi and welcome to the community!

Architecture:
Veeam Backup & Replication 9.5.0.1536
Running in a VM on HyperV 2016 server

May I ask you where does your backup proxy reside? What are your backup proxy settings?

Thanks

[nubee]

I love it when the answer is "do this" ("upgrade configuration version") !
Nobody ever mentioned where this setting is or how to do it.
Is it done from within Veeam?
Is it done from within the guest OS?
Is it done from within powershell?
Is it done on the host?
What program do you have to open?
What do you click?
What do you type?
The instructions for assembling the furniture should not be "assemble the furniture".
Any help would be appreciated.

[Dima P.]

Hello and welcome to the community nubee,

Please take a look at this KB article provided by Microsoft - Upgrade virtual machine version in Hyper-V on Windows 10 or Windows Server. Thank you!

[mkaec]

I love it when the answer is "do this" ("upgrade configuration version") !
Nobody ever mentioned where this setting is or how to do it.
...

I think users in this forum are assumed to be of a higher technical caliber. Dima's link is the first result.

https://www.google.com/search?q=upgrade ... on+version