CBT Fails to Flush on All VMs, Backups taking 12+ hours

[jake1110]

Hi All

I'm currently fighting an issue where EVERY Hyper-V VM we have gives the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental." This of course, means that Veeam has to read the entire contents of the VHD to back it up, resulting in over 12 hour backup windows and killing us. We have a 2008 R2 based cluster that always had this error and a 2012 cluster that never had the issue. While I thought originally that moving the VM's from the old to the new cluster would resolve the issue, it still persists when the VM's are migrated to the working 2012 cluster. I should mention we're a pure Hyper-V shop.

I've tried many solutions with support. We tried removing all the Veeam components from the hosts and re-installing, didn't help. We disabled CBT and then removed the HKLM\System\CurrentControlSet\Services\VeeamFCT\Parameters\CTBD key, re-enabling CBT then ran a job to see if it would reset, didn't work. I also moved a VM to a stand-alone host, then backed it up but it still came up with the error until I completely isolated it on it's own host and own storage target so it wouldn't reference any old backup sets. I don't have time to evict each node, rebuild them, remove backup sets and start all over again. There's a PowerShell command that can be run, but it errors out with Reset-HvVmChangeTracking : Cannot bind parameter, ect, ect.

As i mentioned earlier, Veeam works great on our 2012 cluster, except for the VM's migrated from our old 2008 cluster. We're relatively new to Veeam and this doesn't reflect well on my decision to choose the product over competitors unfortunately. When it works, it works great! We're still working out the kinks on other small problems too, but this is by far the most annoying issue I have. Has anyone else encountered the error before? I was told by support, that removing all the components, the reg key and re-enabling CBT fixes the problem in their lab but it hasn't for me.

Thanks
Jake

[Gostev]

Hi Jake - please include your support case ID, as requested when you click New Topic. Thanks!

[basset]

Hello

I'm expiriencing the same issue at my customer's site.

My case number is 00155669.

[jake1110]

My case number is 00153835.

While I originally thought it was strictly an issue with the 2008 R2 cluster hosts, every single VM we migrated to the "working" 2012 cluster had the issue persist. I'm awaiting another call back to see what my options are.

[mrt]

some, but not all, of my hyper-v vm's are experiencing this too. please update this thread as you learn more, thanks

[GungaMan]

Having the same problem.
Case# 00159458

[jake1110]

I should also note, this only appears to occur on clusters. I don't have many stand alone hosts to test, but every time I manually move a VM to a stand alone host, it backs up fine.

[collinp]

My case # for this is 00153186. Third level support on it since it is related to a month long case - 00121732. Just uploaded a fresh set of logs. In my case, these CBT failures seem to happen on jobs larger than 4 vm's. The integrity of the backups are ok, they just take longer since the backup has to re-scan the entire vm before backing up instead of just changed blocks.

[Gostev]

I will check on the current status for this issues, thanks all for opening support cases as this will raise the priority of this issue!

[ante]

Just wanted to chime in with another identical issue. After a reboot of each cluster node I'm now up to 50% of VMs running with CBT on, but the remaining ones still seem to be hard to get working. Have tried to reset CBT data with the powershell cmdlet but that didn't work either.

I do NOT see any differences if there are 20 VMs in a job or just 2... Still some VMs work and some don't. Always seems to be the same ones that fail, but that's just a gut feeling, haven't yet verified that.

I find it amusing that *all* information available on CBT on Veeam's web & FAQ relates to VMware and not Hyper-V

Hyper-V 2008 R2 SP1, VeeamBnR 6.5.0.106.
Support ID# 00152909.

/Andreas

[ante]

There's a PowerShell command that can be run, but it errors out with Reset-HvVmChangeTracking : Cannot bind parameter, ect, ect.

I also had that issue...

Seems it should be run as such:
PS D:\> $serv = get-vbrserver -name <YourHyperVHost>
PS D:\> Reset-HvVmChangeTracking -server $serv -vmname <YourVMNameHere>

It does not give any console output but $lastexitcode is "0" so hopefully it worked... In my view the documentation on this could be a bit clearer.

/Andreas

[collinp]

Same experience here as with Ante. It is usually the same vm's that fail CBT.

[jake1110]

Andreas - I too got a 0 exit code as well using the code you ran, unfortunately still received the error. Unfortunately for me, over half my VM's have this error, including all of our "heavy hitters." Since it's so close to the Thanksgiving holiday, I'm unable to reboot any servers today but I'll certainly do more testing using that syntax for the reset cmdlet.

[Gostev]

I find it amusing that *all* information available on CBT on Veeam's web & FAQ relates to VMware and not Hyper-V

Hyper-V specific FAQ simply does not exist yet - at this time, I am still in the phase of identifying "frequently" asked questions

[Gostev]

All, we have very little logs submitted for this issue, and cannot investigate it efficiently. If you did not submit the debug logs package, please do. Thank you very much.

[Magz]

I had the same problem after upgrading to 6.5. All the VM-backups in my 2008 R2 Hyper-V cluster failed with the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental.".

After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again

[matizmat]

hi,

Have the same problem, i have submit the log of my backup server (Case # 00161093), all my VM in the 2012 hyper-v cluster are in "Warning" : "Failed to flush change tracking data before snapshot. Using full scan incremental run."

My environment :

1 cluster Windows 2008 R2, Hyper-v (6 physical host), no problem with the backup.
1 cluster Windows 2012 server,Hyper-v (3 physical host), backup during long time and in warning state.

[jake1110]

After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again

Did you do anything else before rebooting, like attempt to manually reset CBT or uninstall/reinstall components? I have an 8 node cluster and working a reboot of all nodes into my day is quite a chore. Has anyone else tried rebooting all their nodes? I'd like to see it work for someone else before I do it, but I'm low on ideas so I might try it.

[Shralok]

Did you do anything else before rebooting, like attempt to manually reset CBT or uninstall/reinstall components? I have an 8 node cluster and working a reboot of all nodes into my day is quite a chore. Has anyone else tried rebooting all their nodes? I'd like to see it work for someone else before I do it, but I'm low on ideas so I might try it.

I'm also interested in a resolution other than "reboot the hosts" since rebooting all 4 of ours takes a few hours with moving VMs back and forth.

My "Warning" message is the same:

Failed to flush change tracking data before snapshot. Using full scan incremental run.

[matizmat]

I had the same problem after upgrading to 6.5. All the VM-backups in my 2008 R2 Hyper-V cluster failed with the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental.".

After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again

I have reboot all my servers (but one by one) because i have some VMs that i can't shutdown ... it don't work.
question : Your solution is stop all server then boot all server in same time ?

[Gostev]

In many cases (but not all), we are seeing this issue being caused by "stuck" old CBT driver from the previous B&R version. All our future updates and releases will include a special procedure during the upgrade to avoid this issue. Meanwhile, rebooting the hosts during your next maintenance window (e.g. patch Tuesday) should help to make CBT working.

[jake1110]

It took me a full day of shuffling around VM's on our "old" 7 node cluster, but I finally did it and had them all reboot. I started first by uninstalling/reinstalling all the Veeam components on each host(just to make myself feel better) and then one by one, rebooted each of them. This appears to have fixed the issue on 90% of them, I'm still waiting on my backups to complete to see if any more appear. So far, only 2 still show the error, which is fine since it used to be 70+. Once I get all the backups done, I'll re-evaluate the situation.

For those of you doing the reboots, you have to restart every host on the cluster. I did individual hosts and CBT was still broken for me. I still find it strange how CBT was still broken after I evicted a node, rebuilt the node as a standalone server, then ran a backup off it and still had the error.

Anyway, things are at least starting to look better!

[jake1110]

I do have one other question if anyone is still watching this thread.

If I migrate VM's from one cluster to another, will this cause the same CBT flush failure? I've been following Microsoft practice of moving CSV's from our 2008 R2 cluster to 2012 cluster and every one I move fails to flush it's CBT data. I assume this is because it's impossible for the new cluster to know what data has changed since it was on the old cluster. I'm guessing the answer has something do to with moving them via Veeam instead, but I was just wondering because it's a lot quicker for me to import them in by simply disconnecting the volume, bringing it up on the new cluster and importing the VM's in failover cluster manager.

[Vitaliy S.]

Yes, if you migrate VMs from one cluster to another, CBT data will be no longer valid and you will have to flush change tracking information for migrated VM. To do this, you need to run these commands on all your Hyper-V nodes:

Code: Select all

sc stop VeeamFCT
sc start VeeamFCT

Keep in mind that this will flush change tracking info for all VMs on all volumes, so the next job run for all VMs will be full.

[matizmat]

I do have one other question if anyone is still watching this thread.

If I migrate VM's from one cluster to another, will this cause the same CBT flush failure? I've been following Microsoft practice of moving CSV's from our 2008 R2 cluster to 2012 cluster and every one I move fails to flush it's CBT data. I assume this is because it's impossible for the new cluster to know what data has changed since it was on the old cluster. I'm guessing the answer has something do to with moving them via Veeam instead, but I was just wondering because it's a lot quicker for me to import them in by simply disconnecting the volume, bringing it up on the new cluster and importing the VM's in failover cluster manager.

jake1110, It's exactly my situation today, all vm on my old CSV 2008 that i have migrate on my CSV 2012 are in warning state with the error "Failed to flush change tracking data before snapshot. Using full scan incremental run."
I have use the export option of hyper-v 2008 an import on hyper-v 2012.

Yes, if you migrate VMs from one cluster to another, CBT data will be no longer valid and you will have to flush change tracking information for migrated VM. To do this, you need to run these commands on all your Hyper-V nodes:

Code: Select all

sc stop VeeamFCT
sc start VeeamFCT

Keep in mind that this will flush change tracking info for all VMs on all volumes, so the next job run for all VMs will be full.

Vitaliy S., i was happy when i have read your post, but it don't work for me. I have apply these command but no change during the backup ... already the warning "failed to flush change tracking data ..."
I have no news from support at this time.

[Vitaliy S.]

Ok, let's wait what our support engineers can say after reviewing the debug log files.

[Gostev]

Well, I would definitely expect the first job run for each VM after resetting CBT to not use CBT data what about the following runs?

[jake1110]

I had the same result as matizmat. The stop and starting of the VeeamFCT service doesn't fix the issue. In the past, I would have to either uninstall/reinstall components and reboot all nodes or disable CBT in the GUI and remove the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VeeamFCT\Parameters\CTDB, then reboot and enable CBT in the GUI again. Not sure what else I can try out short of blowing everything away in backups and host components involved, then rebuilding. Of course, I'm not THAT desperate....yet

Too bad I wasn't in the beta longer than the few days I had it! I had 100+ VM's ready to migrate to 2012 but we're slowing down until we can correct the backup jobs. My cluster has 11 nodes so anything that requires me rebooting or interacting with the hosts takes a long time.

[Gostev]

Just wanted to update that we will be releasing a patch for the issues that cause this error soon (it is in final testing right now).

[mrt]

Just wanted to update that we will be releasing a patch for the issues that cause this error soon (it is in final testing right now).

will the patch just prevent it from happening, or will it fix the problem when it is already happening, or both?