-
- Enthusiast
- Posts: 40
- Liked: 2 times
- Joined: Sep 20, 2012 6:19 pm
- Full Name: Jake Ernst
- Location: Des Moines, IA
- Contact:
CBT Fails to Flush on All VMs, Backups taking 12+ hours
Hi All
I'm currently fighting an issue where EVERY Hyper-V VM we have gives the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental." This of course, means that Veeam has to read the entire contents of the VHD to back it up, resulting in over 12 hour backup windows and killing us. We have a 2008 R2 based cluster that always had this error and a 2012 cluster that never had the issue. While I thought originally that moving the VM's from the old to the new cluster would resolve the issue, it still persists when the VM's are migrated to the working 2012 cluster. I should mention we're a pure Hyper-V shop.
I've tried many solutions with support. We tried removing all the Veeam components from the hosts and re-installing, didn't help. We disabled CBT and then removed the HKLM\System\CurrentControlSet\Services\VeeamFCT\Parameters\CTBD key, re-enabling CBT then ran a job to see if it would reset, didn't work. I also moved a VM to a stand-alone host, then backed it up but it still came up with the error until I completely isolated it on it's own host and own storage target so it wouldn't reference any old backup sets. I don't have time to evict each node, rebuild them, remove backup sets and start all over again. There's a PowerShell command that can be run, but it errors out with Reset-HvVmChangeTracking : Cannot bind parameter, ect, ect.
As i mentioned earlier, Veeam works great on our 2012 cluster, except for the VM's migrated from our old 2008 cluster. We're relatively new to Veeam and this doesn't reflect well on my decision to choose the product over competitors unfortunately. When it works, it works great! We're still working out the kinks on other small problems too, but this is by far the most annoying issue I have. Has anyone else encountered the error before? I was told by support, that removing all the components, the reg key and re-enabling CBT fixes the problem in their lab but it hasn't for me.
Thanks
Jake
I'm currently fighting an issue where EVERY Hyper-V VM we have gives the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental." This of course, means that Veeam has to read the entire contents of the VHD to back it up, resulting in over 12 hour backup windows and killing us. We have a 2008 R2 based cluster that always had this error and a 2012 cluster that never had the issue. While I thought originally that moving the VM's from the old to the new cluster would resolve the issue, it still persists when the VM's are migrated to the working 2012 cluster. I should mention we're a pure Hyper-V shop.
I've tried many solutions with support. We tried removing all the Veeam components from the hosts and re-installing, didn't help. We disabled CBT and then removed the HKLM\System\CurrentControlSet\Services\VeeamFCT\Parameters\CTBD key, re-enabling CBT then ran a job to see if it would reset, didn't work. I also moved a VM to a stand-alone host, then backed it up but it still came up with the error until I completely isolated it on it's own host and own storage target so it wouldn't reference any old backup sets. I don't have time to evict each node, rebuild them, remove backup sets and start all over again. There's a PowerShell command that can be run, but it errors out with Reset-HvVmChangeTracking : Cannot bind parameter, ect, ect.
As i mentioned earlier, Veeam works great on our 2012 cluster, except for the VM's migrated from our old 2008 cluster. We're relatively new to Veeam and this doesn't reflect well on my decision to choose the product over competitors unfortunately. When it works, it works great! We're still working out the kinks on other small problems too, but this is by far the most annoying issue I have. Has anyone else encountered the error before? I was told by support, that removing all the components, the reg key and re-enabling CBT fixes the problem in their lab but it hasn't for me.
Thanks
Jake
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Hi Jake - please include your support case ID, as requested when you click New Topic. Thanks!
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Dec 27, 2011 1:13 pm
- Full Name: Igor Kachuriner
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Hello
I'm expiriencing the same issue at my customer's site.
My case number is 00155669.
I'm expiriencing the same issue at my customer's site.
My case number is 00155669.
-
- Enthusiast
- Posts: 40
- Liked: 2 times
- Joined: Sep 20, 2012 6:19 pm
- Full Name: Jake Ernst
- Location: Des Moines, IA
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
My case number is 00153835.
While I originally thought it was strictly an issue with the 2008 R2 cluster hosts, every single VM we migrated to the "working" 2012 cluster had the issue persist. I'm awaiting another call back to see what my options are.
While I originally thought it was strictly an issue with the 2008 R2 cluster hosts, every single VM we migrated to the "working" 2012 cluster had the issue persist. I'm awaiting another call back to see what my options are.
-
- Enthusiast
- Posts: 53
- Liked: 2 times
- Joined: Feb 10, 2011 7:27 pm
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
some, but not all, of my hyper-v vm's are experiencing this too. please update this thread as you learn more, thanks
-
- Service Provider
- Posts: 3
- Liked: 1 time
- Joined: Jul 04, 2012 2:13 pm
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Having the same problem.
Case# 00159458
Case# 00159458
-
- Enthusiast
- Posts: 40
- Liked: 2 times
- Joined: Sep 20, 2012 6:19 pm
- Full Name: Jake Ernst
- Location: Des Moines, IA
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
I should also note, this only appears to occur on clusters. I don't have many stand alone hosts to test, but every time I manually move a VM to a stand alone host, it backs up fine.
-
- Expert
- Posts: 239
- Liked: 13 times
- Joined: Feb 14, 2012 8:56 pm
- Full Name: Collin P
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
My case # for this is 00153186. Third level support on it since it is related to a month long case - 00121732. Just uploaded a fresh set of logs. In my case, these CBT failures seem to happen on jobs larger than 4 vm's. The integrity of the backups are ok, they just take longer since the backup has to re-scan the entire vm before backing up instead of just changed blocks.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
I will check on the current status for this issues, thanks all for opening support cases as this will raise the priority of this issue!
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Nov 21, 2012 1:46 pm
- Full Name: Andreas Lundgren
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Just wanted to chime in with another identical issue. After a reboot of each cluster node I'm now up to 50% of VMs running with CBT on, but the remaining ones still seem to be hard to get working. Have tried to reset CBT data with the powershell cmdlet but that didn't work either.
I do NOT see any differences if there are 20 VMs in a job or just 2... Still some VMs work and some don't. Always seems to be the same ones that fail, but that's just a gut feeling, haven't yet verified that.
I find it amusing that *all* information available on CBT on Veeam's web & FAQ relates to VMware and not Hyper-V
Hyper-V 2008 R2 SP1, VeeamBnR 6.5.0.106.
Support ID# 00152909.
/Andreas
I do NOT see any differences if there are 20 VMs in a job or just 2... Still some VMs work and some don't. Always seems to be the same ones that fail, but that's just a gut feeling, haven't yet verified that.
I find it amusing that *all* information available on CBT on Veeam's web & FAQ relates to VMware and not Hyper-V
Hyper-V 2008 R2 SP1, VeeamBnR 6.5.0.106.
Support ID# 00152909.
/Andreas
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Nov 21, 2012 1:46 pm
- Full Name: Andreas Lundgren
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
I also had that issue...jake1110 wrote: There's a PowerShell command that can be run, but it errors out with Reset-HvVmChangeTracking : Cannot bind parameter, ect, ect.
Seems it should be run as such:
PS D:\> $serv = get-vbrserver -name <YourHyperVHost>
PS D:\> Reset-HvVmChangeTracking -server $serv -vmname <YourVMNameHere>
It does not give any console output but $lastexitcode is "0" so hopefully it worked... In my view the documentation on this could be a bit clearer.
/Andreas
-
- Expert
- Posts: 239
- Liked: 13 times
- Joined: Feb 14, 2012 8:56 pm
- Full Name: Collin P
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Same experience here as with Ante. It is usually the same vm's that fail CBT.
-
- Enthusiast
- Posts: 40
- Liked: 2 times
- Joined: Sep 20, 2012 6:19 pm
- Full Name: Jake Ernst
- Location: Des Moines, IA
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Andreas - I too got a 0 exit code as well using the code you ran, unfortunately still received the error. Unfortunately for me, over half my VM's have this error, including all of our "heavy hitters." Since it's so close to the Thanksgiving holiday, I'm unable to reboot any servers today but I'll certainly do more testing using that syntax for the reset cmdlet.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Hyper-V specific FAQ simply does not exist yet - at this time, I am still in the phase of identifying "frequently" asked questionsante wrote:I find it amusing that *all* information available on CBT on Veeam's web & FAQ relates to VMware and not Hyper-V
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
All, we have very little logs submitted for this issue, and cannot investigate it efficiently. If you did not submit the debug logs package, please do. Thank you very much.
-
- Service Provider
- Posts: 20
- Liked: 1 time
- Joined: Aug 08, 2012 9:06 am
- Full Name: Magne Stenwig
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
I had the same problem after upgrading to 6.5. All the VM-backups in my 2008 R2 Hyper-V cluster failed with the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental.".
After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again
After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again
-
- Novice
- Posts: 8
- Liked: never
- Joined: Nov 26, 2012 8:53 am
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
hi,
Have the same problem, i have submit the log of my backup server (Case # 00161093), all my VM in the 2012 hyper-v cluster are in "Warning" : "Failed to flush change tracking data before snapshot. Using full scan incremental run."
My environment :
1 cluster Windows 2008 R2, Hyper-v (6 physical host), no problem with the backup.
1 cluster Windows 2012 server,Hyper-v (3 physical host), backup during long time and in warning state.
Have the same problem, i have submit the log of my backup server (Case # 00161093), all my VM in the 2012 hyper-v cluster are in "Warning" : "Failed to flush change tracking data before snapshot. Using full scan incremental run."
My environment :
1 cluster Windows 2008 R2, Hyper-v (6 physical host), no problem with the backup.
1 cluster Windows 2012 server,Hyper-v (3 physical host), backup during long time and in warning state.
-
- Enthusiast
- Posts: 40
- Liked: 2 times
- Joined: Sep 20, 2012 6:19 pm
- Full Name: Jake Ernst
- Location: Des Moines, IA
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Did you do anything else before rebooting, like attempt to manually reset CBT or uninstall/reinstall components? I have an 8 node cluster and working a reboot of all nodes into my day is quite a chore. Has anyone else tried rebooting all their nodes? I'd like to see it work for someone else before I do it, but I'm low on ideas so I might try it.After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again
-
- Enthusiast
- Posts: 51
- Liked: 5 times
- Joined: Aug 29, 2012 5:36 pm
- Full Name: Shralok
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
I'm also interested in a resolution other than "reboot the hosts" since rebooting all 4 of ours takes a few hours with moving VMs back and forth.jake1110 wrote: Did you do anything else before rebooting, like attempt to manually reset CBT or uninstall/reinstall components? I have an 8 node cluster and working a reboot of all nodes into my day is quite a chore. Has anyone else tried rebooting all their nodes? I'd like to see it work for someone else before I do it, but I'm low on ideas so I might try it.
My "Warning" message is the same:
Failed to flush change tracking data before snapshot. Using full scan incremental run.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Nov 26, 2012 8:53 am
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
I have reboot all my servers (but one by one) because i have some VMs that i can't shutdown ... it don't work.Magz wrote:I had the same problem after upgrading to 6.5. All the VM-backups in my 2008 R2 Hyper-V cluster failed with the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental.".
After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again
question : Your solution is stop all server then boot all server in same time ?
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
In many cases (but not all), we are seeing this issue being caused by "stuck" old CBT driver from the previous B&R version. All our future updates and releases will include a special procedure during the upgrade to avoid this issue. Meanwhile, rebooting the hosts during your next maintenance window (e.g. patch Tuesday) should help to make CBT working.
-
- Enthusiast
- Posts: 40
- Liked: 2 times
- Joined: Sep 20, 2012 6:19 pm
- Full Name: Jake Ernst
- Location: Des Moines, IA
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
It took me a full day of shuffling around VM's on our "old" 7 node cluster, but I finally did it and had them all reboot. I started first by uninstalling/reinstalling all the Veeam components on each host(just to make myself feel better) and then one by one, rebooted each of them. This appears to have fixed the issue on 90% of them, I'm still waiting on my backups to complete to see if any more appear. So far, only 2 still show the error, which is fine since it used to be 70+. Once I get all the backups done, I'll re-evaluate the situation.
For those of you doing the reboots, you have to restart every host on the cluster. I did individual hosts and CBT was still broken for me. I still find it strange how CBT was still broken after I evicted a node, rebuilt the node as a standalone server, then ran a backup off it and still had the error.
Anyway, things are at least starting to look better!
For those of you doing the reboots, you have to restart every host on the cluster. I did individual hosts and CBT was still broken for me. I still find it strange how CBT was still broken after I evicted a node, rebuilt the node as a standalone server, then ran a backup off it and still had the error.
Anyway, things are at least starting to look better!
-
- Enthusiast
- Posts: 40
- Liked: 2 times
- Joined: Sep 20, 2012 6:19 pm
- Full Name: Jake Ernst
- Location: Des Moines, IA
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
I do have one other question if anyone is still watching this thread.
If I migrate VM's from one cluster to another, will this cause the same CBT flush failure? I've been following Microsoft practice of moving CSV's from our 2008 R2 cluster to 2012 cluster and every one I move fails to flush it's CBT data. I assume this is because it's impossible for the new cluster to know what data has changed since it was on the old cluster. I'm guessing the answer has something do to with moving them via Veeam instead, but I was just wondering because it's a lot quicker for me to import them in by simply disconnecting the volume, bringing it up on the new cluster and importing the VM's in failover cluster manager.
If I migrate VM's from one cluster to another, will this cause the same CBT flush failure? I've been following Microsoft practice of moving CSV's from our 2008 R2 cluster to 2012 cluster and every one I move fails to flush it's CBT data. I assume this is because it's impossible for the new cluster to know what data has changed since it was on the old cluster. I'm guessing the answer has something do to with moving them via Veeam instead, but I was just wondering because it's a lot quicker for me to import them in by simply disconnecting the volume, bringing it up on the new cluster and importing the VM's in failover cluster manager.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Yes, if you migrate VMs from one cluster to another, CBT data will be no longer valid and you will have to flush change tracking information for migrated VM. To do this, you need to run these commands on all your Hyper-V nodes:
Keep in mind that this will flush change tracking info for all VMs on all volumes, so the next job run for all VMs will be full.
Code: Select all
sc stop VeeamFCT
sc start VeeamFCT
-
- Novice
- Posts: 8
- Liked: never
- Joined: Nov 26, 2012 8:53 am
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
jake1110, It's exactly my situation today, all vm on my old CSV 2008 that i have migrate on my CSV 2012 are in warning state with the error "Failed to flush change tracking data before snapshot. Using full scan incremental run."jake1110 wrote:I do have one other question if anyone is still watching this thread.
If I migrate VM's from one cluster to another, will this cause the same CBT flush failure? I've been following Microsoft practice of moving CSV's from our 2008 R2 cluster to 2012 cluster and every one I move fails to flush it's CBT data. I assume this is because it's impossible for the new cluster to know what data has changed since it was on the old cluster. I'm guessing the answer has something do to with moving them via Veeam instead, but I was just wondering because it's a lot quicker for me to import them in by simply disconnecting the volume, bringing it up on the new cluster and importing the VM's in failover cluster manager.
I have use the export option of hyper-v 2008 an import on hyper-v 2012.
Vitaliy S., i was happy when i have read your post, but it don't work for me. I have apply these command but no change during the backup ... already the warning "failed to flush change tracking data ..."Vitaliy S. wrote:Yes, if you migrate VMs from one cluster to another, CBT data will be no longer valid and you will have to flush change tracking information for migrated VM. To do this, you need to run these commands on all your Hyper-V nodes:Keep in mind that this will flush change tracking info for all VMs on all volumes, so the next job run for all VMs will be full.Code: Select all
sc stop VeeamFCT sc start VeeamFCT
I have no news from support at this time.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Ok, let's wait what our support engineers can say after reviewing the debug log files.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Well, I would definitely expect the first job run for each VM after resetting CBT to not use CBT data what about the following runs?
-
- Enthusiast
- Posts: 40
- Liked: 2 times
- Joined: Sep 20, 2012 6:19 pm
- Full Name: Jake Ernst
- Location: Des Moines, IA
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
I had the same result as matizmat. The stop and starting of the VeeamFCT service doesn't fix the issue. In the past, I would have to either uninstall/reinstall components and reboot all nodes or disable CBT in the GUI and remove the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VeeamFCT\Parameters\CTDB, then reboot and enable CBT in the GUI again. Not sure what else I can try out short of blowing everything away in backups and host components involved, then rebuilding. Of course, I'm not THAT desperate....yet
Too bad I wasn't in the beta longer than the few days I had it! I had 100+ VM's ready to migrate to 2012 but we're slowing down until we can correct the backup jobs. My cluster has 11 nodes so anything that requires me rebooting or interacting with the hosts takes a long time.
Too bad I wasn't in the beta longer than the few days I had it! I had 100+ VM's ready to migrate to 2012 but we're slowing down until we can correct the backup jobs. My cluster has 11 nodes so anything that requires me rebooting or interacting with the hosts takes a long time.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
Just wanted to update that we will be releasing a patch for the issues that cause this error soon (it is in final testing right now).
-
- Enthusiast
- Posts: 53
- Liked: 2 times
- Joined: Feb 10, 2011 7:27 pm
- Contact:
Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours
will the patch just prevent it from happening, or will it fix the problem when it is already happening, or both?Gostev wrote:Just wanted to update that we will be releasing a patch for the issues that cause this error soon (it is in final testing right now).
Who is online
Users browsing this forum: Google [Bot] and 16 guests