Discussions specific to the Microsoft Hyper-V hypervisor
jake1110
Enthusiast
Posts: 40
Liked: 2 times
Joined: Sep 20, 2012 6:19 pm
Full Name: Jake Ernst
Location: Des Moines, IA
Contact:

CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jake1110 » Nov 16, 2012 10:21 pm

Hi All

I'm currently fighting an issue where EVERY Hyper-V VM we have gives the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental." This of course, means that Veeam has to read the entire contents of the VHD to back it up, resulting in over 12 hour backup windows and killing us. We have a 2008 R2 based cluster that always had this error and a 2012 cluster that never had the issue. While I thought originally that moving the VM's from the old to the new cluster would resolve the issue, it still persists when the VM's are migrated to the working 2012 cluster. I should mention we're a pure Hyper-V shop.

I've tried many solutions with support. We tried removing all the Veeam components from the hosts and re-installing, didn't help. We disabled CBT and then removed the HKLM\System\CurrentControlSet\Services\VeeamFCT\Parameters\CTBD key, re-enabling CBT then ran a job to see if it would reset, didn't work. I also moved a VM to a stand-alone host, then backed it up but it still came up with the error until I completely isolated it on it's own host and own storage target so it wouldn't reference any old backup sets. I don't have time to evict each node, rebuild them, remove backup sets and start all over again. There's a PowerShell command that can be run, but it errors out with Reset-HvVmChangeTracking : Cannot bind parameter, ect, ect.

As i mentioned earlier, Veeam works great on our 2012 cluster, except for the VM's migrated from our old 2008 cluster. We're relatively new to Veeam and this doesn't reflect well on my decision to choose the product over competitors unfortunately. When it works, it works great! We're still working out the kinks on other small problems too, but this is by far the most annoying issue I have. Has anyone else encountered the error before? I was told by support, that removing all the components, the reg key and re-enabling CBT fixes the problem in their lab but it hasn't for me.

Thanks
Jake

Gostev
SVP, Product Management
Posts: 24787
Liked: 3516 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Nov 16, 2012 10:52 pm

Hi Jake - please include your support case ID, as requested when you click New Topic. Thanks!

basset
Lurker
Posts: 1
Liked: never
Joined: Dec 27, 2011 1:13 pm
Full Name: Igor Kachuriner
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by basset » Nov 18, 2012 9:40 pm

Hello

I'm expiriencing the same issue at my customer's site.

My case number is 00155669.

jake1110
Enthusiast
Posts: 40
Liked: 2 times
Joined: Sep 20, 2012 6:19 pm
Full Name: Jake Ernst
Location: Des Moines, IA
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jake1110 » Nov 19, 2012 1:42 pm

My case number is 00153835.

While I originally thought it was strictly an issue with the 2008 R2 cluster hosts, every single VM we migrated to the "working" 2012 cluster had the issue persist. I'm awaiting another call back to see what my options are.

mrt
Enthusiast
Posts: 47
Liked: 2 times
Joined: Feb 10, 2011 7:27 pm
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by mrt » Nov 19, 2012 3:18 pm

some, but not all, of my hyper-v vm's are experiencing this too. please update this thread as you learn more, thanks

GungaMan
Service Provider
Posts: 2
Liked: 1 time
Joined: Jul 04, 2012 2:13 pm
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by GungaMan » Nov 19, 2012 3:23 pm

Having the same problem.
Case# 00159458

jake1110
Enthusiast
Posts: 40
Liked: 2 times
Joined: Sep 20, 2012 6:19 pm
Full Name: Jake Ernst
Location: Des Moines, IA
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jake1110 » Nov 19, 2012 4:57 pm

I should also note, this only appears to occur on clusters. I don't have many stand alone hosts to test, but every time I manually move a VM to a stand alone host, it backs up fine.

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by collinp » Nov 19, 2012 6:58 pm

My case # for this is 00153186. Third level support on it since it is related to a month long case - 00121732. Just uploaded a fresh set of logs. In my case, these CBT failures seem to happen on jobs larger than 4 vm's. The integrity of the backups are ok, they just take longer since the backup has to re-scan the entire vm before backing up instead of just changed blocks.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3516 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Nov 19, 2012 9:03 pm

I will check on the current status for this issues, thanks all for opening support cases as this will raise the priority of this issue!

ante
Lurker
Posts: 2
Liked: never
Joined: Nov 21, 2012 1:46 pm
Full Name: Andreas Lundgren
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by ante » Nov 21, 2012 1:53 pm

Just wanted to chime in with another identical issue. After a reboot of each cluster node I'm now up to 50% of VMs running with CBT on, but the remaining ones still seem to be hard to get working. Have tried to reset CBT data with the powershell cmdlet but that didn't work either.

I do NOT see any differences if there are 20 VMs in a job or just 2... Still some VMs work and some don't. Always seems to be the same ones that fail, but that's just a gut feeling, haven't yet verified that.

I find it amusing that *all* information available on CBT on Veeam's web & FAQ relates to VMware and not Hyper-V ;-)

Hyper-V 2008 R2 SP1, VeeamBnR 6.5.0.106.
Support ID# 00152909.

/Andreas

ante
Lurker
Posts: 2
Liked: never
Joined: Nov 21, 2012 1:46 pm
Full Name: Andreas Lundgren
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by ante » Nov 21, 2012 2:19 pm

jake1110 wrote: There's a PowerShell command that can be run, but it errors out with Reset-HvVmChangeTracking : Cannot bind parameter, ect, ect.
I also had that issue...

Seems it should be run as such:
PS D:\> $serv = get-vbrserver -name <YourHyperVHost>
PS D:\> Reset-HvVmChangeTracking -server $serv -vmname <YourVMNameHere>

It does not give any console output but $lastexitcode is "0" so hopefully it worked... In my view the documentation on this could be a bit clearer.

/Andreas

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by collinp » Nov 21, 2012 6:24 pm

Same experience here as with Ante. It is usually the same vm's that fail CBT.

jake1110
Enthusiast
Posts: 40
Liked: 2 times
Joined: Sep 20, 2012 6:19 pm
Full Name: Jake Ernst
Location: Des Moines, IA
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jake1110 » Nov 21, 2012 6:50 pm

Andreas - I too got a 0 exit code as well using the code you ran, unfortunately still received the error. Unfortunately for me, over half my VM's have this error, including all of our "heavy hitters." Since it's so close to the Thanksgiving holiday, I'm unable to reboot any servers today but I'll certainly do more testing using that syntax for the reset cmdlet.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3516 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Nov 21, 2012 7:42 pm

ante wrote:I find it amusing that *all* information available on CBT on Veeam's web & FAQ relates to VMware and not Hyper-V ;-)
Hyper-V specific FAQ simply does not exist yet - at this time, I am still in the phase of identifying "frequently" asked questions ;)

Gostev
SVP, Product Management
Posts: 24787
Liked: 3516 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Nov 22, 2012 10:55 am

All, we have very little logs submitted for this issue, and cannot investigate it efficiently. If you did not submit the debug logs package, please do. Thank you very much.

Magz
Service Provider
Posts: 13
Liked: 1 time
Joined: Aug 08, 2012 9:06 am
Full Name: Magne Stenwig
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Magz » Nov 26, 2012 7:29 am

I had the same problem after upgrading to 6.5. All the VM-backups in my 2008 R2 Hyper-V cluster failed with the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental.".

After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again :-)

matizmat
Novice
Posts: 8
Liked: never
Joined: Nov 26, 2012 8:53 am
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by matizmat » Nov 26, 2012 10:21 am

hi,

Have the same problem, i have submit the log of my backup server (Case # 00161093), all my VM in the 2012 hyper-v cluster are in "Warning" : "Failed to flush change tracking data before snapshot. Using full scan incremental run."

My environment :

1 cluster Windows 2008 R2, Hyper-v (6 physical host), no problem with the backup.
1 cluster Windows 2012 server,Hyper-v (3 physical host), backup during long time and in warning state.

jake1110
Enthusiast
Posts: 40
Liked: 2 times
Joined: Sep 20, 2012 6:19 pm
Full Name: Jake Ernst
Location: Des Moines, IA
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jake1110 » Nov 26, 2012 2:16 pm

After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again
Did you do anything else before rebooting, like attempt to manually reset CBT or uninstall/reinstall components? I have an 8 node cluster and working a reboot of all nodes into my day is quite a chore. Has anyone else tried rebooting all their nodes? I'd like to see it work for someone else before I do it, but I'm low on ideas so I might try it.

Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Shralok » Nov 26, 2012 2:35 pm

jake1110 wrote: Did you do anything else before rebooting, like attempt to manually reset CBT or uninstall/reinstall components? I have an 8 node cluster and working a reboot of all nodes into my day is quite a chore. Has anyone else tried rebooting all their nodes? I'd like to see it work for someone else before I do it, but I'm low on ideas so I might try it.
I'm also interested in a resolution other than "reboot the hosts" since rebooting all 4 of ours takes a few hours with moving VMs back and forth.

My "Warning" message is the same:
Failed to flush change tracking data before snapshot. Using full scan incremental run.

matizmat
Novice
Posts: 8
Liked: never
Joined: Nov 26, 2012 8:53 am
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by matizmat » Nov 26, 2012 3:10 pm

Magz wrote:I had the same problem after upgrading to 6.5. All the VM-backups in my 2008 R2 Hyper-V cluster failed with the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental.".

After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again :-)
I have reboot all my servers (but one by one) because i have some VMs that i can't shutdown ... it don't work.
question : Your solution is stop all server then boot all server in same time ?

Gostev
SVP, Product Management
Posts: 24787
Liked: 3516 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Nov 26, 2012 4:41 pm

In many cases (but not all), we are seeing this issue being caused by "stuck" old CBT driver from the previous B&R version. All our future updates and releases will include a special procedure during the upgrade to avoid this issue. Meanwhile, rebooting the hosts during your next maintenance window (e.g. patch Tuesday) should help to make CBT working.

jake1110
Enthusiast
Posts: 40
Liked: 2 times
Joined: Sep 20, 2012 6:19 pm
Full Name: Jake Ernst
Location: Des Moines, IA
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jake1110 » Nov 27, 2012 1:59 pm

It took me a full day of shuffling around VM's on our "old" 7 node cluster, but I finally did it and had them all reboot. I started first by uninstalling/reinstalling all the Veeam components on each host(just to make myself feel better) and then one by one, rebooted each of them. This appears to have fixed the issue on 90% of them, I'm still waiting on my backups to complete to see if any more appear. So far, only 2 still show the error, which is fine since it used to be 70+. Once I get all the backups done, I'll re-evaluate the situation.

For those of you doing the reboots, you have to restart every host on the cluster. I did individual hosts and CBT was still broken for me. I still find it strange how CBT was still broken after I evicted a node, rebuilt the node as a standalone server, then ran a backup off it and still had the error.

Anyway, things are at least starting to look better!

jake1110
Enthusiast
Posts: 40
Liked: 2 times
Joined: Sep 20, 2012 6:19 pm
Full Name: Jake Ernst
Location: Des Moines, IA
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jake1110 » Nov 30, 2012 4:24 pm

I do have one other question if anyone is still watching this thread.

If I migrate VM's from one cluster to another, will this cause the same CBT flush failure? I've been following Microsoft practice of moving CSV's from our 2008 R2 cluster to 2012 cluster and every one I move fails to flush it's CBT data. I assume this is because it's impossible for the new cluster to know what data has changed since it was on the old cluster. I'm guessing the answer has something do to with moving them via Veeam instead, but I was just wondering because it's a lot quicker for me to import them in by simply disconnecting the volume, bringing it up on the new cluster and importing the VM's in failover cluster manager.

Vitaliy S.
Product Manager
Posts: 22968
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Vitaliy S. » Dec 03, 2012 3:55 pm

Yes, if you migrate VMs from one cluster to another, CBT data will be no longer valid and you will have to flush change tracking information for migrated VM. To do this, you need to run these commands on all your Hyper-V nodes:

Code: Select all

sc stop VeeamFCT
sc start VeeamFCT
Keep in mind that this will flush change tracking info for all VMs on all volumes, so the next job run for all VMs will be full.

matizmat
Novice
Posts: 8
Liked: never
Joined: Nov 26, 2012 8:53 am
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by matizmat » Dec 03, 2012 4:13 pm

jake1110 wrote:I do have one other question if anyone is still watching this thread.

If I migrate VM's from one cluster to another, will this cause the same CBT flush failure? I've been following Microsoft practice of moving CSV's from our 2008 R2 cluster to 2012 cluster and every one I move fails to flush it's CBT data. I assume this is because it's impossible for the new cluster to know what data has changed since it was on the old cluster. I'm guessing the answer has something do to with moving them via Veeam instead, but I was just wondering because it's a lot quicker for me to import them in by simply disconnecting the volume, bringing it up on the new cluster and importing the VM's in failover cluster manager.
jake1110, It's exactly my situation today, all vm on my old CSV 2008 that i have migrate on my CSV 2012 are in warning state with the error "Failed to flush change tracking data before snapshot. Using full scan incremental run."
I have use the export option of hyper-v 2008 an import on hyper-v 2012.
Vitaliy S. wrote:Yes, if you migrate VMs from one cluster to another, CBT data will be no longer valid and you will have to flush change tracking information for migrated VM. To do this, you need to run these commands on all your Hyper-V nodes:

Code: Select all

sc stop VeeamFCT
sc start VeeamFCT
Keep in mind that this will flush change tracking info for all VMs on all volumes, so the next job run for all VMs will be full.
Vitaliy S., i was happy when i have read your post, but it don't work for me. I have apply these command but no change during the backup ... already the warning "failed to flush change tracking data ..."
I have no news from support at this time.

Image

Vitaliy S.
Product Manager
Posts: 22968
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Vitaliy S. » Dec 04, 2012 8:47 am

Ok, let's wait what our support engineers can say after reviewing the debug log files.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3516 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Dec 04, 2012 2:35 pm

Well, I would definitely expect the first job run for each VM after resetting CBT to not use CBT data ;) what about the following runs?

jake1110
Enthusiast
Posts: 40
Liked: 2 times
Joined: Sep 20, 2012 6:19 pm
Full Name: Jake Ernst
Location: Des Moines, IA
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jake1110 » Dec 04, 2012 7:51 pm

I had the same result as matizmat. The stop and starting of the VeeamFCT service doesn't fix the issue. In the past, I would have to either uninstall/reinstall components and reboot all nodes or disable CBT in the GUI and remove the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VeeamFCT\Parameters\CTDB, then reboot and enable CBT in the GUI again. Not sure what else I can try out short of blowing everything away in backups and host components involved, then rebuilding. Of course, I'm not THAT desperate....yet :)

Too bad I wasn't in the beta longer than the few days I had it! I had 100+ VM's ready to migrate to 2012 but we're slowing down until we can correct the backup jobs. My cluster has 11 nodes so anything that requires me rebooting or interacting with the hosts takes a long time.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3516 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Dec 13, 2012 8:23 pm

Just wanted to update that we will be releasing a patch for the issues that cause this error soon (it is in final testing right now).

mrt
Enthusiast
Posts: 47
Liked: 2 times
Joined: Feb 10, 2011 7:27 pm
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by mrt » Dec 13, 2012 8:41 pm

Gostev wrote:Just wanted to update that we will be releasing a patch for the issues that cause this error soon (it is in final testing right now).
will the patch just prevent it from happening, or will it fix the problem when it is already happening, or both?

Post Reply

Who is online

Users browsing this forum: No registered users and 12 guests