CBT Fails to Flush on All VMs, Backups taking 12+ hours

Hyper-V specific discussions

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Magz » Mon Nov 26, 2012 7:29 am

I had the same problem after upgrading to 6.5. All the VM-backups in my 2008 R2 Hyper-V cluster failed with the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental.".

After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again :-)
Magz
Influencer
 
Posts: 10
Liked: never
Joined: Wed Aug 08, 2012 9:06 am
Full Name: Magne Stenwig

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby matizmat » Mon Nov 26, 2012 10:21 am

hi,

Have the same problem, i have submit the log of my backup server (Case # 00161093), all my VM in the 2012 hyper-v cluster are in "Warning" : "Failed to flush change tracking data before snapshot. Using full scan incremental run."

My environment :

1 cluster Windows 2008 R2, Hyper-v (6 physical host), no problem with the backup.
1 cluster Windows 2012 server,Hyper-v (3 physical host), backup during long time and in warning state.
matizmat
Novice
 
Posts: 8
Liked: never
Joined: Mon Nov 26, 2012 8:53 am

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Mon Nov 26, 2012 2:16 pm

After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again


Did you do anything else before rebooting, like attempt to manually reset CBT or uninstall/reinstall components? I have an 8 node cluster and working a reboot of all nodes into my day is quite a chore. Has anyone else tried rebooting all their nodes? I'd like to see it work for someone else before I do it, but I'm low on ideas so I might try it.
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Shralok » Mon Nov 26, 2012 2:35 pm

jake1110 wrote:
After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again


Did you do anything else before rebooting, like attempt to manually reset CBT or uninstall/reinstall components? I have an 8 node cluster and working a reboot of all nodes into my day is quite a chore. Has anyone else tried rebooting all their nodes? I'd like to see it work for someone else before I do it, but I'm low on ideas so I might try it.


I'm also interested in a resolution other than "reboot the hosts" since rebooting all 4 of ours takes a few hours with moving VMs back and forth.

My "Warning" message is the same:
Failed to flush change tracking data before snapshot. Using full scan incremental run.
Shralok
Enthusiast
 
Posts: 51
Liked: 5 times
Joined: Wed Aug 29, 2012 5:36 pm
Full Name: Shralok

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby matizmat » Mon Nov 26, 2012 3:10 pm

Magz wrote:I had the same problem after upgrading to 6.5. All the VM-backups in my 2008 R2 Hyper-V cluster failed with the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental.".

After almost two full weeks troublshooting this with Veeam - support, the solution turned out to be very simple: Boot all the Hyper-V host servers! I have 4 Hyper-V hosts, and booting only two of them did not solve anything on any host, but after I had booted the last one, all VM-backups were successful again :-)

I have reboot all my servers (but one by one) because i have some VMs that i can't shutdown ... it don't work.
question : Your solution is stop all server then boot all server in same time ?
matizmat
Novice
 
Posts: 8
Liked: never
Joined: Mon Nov 26, 2012 8:53 am

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Gostev » Mon Nov 26, 2012 4:41 pm

In many cases (but not all), we are seeing this issue being caused by "stuck" old CBT driver from the previous B&R version. All our future updates and releases will include a special procedure during the upgrade to avoid this issue. Meanwhile, rebooting the hosts during your next maintenance window (e.g. patch Tuesday) should help to make CBT working.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Tue Nov 27, 2012 1:59 pm

It took me a full day of shuffling around VM's on our "old" 7 node cluster, but I finally did it and had them all reboot. I started first by uninstalling/reinstalling all the Veeam components on each host(just to make myself feel better) and then one by one, rebooted each of them. This appears to have fixed the issue on 90% of them, I'm still waiting on my backups to complete to see if any more appear. So far, only 2 still show the error, which is fine since it used to be 70+. Once I get all the backups done, I'll re-evaluate the situation.

For those of you doing the reboots, you have to restart every host on the cluster. I did individual hosts and CBT was still broken for me. I still find it strange how CBT was still broken after I evicted a node, rebuilt the node as a standalone server, then ran a backup off it and still had the error.

Anyway, things are at least starting to look better!
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Fri Nov 30, 2012 4:24 pm

I do have one other question if anyone is still watching this thread.

If I migrate VM's from one cluster to another, will this cause the same CBT flush failure? I've been following Microsoft practice of moving CSV's from our 2008 R2 cluster to 2012 cluster and every one I move fails to flush it's CBT data. I assume this is because it's impossible for the new cluster to know what data has changed since it was on the old cluster. I'm guessing the answer has something do to with moving them via Veeam instead, but I was just wondering because it's a lot quicker for me to import them in by simply disconnecting the volume, bringing it up on the new cluster and importing the VM's in failover cluster manager.
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Vitaliy S. » Mon Dec 03, 2012 3:55 pm

Yes, if you migrate VMs from one cluster to another, CBT data will be no longer valid and you will have to flush change tracking information for migrated VM. To do this, you need to run these commands on all your Hyper-V nodes:
Code: Select all
sc stop VeeamFCT
sc start VeeamFCT

Keep in mind that this will flush change tracking info for all VMs on all volumes, so the next job run for all VMs will be full.
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby matizmat » Mon Dec 03, 2012 4:13 pm

jake1110 wrote:I do have one other question if anyone is still watching this thread.

If I migrate VM's from one cluster to another, will this cause the same CBT flush failure? I've been following Microsoft practice of moving CSV's from our 2008 R2 cluster to 2012 cluster and every one I move fails to flush it's CBT data. I assume this is because it's impossible for the new cluster to know what data has changed since it was on the old cluster. I'm guessing the answer has something do to with moving them via Veeam instead, but I was just wondering because it's a lot quicker for me to import them in by simply disconnecting the volume, bringing it up on the new cluster and importing the VM's in failover cluster manager.


jake1110, It's exactly my situation today, all vm on my old CSV 2008 that i have migrate on my CSV 2012 are in warning state with the error "Failed to flush change tracking data before snapshot. Using full scan incremental run."
I have use the export option of hyper-v 2008 an import on hyper-v 2012.

Vitaliy S. wrote:Yes, if you migrate VMs from one cluster to another, CBT data will be no longer valid and you will have to flush change tracking information for migrated VM. To do this, you need to run these commands on all your Hyper-V nodes:
Code: Select all
sc stop VeeamFCT
sc start VeeamFCT

Keep in mind that this will flush change tracking info for all VMs on all volumes, so the next job run for all VMs will be full.


Vitaliy S., i was happy when i have read your post, but it don't work for me. I have apply these command but no change during the backup ... already the warning "failed to flush change tracking data ..."
I have no news from support at this time.

Image
matizmat
Novice
 
Posts: 8
Liked: never
Joined: Mon Nov 26, 2012 8:53 am

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Vitaliy S. » Tue Dec 04, 2012 8:47 am

Ok, let's wait what our support engineers can say after reviewing the debug log files.
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Gostev » Tue Dec 04, 2012 2:35 pm

Well, I would definitely expect the first job run for each VM after resetting CBT to not use CBT data ;) what about the following runs?
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Tue Dec 04, 2012 7:51 pm

I had the same result as matizmat. The stop and starting of the VeeamFCT service doesn't fix the issue. In the past, I would have to either uninstall/reinstall components and reboot all nodes or disable CBT in the GUI and remove the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VeeamFCT\Parameters\CTDB, then reboot and enable CBT in the GUI again. Not sure what else I can try out short of blowing everything away in backups and host components involved, then rebuilding. Of course, I'm not THAT desperate....yet :)

Too bad I wasn't in the beta longer than the few days I had it! I had 100+ VM's ready to migrate to 2012 but we're slowing down until we can correct the backup jobs. My cluster has 11 nodes so anything that requires me rebooting or interacting with the hosts takes a long time.
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Gostev » Thu Dec 13, 2012 8:23 pm

Just wanted to update that we will be releasing a patch for the issues that cause this error soon (it is in final testing right now).
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby mrt » Thu Dec 13, 2012 8:41 pm

Gostev wrote:Just wanted to update that we will be releasing a patch for the issues that cause this error soon (it is in final testing right now).

will the patch just prevent it from happening, or will it fix the problem when it is already happening, or both?
mrt
Enthusiast
 
Posts: 47
Liked: 2 times
Joined: Thu Feb 10, 2011 7:27 pm

PreviousNext

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 6 guests