CBT Fails to Flush on All VMs, Backups taking 12+ hours

Hyper-V specific discussions

CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Fri Nov 16, 2012 10:21 pm

Hi All

I'm currently fighting an issue where EVERY Hyper-V VM we have gives the error "Failed to flush change tracking data before taking a snapshot. Using full scan incremental." This of course, means that Veeam has to read the entire contents of the VHD to back it up, resulting in over 12 hour backup windows and killing us. We have a 2008 R2 based cluster that always had this error and a 2012 cluster that never had the issue. While I thought originally that moving the VM's from the old to the new cluster would resolve the issue, it still persists when the VM's are migrated to the working 2012 cluster. I should mention we're a pure Hyper-V shop.

I've tried many solutions with support. We tried removing all the Veeam components from the hosts and re-installing, didn't help. We disabled CBT and then removed the HKLM\System\CurrentControlSet\Services\VeeamFCT\Parameters\CTBD key, re-enabling CBT then ran a job to see if it would reset, didn't work. I also moved a VM to a stand-alone host, then backed it up but it still came up with the error until I completely isolated it on it's own host and own storage target so it wouldn't reference any old backup sets. I don't have time to evict each node, rebuild them, remove backup sets and start all over again. There's a PowerShell command that can be run, but it errors out with Reset-HvVmChangeTracking : Cannot bind parameter, ect, ect.

As i mentioned earlier, Veeam works great on our 2012 cluster, except for the VM's migrated from our old 2008 cluster. We're relatively new to Veeam and this doesn't reflect well on my decision to choose the product over competitors unfortunately. When it works, it works great! We're still working out the kinks on other small problems too, but this is by far the most annoying issue I have. Has anyone else encountered the error before? I was told by support, that removing all the components, the reg key and re-enabling CBT fixes the problem in their lab but it hasn't for me.

Thanks
Jake
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Gostev » Fri Nov 16, 2012 10:52 pm

Hi Jake - please include your support case ID, as requested when you click New Topic. Thanks!
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby basset » Sun Nov 18, 2012 9:40 pm

Hello

I'm expiriencing the same issue at my customer's site.

My case number is 00155669.
basset
Lurker
 
Posts: 1
Liked: never
Joined: Tue Dec 27, 2011 1:13 pm
Full Name: Igor Kachuriner

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Mon Nov 19, 2012 1:42 pm

My case number is 00153835.

While I originally thought it was strictly an issue with the 2008 R2 cluster hosts, every single VM we migrated to the "working" 2012 cluster had the issue persist. I'm awaiting another call back to see what my options are.
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby mrt » Mon Nov 19, 2012 3:18 pm

some, but not all, of my hyper-v vm's are experiencing this too. please update this thread as you learn more, thanks
mrt
Enthusiast
 
Posts: 47
Liked: 2 times
Joined: Thu Feb 10, 2011 7:27 pm

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby GungaMan » Mon Nov 19, 2012 3:23 pm

Having the same problem.
Case# 00159458
GungaMan
Lurker
 
Posts: 2
Liked: 1 time
Joined: Wed Jul 04, 2012 2:13 pm

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Mon Nov 19, 2012 4:57 pm

I should also note, this only appears to occur on clusters. I don't have many stand alone hosts to test, but every time I manually move a VM to a stand alone host, it backs up fine.
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby collinp » Mon Nov 19, 2012 6:58 pm

My case # for this is 00153186. Third level support on it since it is related to a month long case - 00121732. Just uploaded a fresh set of logs. In my case, these CBT failures seem to happen on jobs larger than 4 vm's. The integrity of the backups are ok, they just take longer since the backup has to re-scan the entire vm before backing up instead of just changed blocks.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Gostev » Mon Nov 19, 2012 9:03 pm

I will check on the current status for this issues, thanks all for opening support cases as this will raise the priority of this issue!
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby ante » Wed Nov 21, 2012 1:53 pm

Just wanted to chime in with another identical issue. After a reboot of each cluster node I'm now up to 50% of VMs running with CBT on, but the remaining ones still seem to be hard to get working. Have tried to reset CBT data with the powershell cmdlet but that didn't work either.

I do NOT see any differences if there are 20 VMs in a job or just 2... Still some VMs work and some don't. Always seems to be the same ones that fail, but that's just a gut feeling, haven't yet verified that.

I find it amusing that *all* information available on CBT on Veeam's web & FAQ relates to VMware and not Hyper-V ;-)

Hyper-V 2008 R2 SP1, VeeamBnR 6.5.0.106.
Support ID# 00152909.

/Andreas
ante
Lurker
 
Posts: 2
Liked: never
Joined: Wed Nov 21, 2012 1:46 pm
Full Name: Andreas Lundgren

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby ante » Wed Nov 21, 2012 2:19 pm

jake1110 wrote:There's a PowerShell command that can be run, but it errors out with Reset-HvVmChangeTracking : Cannot bind parameter, ect, ect.


I also had that issue...

Seems it should be run as such:
PS D:\> $serv = get-vbrserver -name <YourHyperVHost>
PS D:\> Reset-HvVmChangeTracking -server $serv -vmname <YourVMNameHere>

It does not give any console output but $lastexitcode is "0" so hopefully it worked... In my view the documentation on this could be a bit clearer.

/Andreas
ante
Lurker
 
Posts: 2
Liked: never
Joined: Wed Nov 21, 2012 1:46 pm
Full Name: Andreas Lundgren

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby collinp » Wed Nov 21, 2012 6:24 pm

Same experience here as with Ante. It is usually the same vm's that fail CBT.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Wed Nov 21, 2012 6:50 pm

Andreas - I too got a 0 exit code as well using the code you ran, unfortunately still received the error. Unfortunately for me, over half my VM's have this error, including all of our "heavy hitters." Since it's so close to the Thanksgiving holiday, I'm unable to reboot any servers today but I'll certainly do more testing using that syntax for the reset cmdlet.
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Gostev » Wed Nov 21, 2012 7:42 pm

ante wrote:I find it amusing that *all* information available on CBT on Veeam's web & FAQ relates to VMware and not Hyper-V ;-)

Hyper-V specific FAQ simply does not exist yet - at this time, I am still in the phase of identifying "frequently" asked questions ;)
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Gostev » Thu Nov 22, 2012 10:55 am

All, we have very little logs submitted for this issue, and cannot investigate it efficiently. If you did not submit the debug logs package, please do. Thank you very much.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 3 guests