Discussions specific to the Microsoft Hyper-V hypervisor
Gostev
SVP, Product Management
Posts: 24825
Liked: 3578 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Dec 13, 2012 9:59 pm

This would depend on the specific issue (there are a few different border cases we have missed in testing), but in most cases it should both fix and prevent.

matizmat
Novice
Posts: 8
Liked: never
Joined: Nov 26, 2012 8:53 am
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by matizmat » Dec 14, 2012 9:16 am

Gostev wrote:Just wanted to update that we will be releasing a patch for the issues that cause this error soon (it is in final testing right now).
I look forward to this patch :)

mmihm@salawus.com
Lurker
Posts: 1
Liked: never
Joined: Dec 14, 2012 6:10 pm
Full Name: Mike
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by mmihm@salawus.com » Dec 14, 2012 6:21 pm

Same problem here! Four 2008 R2 hyperv hosts replicating vm's to two 2008 R2 hyperv hosts in our Colo. "Failed to flush tracking data before snapshot". I upgraded to 6.5, same issue. All hosts are rebooted monthly and the issue still occurs.

jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jorgene » Dec 27, 2012 1:18 pm

Any fix for this issue yet? having the same problem on imported VM's in a W2012 Cluster.

Vitaliy S.
Product Manager
Posts: 23016
Liked: 1561 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Vitaliy S. » Dec 27, 2012 1:52 pm

Jorgen, have you had a chance to apply our latest patch to resolve this issue?

jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jorgene » Dec 27, 2012 1:57 pm

Ohh, dowloading and installing now ;) thx.
FYI. It seems like a Move of Virtual Machine Store in a Cluster also makes the CBT break.
I will apply the patch and report back.

jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jorgene » Dec 27, 2012 2:34 pm 2 people like this post

Hi, works correctly after applying the patch. After moving a Virtual Machine Store of a VM to Another LUN/CSV it seems like it takes longer time to backup that VM the first time (probably need to read entire VM file again) but no warnings in backup log file. All seems to work fine after the patch.

Thx !

dan.hoffman
Novice
Posts: 7
Liked: never
Joined: Feb 24, 2012 10:50 pm
Full Name: Dan Hoffman
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by dan.hoffman » Dec 28, 2012 2:17 am

I must have missed something. I installed the patch, and it updated all the hosts just fine. But, I am still getting the problem. I even restarted 2 VMs and one of the hosts to test. Restarted the Veeam server and repository as well. Still getting the same issue when running replications and backups.
Anyone else?

Vitaliy S.
Product Manager
Posts: 23016
Liked: 1561 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Vitaliy S. » Dec 28, 2012 8:17 am

Hi Dan, I guess our support should take a look at your logs after you've applied this patch to see what is the reason for CBT flush failure messages.

jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jorgene » Dec 28, 2012 3:31 pm

Hi again, after adding a new VM (W2008R2) and converting the VHD file to VHDX and adding it to the Cluster backup jobs on that VM now gets the same error again (CBT fails to flush) (i have applied the patch). I made a new job new backup respository but the same error.

I applied the patch again on the Veeam B&R server, and the error dissapears again.
So everytime i import a VM in the cluster i need to reapply the patch, to make the CBT error go away.

dan.hoffman
Novice
Posts: 7
Liked: never
Joined: Feb 24, 2012 10:50 pm
Full Name: Dan Hoffman
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by dan.hoffman » Dec 28, 2012 5:21 pm

Vitaliy S. wrote:Hi Dan, I guess our support should take a look at your logs after you've applied this patch to see what is the reason for CBT flush failure messages.
Just opened a ticket. Thanks.

Gostev
SVP, Product Management
Posts: 24825
Liked: 3578 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Dec 29, 2012 5:23 pm

jorgene wrote:So everytime i import a VM in the cluster i need to reapply the patch, to make the CBT error go away.
This makes no sense to me. How overwriting the existing files with the same ones can change anything at all? I recommend investigating this with the support.

rkovhaev
Veeam Software
Posts: 39
Liked: 21 times
Joined: May 17, 2010 6:49 pm
Full Name: Rustam
Location: hockey night in canada
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by rkovhaev » Dec 31, 2012 11:24 am

Dan, Jorgen,
Could you please let me know your support req. numbers, there is a possibility that old FCT driver is still in memory and we will need to manually load new FCT driver.
Another option would be scheduling a quick downtime for production or live migrate VMs(if it's a cluster) to another node and reboot affected HyperV nodes.

dan.hoffman
Novice
Posts: 7
Liked: never
Joined: Feb 24, 2012 10:50 pm
Full Name: Dan Hoffman
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by dan.hoffman » Dec 31, 2012 5:25 pm

I already tried migrating two affected vms off one host and restarted the hosts then migrated the vms back, it didn't help. I even restarted the VMs just to make sure. Still getting the same error.
case: 00169668

rkovhaev
Veeam Software
Posts: 39
Liked: 21 times
Joined: May 17, 2010 6:49 pm
Full Name: Rustam
Location: hockey night in canada
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by rkovhaev » Jan 02, 2013 1:07 pm

Dan, it looks like you are using cluster disk instead of cluster shared volume (CSV). Cluster disk is not supported yet, support will be added in one of the upcoming releases.

dan.hoffman
Novice
Posts: 7
Liked: never
Joined: Feb 24, 2012 10:50 pm
Full Name: Dan Hoffman
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by dan.hoffman » Jan 04, 2013 7:54 pm

rkovhaev wrote:Dan, it looks like you are using cluster disk instead of cluster shared volume (CSV). Cluster disk is not supported yet, support will be added in one of the upcoming releases.
Odd, it was working great until I upgraded to 6.5 . Will the new release be coming sometime soon?

Gostev
SVP, Product Management
Posts: 24825
Liked: 3578 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Jan 04, 2013 11:19 pm

dan.hoffman wrote:Odd, it was working great until I upgraded to 6.5 . Will the new release be coming sometime soon?
Please note that "in one of the upcoming releases" does not necessarily mean "the next release".
That said, let me get more details on this specific issue with cluster disks.

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by collinp » Jan 07, 2013 4:18 pm

After the patch, there are still CBT issues in my case, although Veeam no longer reports issues in the daily email reports. I am working with tech support on this. A large amount of blocks are still being read (100's of gb's - 80% of the entire .vhd) in order backup only mb's of data for the problem vm's. A symptom that I am seeing when there are cbt issues is an artificially high double-digit dedup ratio.

Here is the screenshot url:
https://picasaweb.google.com/cannonc34/ ... 3830837378

Gostev
SVP, Product Management
Posts: 24825
Liked: 3578 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Jan 07, 2013 8:16 pm

When reporting any technical issues, kindly please include the corresponding support case ID.

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by collinp » Jan 14, 2013 4:16 pm

The case # is 00153186 and all logs have been uploaded. I have not heard back from support yet since installing the patch and following up with support on the 7th. The previous case # was 00121732. I have been having these CBT backup issues with Hyper-v since first using Veeam. After the release of each new filter driver, things have continually been getting better and better. I am now down to a half dozen or so virtual machines where a large amount of data still needs to be read in order to backup only mb's. I'm just trying to do what I can to make the product better. When we begin backing up larger quantities of vm's, we will want the vm's to backup in minutes versus hours.

veremin
Product Manager
Posts: 16913
Liked: 1440 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by veremin » Jan 14, 2013 9:50 pm

Even though the cause of this issue has been already mentioned (usage of clusted disks) I’ve decided to update the topic with the corresponding Veeam KB for the convenience of further readers.

Thanks.

jake1110
Enthusiast
Posts: 40
Liked: 2 times
Joined: Sep 20, 2012 6:19 pm
Full Name: Jake Ernst
Location: Des Moines, IA
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jake1110 » Jan 17, 2013 7:10 pm

Oh this is absolutely killing me.

So I had already manually updated VeeamFCT driver(with support) which fixed some VM's but others were failing to use CBT, luckily my two monster file servers were still working:)

Now I apply the cumulative patch today from 12-24 and while it appears to have fixed most VM's, my two huge file servers now display the dreaded message again....unbelievable. They were just working perfectly for the last month. Nothing has changed whatsoever. I've been fighting this for so long and thought maybe the cumulative patch would fix things. I was also waiting to finalize moving all my VM's to our 2012 cluster.

If I could get 80 - 90% of VM's using CBT(including our two huge file servers) I would be satisfied. Not sure if I should open yet another ticket or try the same troubleshooting as before. Watching 3.5TB of full-incremental scans of my file servers is not fun at all....

MarcPiquer
Novice
Posts: 3
Liked: never
Joined: Dec 23, 2011 8:32 pm
Full Name: Marc Piquer
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by MarcPiquer » Feb 06, 2013 11:15 pm

Hello, there are prevision to post new patch for resolve this issue?

Vitaliy S.
Product Manager
Posts: 23016
Liked: 1561 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Vitaliy S. » Feb 07, 2013 7:46 am

Hello Marc,

Not sure I understand the question, have you applied this patch already?

Thanks!

MarcPiquer
Novice
Posts: 3
Liked: never
Joined: Dec 23, 2011 8:32 pm
Full Name: Marc Piquer
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by MarcPiquer » Feb 07, 2013 11:08 am

Yes, i have applied the patch 6.5.0.128, but the error persist. :(

Vitaliy S.
Product Manager
Posts: 23016
Liked: 1561 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Vitaliy S. » Feb 07, 2013 11:11 am

Then please contact our support team to review your debug log files, maybe the patch wasn't installed correctly.

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by collinp » Feb 08, 2013 7:22 pm 3 people like this post

I just resolved my CBT issues recently. It turns out that we had some vm's that had 'pending' year old chained VMM checkpoints associated with them. The checkpoints were deleted from VMM by the VM administrators but the vm's were never powered off for a long enough period to let the checkpoints merge properly. If you do a find file *.avhd on c:\clusterstorage\ on a hyper-v host you can see if you still have pending snapshots around. I shutdown the 20 or so vm's one at a time and watched from Hyper-v manager and it took 3 hours for all of checkpoints to merge before I could power the vm's back on. Once I cleaned up all the vmm checkpoints, Veeam CBT began working for everything.

I know that Veeam has created it's own CBT filter drive and that theoretically, CBT should still work with Hyper-v checkpoints present. For the most part, this was true in my case except for the checkpoints that were left around for way too long! I believe that VMM checkpoints aren't recommended in productio on 2008 R2, but if they are used they should only be kept for a short period of time.

Gostev
SVP, Product Management
Posts: 24825
Liked: 3578 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by Gostev » Feb 10, 2013 10:06 am

Thanks Collin for taking your time to provide the update, this possible cause is very useful to know.

jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jorgene » Feb 11, 2013 3:18 pm

Hmm.. i just created 3 new VM's in my W2012 Cluster (One W2012 and 2 W2008R2). All new VM's now get Warning "Failed to flush change tracking data before snapshot. Using full scan incremental run" during backup.
First and Second run. Also i had a problem with one of my cluster nodes last week, resulting in that one of the VM's on that node moved to another node. That VM also get this Warning "Failed to flush change tracking data before snapshot. Using full scan incremental run" during backups. But after i reapply the patch 6.5.0.128 on the Veeam B&R Server (not on the Hyper-V Hosts) that backup job run ok again.
I reapplied the patch again on the Veeam B&R server and even rebooted, but the 3 new VM's now all still have Warning message during backup jobs.

jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Post by jorgene » Feb 11, 2013 3:57 pm

Tried a reboot on the Cluster node that holds the running VM's. Still same problem and Warning errors?
What should i do about this, there is no logic in this.

Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests