CBT Fails to Flush on All VMs, Backups taking 12+ hours

Hyper-V specific discussions

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby dan.hoffman » Fri Jan 04, 2013 7:54 pm

rkovhaev wrote:Dan, it looks like you are using cluster disk instead of cluster shared volume (CSV). Cluster disk is not supported yet, support will be added in one of the upcoming releases.

Odd, it was working great until I upgraded to 6.5 . Will the new release be coming sometime soon?
dan.hoffman
Novice
 
Posts: 7
Liked: never
Joined: Fri Feb 24, 2012 10:50 pm
Full Name: Dan Hoffman

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Gostev » Fri Jan 04, 2013 11:19 pm

dan.hoffman wrote:Odd, it was working great until I upgraded to 6.5 . Will the new release be coming sometime soon?

Please note that "in one of the upcoming releases" does not necessarily mean "the next release".
That said, let me get more details on this specific issue with cluster disks.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby collinp » Mon Jan 07, 2013 4:18 pm

After the patch, there are still CBT issues in my case, although Veeam no longer reports issues in the daily email reports. I am working with tech support on this. A large amount of blocks are still being read (100's of gb's - 80% of the entire .vhd) in order backup only mb's of data for the problem vm's. A symptom that I am seeing when there are cbt issues is an artificially high double-digit dedup ratio.

Here is the screenshot url:
https://picasaweb.google.com/cannonc34/ ... 3830837378
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Gostev » Mon Jan 07, 2013 8:16 pm

When reporting any technical issues, kindly please include the corresponding support case ID.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby collinp » Mon Jan 14, 2013 4:16 pm

The case # is 00153186 and all logs have been uploaded. I have not heard back from support yet since installing the patch and following up with support on the 7th. The previous case # was 00121732. I have been having these CBT backup issues with Hyper-v since first using Veeam. After the release of each new filter driver, things have continually been getting better and better. I am now down to a half dozen or so virtual machines where a large amount of data still needs to be read in order to backup only mb's. I'm just trying to do what I can to make the product better. When we begin backing up larger quantities of vm's, we will want the vm's to backup in minutes versus hours.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby v.Eremin » Mon Jan 14, 2013 9:50 pm

Even though the cause of this issue has been already mentioned (usage of clusted disks) I’ve decided to update the topic with the corresponding Veeam KB for the convenience of further readers.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Thu Jan 17, 2013 7:10 pm

Oh this is absolutely killing me.

So I had already manually updated VeeamFCT driver(with support) which fixed some VM's but others were failing to use CBT, luckily my two monster file servers were still working:)

Now I apply the cumulative patch today from 12-24 and while it appears to have fixed most VM's, my two huge file servers now display the dreaded message again....unbelievable. They were just working perfectly for the last month. Nothing has changed whatsoever. I've been fighting this for so long and thought maybe the cumulative patch would fix things. I was also waiting to finalize moving all my VM's to our 2012 cluster.

If I could get 80 - 90% of VM's using CBT(including our two huge file servers) I would be satisfied. Not sure if I should open yet another ticket or try the same troubleshooting as before. Watching 3.5TB of full-incremental scans of my file servers is not fun at all....
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby MarcPiquer » Wed Feb 06, 2013 11:15 pm

Hello, there are prevision to post new patch for resolve this issue?
MarcPiquer
Novice
 
Posts: 3
Liked: never
Joined: Fri Dec 23, 2011 8:32 pm
Full Name: Marc Piquer

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Vitaliy S. » Thu Feb 07, 2013 7:46 am

Hello Marc,

Not sure I understand the question, have you applied this patch already?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby MarcPiquer » Thu Feb 07, 2013 11:08 am

Yes, i have applied the patch 6.5.0.128, but the error persist. :(
MarcPiquer
Novice
 
Posts: 3
Liked: never
Joined: Fri Dec 23, 2011 8:32 pm
Full Name: Marc Piquer

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Vitaliy S. » Thu Feb 07, 2013 11:11 am

Then please contact our support team to review your debug log files, maybe the patch wasn't installed correctly.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby collinp » Fri Feb 08, 2013 7:22 pm 3 people like this post

I just resolved my CBT issues recently. It turns out that we had some vm's that had 'pending' year old chained VMM checkpoints associated with them. The checkpoints were deleted from VMM by the VM administrators but the vm's were never powered off for a long enough period to let the checkpoints merge properly. If you do a find file *.avhd on c:\clusterstorage\ on a hyper-v host you can see if you still have pending snapshots around. I shutdown the 20 or so vm's one at a time and watched from Hyper-v manager and it took 3 hours for all of checkpoints to merge before I could power the vm's back on. Once I cleaned up all the vmm checkpoints, Veeam CBT began working for everything.

I know that Veeam has created it's own CBT filter drive and that theoretically, CBT should still work with Hyper-v checkpoints present. For the most part, this was true in my case except for the checkpoints that were left around for way too long! I believe that VMM checkpoints aren't recommended in productio on 2008 R2, but if they are used they should only be kept for a short period of time.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Gostev » Sun Feb 10, 2013 10:06 am

Thanks Collin for taking your time to provide the update, this possible cause is very useful to know.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jorgene » Mon Feb 11, 2013 3:18 pm

Hmm.. i just created 3 new VM's in my W2012 Cluster (One W2012 and 2 W2008R2). All new VM's now get Warning "Failed to flush change tracking data before snapshot. Using full scan incremental run" during backup.
First and Second run. Also i had a problem with one of my cluster nodes last week, resulting in that one of the VM's on that node moved to another node. That VM also get this Warning "Failed to flush change tracking data before snapshot. Using full scan incremental run" during backups. But after i reapply the patch 6.5.0.128 on the Veeam B&R Server (not on the Hyper-V Hosts) that backup job run ok again.
I reapplied the patch again on the Veeam B&R server and even rebooted, but the 3 new VM's now all still have Warning message during backup jobs.
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jorgene » Mon Feb 11, 2013 3:57 pm

Tried a reboot on the Cluster node that holds the running VM's. Still same problem and Warning errors?
What should i do about this, there is no logic in this.
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

PreviousNext

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 3 guests