CBT Fails to Flush on All VMs, Backups taking 12+ hours

Hyper-V specific discussions

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby tsightler » Mon Feb 11, 2013 3:58 pm

My suggestion would be to open a support case and have them investigate.
tsightler
Veeam Software
 
Posts: 4767
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby rkovhaev » Mon Feb 11, 2013 6:44 pm

jorgene wrote:Tried a reboot on the Cluster node that holds the running VM's. Still same problem and Warning errors?
What should i do about this, there is no logic in this.


Jorgen, could you please let me know your support req. no. I can't find it in our system.
rkovhaev
Veeam Software
 
Posts: 33
Liked: 12 times
Joined: Mon May 17, 2010 6:49 pm
Location: hockey night in canada
Full Name: Rustam

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jorgene » Thu Feb 14, 2013 1:35 pm

Hi, i haven't created a support case for this yet. Thought since many have this problem there was a general solution for this error.
I will create a support case tomorrow when i'm back @ the office.
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jorgene » Thu Feb 14, 2013 9:13 pm

Case Opened: Case ID #: 00181649
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Mon Feb 18, 2013 3:59 pm

collinp wrote:I just resolved my CBT issues recently. It turns out that we had some vm's that had 'pending' year old chained VMM checkpoints associated with them. The checkpoints were deleted from VMM by the VM administrators but the vm's were never powered off for a long enough period to let the checkpoints merge properly. If you do a find file *.avhd on c:\clusterstorage\ on a hyper-v host you can see if you still have pending snapshots around. I shutdown the 20 or so vm's one at a time and watched from Hyper-v manager and it took 3 hours for all of checkpoints to merge before I could power the vm's back on. Once I cleaned up all the vmm checkpoints, Veeam CBT began working for everything.



I wish this was the fix for my CBT woes, but unfortunately we don't take snapshots of VM's except for exceedingly rare occasions for the reason you've mentioned. The snaps grow to become huge and merging takes forever, causing hours of downtime of the VM. Only with Server 2012, have we allowed anyone to take snapshots since it can merge them with the VM powered on. I double checked anyway just to make sure there were no avhd files and there weren't any.

I have another call setup with Support this morning...hoping for the best!
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jorgene » Tue Feb 19, 2013 7:24 am 1 person likes this post

Issue resolved after tons of log files sent to support.
According to support this issue will be resolved in future releases.
My solution to this error was to stop and start VeeamFCT service on all Hyper-V nodes that was connected to the CSV that hold VM's that generated warning messages.
Note: rebooting the nodes didn't relove this error, just the stop and start of the VeeamFCT service when nodes are online.
I will try this again as i will soon migrate/Create more VM's to the cluster.
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Thu Feb 21, 2013 8:19 pm

Yep, the stop/start VeeamFCT driver is something I do all the time. In fact, that was the first solution I was ever offered back in December, but hasn't worked for me. I was on yet another call with Support and that's the only solution I was offered this time around.

I just COMPLETELY rebuilt my Veeam backup server, along with a new database, jobs and uninstall/reinstalled all components on the proxy/repository servers and Hyper-V hosts as well. I completely wiped out all of my backup sets too to make sure nothing was retained.

I ran one test job and immediately received the CBT warning on the first two VM's backed up. I'm hoping this is simply because it's the first time I ran the job, but I doubt that's it.

It's seemingly random which VM's fail and which ones succeed.
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Shralok » Mon Mar 04, 2013 2:33 pm

jorgene wrote:Issue resolved after tons of log files sent to support.
According to support this issue will be resolved in future releases.
My solution to this error was to stop and start VeeamFCT service on all Hyper-V nodes that was connected to the CSV that hold VM's that generated warning messages.
Note: rebooting the nodes didn't relove this error, just the stop and start of the VeeamFCT service when nodes are online.
I will try this again as i will soon migrate/Create more VM's to the cluster.


I am having this issue - but I do not see the VeeamFCT service on any of my nodes. Here is what I do have:
Veeam Backup Hyper-V Integration Service
Veeam Backup Proxy Service
Veeam Installer Service

:? :?:
Shralok
Enthusiast
 
Posts: 51
Liked: 5 times
Joined: Wed Aug 29, 2012 5:36 pm
Full Name: Shralok

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby foggy » Mon Mar 04, 2013 3:32 pm

The VeeamFCT service is not available in the Services snap-in. Try the following in cmd:
Code: Select all
sc stop veeamfct
sc start veeamfct
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Wed Mar 06, 2013 8:56 pm

Update - We had another driver built for us to test with Support, but still did not resolve the problem. I had submitted all of my system logs from two different clusters two weeks ago, hoping something could be spotted that would help develop a fix. As a refresher, we have two completely separate Server 2012 based clusters. We use two proxies, and two repository servers with two volumes a piece. The repository volumes are on Dell Equallogic SAN's, CSV based. We haven't found any issues with the Equallogics or anything out of the ordinary that would break CBT.

Our two huge file servers are still stuck doing full incremental scans. It seems like once a server produces this warning, it gets "stuck" and always stays that way. Newly built VM's tend to suffer the issue as well, which is too bad because I've never seen one actually return to a normal state. We're still trying to understand how this happens. We were presented with three different scenarios that could break CBT, but none of them matched us. One was a new CSV volume being added, which we hadn't done. A second was; adding a new VM onto a CSV while in redirected mode, which we hadn't done either, always used redirected mode. The third scenario was powering on/off the nodes and then immediately run a job, which we never reboot hosts anywhere near on backup times so that didn't pan out either.

I'm still optimistic, but we're over three months with this error and we've only been Veeam customers for six. With offsite replication on the horizon, I'm worried we'll never be able to meet backup windows. I never had problems with 6.1, but I have a feeling that's simply because we hadn't moved our entire environment to Veeam at that time.

For all of you troubleshooting this error, seems like the usual fixes are; start/stop VeeamFCT driver, remove any AVHD files lingering, double check your VeeamFCT.sys driver is the correct version after upgrading to Patch 1. %windir%\system32\drivers\veeamfct.sys is the location. Make sure the version is correct. If you have to manually move the driver, reboot each of the hosts.
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Thu Mar 07, 2013 7:07 pm

Yet another update -

Apparently I was sent the wrong driver by Support, which is why we kept seeing the CBT errors. I have received a new driver and have applied it to two of my three clusters. On initial backups, it appears CBT is working again though it is still running full-incremental scans. I'll update this thread again after my next round of backups. After this run, I should be able to re-run jobs and see that CBT is in fact, tracking changes as expected. After 4 long months, this is definitely the best sign of hope I've seen :)
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Shralok » Thu Mar 21, 2013 7:20 pm

I am running Veeam B&R 6.5.0.128 (64 bit)

My VeeamFCT.sys file has:
File version: 6.5.0.6
Product version: 6.5.0.0

Is this correct? I am having the same CBT errors since patching B&R 6.5

Case # 00191254
Shralok
Enthusiast
 
Posts: 51
Liked: 5 times
Joined: Wed Aug 29, 2012 5:36 pm
Full Name: Shralok

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Shralok » Sun Mar 24, 2013 10:16 pm

Spent the majority of my Saturday following instructions from support on deleting CBT reg keys and then rebooting the entire Hyper-V infrastructure... only to find that this did not resolve the CBT issue.

:evil:
Shralok
Enthusiast
 
Posts: 51
Liked: 5 times
Joined: Wed Aug 29, 2012 5:36 pm
Full Name: Shralok

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby jake1110 » Wed Mar 27, 2013 3:52 pm

Sharlok -

If you look at your "Date Modified" field of the VeeamFCT.sys file, what's the time stamp of it? My file version is 6.5.0.6, but so is my old one so they hadn't yet changed the version number yet. My old driver is from 12/17/2012 and the new one from 3/1/2013.

I have a feeling that the hotfix I received hasn't been released yet, but will be in the next patch.
jake1110
Enthusiast
 
Posts: 40
Liked: 2 times
Joined: Thu Sep 20, 2012 6:19 pm
Location: Des Moines, IA
Full Name: Jake Ernst

Re: CBT Fails to Flush on All VMs, Backups taking 12+ hours

Veeam Logoby Shralok » Wed Mar 27, 2013 5:05 pm

Monday, December 17, 2012, 11:11:00 AM
Shralok
Enthusiast
 
Posts: 51
Liked: 5 times
Joined: Wed Aug 29, 2012 5:36 pm
Full Name: Shralok

PreviousNext

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 8 guests