CBT Fails to Flush on All VMs, Backups taking 12+ hours

[tsightler]

My suggestion would be to open a support case and have them investigate.

[rkovhaev]

Tried a reboot on the Cluster node that holds the running VM's. Still same problem and Warning errors?
What should i do about this, there is no logic in this.

Jorgen, could you please let me know your support req. no. I can't find it in our system.

[jorgene]

Hi, i haven't created a support case for this yet. Thought since many have this problem there was a general solution for this error.
I will create a support case tomorrow when i'm back @ the office.

[jorgene]

Case Opened: Case ID #: 00181649

[jake1110]

I just resolved my CBT issues recently. It turns out that we had some vm's that had 'pending' year old chained VMM checkpoints associated with them. The checkpoints were deleted from VMM by the VM administrators but the vm's were never powered off for a long enough period to let the checkpoints merge properly. If you do a find file *.avhd on c:\clusterstorage\ on a hyper-v host you can see if you still have pending snapshots around. I shutdown the 20 or so vm's one at a time and watched from Hyper-v manager and it took 3 hours for all of checkpoints to merge before I could power the vm's back on. Once I cleaned up all the vmm checkpoints, Veeam CBT began working for everything.

I wish this was the fix for my CBT woes, but unfortunately we don't take snapshots of VM's except for exceedingly rare occasions for the reason you've mentioned. The snaps grow to become huge and merging takes forever, causing hours of downtime of the VM. Only with Server 2012, have we allowed anyone to take snapshots since it can merge them with the VM powered on. I double checked anyway just to make sure there were no avhd files and there weren't any.

I have another call setup with Support this morning...hoping for the best!

[jorgene]

Issue resolved after tons of log files sent to support.
According to support this issue will be resolved in future releases.
My solution to this error was to stop and start VeeamFCT service on all Hyper-V nodes that was connected to the CSV that hold VM's that generated warning messages.
Note: rebooting the nodes didn't relove this error, just the stop and start of the VeeamFCT service when nodes are online.
I will try this again as i will soon migrate/Create more VM's to the cluster.

[jake1110]

Yep, the stop/start VeeamFCT driver is something I do all the time. In fact, that was the first solution I was ever offered back in December, but hasn't worked for me. I was on yet another call with Support and that's the only solution I was offered this time around.

I just COMPLETELY rebuilt my Veeam backup server, along with a new database, jobs and uninstall/reinstalled all components on the proxy/repository servers and Hyper-V hosts as well. I completely wiped out all of my backup sets too to make sure nothing was retained.

I ran one test job and immediately received the CBT warning on the first two VM's backed up. I'm hoping this is simply because it's the first time I ran the job, but I doubt that's it.

It's seemingly random which VM's fail and which ones succeed.

[Shralok]

Issue resolved after tons of log files sent to support.
According to support this issue will be resolved in future releases.
My solution to this error was to stop and start VeeamFCT service on all Hyper-V nodes that was connected to the CSV that hold VM's that generated warning messages.
Note: rebooting the nodes didn't relove this error, just the stop and start of the VeeamFCT service when nodes are online.
I will try this again as i will soon migrate/Create more VM's to the cluster.

I am having this issue - but I do not see the VeeamFCT service on any of my nodes. Here is what I do have:
Veeam Backup Hyper-V Integration Service
Veeam Backup Proxy Service
Veeam Installer Service

[foggy]

The VeeamFCT service is not available in the Services snap-in. Try the following in cmd:

Code: Select all

sc stop veeamfct
sc start veeamfct

[jake1110]

Update - We had another driver built for us to test with Support, but still did not resolve the problem. I had submitted all of my system logs from two different clusters two weeks ago, hoping something could be spotted that would help develop a fix. As a refresher, we have two completely separate Server 2012 based clusters. We use two proxies, and two repository servers with two volumes a piece. The repository volumes are on Dell Equallogic SAN's, CSV based. We haven't found any issues with the Equallogics or anything out of the ordinary that would break CBT.

Our two huge file servers are still stuck doing full incremental scans. It seems like once a server produces this warning, it gets "stuck" and always stays that way. Newly built VM's tend to suffer the issue as well, which is too bad because I've never seen one actually return to a normal state. We're still trying to understand how this happens. We were presented with three different scenarios that could break CBT, but none of them matched us. One was a new CSV volume being added, which we hadn't done. A second was; adding a new VM onto a CSV while in redirected mode, which we hadn't done either, always used redirected mode. The third scenario was powering on/off the nodes and then immediately run a job, which we never reboot hosts anywhere near on backup times so that didn't pan out either.

I'm still optimistic, but we're over three months with this error and we've only been Veeam customers for six. With offsite replication on the horizon, I'm worried we'll never be able to meet backup windows. I never had problems with 6.1, but I have a feeling that's simply because we hadn't moved our entire environment to Veeam at that time.

For all of you troubleshooting this error, seems like the usual fixes are; start/stop VeeamFCT driver, remove any AVHD files lingering, double check your VeeamFCT.sys driver is the correct version after upgrading to Patch 1. %windir%\system32\drivers\veeamfct.sys is the location. Make sure the version is correct. If you have to manually move the driver, reboot each of the hosts.

[jake1110]

Yet another update -

Apparently I was sent the wrong driver by Support, which is why we kept seeing the CBT errors. I have received a new driver and have applied it to two of my three clusters. On initial backups, it appears CBT is working again though it is still running full-incremental scans. I'll update this thread again after my next round of backups. After this run, I should be able to re-run jobs and see that CBT is in fact, tracking changes as expected. After 4 long months, this is definitely the best sign of hope I've seen

[Shralok]

I am running Veeam B&R 6.5.0.128 (64 bit)

My VeeamFCT.sys file has:
File version: 6.5.0.6
Product version: 6.5.0.0

Is this correct? I am having the same CBT errors since patching B&R 6.5

Case # 00191254

[Shralok]

Spent the majority of my Saturday following instructions from support on deleting CBT reg keys and then rebooting the entire Hyper-V infrastructure... only to find that this did not resolve the CBT issue.

[jake1110]

Sharlok -

If you look at your "Date Modified" field of the VeeamFCT.sys file, what's the time stamp of it? My file version is 6.5.0.6, but so is my old one so they hadn't yet changed the version number yet. My old driver is from 12/17/2012 and the new one from 3/1/2013.

I have a feeling that the hotfix I received hasn't been released yet, but will be in the next patch.

[Shralok]

Monday, December 17, 2012, 11:11:00 AM

[jake1110]

It appears you have the old version of that driver.

I would contact Support or your Veeam rep to possibly get the newest version of the driver. I have no idea when the next patch will be available so you can either wait for that or try and get a hold of someone to send it to you. I dealt with this problem for 4 months, so I understand just how painful it is to watch your backups constantly run through their windows.

[Gostev]

We are looking at 2-3 weeks for the next patch. This will include the hotfix Jake is talking about. Until then, the hotfix can be obtained through our technical support.

[jldwyer1420]

Any update on this patch?

[foggy]

Jeff, no need to wait until the patch. You can request the hotfix for the issue through our technical support. Thanks.

[Gostev]

Still looking good on original timelines of 2-3 weeks... the patch release is imminent. The current patch build is currently in RC (Release Candidate) state, when the release can happen any day unless some new bug is found, which would shift the release for a few more days. DR software should not be rushed

[Borsh]

Hello!
I have just updated Veeam 6.5 to patch 3 and have the same warning during replication between clusters Hyper-V 2008 R2

[foggy]

Igor, please contact technical support to verify whether you're experiencing the known issue addressed by the patch, or a different issue. They will also make sure you have the patch installed correctly. Thanks!

[hybrazil]

Hi All!
I running 6.5.0.144 in a hyper-V environment, it's a very simple setup, I only have 11VM's in one cluster running on 2 nodes, I had to change the domain admins password 2 weeks ago and I updated the password in the 'Backup Infrastructure' and the backup jobs in Veeam, however since then I have received the error "Failed to flush change tracking data before snapshot. Using full scan incremental run" on all backup jobs. I tried running the commands,
sc stop veeamfct
sc start veeamfct
on the nodes and VM's and I rebooted all physicall and virtual servers, anybody got any ideas

[foggy]

What about subsequent job runs? Do they utilize CBT?

[hybrazil]

I figured it out! I tried to get the log files and discovered that I could not access the SMB server, so reset the password again in Backup Infrastructure, and rescanned all servers, which worked fine. I then stopped and restarted the VeeamFCT service on the physical servers, and ran the backup jobs.
All jobs completed successfully with no errors.

Hybrazil

[foggy]

Glad you've nailed it so quickly. Thanks for updating the thread; much appreciated.