CBT Fails to Flush on All VMs, Backups taking 12+ hours

[Gostev]

This would depend on the specific issue (there are a few different border cases we have missed in testing), but in most cases it should both fix and prevent.

[matizmat]

Just wanted to update that we will be releasing a patch for the issues that cause this error soon (it is in final testing right now).

I look forward to this patch

[mmihm@salawus.com]

Same problem here! Four 2008 R2 hyperv hosts replicating vm's to two 2008 R2 hyperv hosts in our Colo. "Failed to flush tracking data before snapshot". I upgraded to 6.5, same issue. All hosts are rebooted monthly and the issue still occurs.

[jorgene]

Any fix for this issue yet? having the same problem on imported VM's in a W2012 Cluster.

[Vitaliy S.]

Jorgen, have you had a chance to apply our latest patch to resolve this issue?

[jorgene]

Ohh, dowloading and installing now thx.
FYI. It seems like a Move of Virtual Machine Store in a Cluster also makes the CBT break.
I will apply the patch and report back.

[jorgene]

Hi, works correctly after applying the patch. After moving a Virtual Machine Store of a VM to Another LUN/CSV it seems like it takes longer time to backup that VM the first time (probably need to read entire VM file again) but no warnings in backup log file. All seems to work fine after the patch.

Thx !

[dan.hoffman]

I must have missed something. I installed the patch, and it updated all the hosts just fine. But, I am still getting the problem. I even restarted 2 VMs and one of the hosts to test. Restarted the Veeam server and repository as well. Still getting the same issue when running replications and backups.
Anyone else?

[Vitaliy S.]

Hi Dan, I guess our support should take a look at your logs after you've applied this patch to see what is the reason for CBT flush failure messages.

[jorgene]

Hi again, after adding a new VM (W2008R2) and converting the VHD file to VHDX and adding it to the Cluster backup jobs on that VM now gets the same error again (CBT fails to flush) (i have applied the patch). I made a new job new backup respository but the same error.

I applied the patch again on the Veeam B&R server, and the error dissapears again.
So everytime i import a VM in the cluster i need to reapply the patch, to make the CBT error go away.

[dan.hoffman]

Hi Dan, I guess our support should take a look at your logs after you've applied this patch to see what is the reason for CBT flush failure messages.

Just opened a ticket. Thanks.

[Gostev]

So everytime i import a VM in the cluster i need to reapply the patch, to make the CBT error go away.

This makes no sense to me. How overwriting the existing files with the same ones can change anything at all? I recommend investigating this with the support.

[rkovhaev]

Dan, Jorgen,
Could you please let me know your support req. numbers, there is a possibility that old FCT driver is still in memory and we will need to manually load new FCT driver.
Another option would be scheduling a quick downtime for production or live migrate VMs(if it's a cluster) to another node and reboot affected HyperV nodes.

[dan.hoffman]

I already tried migrating two affected vms off one host and restarted the hosts then migrated the vms back, it didn't help. I even restarted the VMs just to make sure. Still getting the same error.
case: 00169668

[rkovhaev]

Dan, it looks like you are using cluster disk instead of cluster shared volume (CSV). Cluster disk is not supported yet, support will be added in one of the upcoming releases.

[dan.hoffman]

Dan, it looks like you are using cluster disk instead of cluster shared volume (CSV). Cluster disk is not supported yet, support will be added in one of the upcoming releases.

Odd, it was working great until I upgraded to 6.5 . Will the new release be coming sometime soon?

[Gostev]

Odd, it was working great until I upgraded to 6.5 . Will the new release be coming sometime soon?

Please note that "in one of the upcoming releases" does not necessarily mean "the next release".
That said, let me get more details on this specific issue with cluster disks.

[collinp]

After the patch, there are still CBT issues in my case, although Veeam no longer reports issues in the daily email reports. I am working with tech support on this. A large amount of blocks are still being read (100's of gb's - 80% of the entire .vhd) in order backup only mb's of data for the problem vm's. A symptom that I am seeing when there are cbt issues is an artificially high double-digit dedup ratio.

Here is the screenshot url:
https://picasaweb.google.com/cannonc34/ ... 3830837378

[Gostev]

When reporting any technical issues, kindly please include the corresponding support case ID.

[collinp]

The case # is 00153186 and all logs have been uploaded. I have not heard back from support yet since installing the patch and following up with support on the 7th. The previous case # was 00121732. I have been having these CBT backup issues with Hyper-v since first using Veeam. After the release of each new filter driver, things have continually been getting better and better. I am now down to a half dozen or so virtual machines where a large amount of data still needs to be read in order to backup only mb's. I'm just trying to do what I can to make the product better. When we begin backing up larger quantities of vm's, we will want the vm's to backup in minutes versus hours.

[veremin]

Even though the cause of this issue has been already mentioned (usage of clusted disks) I’ve decided to update the topic with the corresponding Veeam KB for the convenience of further readers.

Thanks.

[jake1110]

Oh this is absolutely killing me.

So I had already manually updated VeeamFCT driver(with support) which fixed some VM's but others were failing to use CBT, luckily my two monster file servers were still working:)

Now I apply the cumulative patch today from 12-24 and while it appears to have fixed most VM's, my two huge file servers now display the dreaded message again....unbelievable. They were just working perfectly for the last month. Nothing has changed whatsoever. I've been fighting this for so long and thought maybe the cumulative patch would fix things. I was also waiting to finalize moving all my VM's to our 2012 cluster.

If I could get 80 - 90% of VM's using CBT(including our two huge file servers) I would be satisfied. Not sure if I should open yet another ticket or try the same troubleshooting as before. Watching 3.5TB of full-incremental scans of my file servers is not fun at all....

[MarcPiquer]

Hello, there are prevision to post new patch for resolve this issue?

[Vitaliy S.]

Hello Marc,

Not sure I understand the question, have you applied this patch already?

Thanks!

[MarcPiquer]

Yes, i have applied the patch 6.5.0.128, but the error persist.

[Vitaliy S.]

Then please contact our support team to review your debug log files, maybe the patch wasn't installed correctly.

[collinp]

I just resolved my CBT issues recently. It turns out that we had some vm's that had 'pending' year old chained VMM checkpoints associated with them. The checkpoints were deleted from VMM by the VM administrators but the vm's were never powered off for a long enough period to let the checkpoints merge properly. If you do a find file *.avhd on c:\clusterstorage\ on a hyper-v host you can see if you still have pending snapshots around. I shutdown the 20 or so vm's one at a time and watched from Hyper-v manager and it took 3 hours for all of checkpoints to merge before I could power the vm's back on. Once I cleaned up all the vmm checkpoints, Veeam CBT began working for everything.

I know that Veeam has created it's own CBT filter drive and that theoretically, CBT should still work with Hyper-v checkpoints present. For the most part, this was true in my case except for the checkpoints that were left around for way too long! I believe that VMM checkpoints aren't recommended in productio on 2008 R2, but if they are used they should only be kept for a short period of time.

[Gostev]

Thanks Collin for taking your time to provide the update, this possible cause is very useful to know.

[jorgene]

Hmm.. i just created 3 new VM's in my W2012 Cluster (One W2012 and 2 W2008R2). All new VM's now get Warning "Failed to flush change tracking data before snapshot. Using full scan incremental run" during backup.
First and Second run. Also i had a problem with one of my cluster nodes last week, resulting in that one of the VM's on that node moved to another node. That VM also get this Warning "Failed to flush change tracking data before snapshot. Using full scan incremental run" during backups. But after i reapply the patch 6.5.0.128 on the Veeam B&R Server (not on the Hyper-V Hosts) that backup job run ok again.
I reapplied the patch again on the Veeam B&R server and even rebooted, but the 3 new VM's now all still have Warning message during backup jobs.

[jorgene]

Tried a reboot on the Cluster node that holds the running VM's. Still same problem and Warning errors?
What should i do about this, there is no logic in this.