CBT feature request

[collinp]

Veeam is the greatest backup product that I have every used for backing up VM's. I'm impressed at the amount of features and how quickly new features are introduced (Exchange, SharePoint, Instant Recovery, Offhost proxies, etc). With this being said, there is only one flaw on the Hyper-v implementation. I don't see how I can scale it for large implementations when CBT resets every time we add a host to the cluster, upgrade Veeam editions or implement certain host hotfixes. For a cluster of 200 VM's and 45TB of storage (and growing rapidly), if CBT resets, it takes us 3 days to catch up on backups. It is hard telling the customer that they won't have backups for 3 days. It also puts a lot of pressure on CSV's and could potentially cause outages when CBT is reset and jobs start stacking up. I have to proactively throttle the backups when I see that we lose CBT.

Are there are any plans in the future to try to maintain CBT throughout upgrades and changes to the cluster?

[foggy]

Collin, thank you for the kind words and feedback. There is some place for improvement, probably we could address the following cases in future:

upgrade Veeam editions

However, this will most likely require host reboot. Would it be less painful then re-running full backup?

add a host to the cluster

Regarding this one, I don't think this is possible, since we do not track such events internally and you never know whether some changes were performed to the VMs moved to the new node prior their backup.

implement certain host hotfixes.

Are you saying that applying Windows updates to hosts resets CBT? According to devs, this should not be the case and they would like to take a look at logs containing such occurrences, if possible.

Thanks.

[collinp]

Foggy,

The new node that we added to the cluster had no vm's on it. No vm's were moved to it. We didn't move vm's around. But every vm lost cbt. This has happened twice in a month when we added new hosts.

[foggy]

I'm not saying this is not the case. I'm saying that Veeam B&R cannot guarantee that no changes were performed on the new node, since CBT driver is installed on the hosts on adding them to the Veeam B&R console and is activated on the first job run against them. New nodes are not added to the console automatically.

[collinp]

The new node was joined to the cluster then I added the new host to Veeam Backup and Recovery. The new host had no vm's on it. As the backup jobs started at night I noticed that CBT wasn't working for all VM's. Can CBT fail on a cluster level or does it usually fail per host? It seems to have failed at the cluster level

[foggy]

Yes, it is reset for all nodes in the cluster.

[collinp]

Foggy,

So you don't see a way that I can add a host to a cluster and maintain CBT? If we had funding, we could build out the entire cluster to 8 nodes from the beginning but we have been adding nodes as necessary.

Do you know of a non-standard way of copying the CBT info over to the new host so we don't lose CBT? I suppose I can submit a support ticket to see if there is an undocumented way of getting around this issue.

Thanks again

[foggy]

I'm afraid there is no such a way due to the current CBT mechanism implementation.

[collinp]

It has been 2 years since this post. I still have the same question. How do you upgrade Veeam in a large Hyper-v environment without missing backups due to CBT being reset?

Just to give you an idea of the size of our environment. We now have over 700 Hyper-v VM's, 25 Hyper-v hosts, and full backups totaling more than 200TB. If I were to upgrade Veeam, it would take weeks to catch up on backups in order for CBT to reset.

It seems to me that you could include a feature in Veeam to upgrade CBT one Hyper-v host at a time. This way you can do rolling CBT upgrades.

Collin

[foggy]

Collin, the answer is still the same, since the behavior didn't change. One thing you could consider to do is scheduling an upgrade prior to performing periodic full backups, in case they are performed on some basis. Though, not sure this will tremendously help in such a big environment.

[collinp]

Do you plan on supporting the scaling of large corporate Hyper-v environments in the future by not forcing you to do a full scan backup every time there is an upgrade? My idea was to allow rolling CBT upgrades. Or will the next version of Hyper-v include CBT functionality natively?

[foggy]

I'm not aware of any short-term plans regarding this. Seems it has not generated any considerable number of requests so far.

[chrisdearden]

I suspect it will be a feature of the next version of hyper v - http://www.aidanfinn.com/?p=17505

[nmdange]

Just to give you an idea of the size of our environment. We now have over 700 Hyper-v VM's, 25 Hyper-v hosts, and full backups totaling more than 200TB. If I were to upgrade Veeam, it would take weeks to catch up on backups in order for CBT to reset.

So my environment is about half the size of yours (350 VMs, 15 hosts, 100TB data). After installing v8 update 3, it only took 1 extra day to catch up on the backups. If it really takes weeks to do full backups on 200TB, you must have a bottleneck somewhere in your setup that doesn't exist in mine. Storage network for my Hyper-V clusters is SMB 3.0 using 10Gb RDMA NICs and I use the backup repository server as an off-host backup proxy. What kind of throughput do you see in your backups?

[collinp]

We have a dozen 5TB-10TB VM's that take days to complete. We have 90 backup jobs that average about 8 VM's per backup job. For throughput we see 50 MB/s to 100 MB/s per job. How many concurrent backups do you allow per repository and per host?

[nmdange]

I don't have any limits set for repositories. The "limit max concurrent tasks" and "limit combined data rate" settings on all the repositories are unchecked. The off-host backup proxy has a max of 14 concurrent tasks since the server has a single 14-core CPU. I only have 12 backup jobs with a lot more VMs per job (except for a couple of jobs that have unique settings for specific VMs). You might get better concurrency reducing the number of backup jobs with more VMs per job.

The throughput numbers seem low to me, but that is very much dependent on your environment. Unless there is very little change data, I typically see 500 MB/s or so throughput, occasionally it will be higher, once in awhile it hits 2 GB/s.

[Shestakov]

I suspect it will be a feature of the next version of hyper v

You are right. Microsoft announced it at Ignite.

[collinp]

2 GB/s?? Wow. What does your source storage look like? That is where my bottleneck is.

[nmdange]

My primary storage is a 3x3 storage spaces system (3 SOFS nodes and 3 60-bay SAS JBODs). It is a tiered storage spaces system, so I assume it's reading blocks off the SSDs when it hits the higher throughput values. I also have a smaller secondary 2-node SOFS cluster which is attached to a SAS-based Dell PowerVault MD3220 with 24x 900GB 10K drives.

With SMB, the bottleneck "Source" in Veeam could be in a few different places:
- the physical disks
- the connection between the storage and the Scale Out File servers, particularly if it's iSCSI or Fiber Channel and not SAS
- the Scale Out File Servers themselves, particularly if they are older generation hardware that does not have PCIe 3.0 slots
- the SMB network, if it's not at least 10Gbps with 2 independent paths, using RDMA NICs also helps
- the Hyper-V hosts if you are doing on-host backups. Using an off-host backup proxy would definitely help

[Nail.Mukhametshin]

Hi all!

As we know your CBT mechanism now don't work when any of Windows Failover Cluster node have troubles, have extended maintance period or working with PRO mechanism to save power. See this thread: microsoft-hyper-v-f25/cluster-change-bl ... 26725.html.

Now when we have one node down we experiense extreamly high load of Storage because on every backup Veeam reads all data of VHDs. From this point we cannot mantain cluster in backup time at all because of performance issues.

I assume that this mechanism needs some changes - it must work even nodes not accessible.

The best solution from my point is storing CBT data on cluster resources so this data can be accessible any time when cluster works fine.
Another solution - ability to recreate CBT data when node offline for a long time. So only first backup will reread all data and next backups will use CBT again.

Hope you put in on board.

[foggy]

This will be addressed by the native CBT mechanism introduced in Windows Server 2016.

[Nail.Mukhametshin]

Hi!

Server 2016 is out, so can you confirm that CBT will work now when some of nodes is down? Can you specify requirement for this workin fine?

[foggy]

Resilient CBT will work. But you need to wait for Veeam B&R 9.5.