Discussions specific to the Microsoft Hyper-V hypervisor
Post Reply
collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

CBT feature request

Post by collinp » Dec 05, 2013 3:46 am

Veeam is the greatest backup product that I have every used for backing up VM's. I'm impressed at the amount of features and how quickly new features are introduced (Exchange, SharePoint, Instant Recovery, Offhost proxies, etc). With this being said, there is only one flaw on the Hyper-v implementation. I don't see how I can scale it for large implementations when CBT resets every time we add a host to the cluster, upgrade Veeam editions or implement certain host hotfixes. For a cluster of 200 VM's and 45TB of storage (and growing rapidly), if CBT resets, it takes us 3 days to catch up on backups. It is hard telling the customer that they won't have backups for 3 days. It also puts a lot of pressure on CSV's and could potentially cause outages when CBT is reset and jobs start stacking up. I have to proactively throttle the backups when I see that we lose CBT.

Are there are any plans in the future to try to maintain CBT throughout upgrades and changes to the cluster?

foggy
Veeam Software
Posts: 18253
Liked: 1559 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT feature request

Post by foggy » Dec 05, 2013 1:27 pm

Collin, thank you for the kind words and feedback. There is some place for improvement, probably we could address the following cases in future:
collinp wrote:upgrade Veeam editions
However, this will most likely require host reboot. Would it be less painful then re-running full backup?
collinp wrote:add a host to the cluster
Regarding this one, I don't think this is possible, since we do not track such events internally and you never know whether some changes were performed to the VMs moved to the new node prior their backup.
collinp wrote:implement certain host hotfixes.
Are you saying that applying Windows updates to hosts resets CBT? According to devs, this should not be the case and they would like to take a look at logs containing such occurrences, if possible.

Thanks.

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT feature request

Post by collinp » Dec 05, 2013 3:26 pm

Foggy,

The new node that we added to the cluster had no vm's on it. No vm's were moved to it. We didn't move vm's around. But every vm lost cbt. This has happened twice in a month when we added new hosts.

foggy
Veeam Software
Posts: 18253
Liked: 1559 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT feature request

Post by foggy » Dec 05, 2013 3:40 pm

I'm not saying this is not the case. I'm saying that Veeam B&R cannot guarantee that no changes were performed on the new node, since CBT driver is installed on the hosts on adding them to the Veeam B&R console and is activated on the first job run against them. New nodes are not added to the console automatically.

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT feature request

Post by collinp » Dec 05, 2013 4:00 pm

The new node was joined to the cluster then I added the new host to Veeam Backup and Recovery. The new host had no vm's on it. As the backup jobs started at night I noticed that CBT wasn't working for all VM's. Can CBT fail on a cluster level or does it usually fail per host? It seems to have failed at the cluster level

foggy
Veeam Software
Posts: 18253
Liked: 1559 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT feature request

Post by foggy » Dec 05, 2013 4:02 pm

Yes, it is reset for all nodes in the cluster.

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT feature request

Post by collinp » Dec 05, 2013 5:52 pm

Foggy,

So you don't see a way that I can add a host to a cluster and maintain CBT? If we had funding, we could build out the entire cluster to 8 nodes from the beginning but we have been adding nodes as necessary.

Do you know of a non-standard way of copying the CBT info over to the new host so we don't lose CBT? I suppose I can submit a support ticket to see if there is an undocumented way of getting around this issue.

Thanks again

foggy
Veeam Software
Posts: 18253
Liked: 1559 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT feature request

Post by foggy » Dec 06, 2013 10:56 am

I'm afraid there is no such a way due to the current CBT mechanism implementation.

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT feature request

Post by collinp » Oct 28, 2015 1:01 pm

It has been 2 years since this post. I still have the same question. How do you upgrade Veeam in a large Hyper-v environment without missing backups due to CBT being reset?

Just to give you an idea of the size of our environment. We now have over 700 Hyper-v VM's, 25 Hyper-v hosts, and full backups totaling more than 200TB. If I were to upgrade Veeam, it would take weeks to catch up on backups in order for CBT to reset.

It seems to me that you could include a feature in Veeam to upgrade CBT one Hyper-v host at a time. This way you can do rolling CBT upgrades.

Collin

foggy
Veeam Software
Posts: 18253
Liked: 1559 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT feature request

Post by foggy » Oct 28, 2015 2:13 pm

Collin, the answer is still the same, since the behavior didn't change. One thing you could consider to do is scheduling an upgrade prior to performing periodic full backups, in case they are performed on some basis. Though, not sure this will tremendously help in such a big environment.

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT feature request

Post by collinp » Nov 15, 2015 3:02 pm

Do you plan on supporting the scaling of large corporate Hyper-v environments in the future by not forcing you to do a full scan backup every time there is an upgrade? My idea was to allow rolling CBT upgrades. Or will the next version of Hyper-v include CBT functionality natively?

foggy
Veeam Software
Posts: 18253
Liked: 1559 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT feature request

Post by foggy » Nov 16, 2015 12:02 pm

I'm not aware of any short-term plans regarding this. Seems it has not generated any considerable number of requests so far.

chrisdearden
Expert
Posts: 1530
Liked: 225 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: CBT feature request

Post by chrisdearden » Nov 16, 2015 12:36 pm

I suspect it will be a feature of the next version of hyper v - http://www.aidanfinn.com/?p=17505

nmdange
Expert
Posts: 468
Liked: 113 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: CBT feature request

Post by nmdange » Nov 16, 2015 6:03 pm 1 person likes this post

collinp wrote:Just to give you an idea of the size of our environment. We now have over 700 Hyper-v VM's, 25 Hyper-v hosts, and full backups totaling more than 200TB. If I were to upgrade Veeam, it would take weeks to catch up on backups in order for CBT to reset.
So my environment is about half the size of yours (350 VMs, 15 hosts, 100TB data). After installing v8 update 3, it only took 1 extra day to catch up on the backups. If it really takes weeks to do full backups on 200TB, you must have a bottleneck somewhere in your setup that doesn't exist in mine. Storage network for my Hyper-V clusters is SMB 3.0 using 10Gb RDMA NICs and I use the backup repository server as an off-host backup proxy. What kind of throughput do you see in your backups?

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT feature request

Post by collinp » Nov 18, 2015 1:09 am

We have a dozen 5TB-10TB VM's that take days to complete. We have 90 backup jobs that average about 8 VM's per backup job. For throughput we see 50 MB/s to 100 MB/s per job. How many concurrent backups do you allow per repository and per host?

nmdange
Expert
Posts: 468
Liked: 113 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: CBT feature request

Post by nmdange » Nov 18, 2015 3:21 pm

I don't have any limits set for repositories. The "limit max concurrent tasks" and "limit combined data rate" settings on all the repositories are unchecked. The off-host backup proxy has a max of 14 concurrent tasks since the server has a single 14-core CPU. I only have 12 backup jobs with a lot more VMs per job (except for a couple of jobs that have unique settings for specific VMs). You might get better concurrency reducing the number of backup jobs with more VMs per job.

The throughput numbers seem low to me, but that is very much dependent on your environment. Unless there is very little change data, I typically see 500 MB/s or so throughput, occasionally it will be higher, once in awhile it hits 2 GB/s.

Shestakov
Veeam Software
Posts: 6974
Liked: 711 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: CBT feature request

Post by Shestakov » Nov 18, 2015 6:33 pm

chrisdearden wrote:I suspect it will be a feature of the next version of hyper v
You are right. Microsoft announced it at Ignite.

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: CBT feature request

Post by collinp » Nov 19, 2015 6:07 pm

2 GB/s?? Wow. What does your source storage look like? That is where my bottleneck is.

nmdange
Expert
Posts: 468
Liked: 113 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: CBT feature request

Post by nmdange » Nov 20, 2015 2:54 pm

My primary storage is a 3x3 storage spaces system (3 SOFS nodes and 3 60-bay SAS JBODs). It is a tiered storage spaces system, so I assume it's reading blocks off the SSDs when it hits the higher throughput values. I also have a smaller secondary 2-node SOFS cluster which is attached to a SAS-based Dell PowerVault MD3220 with 24x 900GB 10K drives.

With SMB, the bottleneck "Source" in Veeam could be in a few different places:
- the physical disks
- the connection between the storage and the Scale Out File servers, particularly if it's iSCSI or Fiber Channel and not SAS
- the Scale Out File Servers themselves, particularly if they are older generation hardware that does not have PCIe 3.0 slots
- the SMB network, if it's not at least 10Gbps with 2 independent paths, using RDMA NICs also helps
- the Hyper-V hosts if you are doing on-host backups. Using an off-host backup proxy would definitely help

Nail.Mukhametshin
Enthusiast
Posts: 50
Liked: 3 times
Joined: May 05, 2016 1:15 pm
Full Name: Наиль Мухаметшин
Location: Russia, Moscow
Contact:

[MERGED] Feature Request - Change CBT mechanism in MS Cluste

Post by Nail.Mukhametshin » May 05, 2016 1:34 pm

Hi all!

As we know your CBT mechanism now don't work when any of Windows Failover Cluster node have troubles, have extended maintance period or working with PRO mechanism to save power. See this thread: microsoft-hyper-v-f25/cluster-change-bl ... 26725.html.

Now when we have one node down we experiense extreamly high load of Storage because on every backup Veeam reads all data of VHDs. From this point we cannot mantain cluster in backup time at all because of performance issues.

I assume that this mechanism needs some changes - it must work even nodes not accessible.

The best solution from my point is storing CBT data on cluster resources so this data can be accessible any time when cluster works fine.
Another solution - ability to recreate CBT data when node offline for a long time. So only first backup will reread all data and next backups will use CBT again.

Hope you put in on board.
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab

foggy
Veeam Software
Posts: 18253
Liked: 1559 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT feature request

Post by foggy » May 05, 2016 2:25 pm

This will be addressed by the native CBT mechanism introduced in Windows Server 2016.

Nail.Mukhametshin
Enthusiast
Posts: 50
Liked: 3 times
Joined: May 05, 2016 1:15 pm
Full Name: Наиль Мухаметшин
Location: Russia, Moscow
Contact:

Re: CBT feature request

Post by Nail.Mukhametshin » Oct 13, 2016 12:52 pm

Hi!

Server 2016 is out, so can you confirm that CBT will work now when some of nodes is down? Can you specify requirement for this workin fine?
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab

foggy
Veeam Software
Posts: 18253
Liked: 1559 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT feature request

Post by foggy » Oct 13, 2016 3:16 pm

Resilient CBT will work. But you need to wait for Veeam B&R 9.5.

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests