CBT feature request

Hyper-V specific discussions

CBT feature request

Veeam Logoby collinp » Thu Dec 05, 2013 3:46 am

Veeam is the greatest backup product that I have every used for backing up VM's. I'm impressed at the amount of features and how quickly new features are introduced (Exchange, SharePoint, Instant Recovery, Offhost proxies, etc). With this being said, there is only one flaw on the Hyper-v implementation. I don't see how I can scale it for large implementations when CBT resets every time we add a host to the cluster, upgrade Veeam editions or implement certain host hotfixes. For a cluster of 200 VM's and 45TB of storage (and growing rapidly), if CBT resets, it takes us 3 days to catch up on backups. It is hard telling the customer that they won't have backups for 3 days. It also puts a lot of pressure on CSV's and could potentially cause outages when CBT is reset and jobs start stacking up. I have to proactively throttle the backups when I see that we lose CBT.

Are there are any plans in the future to try to maintain CBT throughout upgrades and changes to the cluster?
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT feature request

Veeam Logoby foggy » Thu Dec 05, 2013 1:27 pm

Collin, thank you for the kind words and feedback. There is some place for improvement, probably we could address the following cases in future:
collinp wrote:upgrade Veeam editions

However, this will most likely require host reboot. Would it be less painful then re-running full backup?

collinp wrote:add a host to the cluster

Regarding this one, I don't think this is possible, since we do not track such events internally and you never know whether some changes were performed to the VMs moved to the new node prior their backup.

collinp wrote:implement certain host hotfixes.

Are you saying that applying Windows updates to hosts resets CBT? According to devs, this should not be the case and they would like to take a look at logs containing such occurrences, if possible.

Thanks.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: CBT feature request

Veeam Logoby collinp » Thu Dec 05, 2013 3:26 pm

Foggy,

The new node that we added to the cluster had no vm's on it. No vm's were moved to it. We didn't move vm's around. But every vm lost cbt. This has happened twice in a month when we added new hosts.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT feature request

Veeam Logoby foggy » Thu Dec 05, 2013 3:40 pm

I'm not saying this is not the case. I'm saying that Veeam B&R cannot guarantee that no changes were performed on the new node, since CBT driver is installed on the hosts on adding them to the Veeam B&R console and is activated on the first job run against them. New nodes are not added to the console automatically.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: CBT feature request

Veeam Logoby collinp » Thu Dec 05, 2013 4:00 pm

The new node was joined to the cluster then I added the new host to Veeam Backup and Recovery. The new host had no vm's on it. As the backup jobs started at night I noticed that CBT wasn't working for all VM's. Can CBT fail on a cluster level or does it usually fail per host? It seems to have failed at the cluster level
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT feature request

Veeam Logoby foggy » Thu Dec 05, 2013 4:02 pm

Yes, it is reset for all nodes in the cluster.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: CBT feature request

Veeam Logoby collinp » Thu Dec 05, 2013 5:52 pm

Foggy,

So you don't see a way that I can add a host to a cluster and maintain CBT? If we had funding, we could build out the entire cluster to 8 nodes from the beginning but we have been adding nodes as necessary.

Do you know of a non-standard way of copying the CBT info over to the new host so we don't lose CBT? I suppose I can submit a support ticket to see if there is an undocumented way of getting around this issue.

Thanks again
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT feature request

Veeam Logoby foggy » Fri Dec 06, 2013 10:56 am

I'm afraid there is no such a way due to the current CBT mechanism implementation.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: CBT feature request

Veeam Logoby collinp » Wed Oct 28, 2015 1:01 pm

It has been 2 years since this post. I still have the same question. How do you upgrade Veeam in a large Hyper-v environment without missing backups due to CBT being reset?

Just to give you an idea of the size of our environment. We now have over 700 Hyper-v VM's, 25 Hyper-v hosts, and full backups totaling more than 200TB. If I were to upgrade Veeam, it would take weeks to catch up on backups in order for CBT to reset.

It seems to me that you could include a feature in Veeam to upgrade CBT one Hyper-v host at a time. This way you can do rolling CBT upgrades.

Collin
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT feature request

Veeam Logoby foggy » Wed Oct 28, 2015 2:13 pm

Collin, the answer is still the same, since the behavior didn't change. One thing you could consider to do is scheduling an upgrade prior to performing periodic full backups, in case they are performed on some basis. Though, not sure this will tremendously help in such a big environment.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: CBT feature request

Veeam Logoby collinp » Sun Nov 15, 2015 3:02 pm

Do you plan on supporting the scaling of large corporate Hyper-v environments in the future by not forcing you to do a full scan backup every time there is an upgrade? My idea was to allow rolling CBT upgrades. Or will the next version of Hyper-v include CBT functionality natively?
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: CBT feature request

Veeam Logoby foggy » Mon Nov 16, 2015 12:02 pm

I'm not aware of any short-term plans regarding this. Seems it has not generated any considerable number of requests so far.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: CBT feature request

Veeam Logoby chrisdearden » Mon Nov 16, 2015 12:36 pm

I suspect it will be a feature of the next version of hyper v - http://www.aidanfinn.com/?p=17505
chrisdearden
Expert
 
Posts: 1529
Liked: 225 times
Joined: Wed Jul 21, 2010 9:47 am
Full Name: Chris Dearden

Re: CBT feature request

Veeam Logoby nmdange » Mon Nov 16, 2015 6:03 pm 1 person likes this post

collinp wrote:Just to give you an idea of the size of our environment. We now have over 700 Hyper-v VM's, 25 Hyper-v hosts, and full backups totaling more than 200TB. If I were to upgrade Veeam, it would take weeks to catch up on backups in order for CBT to reset.


So my environment is about half the size of yours (350 VMs, 15 hosts, 100TB data). After installing v8 update 3, it only took 1 extra day to catch up on the backups. If it really takes weeks to do full backups on 200TB, you must have a bottleneck somewhere in your setup that doesn't exist in mine. Storage network for my Hyper-V clusters is SMB 3.0 using 10Gb RDMA NICs and I use the backup repository server as an off-host backup proxy. What kind of throughput do you see in your backups?
nmdange
Expert
 
Posts: 193
Liked: 57 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: CBT feature request

Veeam Logoby collinp » Wed Nov 18, 2015 1:09 am

We have a dozen 5TB-10TB VM's that take days to complete. We have 90 backup jobs that average about 8 VM's per backup job. For throughput we see 50 MB/s to 100 MB/s per job. How many concurrent backups do you allow per repository and per host?
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 6 guests