Replication using CBT same speed as non CBT ?

Availability for the Always-On Enterprise

Replication using CBT same speed as non CBT ?

Veeam Logoby stevenrodenburg1 » Wed Mar 08, 2017 3:46 pm

Hello,

I have been running everything (backups, replication jobs) **without** using CBT for the last couple of years (because VMware screwed up CBT several times and I lost trust). We have been on B&R 9.5 for a couple of months and on patch 1 since a couple of weeks.

We recently upgraded to vSphere 6.5.0a (hoping CBT is stable now) and carefully re-configured a couple of existing (non CBT) replication jobs of unimportant vm's to using CBT.

I expected these replication-jobs to be faster. But as it turns out, the jobs take the same amount of time. Example: a test SQL server, replicated every hour, always needed between 15 and 17 minutes for the job and after enabling CBT it's the same. So purely speed-wise, no benefits.

I see the job using CBT (it says so in the log). I also see CBT was enabled by looking at those VM's advanced settings. So i assume CBT is active.

Question: should using CBT be faster than Veeam B&R's own mechanism? Or did re-configuring an existing job, while everything was running (i did not shutdown the source VM, or delete the replica or anything like that), cause a sub-optimal condition somehow so Veeam cannot use the new situation to it's full potential?

If CBT is not faster, are there any other benefits of using CBT for backup and replication-jobs? Maybe less IO intensive?
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: Replication using CBT same speed as non CBT ?

Veeam Logoby Gostev » Wed Mar 08, 2017 6:02 pm

In case you are replicating to the VMFS6 datastore, then it might be the known issue I have described in my latest live session on vSphere 6.5 backup gotchas. What does the job show as the bottleneck for sessions with and without CBT enabled?
Gostev
Veeam Software
 
Posts: 21444
Liked: 2362 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Replication using CBT same speed as non CBT ?

Veeam Logoby stevenrodenburg1 » Wed Mar 08, 2017 8:15 pm

Hi Anton,

Source datastore is vSAN and target is an ESXi server with a NFS datastore.

Printout of some random sessions where CBT was not enabled yet:
02.03.2017 10:28:24 :: Load: Source 75% > Proxy 54% > Network 22% > Target 24%
04.03.2017 22:28:07 :: Load: Source 71% > Proxy 64% > Network 29% > Target 27%
05.03.2017 20:25:04 :: Load: Source 86% > Proxy 47% > Network 19% > Target 20%
06.03.2017 07:28:24 :: Load: Source 92% > Proxy 47% > Network 12% > Target 15%

Printout of a couple of today's sessions (CBT):
08.03.2017 08:30:34 :: Load: Source 38% > Proxy 20% > Network 48% > Target 59%
08.03.2017 10:29:11 :: Load: Source 37% > Proxy 22% > Network 33% > Target 57%
08.03.2017 14:28:11 :: Load: Source 50% > Proxy 20% > Network 39% > Target 59%
08.03.2017 20:27:42 :: Load: Source 48% > Proxy 15% > Network 41% > Target 58%

Interesting. All jobs take between 15 and 17 minutes but the difference in "where the load is" is vastly different. What do you make of this?
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: Replication using CBT same speed as non CBT ?

Veeam Logoby Gostev » Wed Mar 08, 2017 11:24 pm

So this tells me CBT does work fine and only return changed blocks, so perhaps the issue is the target datastore performance. Look up the job log to see what transport mode the target proxy is using, try switching it to a different mode (hot add vs. NBD) and see if this changes the statistics?
Gostev
Veeam Software
 
Posts: 21444
Liked: 2362 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Replication using CBT same speed as non CBT ?

Veeam Logoby stevenrodenburg1 » Thu Mar 09, 2017 8:29 pm 1 person likes this post

Ok I changed the single proxy that handles the replication jobs (so source proxy = target proxy) to NBD and hello, the jobs are 3 times as fast :-)
From 15 to 17 minutes down to 5 to 6 minutes. Holy craperoni.

And this is what the bottleneck indicator says now:
09.03.2017 20:15:39 :: Load: Source 91% > Proxy 13% > Network 46% > Target 61%
09.03.2017 21:15:36 :: Load: Source 88% > Proxy 12% > Network 46% > Target 60%
(with CBT)
(Source load is twice as high with NBD as with HotAdd though. The rest is about the same)

Used to be:
08.03.2017 14:28:11 :: Load: Source 50% > Proxy 20% > Network 39% > Target 59%
08.03.2017 20:27:42 :: Load: Source 48% > Proxy 15% > Network 41% > Target 58%
(with CBT)

Well, I guess I have a new favourite Transport mode: NBD
Hot-Add is sooo 2016... ;-)

Question: why is NBD 3x faster (in our VSAN environment at least) than HotAdd? I always thought HotAdd was the king of speed..
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: Replication using CBT same speed as non CBT ?

Veeam Logoby Gostev » Thu Mar 09, 2017 9:19 pm

You'd know already if you watched the recording :wink: for some reason hot add does not play well with sesparse snapshots.
Gostev
Veeam Software
 
Posts: 21444
Liked: 2362 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Replication using CBT same speed as non CBT ?

Veeam Logoby stevenrodenburg1 » Thu Mar 09, 2017 9:31 pm

Ah yes. I forgot about the sesparse thing. I focused on the "NBD now sucks balls because VMware forces NDBSSL" part, hence I did not expect NDB to be so fast.
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: Replication using CBT same speed as non CBT ?

Veeam Logoby stevenrodenburg1 » Fri Mar 10, 2017 10:30 am

Update: I re-configured our backup proxies (not the same as the one doing the replication) to do NBD as well instead of Hotadd. Backup jobs are now 3 to 4 times slower as before. Backup-jobs are not using CBT.

It's the opposite as to what I expected due to: "for some reason hot add does not play well with sesparse snapshots". Or does this not apply when CBT is not used?

This is the overall load of the entire 2.2 TB backup-job (now using NBD): 10.03.2017 06:50:26 :: Load: Source 99% > Proxy 16% > Network 0% > Target 0% (network and target 0% ???)

Replication jobs (NDB + CBT) are still super fast compared to Hotadd.
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: Replication using CBT same speed as non CBT ?

Veeam Logoby Gostev » Fri Mar 10, 2017 4:56 pm

You should only have reconfigured the target proxy, because that where the issue is (writing into the replica VM's sesparse snapshot). By also reconfiguring the source proxy, you are now enjoying that blazing fast NBDSSL performance you noted yourself :wink:
Gostev
Veeam Software
 
Posts: 21444
Liked: 2362 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Replication using CBT same speed as non CBT ?

Veeam Logoby stevenrodenburg1 » Fri Mar 10, 2017 5:35 pm

I'm sorry for causing confusion. I'm talking about the backup jobs (which are other Proxies than those that handle Replication jobs) in my last post.
Replication is super fast now. It's the backup jobs that became 3 to 4 times as slow using NBD + non-CBT. When i switch back to hotadd, they are fast again.

In backups, there are only source proxies ;-)

Another thing I was wondering about: what happens when a backup-chain, that was backed-up without CBT, suddenly starts using CBT ? are the two methods incompatible triggering a full backup or are they compatible and the transition is seamless?
If the latter is the case, can one switch back and forth between the two methods without punishment? I can't find documentation on this exact topic (I guess it's somewhat unusual to do this, but in a lab one can misbehave...)
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: Replication using CBT same speed as non CBT ?

Veeam Logoby Gostev » Fri Mar 10, 2017 8:14 pm

Sure, you can switch freely - this won't trigger a full backup and the transition is seamless, as the only difference is how changed blocks are determined before they are backed up. This is actually explained quite well in the User Guide > https://helpcenter.veeam.com/docs/backu ... tml?ver=95
Gostev
Veeam Software
 
Posts: 21444
Liked: 2362 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], jslic and 55 guests