Replication Performance

VMware specific discussions

Replication Performance

Veeam Logoby tom11011 » Sat Dec 03, 2016 5:05 pm

I would like to attempt to compare our replication performance with others. In my opinion, our performance used to be better in the past but it just seems to take a really long time now and I would like to get some comparisons. I'll try and layout the technical specs below.

Our source and destination servers are Dell PE720, dual six core cpu, 256gigs ram
Vmware vsphere 6.0 Update 2 plus all patches after
Latest Veeam 9.5
Dell PE 3200i SAN at both locations, all drives are 600gig 15K rpm sas
1 gig iscsi (although I'm not sure that matters in this case)
All disk groups are 12 drive disk groups in a raid 10

We are replicating over a 30 Mbps vpn tunnel to a distant site, all 30 Mbps are available, however we rarely use the available bandwidth which I find odd, its more likely to see 15-20 Mbps in use. Source always seems to be our bottleneck according to veeam.

Image
tom11011
Expert
 
Posts: 124
Liked: never
Joined: Wed Dec 01, 2010 8:40 pm
Full Name: Tom

Re: Replication Performance

Veeam Logoby Gostev » Sat Dec 03, 2016 7:16 pm

The issue here is the performance of fetching data from the source SAN... according to your screenshot, something is seriously wrong with the fabric or the SAN itself.
Gostev
Veeam Software
 
Posts: 21621
Liked: 2411 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Replication Performance

Veeam Logoby tom11011 » Sat Dec 03, 2016 7:21 pm

What do you judge that upon?
tom11011
Expert
 
Posts: 124
Liked: never
Joined: Wed Dec 01, 2010 8:40 pm
Full Name: Tom

Re: Replication Performance

Veeam Logoby Gostev » Sat Dec 03, 2016 8:05 pm

By the fact that 20 MB/s is as fast as the backup proxy is able to retrieve the data from SAN... that's too slow even for my 10 year old laptop, let alone SAN with 15K SAS drives. And bottleneck stats confirm that by showing that every other component basically remain idle, just waiting for the source data mover to supply the data.
Gostev
Veeam Software
 
Posts: 21621
Liked: 2411 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Replication Performance

Veeam Logoby tom11011 » Sat Dec 03, 2016 8:49 pm

I think maybe the san is too busy, lots of veeam backups and replications running. I am going to disable all jobs and then just let one replication job run by itself. I'll report back.
tom11011
Expert
 
Posts: 124
Liked: never
Joined: Wed Dec 01, 2010 8:40 pm
Full Name: Tom

Re: Replication Performance

Veeam Logoby tom11011 » Sat Dec 03, 2016 9:19 pm

Here is a replication job running by itself, this looks better.

A few questions,

1.) How does the current performance for a single replication job running rate for my described setup?
2.) if I change my backup jobs to use forward incrementals instead of reverse, what kind of performance impact would one expect to see?
3.) also, will veeam continue to cleanup the old vrb files created by the reverse incremental jobs?
4.) can you contrast the difference between processing rate and speed?

Image
tom11011
Expert
 
Posts: 124
Liked: never
Joined: Wed Dec 01, 2010 8:40 pm
Full Name: Tom

Re: Replication Performance

Veeam Logoby foggy » Mon Dec 05, 2016 2:45 pm

1. Now performance looks more justified.
2. Considering the bottleneck is still source, this will not affect anything.
3. In case of backup method switch to forward incremental, yes.
4. Processing rate is a ratio between the amount of all actually read data and time it took to transfer data to the target. The reading speed in the stats below is the actual average read rate for the disk.
foggy
Veeam Software
 
Posts: 15295
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Replication Performance

Veeam Logoby tom11011 » Mon Dec 05, 2016 3:40 pm

>>>Considering the bottleneck is still source, this will not affect anything.
This was hypothetical, the image above is a replication not a backup, not sure if you based your answer on that.

Thank you to everyone so far. I'm running into a problem with required service level agreements (SLA) that I would like to present in a very generic sense as to get an idea of how it could be handled.

Let's say I have 2 MSSQL 2014 database virtual servers. Each of those servers is configured with 128gigs ram, 8 cores over 2 cpu.

Up until these last months, I was able to manage our SLA which was a 2 hour RPO and a 2 hour RTO.

Each server would basically run a backup to local NAS storage and then a replication to a geographically distant data center. See the original post to get an idea of our technology. Each job for each server was able to finish in about 1 to 1.5 hours on average. But, as our data continues to grow (currently about 750gigs per database server) I am no longer able to meet this SLA.

I'm trying to figure out if I have the right solution or not and am looking for suggestions. What should I be measuring? I have a budget.

Thoughts off the top of my head include the following-

1.) Our gear with the exception of our switches is 10gig ready. Pull the trigger on 10gig switches if it will make a big difference (obviously this doesn't help replications)
2.) Add a second SAN (ie stop adding disk shelves to current san)
3.) Use Veeam to either replicate or backup, but not both, ie use sql mirroring

other?
tom11011
Expert
 
Posts: 124
Liked: never
Joined: Wed Dec 01, 2010 8:40 pm
Full Name: Tom

Re: Replication Performance

Veeam Logoby tsightler » Mon Dec 05, 2016 4:03 pm

To me it looks like CBT is not working even though it says it is using CBT. Look at the amount of data read vs the amount of data transferred, they are massively different, in the first screenshot we read 268GB of data while transferring only 1.8GB. I would suggest trying to reset CBT on these VMs, let it run again. After the CBT reset the first run will not be better, but I'm hoping it will get a lot better on subsequent runs after that.

I guess it could also be possible that something is writing to these blocks even though the actual data is not changing, but that seems less likely.
tsightler
Veeam Software
 
Posts: 4874
Liked: 1821 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Replication Performance

Veeam Logoby tom11011 » Mon Dec 05, 2016 4:09 pm

I would be willing to try that on a job with a single vm. Do I do that from the vsphere power CLI or can I reset it from within the gui somehow?
tom11011
Expert
 
Posts: 124
Liked: never
Joined: Wed Dec 01, 2010 8:40 pm
Full Name: Tom

Re: Replication Performance

Veeam Logoby tom11011 » Mon Dec 05, 2016 4:57 pm

I found veeam KB1113 which leads to vmware kb 2139574. But that script resets cbt on all virtual machines, can anyone point me in the direction of a script for a single vm?
tom11011
Expert
 
Posts: 124
Liked: never
Joined: Wed Dec 01, 2010 8:40 pm
Full Name: Tom

Re: Replication Performance

Veeam Logoby tsightler » Mon Dec 05, 2016 6:20 pm

It looks like those screenshots above are from a replica from backup job, is that right? If so, do can you share the equivalent screenshots from the backup itself? That might give more insight. I can't figure out any scenario where you should be seeing such a high read vs transfer unless CBT is not working properly.
tsightler
Veeam Software
 
Posts: 4874
Liked: 1821 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Replication Performance

Veeam Logoby tom11011 » Mon Dec 05, 2016 6:30 pm

Hi, I'm not sure what you mean by "replica from a backup job". Those pictures are replication jobs.

Here is a screen shot from a backup. Notice the very first line in the log action? "Virtual disk configuration change detected, resetting CBT". All my sql backup jobs say that, but my other backup jobs do not. All jobs have CBT enable, all vm's are running version 7 or higher.

Image
tom11011
Expert
 
Posts: 124
Liked: never
Joined: Wed Dec 01, 2010 8:40 pm
Full Name: Tom

Re: Replication Performance

Veeam Logoby tsightler » Mon Dec 05, 2016 7:01 pm

OK, so now we are getting somewhere. For some reason Veeam is detecting that there has been a change to the VM disk size and is resetting CBT. This might be expected after you resized a disk, but should only happen once after that. Are you getting this message on each and every run?
tsightler
Veeam Software
 
Posts: 4874
Liked: 1821 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Replication Performance

Veeam Logoby tom11011 » Mon Dec 05, 2016 7:06 pm

Yes, each and every run. But only for the db server jobs, not for any other vm.

It is happening on both replication and backup jobs for these particular servers.

These servers have had their disk sizes increased, but the last one was several weeks ago.
tom11011
Expert
 
Posts: 124
Liked: never
Joined: Wed Dec 01, 2010 8:40 pm
Full Name: Tom

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 1 guest