Discussions specific to the VMware vSphere hypervisor
Post Reply
unsichtbarre
Expert
Posts: 130
Liked: 24 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Veem B&R 8 seems to take much more time processing replicas

Post by unsichtbarre » Jan 26, 2015 7:09 pm

We recently upgraded to Veeam B&R v8 and have noticed that it is slowing down our replication jobs significantly. We have gone from surpassing a 30min RPO to requiring greater than 2 hours/job

The slowdown happens during the step(s) "Using target proxy Blah Blah Blah for disk Hard disk X [nbd]" and holds the job for 4-12 minutes per VM. We have parallel processing enabled and enough proxies/CPUs to run multiple disks at a time - and even when only one VM is processing, there is a tremendous delay we did not see before.

If you add 4-12 minutes per VM on a job with 10 VMs/replicas, we are no longer to meet a RPO less than 2 hours, whereas in Veeam 7, we were easily able to meet sub-30 min. RPO

After the data starts to replicate, replication it seems to move as fast as it ever did.

Any ideas?

This case has been sent to Veeam Support and log files have been uploaded.
Case #00733966

Thanks,
-The Invisible Admin-
http://www.johnborhek.com

Vitaliy S.
Product Manager
Posts: 23062
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veem B&R 8 seems to take much more time processing repli

Post by Vitaliy S. » Jan 26, 2015 7:46 pm

Have you tried using hotadd backup mode at the target location to see if it makes any difference or not?

unsichtbarre
Expert
Posts: 130
Liked: 24 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: Veem B&R 8 seems to take much more time processing repli

Post by unsichtbarre » Jan 26, 2015 7:59 pm

We have been burned by hotadd in the past. If possible, we would like to avoid it.

Also, v7 was not configured for hotadd, prior to the upgrade.

THX for the response.
-The Invisible Admin-
http://www.johnborhek.com

Vitaliy S.
Product Manager
Posts: 23062
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veem B&R 8 seems to take much more time processing repli

Post by Vitaliy S. » Jan 26, 2015 8:28 pm

Ok, let's see what our support team figures out after reviewing the debug logs. BTW, does this happen on all target proxy servers?

unsichtbarre
Expert
Posts: 130
Liked: 24 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: Veem B&R 8 seems to take much more time processing repli

Post by unsichtbarre » Jan 27, 2015 12:27 am

At this point, there are two proxies at source side w/4 vCPU each and only the Veeam server at target side w/8vCPU. I can add an additional proxy at the target and test, although the problem occurrs even when only one VM with one disk is running.

The support team has responded and said that "it looks like encryption in enabled." We have double-checked and encryption is not enabled between host/proxy. Since it is a replication, we dont know of any other encryption feature.
-The Invisible Admin-
http://www.johnborhek.com

veremin
Product Manager
Posts: 16998
Liked: 1455 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veem B&R 8 seems to take much more time processing repli

Post by veremin » Jan 27, 2015 7:57 am

Most likely, they're talking about traffic encryption that can be found in the global throttling settings. Thanks.

unsichtbarre
Expert
Posts: 130
Liked: 24 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: Veem B&R 8 seems to take much more time processing repli

Post by unsichtbarre » Jan 28, 2015 6:02 pm

We are still struggling with a halt of about 6 minutes per VM, per job.

Encryption was never enabled anywhere in the GUI, but on the advice of Veeam support we have applied the following Registry key:

DisablePublicIPTrafficEncryption

Type: REG_DWORD
Default value: 0
Description: By default, traffic between different subnets and on public addresses is encrypted. To globally disable network encryption between agents, set to 1.

In the following image, you can clearly see where Veeam just stops for over 6 minutes. In Veeam 7, this job was finalizing both VMs in 3-5 minutes. In our other job with 10 servers, we are adding 60 or more minutes, just waiting around

New logs have been gathered and uploaded to the same case referenced before.

Thanks,

Image
-The Invisible Admin-
http://www.johnborhek.com

unsichtbarre
Expert
Posts: 130
Liked: 24 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: Veem B&R 8 seems to take much more time processing repli

Post by unsichtbarre » Feb 03, 2015 8:32 pm 3 people like this post

Thanks to Ken Sauer at Veeam Support, this is now resolved.

Network Connections > [ALT] > Advanced > Advanced Settings > Adapters and Bindings

Place the Replication network first in the list and the local vSphere Management Network second

Thanks Ken!
-The Invisible Admin-
http://www.johnborhek.com

Post Reply

Who is online

Users browsing this forum: AuGL and 15 guests