Comprehensive data protection for all workloads
Post Reply
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker »

I recently upgraded my repository to 10Gbe speed. Our VMware environment is on Cisco UCS blades with NetApp storage. I'm getting exactly the same backups speeds as I was when my repository was on 1Gb.

I created two jobs with the same VM's in each but going to the different repositories.

My new repository
processing rate is 150 MB/s
Load: Source 99% > Proxy 7% > Network 0% > Target 1%

Old repository
processing rate is 142 MB/s
Load: Source 99% > Proxy 8% > Network 0% > Target 2%

Am I just seeing a limit on how fast the proxy can work in hot add mode regardless of the network bandwidth?
tinto1970
Veeam Legend
Posts: 128
Liked: 38 times
Joined: Sep 26, 2013 8:40 am
Full Name: Alessandro T.
Location: Bologna, Italy
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by tinto1970 »

hi, it seems pretty clear your Bottleneck is Source. So i would not have expected enhancements by upgrading the target.

150MB/s is a not as bad speed in my opinion: it would be nice to have more details about your main storage type (type of disks and so on) and the vSphere infrastructure to evaulate the performance.
Alessandro aka Tinto | VMCE 2024 | Veeam Legend | VCP-DCV 2023 | VVSPHT2023 | vExpert 2024
blog.tinivelli.com
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by Gostev »

Agree, target really did not need the upgrade here with 1% busy. I am not aware of any hidden hot add throttling, but perhaps I could see VMware scheduling storage access equally between all VMs on the datastore, which may lead to this. I assume you don't use Storage QoS, because this will obviously impact hot add I/O performance as well.
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker »

tinto1970 wrote:hi, it seems pretty clear your Bottleneck is Source. So i would not have expected enhancements by upgrading the target.

150MB/s is a not as bad speed in my opinion: it would be nice to have more details about your main storage type (type of disks and so on) and the vSphere infrastructure to evaulate the performance.
More details:

Upgrading the target was intended from the start, (we just got our Veeam license) I used the opportunity to get it on 10Gbe because we already had the switching in place so why not.

Some of the datastores do have SIOC turned on, so I moved some of these test VM's to a datastore with that disabled but not seeing a difference as of yet.

The storage is a NetApp with 32 disks. I've got four hosts with one proxy each in VCenter, version 5.1. Each proxy is assigned 8 CPU and 12gb memory.

Prior to my new repository I've seen the network usage on the repository consistently cresting up to 1Gb and that made me expect it could transfer the data faster once it had more bandwidth. I haven't yet done full environment backups, I've been doing test with 5 to 6 VM's with up to 600gb drives on them. This might show more when I start doing 20 at a time.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by Vitaliy S. »

What about connection to the source hosts? Do you also use 10 Gbe networks? It might be worth checking how network proxy servers perform in your case and then compare these numbers with the hotadd configuration.
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker »

Vitaliy S. wrote:What about connection to the source hosts? Do you also use 10 Gbe networks? It might be worth checking how network proxy servers perform in your case and then compare these numbers with the hotadd configuration.
Yes, the blade chassis is connected to the storage on multiple 10Gbe connections. It's a Cisco UCS blade/NetApp combo. I do know we are behind in firmware and that is scheduled to be upgraded soon. Once that is done we will be connecting the repository on the fiber interconnects which is even closer to the hosts and storage. I'm currently on a 10Gbe switch one hop up from that.

I'll compare the network mode against the hotadd mode today.
tinto1970
Veeam Legend
Posts: 128
Liked: 38 times
Joined: Sep 26, 2013 8:40 am
Full Name: Alessandro T.
Location: Bologna, Italy
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by tinto1970 »

i think you could try to clone one of the VMs you are backing up, to evaulate the performances allowed by your storage architecture.
If you don't have VAAI plugins installed, the time needed by the cloning should be in the same order of the time needed by a full backup, expecially if you clone a VM which is on the NetApp to a datastore located on other storage (i.e. local disks if you have).
Alessandro aka Tinto | VMCE 2024 | Veeam Legend | VCP-DCV 2023 | VVSPHT2023 | vExpert 2024
blog.tinivelli.com
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker »

tinto1970 wrote:i think you could try to clone one of the VMs you are backing up, to evaluate the performances allowed by your storage architecture.
If you don't have VAAI plugins installed, the time needed by the cloning should be in the same order of the time needed by a full backup, especially if you clone a VM which is on the NetApp to a datastore located on other storage (i.e. local disks if you have).
That was a good idea to test.
I cloned a file server with 600GB of data. The cloning took 2 hrs and 32 minutes. To backup the same server, 2 hrs 47 minutes.

It would appear then that is as fast as the storage is going to go.
tinto1970
Veeam Legend
Posts: 128
Liked: 38 times
Joined: Sep 26, 2013 8:40 am
Full Name: Alessandro T.
Location: Bologna, Italy
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by tinto1970 »

PNWMtnBiker wrote:
I cloned a file server with 600GB of data. The cloning took 2 hrs and 32 minutes. To backup the same server, 2 hrs 47 minutes.

It would appear then that is as fast as the storage is going to go.
so Veeam feauture called "forever incremental" and CBT from vSphere will help you to get fast daily backups even if the source is not as fast ;)
Alessandro aka Tinto | VMCE 2024 | Veeam Legend | VCP-DCV 2023 | VVSPHT2023 | vExpert 2024
blog.tinivelli.com
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by Vitaliy S. »

Actually backup job mode will not have any effect on job performance in this case, however CBT and multiple proxy servers is the way to go to make your backups fly.
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker »

Vitaliy S. wrote:Actually backup job mode will not have any effect on job performance in this case, however CBT and multiple proxy servers is the way to go to make your backups fly.
What I've setup is 4 proxies (1 per host) with 8vCPU's and 12gb of memory each. That should let me do 32 simultaneous backups, correct?
ND40oz
Influencer
Posts: 20
Liked: 1 time
Joined: Nov 17, 2010 12:52 am
Full Name: ND40oz
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by ND40oz »

What nic are you using on the proxies, the VMXNET3?

Also, you're going to be limited on parallel processing by the amount of active snapshots on a datastore, the default is 4 but this can be changed in the registry if your storage can handle more. By default, even if you have 4 proxy servers, Veeam will hit the active snapshot limit if you only have one or two datastores even with a single proxy.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by Vitaliy S. »

Yes, you will be backing up 32 virtual disks in parallel, make sure your hosts and source datastores can handle this load. Let us know the results you would have.
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker »

ND40oz wrote:What nic are you using on the proxies, the VMXNET3?

Also, you're going to be limited on parallel processing by the amount of active snapshots on a datastore, the default is 4 but this can be changed in the registry if your storage can handle more. By default, even if you have 4 proxy servers, Veeam will hit the active snapshot limit if you only have one or two datastores even with a single proxy.
Yes, I came across that a couple weeks ago and updated the registry key to allow up to 48.
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker »

Vitaliy S. wrote:Yes, you will be backing up 32 virtual disks in parallel, make sure your hosts and source datastores can handle this load. Let us know the results you would have.
I have all my jobs based on the folders in vcenter and my plan is to enable a job every few days so I can monitor the performance of the SAN, hosts and repository and back it down if I need to. I will definitely update with the results once this starts getting ramped up, which will be soon :D
tdewin
Veeam Software
Posts: 1818
Liked: 655 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by tdewin »

PNWMtnBiker wrote: What I've setup is 4 proxies (1 per host) with 8vCPU's and 12gb of memory each. That should let me do 32 simultaneous backups, correct?
I always advise to make more smaller hot add proxies:
  • Co-scheduling, all those 8 core's need to be scheduled together (https://www.vmware.com/files/pdf/techpa ... d-Perf.pdf). Ok it might be less of an issue when you have 40 threads but still
  • One hot add proxy can process multiple disks if it has enough SCSI slots on the SCSI controller. However there is a limitation, that one hot add proxy can only process one disk of a certain VM at the same time.
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker »

tdewin wrote: I always advise to make more smaller hot add proxies:
  • Co-scheduling, all those 8 core's need to be scheduled together (https://www.vmware.com/files/pdf/techpa ... d-Perf.pdf). Ok it might be less of an issue when you have 40 threads but still
  • One hot add proxy can process multiple disks if it has enough SCSI slots on the SCSI controller. However there is a limitation, that one hot add proxy can only process one disk of a certain VM at the same time.
More smaller proxies is a good point, reading the document now.

I know there is a limit of 15 on the SCSI controller and I could add an additional SCSI controller if necessary. My VCenter has 180 VM's and at the moment I'm not feeling like I have to go there just yet.

A question came to mind, The Max concurrent tasks per proxy. Isn't that limited to cpu's on the proxy? If I have 8 CPU, I can only process 8 disks. If I raise that to use more SCSI connections, then wouldn't I have to raise the cpu's?
veremin
Product Manager
Posts: 20415
Liked: 2302 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by veremin »

A question came to mind, The Max concurrent tasks per proxy. Isn't that limited to cpu's on the proxy? If I have 8 CPU, I can only process 8 disks.
Actually, each task requires a single core. How many cores does the given VM have? Thanks.
tdewin
Veeam Software
Posts: 1818
Liked: 655 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by tdewin »

A question came to mind, The Max concurrent tasks per proxy. Isn't that limited to cpu's on the proxy? If I have 8 CPU, I can only process 8 disks.
Yes but only 1 disk of a certain VM at the same time (Virtual Appliance API limitation). Honestly it is not really a problem since most VM's are single disk
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker »

v.Eremin wrote: Actually, each task requires a single core. How many cores does the given VM have? Thanks.
You're referring to the proxy right? The proxy has 8vCPU's assigned.
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker »

Yes but only 1 disk of a certain VM at the same time (Virtual Appliance API limitation). Honestly it is not really a problem since most VM's are single disk
60% of my VM's have multiple disks.

What I can do is easily enough is to see what result I get with the four proxies (set at 8 cpu) and then add four more proxies but reducing the vCPU's of them all to four and compare.

Since talking with you all it's pretty clear more proxies with less assigned vCPU's each are going to get the job done faster.
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: HotAdd and 10Gbe to repository speeds?

Post by PNWMtnBiker » 3 people like this post

Hi guys, just a follow up...

I tested doubling the amount of proxies but scaling down the vCPU's assigned to each.

So, 8 proxies with 4vCPU each.

I'm getting 352MB/s processing rate now :)
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], Semrush [Bot] and 39 guests