HotAdd and 10Gbe to repository speeds?

[PNWMtnBiker]

I recently upgraded my repository to 10Gbe speed. Our VMware environment is on Cisco UCS blades with NetApp storage. I'm getting exactly the same backups speeds as I was when my repository was on 1Gb.

I created two jobs with the same VM's in each but going to the different repositories.

My new repository
processing rate is 150 MB/s
Load: Source 99% > Proxy 7% > Network 0% > Target 1%

Old repository
processing rate is 142 MB/s
Load: Source 99% > Proxy 8% > Network 0% > Target 2%

Am I just seeing a limit on how fast the proxy can work in hot add mode regardless of the network bandwidth?

[tinto1970]

hi, it seems pretty clear your Bottleneck is Source. So i would not have expected enhancements by upgrading the target.

150MB/s is a not as bad speed in my opinion: it would be nice to have more details about your main storage type (type of disks and so on) and the vSphere infrastructure to evaulate the performance.

[Gostev]

Agree, target really did not need the upgrade here with 1% busy. I am not aware of any hidden hot add throttling, but perhaps I could see VMware scheduling storage access equally between all VMs on the datastore, which may lead to this. I assume you don't use Storage QoS, because this will obviously impact hot add I/O performance as well.

[PNWMtnBiker]

hi, it seems pretty clear your Bottleneck is Source. So i would not have expected enhancements by upgrading the target.

150MB/s is a not as bad speed in my opinion: it would be nice to have more details about your main storage type (type of disks and so on) and the vSphere infrastructure to evaulate the performance.

More details:

Upgrading the target was intended from the start, (we just got our Veeam license) I used the opportunity to get it on 10Gbe because we already had the switching in place so why not.

Some of the datastores do have SIOC turned on, so I moved some of these test VM's to a datastore with that disabled but not seeing a difference as of yet.

The storage is a NetApp with 32 disks. I've got four hosts with one proxy each in VCenter, version 5.1. Each proxy is assigned 8 CPU and 12gb memory.

Prior to my new repository I've seen the network usage on the repository consistently cresting up to 1Gb and that made me expect it could transfer the data faster once it had more bandwidth. I haven't yet done full environment backups, I've been doing test with 5 to 6 VM's with up to 600gb drives on them. This might show more when I start doing 20 at a time.

[Vitaliy S.]

What about connection to the source hosts? Do you also use 10 Gbe networks? It might be worth checking how network proxy servers perform in your case and then compare these numbers with the hotadd configuration.

[PNWMtnBiker]

What about connection to the source hosts? Do you also use 10 Gbe networks? It might be worth checking how network proxy servers perform in your case and then compare these numbers with the hotadd configuration.

Yes, the blade chassis is connected to the storage on multiple 10Gbe connections. It's a Cisco UCS blade/NetApp combo. I do know we are behind in firmware and that is scheduled to be upgraded soon. Once that is done we will be connecting the repository on the fiber interconnects which is even closer to the hosts and storage. I'm currently on a 10Gbe switch one hop up from that.

I'll compare the network mode against the hotadd mode today.

[tinto1970]

i think you could try to clone one of the VMs you are backing up, to evaulate the performances allowed by your storage architecture.
If you don't have VAAI plugins installed, the time needed by the cloning should be in the same order of the time needed by a full backup, expecially if you clone a VM which is on the NetApp to a datastore located on other storage (i.e. local disks if you have).

[PNWMtnBiker]

i think you could try to clone one of the VMs you are backing up, to evaluate the performances allowed by your storage architecture.
If you don't have VAAI plugins installed, the time needed by the cloning should be in the same order of the time needed by a full backup, especially if you clone a VM which is on the NetApp to a datastore located on other storage (i.e. local disks if you have).

That was a good idea to test.
I cloned a file server with 600GB of data. The cloning took 2 hrs and 32 minutes. To backup the same server, 2 hrs 47 minutes.

It would appear then that is as fast as the storage is going to go.

[tinto1970]

I cloned a file server with 600GB of data. The cloning took 2 hrs and 32 minutes. To backup the same server, 2 hrs 47 minutes.

It would appear then that is as fast as the storage is going to go.

so Veeam feauture called "forever incremental" and CBT from vSphere will help you to get fast daily backups even if the source is not as fast

[Vitaliy S.]

Actually backup job mode will not have any effect on job performance in this case, however CBT and multiple proxy servers is the way to go to make your backups fly.

[PNWMtnBiker]

Actually backup job mode will not have any effect on job performance in this case, however CBT and multiple proxy servers is the way to go to make your backups fly.

What I've setup is 4 proxies (1 per host) with 8vCPU's and 12gb of memory each. That should let me do 32 simultaneous backups, correct?

[ND40oz]

What nic are you using on the proxies, the VMXNET3?

Also, you're going to be limited on parallel processing by the amount of active snapshots on a datastore, the default is 4 but this can be changed in the registry if your storage can handle more. By default, even if you have 4 proxy servers, Veeam will hit the active snapshot limit if you only have one or two datastores even with a single proxy.

[Vitaliy S.]

Yes, you will be backing up 32 virtual disks in parallel, make sure your hosts and source datastores can handle this load. Let us know the results you would have.

[PNWMtnBiker]

What nic are you using on the proxies, the VMXNET3?

Also, you're going to be limited on parallel processing by the amount of active snapshots on a datastore, the default is 4 but this can be changed in the registry if your storage can handle more. By default, even if you have 4 proxy servers, Veeam will hit the active snapshot limit if you only have one or two datastores even with a single proxy.

Yes, I came across that a couple weeks ago and updated the registry key to allow up to 48.

[PNWMtnBiker]

Yes, you will be backing up 32 virtual disks in parallel, make sure your hosts and source datastores can handle this load. Let us know the results you would have.

I have all my jobs based on the folders in vcenter and my plan is to enable a job every few days so I can monitor the performance of the SAN, hosts and repository and back it down if I need to. I will definitely update with the results once this starts getting ramped up, which will be soon

[tdewin]

What I've setup is 4 proxies (1 per host) with 8vCPU's and 12gb of memory each. That should let me do 32 simultaneous backups, correct?

I always advise to make more smaller hot add proxies:

• Co-scheduling, all those 8 core's need to be scheduled together (https://www.vmware.com/files/pdf/techpa ... d-Perf.pdf). Ok it might be less of an issue when you have 40 threads but still
• One hot add proxy can process multiple disks if it has enough SCSI slots on the SCSI controller. However there is a limitation, that one hot add proxy can only process one disk of a certain VM at the same time.

[PNWMtnBiker]

I always advise to make more smaller hot add proxies:

• Co-scheduling, all those 8 core's need to be scheduled together (https://www.vmware.com/files/pdf/techpa ... d-Perf.pdf). Ok it might be less of an issue when you have 40 threads but still
• One hot add proxy can process multiple disks if it has enough SCSI slots on the SCSI controller. However there is a limitation, that one hot add proxy can only process one disk of a certain VM at the same time.

More smaller proxies is a good point, reading the document now.

I know there is a limit of 15 on the SCSI controller and I could add an additional SCSI controller if necessary. My VCenter has 180 VM's and at the moment I'm not feeling like I have to go there just yet.

A question came to mind, The Max concurrent tasks per proxy. Isn't that limited to cpu's on the proxy? If I have 8 CPU, I can only process 8 disks. If I raise that to use more SCSI connections, then wouldn't I have to raise the cpu's?

[veremin]

A question came to mind, The Max concurrent tasks per proxy. Isn't that limited to cpu's on the proxy? If I have 8 CPU, I can only process 8 disks.

Actually, each task requires a single core. How many cores does the given VM have? Thanks.

[tdewin]

A question came to mind, The Max concurrent tasks per proxy. Isn't that limited to cpu's on the proxy? If I have 8 CPU, I can only process 8 disks.

Yes but only 1 disk of a certain VM at the same time (Virtual Appliance API limitation). Honestly it is not really a problem since most VM's are single disk

[PNWMtnBiker]

Actually, each task requires a single core. How many cores does the given VM have? Thanks.

You're referring to the proxy right? The proxy has 8vCPU's assigned.

[PNWMtnBiker]

Yes but only 1 disk of a certain VM at the same time (Virtual Appliance API limitation). Honestly it is not really a problem since most VM's are single disk

60% of my VM's have multiple disks.

What I can do is easily enough is to see what result I get with the four proxies (set at 8 cpu) and then add four more proxies but reducing the vCPU's of them all to four and compare.

Since talking with you all it's pretty clear more proxies with less assigned vCPU's each are going to get the job done faster.

[PNWMtnBiker]

Hi guys, just a follow up...

I tested doubling the amount of proxies but scaling down the vCPU's assigned to each.

So, 8 proxies with 4vCPU each.

I'm getting 352MB/s processing rate now