-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
HotAdd and 10Gbe to repository speeds?
I recently upgraded my repository to 10Gbe speed. Our VMware environment is on Cisco UCS blades with NetApp storage. I'm getting exactly the same backups speeds as I was when my repository was on 1Gb.
I created two jobs with the same VM's in each but going to the different repositories.
My new repository
processing rate is 150 MB/s
Load: Source 99% > Proxy 7% > Network 0% > Target 1%
Old repository
processing rate is 142 MB/s
Load: Source 99% > Proxy 8% > Network 0% > Target 2%
Am I just seeing a limit on how fast the proxy can work in hot add mode regardless of the network bandwidth?
I created two jobs with the same VM's in each but going to the different repositories.
My new repository
processing rate is 150 MB/s
Load: Source 99% > Proxy 7% > Network 0% > Target 1%
Old repository
processing rate is 142 MB/s
Load: Source 99% > Proxy 8% > Network 0% > Target 2%
Am I just seeing a limit on how fast the proxy can work in hot add mode regardless of the network bandwidth?
-
- Veeam Legend
- Posts: 128
- Liked: 38 times
- Joined: Sep 26, 2013 8:40 am
- Full Name: Alessandro T.
- Location: Bologna, Italy
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
hi, it seems pretty clear your Bottleneck is Source. So i would not have expected enhancements by upgrading the target.
150MB/s is a not as bad speed in my opinion: it would be nice to have more details about your main storage type (type of disks and so on) and the vSphere infrastructure to evaulate the performance.
150MB/s is a not as bad speed in my opinion: it would be nice to have more details about your main storage type (type of disks and so on) and the vSphere infrastructure to evaulate the performance.
Alessandro aka Tinto | VMCE 2024 | Veeam Legend | VCP-DCV 2023 | VVSPHT2023 | vExpert 2024
blog.tinivelli.com
blog.tinivelli.com
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
Agree, target really did not need the upgrade here with 1% busy. I am not aware of any hidden hot add throttling, but perhaps I could see VMware scheduling storage access equally between all VMs on the datastore, which may lead to this. I assume you don't use Storage QoS, because this will obviously impact hot add I/O performance as well.
-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
More details:tinto1970 wrote:hi, it seems pretty clear your Bottleneck is Source. So i would not have expected enhancements by upgrading the target.
150MB/s is a not as bad speed in my opinion: it would be nice to have more details about your main storage type (type of disks and so on) and the vSphere infrastructure to evaulate the performance.
Upgrading the target was intended from the start, (we just got our Veeam license) I used the opportunity to get it on 10Gbe because we already had the switching in place so why not.
Some of the datastores do have SIOC turned on, so I moved some of these test VM's to a datastore with that disabled but not seeing a difference as of yet.
The storage is a NetApp with 32 disks. I've got four hosts with one proxy each in VCenter, version 5.1. Each proxy is assigned 8 CPU and 12gb memory.
Prior to my new repository I've seen the network usage on the repository consistently cresting up to 1Gb and that made me expect it could transfer the data faster once it had more bandwidth. I haven't yet done full environment backups, I've been doing test with 5 to 6 VM's with up to 600gb drives on them. This might show more when I start doing 20 at a time.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
What about connection to the source hosts? Do you also use 10 Gbe networks? It might be worth checking how network proxy servers perform in your case and then compare these numbers with the hotadd configuration.
-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
Yes, the blade chassis is connected to the storage on multiple 10Gbe connections. It's a Cisco UCS blade/NetApp combo. I do know we are behind in firmware and that is scheduled to be upgraded soon. Once that is done we will be connecting the repository on the fiber interconnects which is even closer to the hosts and storage. I'm currently on a 10Gbe switch one hop up from that.Vitaliy S. wrote:What about connection to the source hosts? Do you also use 10 Gbe networks? It might be worth checking how network proxy servers perform in your case and then compare these numbers with the hotadd configuration.
I'll compare the network mode against the hotadd mode today.
-
- Veeam Legend
- Posts: 128
- Liked: 38 times
- Joined: Sep 26, 2013 8:40 am
- Full Name: Alessandro T.
- Location: Bologna, Italy
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
i think you could try to clone one of the VMs you are backing up, to evaulate the performances allowed by your storage architecture.
If you don't have VAAI plugins installed, the time needed by the cloning should be in the same order of the time needed by a full backup, expecially if you clone a VM which is on the NetApp to a datastore located on other storage (i.e. local disks if you have).
If you don't have VAAI plugins installed, the time needed by the cloning should be in the same order of the time needed by a full backup, expecially if you clone a VM which is on the NetApp to a datastore located on other storage (i.e. local disks if you have).
Alessandro aka Tinto | VMCE 2024 | Veeam Legend | VCP-DCV 2023 | VVSPHT2023 | vExpert 2024
blog.tinivelli.com
blog.tinivelli.com
-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
That was a good idea to test.tinto1970 wrote:i think you could try to clone one of the VMs you are backing up, to evaluate the performances allowed by your storage architecture.
If you don't have VAAI plugins installed, the time needed by the cloning should be in the same order of the time needed by a full backup, especially if you clone a VM which is on the NetApp to a datastore located on other storage (i.e. local disks if you have).
I cloned a file server with 600GB of data. The cloning took 2 hrs and 32 minutes. To backup the same server, 2 hrs 47 minutes.
It would appear then that is as fast as the storage is going to go.
-
- Veeam Legend
- Posts: 128
- Liked: 38 times
- Joined: Sep 26, 2013 8:40 am
- Full Name: Alessandro T.
- Location: Bologna, Italy
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
so Veeam feauture called "forever incremental" and CBT from vSphere will help you to get fast daily backups even if the source is not as fastPNWMtnBiker wrote:
I cloned a file server with 600GB of data. The cloning took 2 hrs and 32 minutes. To backup the same server, 2 hrs 47 minutes.
It would appear then that is as fast as the storage is going to go.
Alessandro aka Tinto | VMCE 2024 | Veeam Legend | VCP-DCV 2023 | VVSPHT2023 | vExpert 2024
blog.tinivelli.com
blog.tinivelli.com
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
Actually backup job mode will not have any effect on job performance in this case, however CBT and multiple proxy servers is the way to go to make your backups fly.
-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
What I've setup is 4 proxies (1 per host) with 8vCPU's and 12gb of memory each. That should let me do 32 simultaneous backups, correct?Vitaliy S. wrote:Actually backup job mode will not have any effect on job performance in this case, however CBT and multiple proxy servers is the way to go to make your backups fly.
-
- Influencer
- Posts: 20
- Liked: 1 time
- Joined: Nov 17, 2010 12:52 am
- Full Name: ND40oz
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
What nic are you using on the proxies, the VMXNET3?
Also, you're going to be limited on parallel processing by the amount of active snapshots on a datastore, the default is 4 but this can be changed in the registry if your storage can handle more. By default, even if you have 4 proxy servers, Veeam will hit the active snapshot limit if you only have one or two datastores even with a single proxy.
Also, you're going to be limited on parallel processing by the amount of active snapshots on a datastore, the default is 4 but this can be changed in the registry if your storage can handle more. By default, even if you have 4 proxy servers, Veeam will hit the active snapshot limit if you only have one or two datastores even with a single proxy.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
Yes, you will be backing up 32 virtual disks in parallel, make sure your hosts and source datastores can handle this load. Let us know the results you would have.
-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
Yes, I came across that a couple weeks ago and updated the registry key to allow up to 48.ND40oz wrote:What nic are you using on the proxies, the VMXNET3?
Also, you're going to be limited on parallel processing by the amount of active snapshots on a datastore, the default is 4 but this can be changed in the registry if your storage can handle more. By default, even if you have 4 proxy servers, Veeam will hit the active snapshot limit if you only have one or two datastores even with a single proxy.
-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
I have all my jobs based on the folders in vcenter and my plan is to enable a job every few days so I can monitor the performance of the SAN, hosts and repository and back it down if I need to. I will definitely update with the results once this starts getting ramped up, which will be soonVitaliy S. wrote:Yes, you will be backing up 32 virtual disks in parallel, make sure your hosts and source datastores can handle this load. Let us know the results you would have.
-
- Veeam Software
- Posts: 1818
- Liked: 655 times
- Joined: Mar 02, 2012 1:40 pm
- Full Name: Timothy Dewin
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
I always advise to make more smaller hot add proxies:PNWMtnBiker wrote: What I've setup is 4 proxies (1 per host) with 8vCPU's and 12gb of memory each. That should let me do 32 simultaneous backups, correct?
- Co-scheduling, all those 8 core's need to be scheduled together (https://www.vmware.com/files/pdf/techpa ... d-Perf.pdf). Ok it might be less of an issue when you have 40 threads but still
- One hot add proxy can process multiple disks if it has enough SCSI slots on the SCSI controller. However there is a limitation, that one hot add proxy can only process one disk of a certain VM at the same time.
-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
More smaller proxies is a good point, reading the document now.tdewin wrote: I always advise to make more smaller hot add proxies:
- Co-scheduling, all those 8 core's need to be scheduled together (https://www.vmware.com/files/pdf/techpa ... d-Perf.pdf). Ok it might be less of an issue when you have 40 threads but still
- One hot add proxy can process multiple disks if it has enough SCSI slots on the SCSI controller. However there is a limitation, that one hot add proxy can only process one disk of a certain VM at the same time.
I know there is a limit of 15 on the SCSI controller and I could add an additional SCSI controller if necessary. My VCenter has 180 VM's and at the moment I'm not feeling like I have to go there just yet.
A question came to mind, The Max concurrent tasks per proxy. Isn't that limited to cpu's on the proxy? If I have 8 CPU, I can only process 8 disks. If I raise that to use more SCSI connections, then wouldn't I have to raise the cpu's?
-
- Product Manager
- Posts: 20415
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
Actually, each task requires a single core. How many cores does the given VM have? Thanks.A question came to mind, The Max concurrent tasks per proxy. Isn't that limited to cpu's on the proxy? If I have 8 CPU, I can only process 8 disks.
-
- Veeam Software
- Posts: 1818
- Liked: 655 times
- Joined: Mar 02, 2012 1:40 pm
- Full Name: Timothy Dewin
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
Yes but only 1 disk of a certain VM at the same time (Virtual Appliance API limitation). Honestly it is not really a problem since most VM's are single diskA question came to mind, The Max concurrent tasks per proxy. Isn't that limited to cpu's on the proxy? If I have 8 CPU, I can only process 8 disks.
-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
You're referring to the proxy right? The proxy has 8vCPU's assigned.v.Eremin wrote: Actually, each task requires a single core. How many cores does the given VM have? Thanks.
-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
60% of my VM's have multiple disks.Yes but only 1 disk of a certain VM at the same time (Virtual Appliance API limitation). Honestly it is not really a problem since most VM's are single disk
What I can do is easily enough is to see what result I get with the four proxies (set at 8 cpu) and then add four more proxies but reducing the vCPU's of them all to four and compare.
Since talking with you all it's pretty clear more proxies with less assigned vCPU's each are going to get the job done faster.
-
- Enthusiast
- Posts: 63
- Liked: 8 times
- Joined: Jan 16, 2014 11:12 pm
- Full Name: Jon Dufour
- Contact:
Re: HotAdd and 10Gbe to repository speeds?
Hi guys, just a follow up...
I tested doubling the amount of proxies but scaling down the vCPU's assigned to each.
So, 8 proxies with 4vCPU each.
I'm getting 352MB/s processing rate now
I tested doubling the amount of proxies but scaling down the vCPU's assigned to each.
So, 8 proxies with 4vCPU each.
I'm getting 352MB/s processing rate now
Who is online
Users browsing this forum: Google [Bot] and 42 guests