Maximum number of simultaneously jobs

[liviu.tutuianu]

Hello to all,

I have one question that I think it will get a quick answer. . What is the Maximum number of replication jobs that can be run on a single VBR?

Environment:
Locations: 21 (20 Stores and 1 HO)
ESXs: in every store we have 2 ESXs
Proxies: in every store, we have a dedicated proxy
Repositories: in every store we have a dedicated repository, on the second ESX
VBRs: only one VBR in HO (headoffice) (4 CPUs, 8 GB RAM)

For each store, we have a replication job configured on the HO VBR. This job replicates a VM from ESX 1 to ESX 2. The proxy server is on ESX 1. The Repository is on ESX 2. The replication is being made over the LAN.

So, we have 20 replication jobs (from each store to HO) that need to be run simultaneously. The problem is that we need to do this between 00:00-6:00. We observed that only 4 job run simultaneously and all the other just keep waiting until one of these jobs finishes...and after that starts. Is there a maximum number of jobs that can be run per CPU or how can we find if there is another bottleneck?

Thank you in advance.

[ZachW]

Please jump into the proxy settings of the proxy(s) that are being use for this replication job and increase the amount of concurrent tasks based on the amount of CPU cores on the VM. It is recommended to have 2 cores per job running.

[liviu.tutuianu]

Hello Zach,

We will make these changes on all the jobs and will come back with a feedback in the next days.

Thank you for the quick answer.

[chrisdearden]

Can you confirm which is the case ?

For each store, we have a replication job configured on the HO VBR. This job replicates a VM from ESX 1 to ESX 2. The proxy server is on ESX 1. The Repository is on ESX 2. The replication is being made over the LAN.

So, we have 20 replication jobs (from each store to HO) that need to be run simultaneously.

Are you replicating within Each branch ?
As Zach said - it seems like you are hitting the maximum number of jobs that a given proxy or repository can process concurrently.

However from what you have initially said, this should not be a problem as you only have 1 job per site. What you need to ensure is that each Site is only using resources form that site ( no HO repository or Proxy )

[liviu.tutuianu]

Hello Chris,

For each branch, we use a dedicated Proxy, and a dedicated Repository (both located on the ESXs from branch). The replication takes place in branch, between the two ESXs from there. The only common point is the HO VBR from where the replication jobs are being managed. So every proxy processes only one job. Now, all proxies have 2 concurrent tasks set.

We get the same error as yesterday: "....waiting for backup infrastructure resources availability...". And my conclusion is that the bottleneck should be at the VBR's level.

Any idea? Soon, we will have around 50 branches managed by this VBR. And the same situation will be also in other countries where we implement VEEAM.

[foggy]

Make sure that the local proxy servers are effectively used during processing. Right-click the job, select Statistics and select the VM to the left - in the right panel you will see the name of the proxy that is used for processing this particular VM.

[vMO]

As replication is going from proxy to proxy, on which of the two esx host's is your proxy?

How many other backup jobs are runing in HQ at the same time? There is a min. of RAM needed on VBR server for each job running simultaneously, but actually i don't know the exact nummber, but will come back with that. As first step you can monitor RAM on VBR or increase memory.

[liviu.tutuianu]

Hello vMO,

In the first post I tried to explain the VEEAM infrastructure. I know that for every job, we need around 400 MB RAM. On the HO VBR we have 8 GB RAM and we tried to run initially 20 job, then reducing the number to 8. In each branch we have only one proxy server (located on ESX1) because we have only one job that replicates the machines from ESX1 to ESX2.

Right now, I HO we cannot run more then 4 simultaneously jobs. But, as I told before, on the HO VBR we don't copy anything. All processes happen in each branch, using their local resources (2 ESXs, 1 repository, 1 proxy server). I think that finding the bottleneck for this error "...waiting for backup infrastructure resources availability..." will solve our problem.

And, for the main question: The number of CPU's from a VBR limits the maximum number of jobs that can be run simultaneously? anybody of you has the answer?

Thank you,
Liviu

[foggy]

Liviu, backup server CPU does not really matter, what is important, is the amount of CPU cores on the proxy server as all the processing load is handled by the proxy (unless the default proxy server, which is a part of the Veeam B&R installation, is used). And, as Zach has already stated above, it is recommended to have 2 cores per job running on the proxy.

Actually, it seems to be an issue with duplicate moref IDs that different VMs could have on different ESXi hosts. Veeam B&R erroneously treats such VMs as the same and if the VM with this ID is already processed by some job, it waits for its availability to start another task over the same VM. If you check the job statistics and select the VM to the left, you should see the corresponding message. The issue will be addressed in the next public hotfix.

Thanks for pointing that out!

[rbrambley]

Looks like I am replying a week late. Let us know if you made any changes to get more replica jobs running simultaneously.

Short answer to your main question - No, the number of CPUs on the VBR server (where Veeam is installed) does not limit the number of simultaneous jobs UNLESS you are actually using the VBR as a proxy. It is a proxy (and repository btw) by default. You do not want to use the VBR server as a proxy.

The amount of VBR RAM can limit SIMULTANEOUS jobs, however. You have 20 jobs. Each job will take up close to 512 mb of RAM when running on the VBR server (where Veeam is installed). You have 8GB of RAM in your VBR server so you can run a max of 16 simultaneous jobs - loosely calculated and rounding up. Think of your VBR in HO as the "brains". If you overload your brains it will bog down managing your jobs.

There is no limit to total number of VBR jobs - just simultaneous based on VM/Server RAM. For example, if you have to keep VBR at 8G RAM then running 10 jobs at 12 am and 10 at 3 am would be fine (5GB simultaneously).

BUT, I don't think you are reaching that point based on what you describe. Your message is that Veeam is waiting on resource availability. If I understand your set up right, you are doing local replication from ESX1 to ESX2 at 20 different branches. Each branch has it's own proxy on ESX1.

Resource availablity oversimplified means that the Veeam proxies have all the jobs they can handle - i.e. each proxy's max concurrent task settings (the Proxy rule of thumb is 1 task per 2 CPU). So, if a proxy has a setting of 4 concurrent tasks and you schedule 20 jobs to start at the same time then the first 4 will run but the other 16 will queue until a job finishes making room for another one to run. You could always give the proxy more cpus and change the concurrent task setting or you could add more proxies. Either way Veeam can use the proxy pool of resources as more "muscle" to do the job.

Since Veeam replication goes from a source proxy to a target proxy, I would expect that you would have 20 jobs kickoff and be running at each branch. Yes, it sounds like your VBR would struggle, but jobs should be running. You say only 4 run and every other jobs waits. This makes me think that you possibly have misconfigured each job to involve the VBR server (as a proxy perhaps? It is a proxy by default). Be sure that the jobs have the local branch proxy configured as both the source and target proxy.

Check the Job Statistics to see which proxies are actually being used for each job. if you are using automatic mode for proxies I would recommend you switch and manually specify local proxies for each respective branch.

Make sure that each branch proxy has as many vCPUs as you can afford. Set the local proxy concurrent tasks accordingly. Those should be where your replication path is established. Your certainly do not want to have a replication path that goes from branch to HO back to branch - multiplied by 20!

[liviu.tutuianu]

Hello,

Excellent answer. I will check all that you said in this post. I think that we set the VBR from the HQ as the destination proxy for every store and this could be the bottleneck.

Thank you,
Liviu

[chrisdearden]

Hello,

Excellent answer. I will check all that you said in this post. I think that we set the VBR from the HQ as the destination proxy for every store and this could be the bottleneck.

Thank you,
Liviu

Its worth checking - with replication , the target proxy isn't as heavily utilised as the source proxy , so should be able to handle more than 1 job per 2 vCPU/core - In that instance , you could override the max number of jobs that the default proxy handles ( if it is only ever a replication target proxy )

remeber if you are replicating within the remote site between two standalone hosts , you ought to have a proxy on each host.

[Borg]

Where are the proxy settings ?
I just want to increase from 2 to 3 tasks.

[foggy]

Right-click the proxy under Storage Infrastructure > Backup Proxies, click Properties and select the Server step.

[Borg]

I do not have anything under Storage Infrastructure because for the current servers we use the server's own internal storage and the backups are on a backup server with it's own storage, so there's nothing like HP3Par, Dell or other shared storage in the infrastructure at this point.

-Edit-
I found the setting under Backup Infrastructutre --> Backup Proxy.
Thank you.

[foggy]

Sorry, of course I meant Backup Infrastructure node.