Concurrent tasks settings

[gberkshier-cefcu]

We're trying to optimize settings in our environment. Things have grown rapidly, and now things are a bit out of control and performance is erratic.

We have a Veeam server and several proxies at our main location. The backup repository is a Synology NAS that we are backing up to directly. Early on, we had this connected to the Veeam server directly via iSCSI, but there was a patch in the past that Synology pushed out that completely killed the performance on that connection, so I scrapped it all, re-configured to just send data directly to the NAS, and that has been the way we've used it ever since. I believe the iSCSI issue was resolved in a subsequent patch. I'd like to some day re-do all of this and go back to the iSCSI connection so I can take advantage of REFS but don't want to do that until I can get the server updated to a new version of the OS.

So my question is - we are told to set concurrent tasks on proxies and repositories, based on the CPU count. Does this come into play when using a Synology NAS with CIFS shares? If we follow the logic of setting tasks based on number of CPUs and RAM, this logic doesn't match, and Veeam doesn't install any services on it like it would on a Windows-based repository. I can see that the NAS has 128GB of RAM in the performance monitoring, but I have no idea how many CPUs are in it. Even if I did, they don't seem to come into play anyway. CPU and RAM never seem to be pegged much while backups are running. The network traffic, of course, does, and we have 10Gb adapters in them. The additional wrinkle in this is that the Synology storage has a limit of 200TB per volume, due to how the RAID was setup, so when I added storage to it, I had to create a new volume on the same NAS, and then add it as an additional repository in Veeam, and then create a SOR. This seems to me like it could be simplified if I just scrapped it all, re-did the storage to overcome the space limitation (I think I can do that?) and then connect to a separate server with iSCSI and an REFS-formatted disk as a new repository. But, since I have jobs running and stuff being stored, I'm pretty much out of luck doing that for now.

Any tips on this?

[foggy]

Hi Glenn, your understanding is correct, the concurrent tasks requirements/recommendations apply to the repository server where the Veeam data mover agent is running, not NAS itself.

[gberkshier-cefcu]

So what should I have my NAS repository set to for concurrent tasks? Taking into account that I currently have two RAID volumes that I'm hitting with all of the proxies. I believe each volume/repository got set to 15 concurrent tasks. Does anyone have a general rule of thumb I should follow here? Obviously I can experiment, but wasn't sure what I should try. In the Veeam jobs, more times than not, the target is the bottleneck, and I do have a 10Gb connection. But, we do have a lot of jobs running, too.

[foggy]

Since it is currently connected via CIFS, what server is assigned a gateway role, the backup server itself? You should respect the other roles assigned to the same server then and keep in mind the total number of tasks for it.

[gberkshier-cefcu]

I forgot about checking the gateway for the repositories. They've been manually set to the same server. I could set that to auto, but we have our Veeam network spread out across our main site and DR as one logical network, so I'm not sure if I want to set that to auto. I don't want the backup job to be automatically assigned to a Veeam proxy/gateway at our DR site for our local site inadvertently. Now I'm wondering if I need to create more gateways, along with the extra proxies I just setup.

[foggy]

With automatic mode, the gateway is one of the proxy servers used in the job, so it is worth trying - will take the load of the backup server if it is an issue.

[gberkshier-cefcu]

So my question still remains, though. My repositories aren't Windows-based. They strict CIFS shares on the network. Looking at the repository properties, they're both currently set to 15 simultaneous tasks. I'm not sure who set them that high, or when this was done. Maybe I did when these first got setup, I don't remember. What should they be set to? I can't go off the recommendation of 1 task per CPU core, since I don't have that info. Even if I did, the CPU's aren't really being tapped for transferring this data, correct? At this end, the issue seems to be network traffic. Even with the network connection being 10Gb, the target is the bottleneck in pretty much every job we have. Should the concurrent tasks be set lower? If so, to what? Do I just keep playing with the concurrent tasks until I find a sweet spot that works? I just don't know if this setting is what it should be. And because I'm dealing with the same NAS with different RAID volumes that are in the same SOR, that really means that everything is hitting it on the separate repositories, each with 15 concurrent tasks, so that really means this one repository is getting slammed with 30 concurrent tasks for backing up.

[foggy]

As mentioned, this setting and the recommendation of 1 task per CPU core apply to the server running the Veeam data move agent, which is the gateway server in the case of CIFS repository. You should set it to the value that being summed up with the similar setting for the proxy server running on the same server (if the case) will not exceed the number of its CPU cores. If you're concerned with the incoming load on the NAS itself, then you should use the ingestion rate setting to limit it.

[gberkshier-cefcu]

So if I set it to 8 (the number of cores in my gateway), should I cut that down to 4 for each repository, since they are both using the same gateway and NAS, and going through the same gateway? Maybe I should setup a second gateway so each repository has its own?

[foggy]

Do you have the per-VM backup chains option enabled for the repositories? In this case and if auto mode is enabled, Veeam B&R should select a separate gateway per every VM in the job, so already use multiple gateways.

[gberkshier-cefcu]

Ok, I have per-vm backups set. What about my other question? I still don't know what I need to set concurrent tasks to for the repositories - the original question of this post. I have one NAS. The NAS has two RAID volumes, due to the 200TB limit on the volume size. I have both of these repositories set as extents in a SOR. I'm still not clear what I should set this to.

[foggy]

If the number of cores on the server used both as a proxy and a repository/gateway is 8, I'd set them to 4 for both roles, to avoid resource contention. Also, keep an eye on the storage itself, it should be able to keep up with this number of tasks in terms of data ingestion.