Hot Add best practices

[preetam_de]

Hallo All,

this is the second post from me in a day/hour.
Hope this is fine.

We have deployed veeam B&R.
We are finding very low throughput when we use Transport mode Automatic (which is most of the case hot-add).
Every time we replicate I see target as bottleneck.
Then we moved from Automatic to network based transport mode
Throughput increased big time i.e. 120 MBps to 310 Mbps

what I observed is, Veeam is hot adding disk under scsi0.1 and etc.
Windows 2019 VM is actually configured by default to use SAS Controller?

Do you think it will make difference if i change SCSI controller to paravirtual?
Are we missing anything here?

Veeam Backup Server
vCPU: 4
RAM: 16 GB
Windows 2019

We have looked at Storage side and we looked at vmware side but we cannot find any hints as to why hot add is so slow.
Our storage All flash storage

[HannesK]

Hello,
the question is, how the bottleneck analysis really looks like and what kind of environment you have. For basic understanding I recommend checking out the FAQ

I don't know how many VMs you have. If the backup server is the only Veeam machine you have, then I guess it should be larger. 2 CPU & 8 GB RAM are for the OS. There is also as SQL server installed with Veeam. That leaves almost nothing for the Veeam software in your environment. I would at least double the resources. Please check the release notes for minimum configuration requirements.

As you mention only one backup server (no additional proxies)... did you adjust the concurrent tasks for the proxy? Per default they are only two at the backup server. https://helpcenter.veeam.com/docs/backu ... ml?ver=100

How many VMs / disk do you process in parallel? Classic customer situation I remember is: "we tested with one VM".

I'm assume, that the SCSI controller is the smallest tuning opportunity.

Best regards,
Hannes

[preetam_de]

Hi,
Thanks a lot for directions.
I know bit difficult to find the bottleneck here.
We spend good time yesterday on it with our Storage expert.
we tried to simulate hot add e.g. adding the snapshot disk to the veeam B&R and copied a 3 GB ISO file. It went in seconds.

We already increased the concurrent tasks to 4.
we have
1. vcenter 6.7u3
2. esxi 6.5u3
3. Veeam B&R 10 has 4 vCPU and 16 GB RAM configured.
It is the only server doing all the roles.
we will deploy additional proxy today to check this. But since we are backing up only 5 machine for testing, and each VM has 3 disks.
But the throughout is not above 120Mbps as compared network which is 200 plus
SQL server is inbuilt i.e. express
We have all flash array.

[HannesK]

looks undersized, yes. The tasks also exists for the repository. And there you can also configure per-vm chains for more parallel performance on writing to the target.https://helpcenter.veeam.com/docs/backu ... ml?ver=100

additionally to what I already said... if you are talking about incremental backups, then the hot-add process can take longer than the backup itself... it depends on the change rate.

We have all flash array.

as backup target? how is it connected to the backup server? I assume, that you are not backing up to the production storage.

if the target bottleneck value is not permanently between 95-99%, then it is fine normally.

[preetam_de]

I did not replied further because it was clear more things were involved here.
A update - we know the cause now. It is write cache not working inspite of being checked on array.
We have Dell SC 5020 Active-Active Storage.
We are working on it.
Happy to see Veeam's technical people jumping in and helping me further. It means a lot.