slow backup copies from Azure to AWS: Case # 06356038

[rtheseeker]

Backup performace is fine; the whole infrastructure is in Azure (VBO server and proxies)
We upgraded the disk on proxies on 3rd oct and since then having been expriencing really slow backup copy performance; the backup copy used to take 3 hours before however now it is taking days; we have not had a sucessful copy since. The size of the data is same. As per VEEAM One report the resource utilization on proxies and VBO during the backup copy is minimal; the resource spike only occurs during backup job and is expected.

Raised a support case 06356038
Been advised to update the .net version and patch VBO to the most recent version
Proxy Threads as suggested earlier to 128

Everything was fine till 3rd OCT; we have already rebooted the whole intrastructure once.
Disabled all the backup copy job and tried running only one backup copy per proxy. Each proxy except for one processes 2 backup copy jobs
Also, one of the proxies only processes one backup copy job;

Also I have been advised
"restarting proxy in between backup or copy jobs would definitely affect and cause slowness, as data integrity and verification has to be performed"

Does it mean we should let the copy finish irrespective of time it takes and subsqeunt backup copies should run normally

[Pagrossman]

If you upgraded to the latest version, so there is known issue that slow down copy jobs...
veeam-backup-for-microsoft-365-f47/copy ... 90486.html

[rtheseeker]

I flagged this to the support person however he had advised me to upgrade office365 to the latest version

[rtheseeker]

have updated the evnrioment however have been advised to check with AWS support as well; also being advised to increase thrads on proxy to 128
"
Also, could you please let me know if you've looked into the other suggestions that I mentioned earlier (increasing proxy threads, opening a case with AWS/S3 support)"

[Polina]

Hi All,

To give an update on this: the issue with copying across different platforms is being investigated by RnD. For the issue Jan is referring to, RnD is preparing a hotfix.

Thanks

[rtheseeker]

got given a hotfix today to install; going to do that and update

[rtheseeker]

unfotrunetly the fix did not work; also the backup copy behaviour is puzzeling me; after taking long time; the subsquent backup copies in far less time across proxies; however the issue occured again although nothing changed; although the copy did not take as long as before but it still took over a day on all the backup copy jobs; support has advised me to increase the cpu/memory on the proxies however during the backup copy job these are minimalilly used; there is no resource contention(Veeam one daily/weekly performance data). Looking at the job history on all the backup copy jobs I see the bottleneck as source; if the issue was with proxies would that not appear as the bottleneck. Also, all the data is kept on Azure Blob and copied to S3.
for example on one job it transferred 491GB of data in 14 hours with read and wirte speed of 20.x MB/s processing speed 9.5MB/s and the previous job took 450 hours to transfer 371GB of data with processing rate of 240KB/s both are showing bottleneck as source
Also, just to understand the backup copy process Say if the backup completed but backup copy did not finish for days however the subsequent backup copies finished; would we still have all the restore points on the secondary copy; say if primary copy was unavailable would we be able to restore from secondary copy

[rtheseeker]

another thing I noticed everytime we reboot the proxies the subsequent jobs go really slow. We had rebooted few servers due to windows patching and subsquent backup copies go really slow; however once the first backup copy after reboot is complete; the next backup copy seem to stay normal. What could be causing this