Really slow merging to cloud storage

[33minutesinheaven]

Hey, i am seeing this problem with one of my jobs and want to know if there is some way to correct it. Basically i want to restart the machine but cant because this stupid job runs for days stuck at 98%...

why? is it a bug or what?
And why is the backup bottleneck "throttling"? there is no schedule on the job, so why is it throttling?

man i cant seem to be able to attach any images... how do you do that in a forums post? here is the text.

Job started at 2020-03-26 2:00:27 AM
New copy interval started
Building list of machines to process 00:02
Waiting for new restore points 01:49:56
VM size: 8.0 TB
Changed block tracking is enabled
Processing FS1 06:22:27
Merging oldest incremental backup into full backup file (98% done) [fast clone] 27:27:41
Required backup infrastructure resources have been assigned 19:29
Restore point 2020-03-12 2:00:00 AM created successfully 18:18
Creating restore point 2020-03-19 2:00:00 AM 27:07:41

even in the details tab it says that its finished this morning at 10:!5 am but the parent is stuck on merging. i was hoping to be able to run the upgrade to version 10 today. sigh.

Restore point is locked by job 01:49:57
Copying restore point 2020-03-26 12:41:35 AM from backup repository TeraStation Backup 62TB
Queued for processing at 2020-03-26 3:53:12 AM
Required backup infrastructure resources have been assigned 39:55
VM processing started at 2020-03-26 4:33:23 AM
VM size: 8.0 TB
Storage initialized 00:28
Network traffic will be encrypted
Saving FS1-NEW.vmx 00:01
Saving FS1-NEW.vmxf 00:00
Saving FS1-NEW.nvram 00:00

Saving GuestIndexData.zip 00:12
Saving GuestMembers.xml 00:00
Hard disk 1 (80.0 GB) 24.8 GB read at 24 MB/s 22:30
Hard disk 2 (1.4 TB) 54.2 GB read at 13 MB/s 01:14:52
Hard disk 3 (3.5 TB) 26.2 GB read at 14 MB/s 35:35
Hard disk 4 (2.9 TB) 58.9 GB read at 6 MB/s 02:42:29
Network traffic throttling is enabled
Hard disk 5 (1.7 TB) 1.2 GB read at 64 MB/s 02:22
Hard disk 6 (1.6 TB) 17.8 GB read at 8 MB/s 37:28
Finalizing 00:07
Busy: Source 29% > Proxy 48% > Network 85% > Target 0%
Primary bottleneck: Throttling
Network traffic verification detected no corrupted blocks
Processing finished at 2020-03-26 10:15:41 AM

[veremin]

Just to be sure you're using cloud repositories enabled by your Service Provider?

If so, the throttling means there is a processing limit that SP has set for your account or you've configured global throttling rules yourself.

Thanks!

[poulpreben]

This sounds like the throttling applied by Veeam at your service provider’s Cloud Connect instance.

If it is indeed a service provider, ask them to log a case and increase the relevant registry keys for quota allocation during fast clone. We increased these limits by 50x the default to ensure our tenants’ fast clone operations complete within the SLA.

[33minutesinheaven]

can you confirm that this is the cause the specific slow merge scenario i had reported? it did eventually finish merging last week.

And yes i do have throttling rules set to 40mbps during the day. So that does explain it if the slow merge is caused by this. please do confirm.

This is only throttled to half our available bandwidth though. So worst case if it was no throttling it would still take like 20 hours to merge...

does the upgrade to version 10 address this at all?

[poulpreben]

The symptoms you describe seem like you are hitting another limit that is outside of the bandwidth throttling. The merge process doesn’t transfer anything across the wire.

The registry setting that I describe adjusts how Cloud Connect ensures you do not exceed your allowed storage quota at the provider. When you upload a chunk of data, Cloud Connect periodically updates the corresponding database records on the amount of consumed storage. If the updates amount exceeds your allowed quota, the next chunk of data isn’t admitted.

The symptom that you are experiencing is happening because the quota update is a synchronous operation, meaning that the data transfer (or merge) waits for the database to update before it continues. The default quota update is ridiculously small — like 10 MB (maybe 100, don’t remember). Since ReFS merges or synthetic full generation performs at over 3 GB/s, this whole process introduces too much latency. We simply increased the registry setting so that it allocates 16 GB at a time.

The setting has to be implemented by your provider.

(Disclaimer: This is purely based on my own reverse engineering of how this works, so it may not be entirely accurate)

[33minutesinheaven]

oh the provider is whipcord. i doubt they are running windows with a registry...

what registry setting are you referring to? i dont see it above.

if you mean this, i dont have a cloud connect item on the left hand side...
https://helpcenter.veeam.com/docs/backu ... ml?ver=100