Merging takes for ever

[MS84]

Hi,

My backup copy job is merging for a very long time, so maybe I'm not using the proper backup method?
04-02-2020 04:05:21 :: Merging oldest restore point into full backup file [GFS] (46% done) 78:20:42

My setup:
I have 2 locations with a 500Mbit MPLS connection between.
Location A:
1x Synology NAS with 36TB storage as repository
2x ESXI hosts
15VM Total size: 6,76TB

Veeam Backup & Replication 9.5 Update 4

Location B:
1x Synology NAS with 22TB storage as repository

Every night (00:01) my backup job (Incremental) starts on location A. First it's backing up to my NAS on location B. This takes a couple of hours. Perfect.
After my normal backup job finishes, my backup copy job starts to make a copy from my NAS on location B to my NAS on location A. This takes a couple of hours. Perfect.

Every saturday my backup job creates a "Active full backup". When my backup copy job starts after this backup, it takes for ages to finish because of merging [GFS].
Backup job settings:



Backup copy job settings:


My wish is to have a fresh backup on both locations every night. Will it help If I just marked the "Read the entire restore point from source backup instead of synthesizing it from increments" checkbox?

Any sugestions?

Best regards,

[Egor Yakovlev]

Welcome to the Forums!
All Synthetic operations are done on a Gateway Server specified in your NAS Repository properties window. By default it is Veeam Server. So chain transformations on SiteB NAS are roaming home to SiteA via 62MB\s link of yours and then coming back again to land on SiteB NAS.
Please introduce additional Gateway server on SiteB, set it under SiteB NAS Repository settings and observe the change!
Cheers!

[MS84]

Hi Egor,
Thank you very much

The "mergning" is beeing done on the NAS on Location A where the Veeam server also is located.

So the 500Mbit WAN link is not the bottleneck. I just don't understand the poor performance. The NAS is only transmitting around 50Mbit up and down when merging. As you can see in the above screenshot, it's performaing around 450Mbit when backing up.

[tsightler]

Actually, your WAN link is at least part of the bottleneck, as you seem to missing is the difference between throughput and latency on your link. Sending incremental backup from one site to the other is a single stream, so it's able to use all of your bandwidth, but performing a merge requires reading random blocks of data from the remote, processing them, and sending them back to the remote. Even at 500Mbps, the latency of the link between two sites is likely 10x ore more what a local link would be and will have a negative impact on performance. That's why Veeam has the ability to define a gateway on the remote side, not because of bandwidth, but to mitigate the impact that latency has on such a process.

Now, that might not be the only issue, this random I/O is also much more difficult on you NAS as well, because read/write from the same device obviously creates a much higher I/O burden than read alone, and we really don't know what your NAS is capable of from an IOP perspective since you're not doing merges locally either, but certainly the best next step is to at least test a gateway on the remote site if that is possible.

As an alternative, and to answer your original question, you can use the option you highlighted "read entire restore point from source" and in this case it will probably be singificantly faster than synthetically building a restore point across the WAN, because currently it is reading and writing all of the data for a full backup from SiteB back to SiteB via the WAN, while with the new option it will at least stream the full from Site A to Site B again which will use your full bandwidth. My guess is that will take maybe 20 hours or so but that's quite a bit faster than 80+ hours!

[MS84]

Hi tsightler,

Again: It's not merging on location B, but on location A. The same location as my Veeam server.

I do normal backup to location B first. Then my backup copy job, creates a copy on location A. The same as my Server.

Like this:


Do you still think a gateway on location B is the best option? I'ts no problem for me to set it up, I just don't understand the meaning of it.

20 hours is good enough for me

[tsightler]

Ok, I see, I somehow missed that in your first post. So you are running your backup directly to site B and then a backup copy job back to A. In that case it's almost certainly the second issue I mentioned, that the NAS simply isn't capable of delivering the required IOPS needed to perform a full merge, basically building 100% of the full backup via random I/O (synthetic merges need at least 10x the IOPS of active full). Those Synology boxes are typically just software RAID from Linux so are generally some of the worst possible performance for that particular workload.

So I'd say you could use the "read entire backup from full", that should basically take the same time as the active full backup, or just run two backup jobs, one local and one remote.

[MS84]

Hi tsightler

I'll give the "read entire backup from full" a try

And no need for considering "revers incremental" or any other methods?

[cdickerson]

Could be this advanced SMB setting on the Synology.
veeam-agent-for-windows-f33/solved-veea ... 63558.html