Does restore point count slow down replication jobs?

[jbarrow.viracoribt]

We've been asked to investigate why our replication jobs take so long to process.

After chatting with support, it seems that a great deal of time is spent removing restore points.

Here is an example:


The above job took 36:43 to run in total but a great deal of time looks spent just doing retention work. Right now we're configured to keep 12 restore points for our replicas.

So I'm wondering, does the amount of replica restore points have a direct effect on the time it takes for these jobs to complete? For instance, I was to reduce this job down to say 1, just keeping the DR site as current as possible, would future runs of this job and the time it takes to apply the retention policy on the DR site go down by a large amount or is it mostly storage related so it would stay the same?

Case: 01002324

[jbarrow.viracoribt]

Also, what determines how many of the "applying retention policy" tasks can be going at once? They don't seem to go through one-by-one and they don't seem to have a direct match up to proxy slots.

[jbarrow.viracoribt]

But if I'm keeping 12, and the one it's removing is the oldest one in the chain, you're telling me that doesn't take any longer for VMware to handle/process vs if there was only one in the chain?

For instance, if the below VM was just replicated, and it went to remove a restore point, it would be removing the oldest one (8-31-2015 1:16:18PM), at the top of the list keeping all of the other ones. Having all these additional snapshots existing does not effect the speed of that removal at all?

[PTide]

I had double-checked something and had to delete my previous post not to confuse other people who might find this thread in future.

So, my first thought was that deep snapshot tree should not cause significant delays as there is always the only snapshot to be deleted and another one to be relinked. However, on practice it takes longer for VMware to prune snapshots that fall under retention if the tree is deep. Sorry for misguidance.

"what determines how many of the "applying retention policy" tasks can be going at once

If VMs are replicated to a standalone ESXi host then only two of them can be processed simultaneously. If VMs are replicated to cluster then only four can be processed simultaneously, no matter how many hosts are in the cluster.

Thank you.

[Vitaliy S.]

On a side note, can you take a look at datastore performance when snapshot is being committed (restore point gets removed)? Do you see any high latency, IOPs rate?

P.S. This can be checked with a free version of Veeam ONE.

[jbarrow.viracoribt]

Changing from the 12 replication restore points down to 2 seems to have made a pretty big difference in our job completion times. We shaved about 15 minutes (or so) on average off of this one jobs run time by making this change.



Before:


After:

[PTide]

Thanks for sharing Jonathan, very good illustration!

[Vitaliy S.]

Veeam ONE backup report spotted!