After chatting with support, it seems that a great deal of time is spent removing restore points.
Here is an example:

The above job took 36:43 to run in total but a great deal of time looks spent just doing retention work. Right now we're configured to keep 12 restore points for our replicas.
So I'm wondering, does the amount of replica restore points have a direct effect on the time it takes for these jobs to complete? For instance, I was to reduce this job down to say 1, just keeping the DR site as current as possible, would future runs of this job and the time it takes to apply the retention policy on the DR site go down by a large amount or is it mostly storage related so it would stay the same?
Case: 01002324