Applying Retention Policy Slow

Availability for the Always-On Enterprise

Applying Retention Policy Slow

Veeam Logoby adapterer » Mon Jul 10, 2017 4:28 am

Hi,

I'm having a problem with a replication job not meeting RPO due to long snapshot deletion/merge times aka "Applying retention policy" segment of the job. Maybe this is more of a question for VMWare but perhaps someone has been down this path.

I have a client VM of about 3TB which has 34 VM disks attached. The snapshot removal time ranges from 40 mins to 8 hrs and the job is supposed to run every 2 hours so we are not meeting RPO. The disk subsystem I have never seen above about 30% busy, and is capable of much higher IOPS/throughput so it makes me suspect the bottleneck is elsewhere. The storage is iSCSI with 10GBe networking, again I can't see a bottleneck on the network. So, I have some questions:

1. Will VMWare try to 'merge' all 34 VM disks at once?
2. If so, is there a way to limit the number of disks merged at once?
3. Is it possible this is causing odd performance limitations with iSCSI storage and 'iSCSI disk reservations', i.e. the storage being locked for write whilst it's waiting for another to complete (for reference this VM is on it's own datastore by itself)

Finding this bottleneck is driving me nuts.. thanks in advance!
adapterer
Service Provider
 
Posts: 184
Liked: 28 times
Joined: Mon Oct 12, 2015 11:24 pm

Re: Applying Retention Policy Slow

Veeam Logoby adapterer » Tue Jul 11, 2017 11:57 pm

Has anyone dealt with VM's with lots of VM disks attached?
adapterer
Service Provider
 
Posts: 184
Liked: 28 times
Joined: Mon Oct 12, 2015 11:24 pm

Re: Applying Retention Policy Slow

Veeam Logoby dellock6 » Wed Jul 12, 2017 11:35 am

Since vSphere 6.0 disks are managed in parallel:
https://www.virtualtothecore.com/en/vsp ... hing-past/

I don't know how many disks are consolidated at once, but in the tests that Tom did, 15 disks were all processed at the same time indeed. No idea if this option can be limited, as you said is something that you may ask to VMware, unless someone here knows.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5136
Liked: 1377 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Applying Retention Policy Slow

Veeam Logoby adapterer » Thu Jul 13, 2017 12:47 am

Thanks Luca, that really helps.

I have engaged VMWare support - I'm scratching my head on this one. I can see the merge happening and review ESXTOP stats - only getting 10MB/s read/write from the host. Yet, if I run the VMWare IO analyzer appliance simultaneously (same host, same datastore, no other VM's) I can get 300MB/s + for 512k random read/writes). Doesnt appear that host,network or storage are running out of headroom :(
adapterer
Service Provider
 
Posts: 184
Liked: 28 times
Joined: Mon Oct 12, 2015 11:24 pm


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: DaveWatkins, Google Feedfetcher, sg1 and 5 guests