Backup Job Duration increasing with multible Snapshots

[Eric_Cartman]

Hello!

I've been noticing a stange behaviour within my Veeam B&R V11 Backup Jobs.
We have multible VM Backup Jobs enabled where the duration of incremental as full backups are very similar.
But not if one or more VMs are having Snapshots attached.
Then the duration "explodes" and is taking a very very VERY long time to finish - especially the time between preparing the VSS and starting the process of snapshoting.
There's no other job running, no extra network traffic, plenty of ressources available...

09.08.2021 08:58:21 :: Queued for processing at 09.08.2021 08:58:21
09.08.2021 08:58:21 :: Required backup infrastructure resources have been assigned
09.08.2021 08:58:26 :: All VMs have been prepared for storage snapshot
09.08.2021 10:04:25 :: Creating storage snapshots for backup

or

08.08.2021 20:13:27 :: Queued for processing at 08.08.2021 20:13:27
08.08.2021 20:13:27 :: Required backup infrastructure resources have been assigned
08.08.2021 20:13:32 :: All VMs have been prepared for storage snapshot
09.08.2021 00:33:58 :: Creating storage snapshots for backup

I also contacted the VEEAM Support with this issue a time ago.
They've checked the logs and instructed me to delete all snapshots possible BEFORE the backup job starts but for us as a company this is not a practical solution.

So I'm asking if anyone else is noticing this problem?
And did you solve it?

Thanks!
Florian

[Vitaliy S.]

Florian,

Can you please tell us why do you keep snapshots on productions VMs? And how many of those do you currently have (on average for a single VM)? Just trying to understand the scenario to find a suitable workaround for you.

Thanks!

[PetrM]

And mind me asking how our engineers have come to the conclusion that the issue is related to the presence of snapshots on production VMs? As far as I see the problem is that there is 2 hours gap between preparation of VMs for storage snapshot creation and creation itself. You may just share a support case ID for our reference.

Thanks!

[Eric_Cartman]

Florian,

Can you please tell us why do you keep snapshots on productions VMs? And how many of those do you currently have (on average for a single VM)? Just trying to understand the scenario to find a suitable workaround for you.

Thanks!

We create snapshots before updating or installing M$ KBs on our SCCM or other critical VMs in out productive system.
For your understanding we're a hospital so almost all systems are critical...
The snapshots are deleted within 72h but for the time beeing it radically slows the backup process down which gives me as person in charge for a flawless backup a headache

[Eric_Cartman]

And mind me asking how our engineers have come to the conclusion that the issue is related to the presence of snapshots on production VMs? As far as I see the problem is that there is 2 hours gap between preparation of VMs for storage snapshot creation and creation itself. You may just share a support case ID for our reference.

Thanks!

Case #04774832 - it's in German - sorry
But thank you for looking into it

[karsten123]

better do quick backup and quick rollback if necessary. snapshots are for doing backups

[Vitaliy S.]

Florian, for the sake of the experiment, do you have the same job logs snippet when you didn't have snapshots on VMs (need to understand the difference in time for the VM preparation process)? Also, do you see the same behavior for all VMs or just for some particular ones?

[soncscy]

I can possibly explain this if it's like cases I've had in the past, and it's a VMware issue, not a Veeam issue:

https://kb.vmware.com/s/article/1007331

The important part is here:

>Normally, the lease block information contains the mapping for the region requested. If the disk is severely fragmented, only a portion of the actual mapping may be returned and additional mapping requests are needed.
If a virtual machine has a previous snapshot, the disk might also be considerably fragmented. The mapping information might not span the entire disk and additional mapping requests are again required.

Basically, if I get it right, you're doing storage snapshot backups, right? When fetching the data like this from VMware, it's not guaranteed that a single call gets all regions of the VM disk(s) because of fragmentation, and with snapshots this gets even worse. (Edit: to clarify, I mean it can get really bad. I think for one client, once we found the root cause and how it shows in the veeam logs, grepping the disk region mapping string returned a million + results for just one job)

For my clients, we just are judicious about pre-existing snapshots and are adamant that clients rely on ad-hoc backups for such upgrade testing or to not test on production environments and instead spin up an isolated machine. Instant Recovery and the sandboxing Veeam has makes this __very__ convenient and you get the exact same results of testing without impacting production/backups schedule.

[Eric_Cartman]

Florian, for the sake of the experiment, do you have the same job logs snippet when you didn't have snapshots on VMs (need to understand the difference in time for the VM preparation process)? Also, do you see the same behavior for all VMs or just for some particular ones?

Yes we have a gap of a few minutes between the preparation and the snapshot creation when there's absolutely no VM snapshot on any machine but to be honest i can't make out one single machine...
It's more like a "All or no one" behavior...

[Eric_Cartman]

I can possibly explain this if it's like cases I've had in the past, and it's a VMware issue, not a Veeam issue:

https://kb.vmware.com/s/article/1007331

Thanks for the link but if that would be the case there's no solution mentioned how to fix it.
And to eliminate all VM snapshots and just use VEEAM instead is also not a solution because we have multible - non IT - users using the Vsphere as well and we don't want to compromize our backup security

[PetrM]

By the way, I've reviewed the support case, according to our engineer an excessive number of MapDiskRegion requests triggers the performance issue so looks like it matches the scenario from KB provided by Harvey. The only possible workaround which comes to my mind so far is to schedule jobs to run when there are no snapshots on production VMs.

Thanks!

[soncscy]

Why not just introduce the benefit of lab requests into your company's workflow?

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Seems this really is exactly what you're looking for, but trust me when I say I understand it's far easier said than done, and no department likes "losing" capabilities they previously had.

But long term, I think that "we remove the risk of additional snapshot issues on production/testing on production while also maintaining our backup schedule" is far more persuasive than "we don't wanna" when it comes to the powers that be...

But good luck; if you really do have this, there's nothing to do from a backup side except not use VMware's direct san.

[Eric_Cartman]

Thank you all for contributing - I will take it to my superiors and let them decide how we go on from here...