-
- Veeam ProPartner
- Posts: 299
- Liked: 43 times
- Joined: Dec 03, 2015 3:41 pm
- Location: UK
- Contact:
Ten minute backup delay, with VM snapshots
I've been testing out IBM Storage Integration in 9.5 u3, and so far everything is working well first time.
One issue I've noticed however, is that any VM with an existing VMware snapshot experiences an 8-13 minute delay during the initial backup stage.
This only happens with Storage Integration enabled, and only on VMs with snapshots. The delay happens at the "Collecting disk files location data" stage - and holds up all other VMs from progressing in the same storage snapshot.
Before I submit a support ticket - is this normal, or does anyone have an solution for it?
One issue I've noticed however, is that any VM with an existing VMware snapshot experiences an 8-13 minute delay during the initial backup stage.
This only happens with Storage Integration enabled, and only on VMs with snapshots. The delay happens at the "Collecting disk files location data" stage - and holds up all other VMs from progressing in the same storage snapshot.
Before I submit a support ticket - is this normal, or does anyone have an solution for it?
-
- Chief Product Officer
- Posts: 31460
- Liked: 6648 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Ten minute backup delay, with VM snapshots
I don't think it's normal and would suspect either a vCenter API performance issue, or some unoptimal code on our side not handling this scenario very well - we'll need to see the debug logs to understand what is taking the time. Thanks!
-
- Veeam ProPartner
- Posts: 299
- Liked: 43 times
- Joined: Dec 03, 2015 3:41 pm
- Location: UK
- Contact:
Re: Ten minute backup delay, with VM snapshots
Thanks for the reply. It's still a new system - so the issue could be our side.
Will open a call and send the debug logs now.
Will open a call and send the debug logs now.
-
- Influencer
- Posts: 22
- Liked: 3 times
- Joined: Aug 06, 2019 2:02 am
- Full Name: Nathan Shaw
- Contact:
Re: Ten minute backup delay, with VM snapshots
Hi did you get to the bottom of this? Have seen same thing - only using storage integration and when SAN snapshots are present. Up to 45 min delay for a single VM backup job. Test environment consists of one VM, one VMDK, one datastore, one LUN. Can add a 2nd VM to same datastore and back it up in 2 minutes.
-
- VP, Product Management
- Posts: 27055
- Liked: 2710 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Ten minute backup delay, with VM snapshots
Nathan, I believe this is a bit different issue since you're seeing the delay when SAN snapshots are present while OP was referencing the vSphere snapshot. In this situation, it's better to contact our support team to validate the setup and review the debug log files.
-
- Veeam ProPartner
- Posts: 299
- Liked: 43 times
- Joined: Dec 03, 2015 3:41 pm
- Location: UK
- Contact:
Re: Ten minute backup delay, with VM snapshots
I had to check the support calls for this, as we experienced two similar issues.
One caused a 20-45 minute per disk delay on Backup Copy Jobs (fixed with a reg key), and one caused a 10 minute delay with VM snapshots (this one).
Your issue sounds different to the one we experienced, but for others reading this thread with this issue - I believe it was diagnosed as fragmentation on the VM snapshot file.
A few fixes were tried, but ultimately the best resolution was a policy change of not allowing/keeping snapshots >24 hours.
Veeam Quick Backups had already replaced the vast majority of our manual VM snapshot usage, and the rest was just educating support staff about best practice usage.
One caused a 20-45 minute per disk delay on Backup Copy Jobs (fixed with a reg key), and one caused a 10 minute delay with VM snapshots (this one).
Your issue sounds different to the one we experienced, but for others reading this thread with this issue - I believe it was diagnosed as fragmentation on the VM snapshot file.
A few fixes were tried, but ultimately the best resolution was a policy change of not allowing/keeping snapshots >24 hours.
Veeam Quick Backups had already replaced the vast majority of our manual VM snapshot usage, and the rest was just educating support staff about best practice usage.
-
- Influencer
- Posts: 22
- Liked: 3 times
- Joined: Aug 06, 2019 2:02 am
- Full Name: Nathan Shaw
- Contact:
Re: Ten minute backup delay, with VM snapshots
Yes, co-incidence for us, was indeed happening only when VMware snapshot present.Vitaliy S. wrote: ↑Aug 16, 2020 9:20 am Nathan, I believe this is a bit different issue since you're seeing the delay when SAN snapshots are present while OP was referencing the vSphere snapshot. In this situation, it's better to contact our support team to validate the setup and review the debug log files.
-
- Influencer
- Posts: 22
- Liked: 3 times
- Joined: Aug 06, 2019 2:02 am
- Full Name: Nathan Shaw
- Contact:
Re: Ten minute backup delay, with VM snapshots
I’ve also logged various support cases over this issue, as it can cause delays so long that your backups don’t even start within a daily window. Would love Veeam to workaround this by switching to non-SAN backup for a given VM if VMware snapshots are present for that VM. Unfortunately, support teams feedback is always ‘not our fault, it’s VMware API, too bad’ then we close the case
-
- Veeam Software
- Posts: 934
- Liked: 287 times
- Joined: Jan 31, 2011 11:17 am
- Full Name: Max
- Contact:
Re: Ten minute backup delay, with VM snapshots
Snapshots should only be active temporarily. If you experience this problem, your snapshots probably grew too big or were active too long? Why can't you just clean them up?
-
- Influencer
- Posts: 22
- Liked: 3 times
- Joined: Aug 06, 2019 2:02 am
- Full Name: Nathan Shaw
- Contact:
Re: Ten minute backup delay, with VM snapshots
They are active temporarily. VMware say don’t keep them longer than 72 hours. However, Veeam backups can be crippled by the existence of a single snapshot that is only hours old. Veeam docs say don’t use SAN backups if it breaks. Not a very premium feature as it stands.
-
- Chief Product Officer
- Posts: 31460
- Liked: 6648 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Ten minute backup delay, with VM snapshots
That is a really weird recommendation. Do you happen to have a link to this statement? Just feels like a totally random and unjustified number, so might be a good topic to discuss with VMware. The very reason we added backup from storage snapshots is because there were so many customers with I/O intensive VMs for which having a snapshot present for longer than a few minutes caused huge issues with the commit phase... but 72 hours? Where this number is even coming from, when one VM will write at 1KB/s and another at 1GB/s, resulting in dramatically different snapshot sizes.
This also begs for a question though, if you ARE fine keeping snapshots for days, then what is the reason to do backup from storage snapshots to start with? When clearly you don't have the issue this functionality is designed to solve in the first place.
-
- Chief Product Officer
- Posts: 31460
- Liked: 6648 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Ten minute backup delay, with VM snapshots
@Andreas Neufert btw should we add an option to automatically skip VMs with snapshots from BfSS processing? What do you think?
-
- VP, Product Management
- Posts: 6707
- Liked: 1401 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Ten minute backup delay, with VM snapshots
Let me research first the root cause a bit myself.
-
- VP, Product Management
- Posts: 6707
- Liked: 1401 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Ten minute backup delay, with VM snapshots
Likely root cause identified. Nathan and Max, is there a way you could open a support case to share logs? (Please add a note which Job and VM processing at what time was delayed). Please share here the ID.
This delay issue did not pop up for a longer time and maybe we have optimized things in later versions. Are you on any newer release (v11 or v11a)?
This delay issue did not pop up for a longer time and maybe we have optimized things in later versions. Are you on any newer release (v11 or v11a)?
-
- Veeam Software
- Posts: 934
- Liked: 287 times
- Joined: Jan 31, 2011 11:17 am
- Full Name: Max
- Contact:
Re: Ten minute backup delay, with VM snapshots
@Andreas: We've had an open case, but didn't work on it as the customer discovered the snapshots and resolved the problem by cleaning them up. VBR build was my first assumption, but the problem persisted on the most current build (P20211211). If you want to check the logs, here's the case ID: #01975070
I have also never seen this that extreme before. It could be a combination of Nimble Storage (Flash+Dedup?), Storage Snapshots and existing (old) VMs snapshots.
I have also never seen this that extreme before. It could be a combination of Nimble Storage (Flash+Dedup?), Storage Snapshots and existing (old) VMs snapshots.
-
- Influencer
- Posts: 22
- Liked: 3 times
- Joined: Aug 06, 2019 2:02 am
- Full Name: Nathan Shaw
- Contact:
Re: Ten minute backup delay, with VM snapshots
Yeah I agree 72 hours is quite arbitrary & not useful for high-change VMs, but source here anyway: https://kb.vmware.com/s/article/1025279Gostev wrote: ↑Jan 21, 2022 8:31 pm That is a really weird recommendation. Do you happen to have a link to this statement? Just feels like a totally random and unjustified number, so might be a good topic to discuss with VMware. The very reason we added backup from storage snapshots is because there were so many customers with I/O intensive VMs for which having a snapshot present for longer than a few minutes caused huge issues with the commit phase... but 72 hours? Where this number is even coming from, when one VM will write at 1KB/s and another at 1GB/s, resulting in dramatically different snapshot sizes.
This also begs for a question though, if you ARE fine keeping snapshots for days, then what is the reason to do backup from storage snapshots to start with? When clearly you don't have the issue this functionality is designed to solve in the first place.
For us, no, we don’t want multi-day snapshots at all, and any snapshots are usually a surprise, we always aim to have no snapshots at all, but the reality is that sometimes they do get created by admins, and not purged before the backup run.
My feeling is this is worse with thin-provisioned disks, but I don’t have any hard stats to back that up.
-
- Influencer
- Posts: 22
- Liked: 3 times
- Joined: Aug 06, 2019 2:02 am
- Full Name: Nathan Shaw
- Contact:
Re: Ten minute backup delay, with VM snapshots
Also case 05185300.
-
- VP, Product Management
- Posts: 6707
- Liked: 1401 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Ten minute backup delay, with VM snapshots
Just to update here. We are working with VMware on the question on how the used API can be improved that we use for this. As well we have some additional ideas on our side.
I will update you all here, but as it is working as designed for now, there is no short term fix.
I suggest to use Veeam ONE to monitor snapshot usage and lifetime.
As well I suggest to implement the Veeam Enterprise Manager and implement the Veeam vcenter plug-in.
Instead of letting anyone create snapshots, let them create Quick Backups from the vcenter instead. It is as well very fast (incremental backup to the last restore point) and is a real backup of that restore point that do not affect the performance of the datastore/VM/backup or bring the environment at risk because of growing snapshots consuming all space of the datastore. This feature was introduced exactly to address this some years ago when a customer automated exactly this and we found it was a brilliant idea and implemented it to the product.
More info:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110
I will update you all here, but as it is working as designed for now, there is no short term fix.
I suggest to use Veeam ONE to monitor snapshot usage and lifetime.
As well I suggest to implement the Veeam Enterprise Manager and implement the Veeam vcenter plug-in.
Instead of letting anyone create snapshots, let them create Quick Backups from the vcenter instead. It is as well very fast (incremental backup to the last restore point) and is a real backup of that restore point that do not affect the performance of the datastore/VM/backup or bring the environment at risk because of growing snapshots consuming all space of the datastore. This feature was introduced exactly to address this some years ago when a customer automated exactly this and we found it was a brilliant idea and implemented it to the product.
More info:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110
Who is online
Users browsing this forum: Bing [Bot] and 91 guests