Host-based backup of VMware vSphere VMs.
Post Reply
raphael@schitz.net
Enthusiast
Posts: 71
Liked: 2 times
Joined: Jul 25, 2009 12:14 am
Contact:

Disable the waiting of snapshot removal task

Post by raphael@schitz.net »

Is there any way to disable the waiting for snapshot removal task in order to speed up veeam jobs ?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Hi Raphael, snashot removal task is triggered at the end of the backup/replication job. Could you please explain me how this feature would help you in your current deployment?
raphael@schitz.net
Enthusiast
Posts: 71
Liked: 2 times
Joined: Jul 25, 2009 12:14 am
Contact:

Re: Snapshot removal issues of a large VM

Post by raphael@schitz.net »

Vitaliy S. wrote:Hi Raphael, snashot removal task is triggered at the end of the backup/replication job. Could you please tell how that would help with the issue discussed in this thread?
Hi Vitaly,

Sorry, i did not specify in case you got a job with a lot of VMs.
In our (my customer) case, we set up job backing up resource pools where hundreds of vm could live so snapshot removal is consuming a lot of time on those jobs.
It really could be nice if we can have an option like "don't wait for snapshot revmoval task end (i know the risk on my storage array, etc...)" on jobs.
beside the use case, is it possible anyway ?

Thanks for your time.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Thanks for clarifications, but I see a huge negative effect which will level off the benefit of increased backup job speed.

What would happen to your storage when all your VMs will start committing their snapshots (might even happen at the same time)? This will impact all VMs running on this datastore and can even lower their performance. I do not believe that this is something you're after with this feature request.

That said, If you want to have smaller backup window, I would recommend to split your job into a couple smaller ones and use multiple proxy servers to process VMs concurrently.

Hope this makes sense.
raphael@schitz.net
Enthusiast
Posts: 71
Liked: 2 times
Joined: Jul 25, 2009 12:14 am
Contact:

Re: Disable the waiting of snapshot removal task

Post by raphael@schitz.net »

Thanks for your feedback Vitaly,

Actually in our case (referring to the vcenter task timestamps), the snapshot of the vm #1 would be commited while the vm #2 would be backed up and so on. It would easily be done before the next vm backup finish (again in our case but i know some customers with the same behavior).
Snapshot commit are not extremely long but even 1 or 2 minutes per VM could saves hours when you got 100+ to backup.

About the backup window, we really tried many things but (for us) the resource pool backups are the best way since we don't have to setup backup for new vms, if they're deployed in a production level pool, it will be in the next pool backup.

Here is an example of an SQL server located on a high end array and still the performances are quite good, the snapshot removal takes few minutes :

Image

Image

Image

It would be really great if you could consider this kind of case :D

Thanks
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Disable the waiting of snapshot removal task

Post by Vitaliy S. » 1 person likes this post

Ok, thanks for the feedback, much appreciated!
raphael@schitz.net
Enthusiast
Posts: 71
Liked: 2 times
Joined: Jul 25, 2009 12:14 am
Contact:

Re: Disable the waiting of snapshot removal task

Post by raphael@schitz.net »

Hi Vitaliy,
With those elements, could you tell me if this is something you'd consider too add in the future (even with a registry key) ?
Thanks
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Disable the waiting of snapshot removal task

Post by Vitaliy S. »

Hi Raphael, this depends on the demand and our current priorities for the next releases. Thanks!
jyarborough
Veeam ProPartner
Posts: 31
Liked: never
Joined: Apr 03, 2010 2:23 am
Full Name: John Yarborough
Contact:

Re: Disable the waiting of snapshot removal task

Post by jyarborough »

I could see a huge benefit in this. However, I think some intelligence around it would be necessary. For example, inside a single job with multiple VMs, when one finishes the actual backup and goes to commit the snapshot, it would let the next job in line fire off and get started. However, the intelligence would need to be in checking to see how many commits are occurring, or how far behind the job is and only allow like 1 (or x) concurrent commits before it allows the next job to start.
raphael@schitz.net
Enthusiast
Posts: 71
Liked: 2 times
Joined: Jul 25, 2009 12:14 am
Contact:

Re: Disable the waiting of snapshot removal task

Post by raphael@schitz.net »

jyarborough wrote:I could see a huge benefit in this. However, I think some intelligence around it would be necessary. For example, inside a single job with multiple VMs, when one finishes the actual backup and goes to commit the snapshot, it would let the next job in line fire off and get started. However, the intelligence would need to be in checking to see how many commits are occurring, or how far behind the job is and only allow like 1 (or x) concurrent commits before it allows the next job to start.
I agree but it could be also heuristicly based on the last commit time.
I also think it could be nice for "powerusers" like us to do the intelligence our self. In our example, i know the storage side can easily handle multiple snapshots commit simultaneously.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Disable the waiting of snapshot removal task

Post by Gostev »

OK, so I don't really see how this would help when you are running multiple concurrent jobs, which is what most of our customers do. Storage is never idle in that case anyway. Are we talking single job scenario here?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Disable the waiting of snapshot removal task

Post by Gostev »

raphael@schitz.net wrote:i know the storage side can easily handle multiple snapshots commit simultaneously.
No doubt about that, I see you have 3PAR - enough said... Ferrari of storage.
raphael@schitz.net
Enthusiast
Posts: 71
Liked: 2 times
Joined: Jul 25, 2009 12:14 am
Contact:

Re: Disable the waiting of snapshot removal task

Post by raphael@schitz.net »

Gostev wrote:OK, so I don't really see how this would help when you are running multiple concurrent jobs, which is what most of our customers do. Storage is never idle in that case anyway. Are we talking single job scenario here?
It's more about reducing the time taken by side ops like creating/commiting snapshots for long jobs. In our case, we backup resource pools everydays and with hundred of vms it's sometimes difficult to backup the hole thing in a night. We add some scheduled scripts to stop and retry jobs when we want to but it could be very nice to explore anything that could lead to a shorter backup window.

Thanks
raphael@schitz.net
Enthusiast
Posts: 71
Liked: 2 times
Joined: Jul 25, 2009 12:14 am
Contact:

Re: Disable the waiting of snapshot removal task

Post by raphael@schitz.net »

Gostev wrote: No doubt about that, I see you have 3PAR - enough said... Ferrari of storage.
sure 8)
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Disable the waiting of snapshot removal task

Post by yizhar »

HI.

An alternate (and I think also better) solution for this scenario, you may consider it as a feature request:

Add an option to the Veeam Job properties, specifiying how many VM can be backuped up simultaneously, which will default to 1.
An administrator which has many relatively small VMs to backup in a single job, will be able to specify 2 concurrent VMs (or a bit more), which will be more efficient, and will not require using a separate job.
This will be similar to other solutions such as VDR/VDP which backup several VMs concurrently.

Yizhar
emtunc
Novice
Posts: 6
Liked: never
Joined: Feb 12, 2013 12:06 pm
Full Name: Mikail Tunç
Contact:

[MERGED] Asynchronous removing and creating snapshots?

Post by emtunc »

In terms of performance of Veeam jobs, is there a way to perform the removal and creation of snapshots in a more 'optimal' way?

For example, sometimes the snapshot removal process takes quite a long while to complete... sometimes a few minutes, almost an hour or more.

During this time, Veeam and the target backup destination are pretty much sitting idle. My thought was that this process could maybe be optimised to deal with this idle time. Something like this:

1. Create snapshot, backup VM1
2. Remove VM1 snapshot and continue to create snapshot of next VM2 in parallel so that the job continues rather than waiting for current snapshot removal to complete.

OR

1. Create snapshot, backup VM1
2. After VM 1 has finished backing up, create snapshot of VM2 and start backing up VM2
3. Whilst VM2 is in the process of backing up, initiate background snapshot removal of VM1.

Perhaps there are some technical limitations that would mean this would not be possible but worth bringing up anyway.
rolandk
Influencer
Posts: 16
Liked: never
Joined: Jul 15, 2009 11:04 am
Full Name: RolandK
Contact:

Re: Disable the waiting of snapshot removal task

Post by rolandk »

it`s even worse - i had my veeam backup completely broken, as no job did ever complete because of snapshot removal tasks.

after digging into the issue for a while (as there where rpc errors i sat with the firewall admin to analyse stale connections, firewall timeouts etc), i found that it was the backup job just waiting for some background task to try removing non-existant snapshots -(of VMs not in the current backup job) and then eventually time-out.

the real problem i see is not the background removal task itself,

the problem i see is, that it`s nowhere shown to the end user, that such tasks are in progress
and
that a running task which should back up one VM for example just sits and wait because some hidden backup removal tasks are in progress and never complete
and
that such background operation does have no internal timeout but blocks the backup processing as a whole.

if someone from veeam want`s to have a look at this - just take a look at Case # 00445861

i think this should need improvement.
rolandk
Influencer
Posts: 16
Liked: never
Joined: Jul 15, 2009 11:04 am
Full Name: RolandK
Contact:

Re: Disable the waiting of snapshot removal task

Post by rolandk »

for example - i was trying to backup "s-ora-qs01" VM (backupjob with one single VM), veeam was just sitting there and showing

VM processing started at <......>
VM size: 40,0 GB (36 GB used)

and got stuck at this action, showing absolutely no progress.

Looking under the hood , the log showed, that Veeam was constantly being busy with trying to delete non-existant snapshots (see below) on 3 completely different VMs on a completely different ESX server, which had been removed manually, because some weird error created a maximum number of snapshots .

so, how can it be, that a backup task of VM "A" got completely get`s stuck, showing no info/progress to the user what`s happening , because some background removal task does operations on "VM B", "VM C" and "VM D" , which are not related to the current backup task of "VM A" at all ?

This looks like erratic or at least sub-optimal behaviour to me, at least it would be helpful if background-removal-activity/progress would be logged somewhere in the gui.

regards
roland
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Disable the waiting of snapshot removal task

Post by Gostev » 1 person likes this post

Thanks for posting the support case number, we will take a look.
rolandk
Influencer
Posts: 16
Liked: never
Joined: Jul 15, 2009 11:04 am
Full Name: RolandK
Contact:

Re: Disable the waiting of snapshot removal task

Post by rolandk »

thanks!
rolandk
Influencer
Posts: 16
Liked: never
Joined: Jul 15, 2009 11:04 am
Full Name: RolandK
Contact:

Re: Disable the waiting of snapshot removal task

Post by rolandk »

one more note:
as i also had the problem that i could not stop the "hung" backup tasks, i just came across http://www.veeam.com/kb1727 - and i am even more curious: if there are background tasks which prevent tasks being aborted, why can`t you at least inform the user that they are happening and that they need to be finished until the job can be aborted? or - even better: what keeps you away from aborting tasks even if snapshot removal has been triggered by veeam in virtual center? does that need to be a "synchronous" event?
rolandk
Influencer
Posts: 16
Liked: never
Joined: Jul 15, 2009 11:04 am
Full Name: RolandK
Contact:

Re: Disable the waiting of snapshot removal task

Post by rolandk »

here is some excerpt from the log, i thought i posted it on wednesday. we can see that some action fail and veeam is retrying again and again and spends quite some time on this....
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Disable the waiting of snapshot removal task

Post by Vitaliy S. »

Hi Roland, please avoid posting log snippets, as it is explained when you click New Topic. Thanks for your feedback on possible improvements of tasks handling logic.
rolandk
Influencer
Posts: 16
Liked: never
Joined: Jul 15, 2009 11:04 am
Full Name: RolandK
Contact:

Re: Disable the waiting of snapshot removal task

Post by rolandk »

ah, sorry - as i did not open a new thread for a while i didn`t know that logs are not wanted here. no problem
Post Reply

Who is online

Users browsing this forum: Lewpy and 47 guests