Disable the waiting of snapshot removal task

[raphael@schitz.net]

Is there any way to disable the waiting for snapshot removal task in order to speed up veeam jobs ?

[Vitaliy S.]

Hi Raphael, snashot removal task is triggered at the end of the backup/replication job. Could you please explain me how this feature would help you in your current deployment?

[raphael@schitz.net]

Hi Raphael, snashot removal task is triggered at the end of the backup/replication job. Could you please tell how that would help with the issue discussed in this thread?

Hi Vitaly,

Sorry, i did not specify in case you got a job with a lot of VMs.
In our (my customer) case, we set up job backing up resource pools where hundreds of vm could live so snapshot removal is consuming a lot of time on those jobs.
It really could be nice if we can have an option like "don't wait for snapshot revmoval task end (i know the risk on my storage array, etc...)" on jobs.
beside the use case, is it possible anyway ?

Thanks for your time.

[Vitaliy S.]

Thanks for clarifications, but I see a huge negative effect which will level off the benefit of increased backup job speed.

What would happen to your storage when all your VMs will start committing their snapshots (might even happen at the same time)? This will impact all VMs running on this datastore and can even lower their performance. I do not believe that this is something you're after with this feature request.

That said, If you want to have smaller backup window, I would recommend to split your job into a couple smaller ones and use multiple proxy servers to process VMs concurrently.

Hope this makes sense.

[raphael@schitz.net]

Thanks for your feedback Vitaly,

Actually in our case (referring to the vcenter task timestamps), the snapshot of the vm #1 would be commited while the vm #2 would be backed up and so on. It would easily be done before the next vm backup finish (again in our case but i know some customers with the same behavior).
Snapshot commit are not extremely long but even 1 or 2 minutes per VM could saves hours when you got 100+ to backup.

About the backup window, we really tried many things but (for us) the resource pool backups are the best way since we don't have to setup backup for new vms, if they're deployed in a production level pool, it will be in the next pool backup.

Here is an example of an SQL server located on a high end array and still the performances are quite good, the snapshot removal takes few minutes :







It would be really great if you could consider this kind of case

Thanks

[Vitaliy S.]

Ok, thanks for the feedback, much appreciated!

[raphael@schitz.net]

Hi Vitaliy,
With those elements, could you tell me if this is something you'd consider too add in the future (even with a registry key) ?
Thanks

[Vitaliy S.]

Hi Raphael, this depends on the demand and our current priorities for the next releases. Thanks!

[jyarborough]

I could see a huge benefit in this. However, I think some intelligence around it would be necessary. For example, inside a single job with multiple VMs, when one finishes the actual backup and goes to commit the snapshot, it would let the next job in line fire off and get started. However, the intelligence would need to be in checking to see how many commits are occurring, or how far behind the job is and only allow like 1 (or x) concurrent commits before it allows the next job to start.

[raphael@schitz.net]

I could see a huge benefit in this. However, I think some intelligence around it would be necessary. For example, inside a single job with multiple VMs, when one finishes the actual backup and goes to commit the snapshot, it would let the next job in line fire off and get started. However, the intelligence would need to be in checking to see how many commits are occurring, or how far behind the job is and only allow like 1 (or x) concurrent commits before it allows the next job to start.

I agree but it could be also heuristicly based on the last commit time.
I also think it could be nice for "powerusers" like us to do the intelligence our self. In our example, i know the storage side can easily handle multiple snapshots commit simultaneously.

[Gostev]

OK, so I don't really see how this would help when you are running multiple concurrent jobs, which is what most of our customers do. Storage is never idle in that case anyway. Are we talking single job scenario here?

[Gostev]

i know the storage side can easily handle multiple snapshots commit simultaneously.

No doubt about that, I see you have 3PAR - enough said... Ferrari of storage.

[raphael@schitz.net]

OK, so I don't really see how this would help when you are running multiple concurrent jobs, which is what most of our customers do. Storage is never idle in that case anyway. Are we talking single job scenario here?

It's more about reducing the time taken by side ops like creating/commiting snapshots for long jobs. In our case, we backup resource pools everydays and with hundred of vms it's sometimes difficult to backup the hole thing in a night. We add some scheduled scripts to stop and retry jobs when we want to but it could be very nice to explore anything that could lead to a shorter backup window.

Thanks

[raphael@schitz.net]

No doubt about that, I see you have 3PAR - enough said... Ferrari of storage.

sure

[yizhar]

HI.

An alternate (and I think also better) solution for this scenario, you may consider it as a feature request:

Add an option to the Veeam Job properties, specifiying how many VM can be backuped up simultaneously, which will default to 1.
An administrator which has many relatively small VMs to backup in a single job, will be able to specify 2 concurrent VMs (or a bit more), which will be more efficient, and will not require using a separate job.
This will be similar to other solutions such as VDR/VDP which backup several VMs concurrently.

Yizhar

[emtunc]

In terms of performance of Veeam jobs, is there a way to perform the removal and creation of snapshots in a more 'optimal' way?

For example, sometimes the snapshot removal process takes quite a long while to complete... sometimes a few minutes, almost an hour or more.

During this time, Veeam and the target backup destination are pretty much sitting idle. My thought was that this process could maybe be optimised to deal with this idle time. Something like this:

1. Create snapshot, backup VM1
2. Remove VM1 snapshot and continue to create snapshot of next VM2 in parallel so that the job continues rather than waiting for current snapshot removal to complete.

OR

1. Create snapshot, backup VM1
2. After VM 1 has finished backing up, create snapshot of VM2 and start backing up VM2
3. Whilst VM2 is in the process of backing up, initiate background snapshot removal of VM1.

Perhaps there are some technical limitations that would mean this would not be possible but worth bringing up anyway.

[rolandk]

it`s even worse - i had my veeam backup completely broken, as no job did ever complete because of snapshot removal tasks.

after digging into the issue for a while (as there where rpc errors i sat with the firewall admin to analyse stale connections, firewall timeouts etc), i found that it was the backup job just waiting for some background task to try removing non-existant snapshots -(of VMs not in the current backup job) and then eventually time-out.

the real problem i see is not the background removal task itself,

the problem i see is, that it`s nowhere shown to the end user, that such tasks are in progress
and
that a running task which should back up one VM for example just sits and wait because some hidden backup removal tasks are in progress and never complete
and
that such background operation does have no internal timeout but blocks the backup processing as a whole.

if someone from veeam want`s to have a look at this - just take a look at Case # 00445861

i think this should need improvement.

[rolandk]

for example - i was trying to backup "s-ora-qs01" VM (backupjob with one single VM), veeam was just sitting there and showing

VM processing started at <......>
VM size: 40,0 GB (36 GB used)

and got stuck at this action, showing absolutely no progress.

Looking under the hood , the log showed, that Veeam was constantly being busy with trying to delete non-existant snapshots (see below) on 3 completely different VMs on a completely different ESX server, which had been removed manually, because some weird error created a maximum number of snapshots .

so, how can it be, that a backup task of VM "A" got completely get`s stuck, showing no info/progress to the user what`s happening , because some background removal task does operations on "VM B", "VM C" and "VM D" , which are not related to the current backup task of "VM A" at all ?

This looks like erratic or at least sub-optimal behaviour to me, at least it would be helpful if background-removal-activity/progress would be logged somewhere in the gui.

regards
roland

[Gostev]

Thanks for posting the support case number, we will take a look.

[rolandk]

thanks!

[rolandk]

one more note:
as i also had the problem that i could not stop the "hung" backup tasks, i just came across http://www.veeam.com/kb1727 - and i am even more curious: if there are background tasks which prevent tasks being aborted, why can`t you at least inform the user that they are happening and that they need to be finished until the job can be aborted? or - even better: what keeps you away from aborting tasks even if snapshot removal has been triggered by veeam in virtual center? does that need to be a "synchronous" event?

[rolandk]

here is some excerpt from the log, i thought i posted it on wednesday. we can see that some action fail and veeam is retrying again and again and spends quite some time on this....

[Vitaliy S.]

Hi Roland, please avoid posting log snippets, as it is explained when you click New Topic. Thanks for your feedback on possible improvements of tasks handling logic.

[rolandk]

ah, sorry - as i did not open a new thread for a while i didn`t know that logs are not wanted here. no problem