Host-based backup of Microsoft Hyper-V VMs.
Post Reply
Nail.Mukhametshin
Enthusiast
Posts: 51
Liked: 3 times
Joined: May 05, 2016 1:15 pm
Full Name: Наиль Мухаметшин
Location: Russia, Moscow
Contact:

Feature Request - Ignore VSS errors for Hyper-V quiescence

Post by Nail.Mukhametshin »

Use case:
Hyper-V VSS Writer detects Online backup for a VM.
The VM is backed up using Hyper-V quiescence. Guest Processing using Veeam Agent cannot be enabled.
During the backup of the VM an error takes place inside the guest.
Backup fails.

Desired outcome:
Instead of backup failure, crash-consistent backup of the VM is attempted.

Case # 01902316
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Mike Resseler »

Hi,
So if I understand it correctly, you want to take a crash-consistent backup when something goes wrong in the VM. I feel this is kind of dangerous. If the backup doesn't fail, you will not be notified and you might have only crash-consistent backups for a long time. At the moment of restore, you could run into a situation that you might be able to recover, but have bad data or the application fails...

But in essence, (again, if I understand correctly) what we have for application consistency (first try that, then go to Hyper-V quiescence if previous fails) you want to have that also for Hyper-V quiescence - crash-consistent (maybe with a warning or so?)
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by nmdange »

If you have an issue with Hyper-V quiescence, you can work around it by disabling the "Backup (Volume Shadow Copy)" option in the Hyper-V Integration Services settings of the VM. I've used this as a workaround when a VM has VSS issues. I don't think it's a good idea to have Veeam do this automatically. You should fix the issue in the guest causing the problem, and if you can't, manually workaround the issue.
Nail.Mukhametshin
Enthusiast
Posts: 51
Liked: 3 times
Joined: May 05, 2016 1:15 pm
Full Name: Наиль Мухаметшин
Location: Russia, Moscow
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Nail.Mukhametshin »

Hi Mike!

We have a situation where we work as Service Providers for our internal clients, so we can't track what clients do in their VMs and they can create and delete VMs via self-service portal without our notifications from one hand.

At other hand we have situation where we try to backup our clusters with 200+ VMs with one Job per cluster. 95% of VMs have a successesfull backup with Hyper-V quiscense enabled and other 5% can fail.

In this situation we have a Job that will never have succeded, so never delete outdated backups. As workaround we now backup ALL VMs as Crash Consistent backup, but we want take VSS consistent backup if it possible and take Crash Consistent backup if not. An addition if in a Job will be detected VMs that was falled back to Crash Consistent backup instead of VSS Consistent rise a Warning, so we can tell our clients to troubleshoot errors in theid VMs if they want VSS Consistent backup.

Please do not offer to create a separate Job for failed VMs or change VM template to disable Backup options in Hyper-V tools. All of this is workarounds work for a single VMs that works for a years, we have a hundreds of VMs that can be recreated in couple of days.
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Mike Resseler »

Hi Nail,

Let me consult with our engineers. Unfortunately I don't have an answer at this point in time

Brgds,

Mike
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Mike Resseler »

Hi Nail,
Could you check if this solution solves your problem?

https://helpcenter.veeam.com/backup/hyp ... hv_hv.html

"4. <...>you do not want to suspend VMs in the job, select the Take crash consistent backup instead of suspending VM check box. In this case, Veeam Backup & Replication will create a crash-consistent backup of a VM."

Thanks
Mike
Nail.Mukhametshin
Enthusiast
Posts: 51
Liked: 3 times
Joined: May 05, 2016 1:15 pm
Full Name: Наиль Мухаметшин
Location: Russia, Moscow
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Nail.Mukhametshin »

I have this option checked in all my backups - you can find this in a logs that attached to a support case # 01902316

Support engeneer said that this option is for VMs that have Offline backup caption. I tied to say him that this option as having a bug (don't backup my VMs if VSS Consistent failed), but he told that this is not the same behavior.
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Nail.Mukhametshin
Enthusiast
Posts: 51
Liked: 3 times
Joined: May 05, 2016 1:15 pm
Full Name: Наиль Мухаметшин
Location: Russia, Moscow
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Nail.Mukhametshin »

If you want we can make webex meeting and I'll show you all you want and we can discuss everything. My working hours 11:00-17:00 (GMT+3) Mon-Fri
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Mike Resseler »

I'm afraid that our solution will always first try to use the Hyper-V quiescence because that is your first choice. When it fails, we don't do an automatic failover to a crash-consistent backup. In my opinion, this actually makes sense since I wouldn't want to have some of my VMs being crash-consistent while I expect them to be quiescence...

However, in your specific use case, I might understand that you are in a different situation. Let me try to summarize if I understand it correctly and see if we can take this to the developers team as a potential feature request:

Description: You want every VM to try Hyper-V quiescence first, but take a crash-consistent backup (no freeze!) when that fails

Reason is that you don't have control over what is happening inside the guests but you still want to provide the customer with a backup, even if it is only crash-consistent. Any other (business) reasons? Are there SLA's involved?

How would you like to get notified. Should the job be in a warning state with a notification? A success but with a report in Veeam ONE (for example) that shows the VM's that used the crash-consistent method? Or you don't care at all about the notification (imho, I would like to know this for sure...)

Anything else I am missing? (I am brainstorming here, so whatever information / idea you have can be useful for our teams...)

Thanks

Mike
Nail.Mukhametshin
Enthusiast
Posts: 51
Liked: 3 times
Joined: May 05, 2016 1:15 pm
Full Name: Наиль Мухаметшин
Location: Russia, Moscow
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Nail.Mukhametshin » 1 person likes this post

Mike Resseler wrote:Description: You want every VM to try Hyper-V quiescence first, but take a crash-consistent backup (no freeze!) when that fails

Reason is that you don't have control over what is happening inside the guests but you still want to provide the customer with a backup, even if it is only crash-consistent.
Yes, you are absolutely right!
Mike Resseler wrote:Any other (business) reasons? Are there SLA's involved?
We have a SLA with 24 hours RTO/RPO without specification of backup type, but we try to do our best and provide a VSS Consistent backup because it can reduce the overall downtime of our services. Some of services on these VMs on our full support so we interested to recover consistent VMs in the first place.
Mike Resseler wrote:How would you like to get notified. Should the job be in a warning state with a notification? A success but with a report in Veeam ONE (for example) that shows the VM's that used the crash-consistent method? Or you don't care at all about the notification (imho, I would like to know this for sure...)
I think that per VM warning in a Job and Job in a warning state will be best solution to find out VMs with problems.
Mike Resseler wrote:Anything else I am missing? (I am brainstorming here, so whatever information / idea you have can be useful for our teams...)
May be also specify witch type of backup was taken on a restore page where we select restore point (thas will be WOW :wink: ).
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Mike Resseler »

Thanks for all the information. Not going to promise anything at all (I wouldn't even be able to even if I wanted ;-)) but this certainly will be looked at by the R&D team
cheers
Mike
Nail.Mukhametshin
Enthusiast
Posts: 51
Liked: 3 times
Joined: May 05, 2016 1:15 pm
Full Name: Наиль Мухаметшин
Location: Russia, Moscow
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Nail.Mukhametshin »

Can you reply here when you will have any information about this request? Planned(may be with versions)/reviewed/discarded/etc... All information intrested ;)
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Mike Resseler »

Hi Nail,

These are things that happen mostly in the background but if it is finally decided I will certainly come back and ask for more information ;-)
Neturius
Lurker
Posts: 2
Liked: never
Joined: Nov 08, 2017 12:12 pm
Full Name: Peter Frentin
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Neturius »

Hi Mike,

Can you please give an update on this request? We just had a case where crash consistent backups were taken of a VM for a long time because of an VSS issue inside the machine without recognition. This caused some trouble now when we had to restore a backup. We run the backups with the options "Enable Hyper-V guest quiscence" and "Take crash consistent backup instead of suspending VM". That Veeam is offering this fallback is perfect. However, if VSS fails it should generate a warning. Instead by now it justs logs

... Creating snapshot Microsoft Software Shadow Copy provider 1.0 (mode: Crash consistent)

with everything green. Thus the cause for the VSS issue keeps unexplored forever.

Thanks for you help...
Peter
BCT-Tech
Service Provider
Posts: 78
Liked: 1 time
Joined: Mar 20, 2015 6:12 pm
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by BCT-Tech »

+1
Rumple
Service Provider
Posts: 81
Liked: 14 times
Joined: Mar 10, 2010 7:50 pm
Full Name: Mark Hodges
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Rumple »

as a service provider I will second this motion...
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Post by Mike Resseler »

Hi all,

Still no further feedback on this. But I will put it back on the table.

Would a report makes sense (something you can run weekly or so?)
Nail.Mukhametshin
Enthusiast
Posts: 51
Liked: 3 times
Joined: May 05, 2016 1:15 pm
Full Name: Наиль Мухаметшин
Location: Russia, Moscow
Contact:

Re: Feature Request - Ignore VSS errors for Hyper-V quiescence

Post by Nail.Mukhametshin »

Glad to see here some of SPs that experience same troubles that me =)
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests