Feature Request - Ignore VSS errors for Hyper-V quiescence

Hyper-V specific discussions

Feature Request - Ignore VSS errors for Hyper-V quiescence

Veeam Logoby Nail.Mukhametshin » Thu Oct 13, 2016 12:57 pm

Use case:
Hyper-V VSS Writer detects Online backup for a VM.
The VM is backed up using Hyper-V quiescence. Guest Processing using Veeam Agent cannot be enabled.
During the backup of the VM an error takes place inside the guest.
Backup fails.

Desired outcome:
Instead of backup failure, crash-consistent backup of the VM is attempted.

Case # 01902316
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Nail.Mukhametshin
Enthusiast
 
Posts: 27
Liked: 2 times
Joined: Thu May 05, 2016 1:15 pm
Location: Russia, Moscow
Full Name: Наиль Мухаметшин

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Mike Resseler » Fri Oct 14, 2016 5:17 am

Hi,
So if I understand it correctly, you want to take a crash-consistent backup when something goes wrong in the VM. I feel this is kind of dangerous. If the backup doesn't fail, you will not be notified and you might have only crash-consistent backups for a long time. At the moment of restore, you could run into a situation that you might be able to recover, but have bad data or the application fails...

But in essence, (again, if I understand correctly) what we have for application consistency (first try that, then go to Hyper-V quiescence if previous fails) you want to have that also for Hyper-V quiescence - crash-consistent (maybe with a warning or so?)
Mike Resseler
Veeam Software
 
Posts: 3382
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby nmdange » Fri Oct 14, 2016 1:34 pm

If you have an issue with Hyper-V quiescence, you can work around it by disabling the "Backup (Volume Shadow Copy)" option in the Hyper-V Integration Services settings of the VM. I've used this as a workaround when a VM has VSS issues. I don't think it's a good idea to have Veeam do this automatically. You should fix the issue in the guest causing the problem, and if you can't, manually workaround the issue.
nmdange
Expert
 
Posts: 233
Liked: 60 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Nail.Mukhametshin » Fri Oct 14, 2016 3:34 pm

Hi Mike!

We have a situation where we work as Service Providers for our internal clients, so we can't track what clients do in their VMs and they can create and delete VMs via self-service portal without our notifications from one hand.

At other hand we have situation where we try to backup our clusters with 200+ VMs with one Job per cluster. 95% of VMs have a successesfull backup with Hyper-V quiscense enabled and other 5% can fail.

In this situation we have a Job that will never have succeded, so never delete outdated backups. As workaround we now backup ALL VMs as Crash Consistent backup, but we want take VSS consistent backup if it possible and take Crash Consistent backup if not. An addition if in a Job will be detected VMs that was falled back to Crash Consistent backup instead of VSS Consistent rise a Warning, so we can tell our clients to troubleshoot errors in theid VMs if they want VSS Consistent backup.

Please do not offer to create a separate Job for failed VMs or change VM template to disable Backup options in Hyper-V tools. All of this is workarounds work for a single VMs that works for a years, we have a hundreds of VMs that can be recreated in couple of days.
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Nail.Mukhametshin
Enthusiast
 
Posts: 27
Liked: 2 times
Joined: Thu May 05, 2016 1:15 pm
Location: Russia, Moscow
Full Name: Наиль Мухаметшин

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Mike Resseler » Fri Oct 14, 2016 7:42 pm

Hi Nail,

Let me consult with our engineers. Unfortunately I don't have an answer at this point in time

Brgds,

Mike
Mike Resseler
Veeam Software
 
Posts: 3382
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Mike Resseler » Fri Oct 14, 2016 10:28 pm

Hi Nail,
Could you check if this solution solves your problem?

https://helpcenter.veeam.com/backup/hyp ... hv_hv.html

"4. <...>you do not want to suspend VMs in the job, select the Take crash consistent backup instead of suspending VM check box. In this case, Veeam Backup & Replication will create a crash-consistent backup of a VM."

Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 3382
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Nail.Mukhametshin » Fri Oct 14, 2016 11:39 pm

I have this option checked in all my backups - you can find this in a logs that attached to a support case # 01902316

Support engeneer said that this option is for VMs that have Offline backup caption. I tied to say him that this option as having a bug (don't backup my VMs if VSS Consistent failed), but he told that this is not the same behavior.
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Nail.Mukhametshin
Enthusiast
 
Posts: 27
Liked: 2 times
Joined: Thu May 05, 2016 1:15 pm
Location: Russia, Moscow
Full Name: Наиль Мухаметшин

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Nail.Mukhametshin » Fri Oct 14, 2016 11:42 pm

If you want we can make webex meeting and I'll show you all you want and we can discuss everything. My working hours 11:00-17:00 (GMT+3) Mon-Fri
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Nail.Mukhametshin
Enthusiast
 
Posts: 27
Liked: 2 times
Joined: Thu May 05, 2016 1:15 pm
Location: Russia, Moscow
Full Name: Наиль Мухаметшин

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Mike Resseler » Sat Oct 15, 2016 5:08 am

I'm afraid that our solution will always first try to use the Hyper-V quiescence because that is your first choice. When it fails, we don't do an automatic failover to a crash-consistent backup. In my opinion, this actually makes sense since I wouldn't want to have some of my VMs being crash-consistent while I expect them to be quiescence...

However, in your specific use case, I might understand that you are in a different situation. Let me try to summarize if I understand it correctly and see if we can take this to the developers team as a potential feature request:

Description: You want every VM to try Hyper-V quiescence first, but take a crash-consistent backup (no freeze!) when that fails

Reason is that you don't have control over what is happening inside the guests but you still want to provide the customer with a backup, even if it is only crash-consistent. Any other (business) reasons? Are there SLA's involved?

How would you like to get notified. Should the job be in a warning state with a notification? A success but with a report in Veeam ONE (for example) that shows the VM's that used the crash-consistent method? Or you don't care at all about the notification (imho, I would like to know this for sure...)

Anything else I am missing? (I am brainstorming here, so whatever information / idea you have can be useful for our teams...)

Thanks

Mike
Mike Resseler
Veeam Software
 
Posts: 3382
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Nail.Mukhametshin » Sat Oct 15, 2016 9:43 am 1 person likes this post

Mike Resseler wrote:Description: You want every VM to try Hyper-V quiescence first, but take a crash-consistent backup (no freeze!) when that fails

Reason is that you don't have control over what is happening inside the guests but you still want to provide the customer with a backup, even if it is only crash-consistent.


Yes, you are absolutely right!

Mike Resseler wrote:Any other (business) reasons? Are there SLA's involved?


We have a SLA with 24 hours RTO/RPO without specification of backup type, but we try to do our best and provide a VSS Consistent backup because it can reduce the overall downtime of our services. Some of services on these VMs on our full support so we interested to recover consistent VMs in the first place.

Mike Resseler wrote:How would you like to get notified. Should the job be in a warning state with a notification? A success but with a report in Veeam ONE (for example) that shows the VM's that used the crash-consistent method? Or you don't care at all about the notification (imho, I would like to know this for sure...)


I think that per VM warning in a Job and Job in a warning state will be best solution to find out VMs with problems.

Mike Resseler wrote:Anything else I am missing? (I am brainstorming here, so whatever information / idea you have can be useful for our teams...)


May be also specify witch type of backup was taken on a restore page where we select restore point (thas will be WOW :wink: ).
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Nail.Mukhametshin
Enthusiast
 
Posts: 27
Liked: 2 times
Joined: Thu May 05, 2016 1:15 pm
Location: Russia, Moscow
Full Name: Наиль Мухаметшин

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Mike Resseler » Mon Oct 17, 2016 5:23 am

Thanks for all the information. Not going to promise anything at all (I wouldn't even be able to even if I wanted ;-)) but this certainly will be looked at by the R&D team
cheers
Mike
Mike Resseler
Veeam Software
 
Posts: 3382
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Nail.Mukhametshin » Wed Nov 09, 2016 12:57 pm

Can you reply here when you will have any information about this request? Planned(may be with versions)/reviewed/discarded/etc... All information intrested ;)
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
Nail.Mukhametshin
Enthusiast
 
Posts: 27
Liked: 2 times
Joined: Thu May 05, 2016 1:15 pm
Location: Russia, Moscow
Full Name: Наиль Мухаметшин

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Mike Resseler » Wed Nov 09, 2016 2:11 pm

Hi Nail,

These are things that happen mostly in the background but if it is finally decided I will certainly come back and ask for more information ;-)
Mike Resseler
Veeam Software
 
Posts: 3382
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Feature Request - Ignore VSS errors for Hyper-V quiescen

Veeam Logoby Neturius » Wed Nov 08, 2017 12:45 pm

Hi Mike,

Can you please give an update on this request? We just had a case where crash consistent backups were taken of a VM for a long time because of an VSS issue inside the machine without recognition. This caused some trouble now when we had to restore a backup. We run the backups with the options "Enable Hyper-V guest quiscence" and "Take crash consistent backup instead of suspending VM". That Veeam is offering this fallback is perfect. However, if VSS fails it should generate a warning. Instead by now it justs logs

... Creating snapshot Microsoft Software Shadow Copy provider 1.0 (mode: Crash consistent)

with everything green. Thus the cause for the VSS issue keeps unexplored forever.

Thanks for you help...
Peter
Neturius
Lurker
 
Posts: 1
Liked: never
Joined: Wed Nov 08, 2017 12:12 pm
Full Name: Peter Frentin


Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 1 guest