Feature Request - Ignore VSS errors for Hyper-V quiescence

[Nail.Mukhametshin]

Use case:
Hyper-V VSS Writer detects Online backup for a VM.
The VM is backed up using Hyper-V quiescence. Guest Processing using Veeam Agent cannot be enabled.
During the backup of the VM an error takes place inside the guest.
Backup fails.

Desired outcome:
Instead of backup failure, crash-consistent backup of the VM is attempted.

Case # 01902316

[Mike Resseler]

Hi,
So if I understand it correctly, you want to take a crash-consistent backup when something goes wrong in the VM. I feel this is kind of dangerous. If the backup doesn't fail, you will not be notified and you might have only crash-consistent backups for a long time. At the moment of restore, you could run into a situation that you might be able to recover, but have bad data or the application fails...

But in essence, (again, if I understand correctly) what we have for application consistency (first try that, then go to Hyper-V quiescence if previous fails) you want to have that also for Hyper-V quiescence - crash-consistent (maybe with a warning or so?)

[nmdange]

If you have an issue with Hyper-V quiescence, you can work around it by disabling the "Backup (Volume Shadow Copy)" option in the Hyper-V Integration Services settings of the VM. I've used this as a workaround when a VM has VSS issues. I don't think it's a good idea to have Veeam do this automatically. You should fix the issue in the guest causing the problem, and if you can't, manually workaround the issue.

[Nail.Mukhametshin]

Hi Mike!

We have a situation where we work as Service Providers for our internal clients, so we can't track what clients do in their VMs and they can create and delete VMs via self-service portal without our notifications from one hand.

At other hand we have situation where we try to backup our clusters with 200+ VMs with one Job per cluster. 95% of VMs have a successesfull backup with Hyper-V quiscense enabled and other 5% can fail.

In this situation we have a Job that will never have succeded, so never delete outdated backups. As workaround we now backup ALL VMs as Crash Consistent backup, but we want take VSS consistent backup if it possible and take Crash Consistent backup if not. An addition if in a Job will be detected VMs that was falled back to Crash Consistent backup instead of VSS Consistent rise a Warning, so we can tell our clients to troubleshoot errors in theid VMs if they want VSS Consistent backup.

Please do not offer to create a separate Job for failed VMs or change VM template to disable Backup options in Hyper-V tools. All of this is workarounds work for a single VMs that works for a years, we have a hundreds of VMs that can be recreated in couple of days.

[Mike Resseler]

Hi Nail,

Let me consult with our engineers. Unfortunately I don't have an answer at this point in time

Brgds,

Mike

[Mike Resseler]

Hi Nail,
Could you check if this solution solves your problem?

https://helpcenter.veeam.com/backup/hyp ... hv_hv.html

"4. <...>you do not want to suspend VMs in the job, select the Take crash consistent backup instead of suspending VM check box. In this case, Veeam Backup & Replication will create a crash-consistent backup of a VM."

Thanks
Mike

[Nail.Mukhametshin]

I have this option checked in all my backups - you can find this in a logs that attached to a support case # 01902316

Support engeneer said that this option is for VMs that have Offline backup caption. I tied to say him that this option as having a bug (don't backup my VMs if VSS Consistent failed), but he told that this is not the same behavior.

[Nail.Mukhametshin]

If you want we can make webex meeting and I'll show you all you want and we can discuss everything. My working hours 11:00-17:00 (GMT+3) Mon-Fri

[Mike Resseler]

I'm afraid that our solution will always first try to use the Hyper-V quiescence because that is your first choice. When it fails, we don't do an automatic failover to a crash-consistent backup. In my opinion, this actually makes sense since I wouldn't want to have some of my VMs being crash-consistent while I expect them to be quiescence...

However, in your specific use case, I might understand that you are in a different situation. Let me try to summarize if I understand it correctly and see if we can take this to the developers team as a potential feature request:

Description: You want every VM to try Hyper-V quiescence first, but take a crash-consistent backup (no freeze!) when that fails

Reason is that you don't have control over what is happening inside the guests but you still want to provide the customer with a backup, even if it is only crash-consistent. Any other (business) reasons? Are there SLA's involved?

How would you like to get notified. Should the job be in a warning state with a notification? A success but with a report in Veeam ONE (for example) that shows the VM's that used the crash-consistent method? Or you don't care at all about the notification (imho, I would like to know this for sure...)

Anything else I am missing? (I am brainstorming here, so whatever information / idea you have can be useful for our teams...)

Thanks

Mike

[Nail.Mukhametshin]

Description: You want every VM to try Hyper-V quiescence first, but take a crash-consistent backup (no freeze!) when that fails

Reason is that you don't have control over what is happening inside the guests but you still want to provide the customer with a backup, even if it is only crash-consistent.

Yes, you are absolutely right!

Any other (business) reasons? Are there SLA's involved?

We have a SLA with 24 hours RTO/RPO without specification of backup type, but we try to do our best and provide a VSS Consistent backup because it can reduce the overall downtime of our services. Some of services on these VMs on our full support so we interested to recover consistent VMs in the first place.

How would you like to get notified. Should the job be in a warning state with a notification? A success but with a report in Veeam ONE (for example) that shows the VM's that used the crash-consistent method? Or you don't care at all about the notification (imho, I would like to know this for sure...)

I think that per VM warning in a Job and Job in a warning state will be best solution to find out VMs with problems.

Anything else I am missing? (I am brainstorming here, so whatever information / idea you have can be useful for our teams...)

May be also specify witch type of backup was taken on a restore page where we select restore point (thas will be WOW ).

[Mike Resseler]

Thanks for all the information. Not going to promise anything at all (I wouldn't even be able to even if I wanted ) but this certainly will be looked at by the R&D team
cheers
Mike

[Nail.Mukhametshin]

Can you reply here when you will have any information about this request? Planned(may be with versions)/reviewed/discarded/etc... All information intrested

[Mike Resseler]

Hi Nail,

These are things that happen mostly in the background but if it is finally decided I will certainly come back and ask for more information

[Neturius]

Hi Mike,

Can you please give an update on this request? We just had a case where crash consistent backups were taken of a VM for a long time because of an VSS issue inside the machine without recognition. This caused some trouble now when we had to restore a backup. We run the backups with the options "Enable Hyper-V guest quiscence" and "Take crash consistent backup instead of suspending VM". That Veeam is offering this fallback is perfect. However, if VSS fails it should generate a warning. Instead by now it justs logs

... Creating snapshot Microsoft Software Shadow Copy provider 1.0 (mode: Crash consistent)

with everything green. Thus the cause for the VSS issue keeps unexplored forever.

Thanks for you help...
Peter

[BCT-Tech]

+1

[Rumple]

as a service provider I will second this motion...

[Mike Resseler]

Hi all,

Still no further feedback on this. But I will put it back on the table.

Would a report makes sense (something you can run weekly or so?)

[Nail.Mukhametshin]

Glad to see here some of SPs that experience same troubles that me =)