2008 R2 VM gets put in Saved State during hypervisor backup

[exportgoldman]

Hello Everyone,

I'm usually pretty good at troubleshooting this, but I'm stuck on this one.

I have a 2008 R2 SP1 Hypervisor which gets backups using ShadowProtect, with around 15 VM's.

All the VM's except 1 do online backups, we were testing Veeam and this may be unrelated, but I uninstalled the software within the VM when we found this problem but cannot get the VM to not backup using saved state.

I have put the VHD in a new VM, I have re-registered the VSS DLL's in the VM as per the Symantec article (http://www.symantec.com/connect/downloa ... erver-2008)

The strange thing is I get absolutely no errors on the hypervisor, or guest regarding VSS, it's like the VSS signal is not getting through to this VM. I don't know how to check this, but I have turned off and then back on the VSS heartbeat setting in the integration options in the VM Settings.

I have also re-installed the Integration components in the VM.

vssadmin list writers shows no problems and everything is stable/no error. on the guest VM

vssadmin list providers only lists the Microsoft provider. on the guest VM

I do get a vssadmin list writers error on HyperV with Unexpected error but can't find any logs for it.

I originally posted this on the Microsoft forums but got sent over here.

[Gostev]

Hi Tyler, it is not clear if you are having the issue with ShadowProtect, Symantec or Veeam?

[exportgoldman]

Hi Tyler, it is not clear if you are having the issue with ShadowProtect, Symantec or Veeam?

The problem is after we installed Veeam.

[Gostev]

I understand that, but what product's backup process puts the VM in the saved state?

[exportgoldman]

I understand that, but what product's backup process puts the VM in the saved state?

It was Veeam, then when we removed it, and put ShadowProtect back on on the Hypervisor level it's ShadowProtect. Something Veeam did broke the VSS stack, so HyperV shuts down the VM as it can't pass the VSS messages up to the VM. I have two machines doing this now.

[Gostev]

Veeam does not put anything in the VSS stack of the processed VM. Unlike Symantec, for example - which does install its own VSS provider (which is well known to mess up VSS processing for Veeam as well, by the way).

[exportgoldman]

So basically your saying it's not your products fault and it's something else huh?

Interesting since we have two different sites we put Veeam on and now this is happening. Shame it took a month of phone calls and posts in the forums to get fobbed off.

[exportgoldman]

Failing any actual help from Veeam, I have found that re-installing Veeam and then setting "Enable application-aware image processing" and setting a username and password now allows VSS to get a online snapshot of this VM.

I assume I have to backout these changes somehow before uninstalling Veeam, otherwise it cripples VSS getting a backup of this VM.

Documenting for others.

[Gostev]

First, it is simply not fair to say you did not get any help from Veeam, because I have been answering all your questions above promptly.

Second, I can only repeat that Veeam does not put anything permanent in the VSS stack of the processed VM. The setting you have referred to is not used unless you actually run the Veeam job - in which case, temporary run-time VSS coordination process is injected into the guest by the job. This process only exists in the guest while Veeam is backing up the specific VM, and is removed immediately as soon as the backup is taken. Thus, there is literally nothing VSS-related to "backout" from guest before uninstalling Veeam. And all job settings are only used in the run-time too (when you actually run the job).

[exportgoldman]

First, it is simply not fair to say you did not get any help from Veeam, because I have been answering all your questions above promptly.

Second, I can only repeat that Veeam does not put anything permanent in the VSS stack of the processed VM. The setting you have referred to is not used unless you actually run the Veeam job - in which case, temporary run-time VSS coordination process is injected into the guest by the job. This process only exists in the guest while Veeam is backing up the specific VM, and is removed immediately as soon as the backup is taken. Thus, there is literally nothing VSS-related to "backout" from guest before uninstalling Veeam. And all job settings are only used in the run-time too (when you actually run the job).

I'm not sure exactly how your software works, but we had that setting enabled, and uninstalled your software and the problems as my original description occurred, which were the VM was put in a saved state before ANY VSS backup from any VSS backup program. Reinstalling Veeam, then setting up a job with those settings again, running a backup job, then disabling those settings in Veeam allowed other VSS Backup programs to then not put the VM into saved state while performing a backup.

I then uninstalled the Veeam bits one by one running a test each time and it worked fine.

I am only informing you of this, as it may be a bug in the uninstaller or how your product works as I had this happen on two different client sites which were running trials of Veeam, and doing the above solved it.

I don't think it's unfair to say this took too long to resolve - you may answer quickly but not with anything useful sorry. But putting that aside, this appears to be the solution.

I don't know what exactly Veeam changes when this option is set, but it's not in the VM config itself as taking the VHD and assigning it to another VM has the same fault, so the setting appears to be put inside the VHD itself - I could do further investigation with Process Monitor to fully conclude what setting it changed for the worst but I think this is a good enough resolution for others to follow with the same problem.

[Gostev]

I don't think it's unfair to say this took too long to resolve - you may answer quickly but not with anything useful sorry.

Reading through our discussion above, I have answered your specific question, and you never came back to ask more, or to say they are not useful/sufficient, or even to say "thanks". This usually indicates the issue is resolved, but if not - I cannot read your mind? Then 1 month later, you post all of a sudden that you are not getting any help from Veeam. Huh? If you'd ask more questions, I would answer. If you'd posted your support case ID, I would monitor it, and drive to resolution. However, if you are completely quiet after my last response, how can I help you?

Now, note that I am not a technical support person - I am just telling you exactly how the product works, and how it may or may not relate to the issue you are experiencing. To really troubleshoot an issue, you should open the support case and have our technical staff investigate this more closely with the logs and/or webex. As explained when you click New Topic, this is simply not a support forum:

Please appreciate that this forum is not intended to provide technical support, and as such is not monitored by support staff. The product management team behind this forum is unable to provide assistance troubleshooting complex technical issues, so kindly please set your expectations accordingly.

Putting that aside, I just want to make it clear that the solution that you have found does not seem right at all. Selecting that checkbox does not make any difference to guests if the job is stopped, as what is controlled on this step is job's run-time guest processing logic.

If you want to find the real reason, you should open a support case and have it escalated into R&D for investigation. However, if you have already formed your opinion, it is fine as well - I just don't agree with it, because it does not make any sense.

And no, this is not an "uninstaller bug" either - since we literally do not install anything permanent into the guest, there is absolutely nothing for us to uninstall from there during product uninstallation.