Job warnings after upgrade to v7

[sho]

I thought I'd post about my experience and findings of upgrading our B&R v6.5 installation to v7.

Our setup is pretty basic as follows....

• Veeam running on a physical 2008R2 box backing up to local storage

• No separate backup proxy servers

• One backup job to backup all 24 VMs running on a 2008R2 Hyper-V cluster

• VMs are a mix of Svr 2008R2 and 2012

After upgrading B&R to v7 (very straight forward process) I found that I was getting the following warning on all of my 2012 VMs in the B&R backup job log...

Guest processing skipped (check guest OS VSS state and integration components version)

Looking in the Application Event Log on the affected VMs I see the following error...

Code: Select all

Log Name:      Application
Source:        vmicvss
Date:          20/08/2013 21:14:39
Event ID:      13
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ****************.internal
Description:
Windows cannot perform an online backup of this system because scoped snapshots are enabled. To resolve this, disable scoped snapshots by creating the following registry value on this computer:
 	PATH: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SystemRestore\
 	DWORD: ScopeSnapshots
 	Value: 0
 Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="vmicvss" />
    <EventID Qualifiers="49154">13</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2013-08-20T20:14:39.000000000Z" />
    <EventRecordID>3860</EventRecordID>
    <Channel>Application</Channel>
    <Computer>TGS-SCVMM.tgs.internal</Computer>
    <Security />
  </System>
  <EventData>
  </EventData>
</Event>

Interesting to note that these errors also exist before I performed the v7 upgrade to B&R but no warnings were ever shown in v6.5.

I have today added the registry key given in the above error to one of my 2012 VMs and will see what change this brings when the backup job next runs.

[sho]

Well after applying the registry entry given above to a test 2012 guest VM, backup of this VM completed successfully. Therefore yesterday I applied the registry entry to all of my 2012 guest VMs and this morning Veeam is showing all backups as being successful.

Interesting to note that before applying these changes backup jobs would sometimes complete with warnings and other times complete with errors.

[foggy]

Stuart, thanks for sharing your observations, much appreciated. Looks like proper handling of such situations was implemented in v7, while previous versions did not include this logic and hence did not produce any warnings.

[pthmpson]

I got the same error message (check guest OS VSS state and integration components version) after upgrading to Veeam v7.
I just did what the error message said and upgraded the Integration Components on each of my Hyper-V VMs to v6.2.9200.16433 - all of my backups worked after that.

Phil

[dockx]

Hi,

Another problem arose: trying to backup an Exchange 2013 VM but getting following error (while enable application-aware image processing is on). Hyper-V integration services is installed on the Hyper-V host.

Guest processing skipped (check guest OS VSS state and integration components version)
Unable to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Crash consistent). Details: Guest processing skipped (check guest OS VSS state and integration components version)
Error: Guest processing skipped (check guest OS VSS state and integration components version)

If I go into advanced settings and and turn on 'Disable application procession', then Veeam backups the VM, but the logs aren't truncated.

Do I need to install something on the Exchange virtual machine?

Any golden advice?

Kind regards,
D.

[foggy]

Does the VM guest OS have dynamic disks by any chance?

[dockx]

No, fixed .vhd.

[foggy]

Do you have any related messages in the Windows Event Log? What are the OS versions of both Hyper-V host and guest VM?

[dockx]

The VEEAM host is server 2012 and has the following in Event Viewer:
Error Session Backup Exchange has been completed.

Exchange (guest) is a server 2012 and has following in the Event Viewer:

Code: Select all

Windows cannot perform an online backup of this system because scoped snapshots are enabled. To resolve this, disable scoped snapshots by creating the following registry value on this computer:
 	PATH: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SystemRestore\
 	DWORD: ScopeSnapshots
 	Value: 0
 Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.

Hyper-V host is a 2008R2 Enterprise server and has no relevant events in the log...

Maybe it has something to do with those scoped snapshots?

[veremin]

According to our customers’ experience, applying registry key solves the issue, indeed. Thanks.

[dockx]

I've added the registry key and it seems to be working.

http://blog.osmicro.org/hyper-v-vss-sna ... ved-state/

Thanks all.

[foggy]

You're welcome. We're currently preparing the KB article for this issue.

[mathao]

When will the KB be available? We have the same issue with servers 2003's VMs since the upgrade.

Unable to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Crash consistent). Details: Guest processing skipped (check guest OS VSS state and integration components version)

[Vitaliy S.]

Have you tried to apply the suggestions mentioned above?

[Simonx182]

I have the same problem on 1 VM of our Hyper-V 2008 R2 infrastructure.
The failing VM runs Server 2008 R2 and has Fileserver (DFS) and DHCP Role on it. All other vm's with 2008 R2 run properly.
Already tried to add the registry key to the failing guest VM, as mentioned before, no change. Do I have to reboot the guest VM after creating the registry key?

In the meantime I've configured that functional VSS processing is not required for this VM, so it creates at least a basic backup.
What else can I do to solve this problem ASAP? Open ticket with Veeam Support?

Regards Simon

[foggy]

Simon, with all technical issues it is always recommended to open a case with Veeam support, since most likely logs investigation is required.

That said, I'm not sure you're experiencing the same issue discussed in this thread as your guest OS is Windows 2008 R2, not 2012 as in the cases reported above. What specific error or warning do you get in the job session log? Does the job fail to backup the VM? What Veeam B&R version are you on and does the VM guest OS have dynamic disks by any chance?

[Simonx182]

Error message is the following:

Unable to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Crash consistent). Details: Guest processing skipped (check guest OS VSS state and integration components version)
Veeam B&R Version 7.0.0.715 . No dynamic disks.

I've opened a case about 30 minutes ago for that case including all log files. We'll se what comes out there.
Thank you.

[veremin]

It will be appreciated if you post your ticket number here, so that, we can follow the case and update the topic once the resolution is found. Thanks.

[Simonx182]

Here is my Case Number: # 00464858

[Simonx182]

My "Unable to create snapshot" problem with the 2008 R2 Server VM and Hyper-V Server is now solved. Thanks to the great Veeam support engineer we found the problem quickly: It was the the configured ShadowCopies location on a partition of the failing VM. The Shadow Copies must be located on the same drive. After changing the location everything worked as it should.

The support engineer gave me this tecnical explanation:

the hyper-v vss writer has problems with this configuration - not the Veeam, since Veeam plays a role of the vss software requester and only triggers VSS provider and VSS writers (since, the same issue you'll see if you use any other backup solution - Microsoft DPM for example).

So, the shadow copy location should be on the same drive - this is a Microsoft limitation.

In the Veeam 6th we skipped guest processing (if it fails) and didn't crash the backup job itself - that was not a good solution. In the same situation the job will fail now.

Thank you for the great and fast support.

Regards
Simon

[foggy]

Simon, thanks for updating the thread with resolution, much appreciated.

[jorgene]

Hi,

Using Veeam 7 B&R for Hyper-V.

When trying to backup a W2012 VM i get this error.

Unable to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Crash consistent). Details: Guest processing skipped (check guest OS VSS state and integration components version)
Error: Guest processing skipped (check guest OS VSS state and integration components version)

I Think it's natural since W2012 have newer version of integration Components natively then W2008R2 has.
One workaround i have done is that i configure "Enable Hyper-V guest quiescence" on the backup job and remove "Enable application-aware image processing option" in the Job.
Then Veeam will take the VM into saved state and can successfully perform a backup of the VM.

But is there a way to take a backup with Enable application-aware image processing in this scenario?
(anyone aware of any updated integration Components to download and install on W2008R2 to match the version on a W2012 guest)?

[foggy]

Jorgen, please try the suggestions mentioned above in this thread. Should help.

[JohnV]

Jorgen,
I have the exact same problem. I am demoing v7 (not yet a Veeam customer) and I cannot get application aware processing to work.

My entire environment is 2012 r2. I have Veeam running on 2012 r2 in a VM and no matter what I do when I enable application aware image processing I get the same error. I am not getting any VSS errors on the VMs, host or proxies.

If I set the job to ignore application processing errors then it reverts back to pausing the VM and backups complete without error. But with it set to require success then it fails. I can't tell if it is even asking VSS to take a snapshot of any of the applications. It all happens so fast that it seems like either a firewall or a permissions error.

Sure wish this thing worked out of the box. Looks like I can't get off of BackupExec just yet.

Bummer.

JV

[foggy]

John, this thread is basically discussing the issue with backing up Windows 2012 guest running on Windows 2008 R2 host, which is not your case, if I understand you correctly. So I recommend you contacting technical support for investigation.

Btw, is there a chance that the failed VM has dynamic disks?

[JohnV]

Foggy,
I originally had v7r2 installed on a 2008R2 box and was getting the exact same error. None of the registry settings mentioned had any effect on it.

I moved v7r2 to a 2012r2 VM with no change in the error.

To answer your question none of my VM's have 'dynamic' disks they are all 'basic' disks. There are 'dynamically expanding virtual disks' in these VM's but no one is saying that is a problem.

[foggy]

I originally had v7r2 installed on a 2008R2 box and was getting the exact same error. None of the registry settings mentioned had any effect on it.
I moved v7r2 to a 2012r2 VM with no change in the error.

Yes, because, once again, the original issue talks about backing up Windows 2012 VM running on Windows 2008 R2 Hyper-V host. (Veeam B&R server OS is out of the picture.)

To answer your question none of my VM's have 'dynamic' disks they are all 'basic' disks. There are 'dynamically expanding virtual disks' in these VM's but no one is saying that is a problem.

Ok, then please provide logs to technical support for investigation.

[Andreas Neufert]

Code: Select all

03.02.2014 12:21:14 :: Unable to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Crash consistent). Details: Guest processing skipped (check guest OS VSS state and integration components version)

Just wanted to add here that when you run Win2012 on Win2012R2 Hyper-V you have to update integration components as well. Normally this is automatically done by Windows Update, but for example if you can not update the VMs (lab environment) the easiest way is to mount c:\windows\system32\vmguest.iso to the VMs and start support\x86\setup.exe to update the integration services.
After that the backups run fine.