Application Aware backups fail using Hardware VSS provider

[RossCrawford]

Hi All,

Support Case ID: 00983814

Recently purchased Veeam so this isn't an issue that has just happened.

We use Hitachi HUS110 storage and a Hyper-V 2012 R2 cluster with 2 nodes.

I have been testing backups and there is strange behaviour when using the Hitachi HW VSS Provider and application aware enabled in the Veeam job. For example I have a 2012 R2 VM that fails when the Windows Firewall is enabled, but works every time when it's disabled, I then have another 2012 R2 VM that doesn't work at all, and another that works with the Firewall enabled. The Firewall settings were all the same.

In the past I've been able to backup some 2008 R2 VMs (up to 6 at once) using this method, but now they all fail so I have a test job that only has one VM at a time to test. These are all off-host with Direct FC to the HUS110, it also fails on-host with Hitachi HW VSS but works with Microsoft Software VSS.

Main error I receive is this:

Code: Select all

06/08/2015 09:12:49 :: Unable to create snapshot Hitachi VSS Hardware Provider on Hyper-V Off-Host Backup Proxy (mode: Veeam application-aware processing). Details: Unknown status of async operation
The shadow copy provider timed out while holding writes to the volume being shadow copied. This is probably due to excessive activity on the volume by an application or a system service. Try again later when activity on the volume is reduced.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks. 

It's very frustrating to troubleshoot as there are so many different reasons from what I've seen online. I don't think it's taking a long time to take the snapshot as the error may suggest, or that there is high I/O..I just think it's unable to use the in VM VSS. Another note, if I disable Application Aware on the Veeam job every backup works.

Thanks
Ross

[Shestakov]

Hello Ross,
Looks like you were using storage snapshots which are not supported for Hitachi so far.

Also note that to create and manage volume shadow copies on the shared storage, you must install a VSS hardware provider that supports transportable shadow copies on the off-host proxy and the Hyper-V host. The VSS hardware provider is usually distributed as a part of client components supplied by the storage vendor.
I would also keep working with the support team.

Thanks!

[RossCrawford]

Hi Shestakov,

Ah ok so as they are not support I assume there may be an issue with the Hitachi snapshot and in guest VSS.

Without Application aware enabled on the Veeam job (just crash consistent) the off-host proxy with Hitachi HW VSS Provider takes a snapshot and does a successful backup, it's only when I tick application aware.

Thanks
Ross

[Shestakov]

Ross,
You can take application-aware snapshots(backups) with Veeam, but not from storage snapshots.
Please also read requirements for a machine performing the role of an off-host backup proxy.
Thanks!

[RossCrawford]

Thanks, is this just a limitation for Hitachi then as I thought you could do this with an off-host direct SAN or is it just with HP,EMC, Netapp for now.

I have everything in place for off-host to work but if that's the case I will have to use on-host without Hareware VSS provider.

[Shestakov]

Note that application-aware processing is supported for Microsoft Windows 2003 and later. Backup with application-aware image processing requires that your guest OS has Hyper-V Integration Services installed.

[RossCrawford]

That's not a problem these are all 2008 R2 and 2012 R2 VMs with the latest integration services installed from the 2012 R2 hosts.

[Shestakov]

Ross,
I apologize for the misleading, I was wrong.
Storage snapshots are not supported for Hitachi applies only to vSphere infrastructure, while works for Hyper-V.

[Vitaliy S.]

06/08/2015 09:12:49 :: Unable to create snapshot Hitachi VSS Hardware Provider on Hyper-V Off-Host Backup Proxy (mode: Veeam application-aware processing). Details: Unknown status of async operation
The shadow copy provider timed out while holding writes to the volume being shadow copied. This is probably due to excessive activity on the volume by an application or a system service. Try again later when activity on the volume is reduced.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.

Do you have the same error when using on-host proxy with Hitachi VSS and application-aware image processing option enabled?

[RossCrawford]

Ross,
I apologize for the misleading, I was wrong.
Storage snapshots are not supported for Hitachi applies only to vSphere infrastructure, while works for Hyper-V.

No problem thanks for the correction.

Do you have the same error when using on-host proxy with Hitachi VSS and application-aware image processing option enabled?

Yes same problem with on host and hardware VSS, strange thing I found a hotfix from Microsoft and installed it in one of the 2008 R2 VMs, rebooted and it now backs up. However I did the same on another VM and it doesn't work.

I also have 2 identical web server VMs on 2008 R2 so application aware isn't really needed but as a test I tried it and it backs up one of them and not the other.

[Vitaliy S.]

Ok, it seems like the issue does not depend on the AAIP option. If all VMs are on the same MS patch level as well as Hyper-V hosts are up-to-date, then let's focus on Hitachi VSS provider troubleshooting. Do you have a support cased opened with Hitachi? Our support team can arrange a WebEx for a joint troubleshooting session.

[RossCrawford]

Hi Vitaliy,

Thanks, I have just opened a ticket with Hitachi and waiting to hear back.

I will update when I receive a case ID.

Thanks
Ross

[RossCrawford]

I think I've found part of the issue here. It appears to be with the SQL part of application aware processing, my two identical web servers I mentioned where one backed up with appware fully enabled and the other didn't, well the one that didn't has SQL reporting installed (DB is on another server). When I had truncate logs enabled it didn't like it and failed, so I left appware enabled require success but unticked truncate and ticked copy only...it then worked using the MS Software provider but still failed with the Hitachi Provider. The Hitachi Provider works with some other VMs which makes me think it's VM or job related.

[Vitaliy S.]

Ross, hardware VSS should work identically with all VMs, so I would bet on VM configuration. Do you have any news from Hitachi support?

[RossCrawford]

Hi Vitaliy,

I think it is VM configuration, but not sure what. For example I have a 2008 R2 VM that backed up fine with Hardware VSS full app aware. Then I had an an identical 2008 R2 VM that wouldn't back up at all with Hardware VSS until I converted it to a VHDX drive. I assume there is an underlying issue with disk signature or something with some VMs that have been made with a template, although it's not consistent across them all.

Only other issue which isn't a big deal is that some VMs that don't have SQL on fail to backup with Truncate SQL set, I can turn this to copy SQL logs only as SQL isn't even installed on it and it will back up just fine.

[Vitaliy S.]

What about the 'vssadmin list writers' command output when the backup job fails? Can you run it inside the VM and in the Hyper-V host?

[RossCrawford]

Well I don't know what I've done this afternoon other than reboot the Veeam server since making any of the previously mentioned changes but now it's working fine!

(It was saying error timed out on the Hyper-V VSS Writer on the host, I think event ID 8 or 9)

Thanks for help but I believe it should be solved now!

[Vitaliy S.]

Ok, please keep us updated on how it runs for the next week. Glad that you're up and running now.

[RossCrawford]

Thanks Vitaliy will do!

[Phreaker]

Is there any support date set for Hitachi SAN?

[Vitaliy S.]

Alan, can you please clarify your request? Any vendor should be supported by Veeam B&R, since we are using VSS writers coming from the storage vendors.