Application Aware backups fail using Hardware VSS provider

Hyper-V specific discussions

Application Aware backups fail using Hardware VSS provider

Veeam Logoby RossCrawford » Thu Aug 06, 2015 10:10 am

Hi All,

Support Case ID: 00983814

Recently purchased Veeam so this isn't an issue that has just happened.

We use Hitachi HUS110 storage and a Hyper-V 2012 R2 cluster with 2 nodes.

I have been testing backups and there is strange behaviour when using the Hitachi HW VSS Provider and application aware enabled in the Veeam job. For example I have a 2012 R2 VM that fails when the Windows Firewall is enabled, but works every time when it's disabled, I then have another 2012 R2 VM that doesn't work at all, and another that works with the Firewall enabled. The Firewall settings were all the same.

In the past I've been able to backup some 2008 R2 VMs (up to 6 at once) using this method, but now they all fail so I have a test job that only has one VM at a time to test. These are all off-host with Direct FC to the HUS110, it also fails on-host with Hitachi HW VSS but works with Microsoft Software VSS.

Main error I receive is this:
Code: Select all
06/08/2015 09:12:49 :: Unable to create snapshot Hitachi VSS Hardware Provider on Hyper-V Off-Host Backup Proxy (mode: Veeam application-aware processing). Details: Unknown status of async operation
The shadow copy provider timed out while holding writes to the volume being shadow copied. This is probably due to excessive activity on the volume by an application or a system service. Try again later when activity on the volume is reduced.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.


It's very frustrating to troubleshoot as there are so many different reasons from what I've seen online. I don't think it's taking a long time to take the snapshot as the error may suggest, or that there is high I/O..I just think it's unable to use the in VM VSS. Another note, if I disable Application Aware on the Veeam job every backup works.

Thanks
Ross
RossCrawford
Influencer
 
Posts: 10
Liked: never
Joined: Thu Aug 06, 2015 9:33 am
Full Name: Ross James Crawford

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby Shestakov » Thu Aug 06, 2015 10:59 am

Hello Ross,
Looks like you were using storage snapshots which are not supported for Hitachi so far.

Also note that to create and manage volume shadow copies on the shared storage, you must install a VSS hardware provider that supports transportable shadow copies on the off-host proxy and the Hyper-V host. The VSS hardware provider is usually distributed as a part of client components supplied by the storage vendor.
I would also keep working with the support team.

Thanks!
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby RossCrawford » Thu Aug 06, 2015 11:02 am

Hi Shestakov,

Ah ok so as they are not support I assume there may be an issue with the Hitachi snapshot and in guest VSS.

Without Application aware enabled on the Veeam job (just crash consistent) the off-host proxy with Hitachi HW VSS Provider takes a snapshot and does a successful backup, it's only when I tick application aware.

Thanks
Ross
RossCrawford
Influencer
 
Posts: 10
Liked: never
Joined: Thu Aug 06, 2015 9:33 am
Full Name: Ross James Crawford

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby Shestakov » Thu Aug 06, 2015 11:12 am

Ross,
You can take application-aware snapshots(backups) with Veeam, but not from storage snapshots.
Please also read requirements for a machine performing the role of an off-host backup proxy.
Thanks!
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby RossCrawford » Thu Aug 06, 2015 11:17 am

Thanks, is this just a limitation for Hitachi then as I thought you could do this with an off-host direct SAN or is it just with HP,EMC, Netapp for now.

I have everything in place for off-host to work but if that's the case I will have to use on-host without Hareware VSS provider.
RossCrawford
Influencer
 
Posts: 10
Liked: never
Joined: Thu Aug 06, 2015 9:33 am
Full Name: Ross James Crawford

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby Shestakov » Thu Aug 06, 2015 11:30 am

Note that application-aware processing is supported for Microsoft Windows 2003 and later. Backup with application-aware image processing requires that your guest OS has Hyper-V Integration Services installed.
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby RossCrawford » Thu Aug 06, 2015 11:51 am

That's not a problem these are all 2008 R2 and 2012 R2 VMs with the latest integration services installed from the 2012 R2 hosts.
RossCrawford
Influencer
 
Posts: 10
Liked: never
Joined: Thu Aug 06, 2015 9:33 am
Full Name: Ross James Crawford

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby Shestakov » Fri Aug 07, 2015 4:48 pm

Ross,
I apologize for the misleading, I was wrong.
Storage snapshots are not supported for Hitachi applies only to vSphere infrastructure, while works for Hyper-V.
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby Vitaliy S. » Fri Aug 07, 2015 5:03 pm

RossCrawford wrote:06/08/2015 09:12:49 :: Unable to create snapshot Hitachi VSS Hardware Provider on Hyper-V Off-Host Backup Proxy (mode: Veeam application-aware processing). Details: Unknown status of async operation
The shadow copy provider timed out while holding writes to the volume being shadow copied. This is probably due to excessive activity on the volume by an application or a system service. Try again later when activity on the volume is reduced.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.

Do you have the same error when using on-host proxy with Hitachi VSS and application-aware image processing option enabled?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby RossCrawford » Sun Aug 09, 2015 12:46 pm

Shestakov wrote:Ross,
I apologize for the misleading, I was wrong.
Storage snapshots are not supported for Hitachi applies only to vSphere infrastructure, while works for Hyper-V.


No problem thanks for the correction.

Vitaliy S. wrote:
RossCrawford wrote:06/08/2015 09:12:49 :: Unable to create snapshot Hitachi VSS Hardware Provider on Hyper-V Off-Host Backup Proxy (mode: Veeam application-aware processing). Details: Unknown status of async operation
The shadow copy provider timed out while holding writes to the volume being shadow copied. This is probably due to excessive activity on the volume by an application or a system service. Try again later when activity on the volume is reduced.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.

Do you have the same error when using on-host proxy with Hitachi VSS and application-aware image processing option enabled?


Yes same problem with on host and hardware VSS, strange thing I found a hotfix from Microsoft and installed it in one of the 2008 R2 VMs, rebooted and it now backs up. However I did the same on another VM and it doesn't work.

I also have 2 identical web server VMs on 2008 R2 so application aware isn't really needed but as a test I tried it and it backs up one of them and not the other.
RossCrawford
Influencer
 
Posts: 10
Liked: never
Joined: Thu Aug 06, 2015 9:33 am
Full Name: Ross James Crawford

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby Vitaliy S. » Sun Aug 09, 2015 7:51 pm

Ok, it seems like the issue does not depend on the AAIP option. If all VMs are on the same MS patch level as well as Hyper-V hosts are up-to-date, then let's focus on Hitachi VSS provider troubleshooting. Do you have a support cased opened with Hitachi? Our support team can arrange a WebEx for a joint troubleshooting session.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby RossCrawford » Mon Aug 10, 2015 10:39 am

Hi Vitaliy,

Thanks, I have just opened a ticket with Hitachi and waiting to hear back.

I will update when I receive a case ID.

Thanks
Ross
RossCrawford
Influencer
 
Posts: 10
Liked: never
Joined: Thu Aug 06, 2015 9:33 am
Full Name: Ross James Crawford

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby RossCrawford » Tue Aug 11, 2015 5:52 pm

I think I've found part of the issue here. It appears to be with the SQL part of application aware processing, my two identical web servers I mentioned where one backed up with appware fully enabled and the other didn't, well the one that didn't has SQL reporting installed (DB is on another server). When I had truncate logs enabled it didn't like it and failed, so I left appware enabled require success but unticked truncate and ticked copy only...it then worked using the MS Software provider but still failed with the Hitachi Provider. The Hitachi Provider works with some other VMs which makes me think it's VM or job related.
RossCrawford
Influencer
 
Posts: 10
Liked: never
Joined: Thu Aug 06, 2015 9:33 am
Full Name: Ross James Crawford

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby Vitaliy S. » Fri Aug 14, 2015 1:57 pm

Ross, hardware VSS should work identically with all VMs, so I would bet on VM configuration. Do you have any news from Hitachi support?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Application Aware backups fail using Hardware VSS provid

Veeam Logoby RossCrawford » Fri Aug 14, 2015 2:11 pm

Hi Vitaliy,

I think it is VM configuration, but not sure what. For example I have a 2008 R2 VM that backed up fine with Hardware VSS full app aware. Then I had an an identical 2008 R2 VM that wouldn't back up at all with Hardware VSS until I converted it to a VHDX drive. I assume there is an underlying issue with disk signature or something with some VMs that have been made with a template, although it's not consistent across them all.

Only other issue which isn't a big deal is that some VMs that don't have SQL on fail to backup with Truncate SQL set, I can turn this to copy SQL logs only as SQL isn't even installed on it and it will back up just fine.
RossCrawford
Influencer
 
Posts: 10
Liked: never
Joined: Thu Aug 06, 2015 9:33 am
Full Name: Ross James Crawford

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: Bing [Bot] and 7 guests