No offhost Hyper-V backup proxies are available

[foggy]

Leo, please make sure you've followed the system requirements described in provider installation guide (p.11). Looks like configuration issue (provider cannot detect HBA card).

[LinkeLoe]

I have followed the installation guide, but I think the installation guide assumes that fibre channel is being used... We only use iSCSI, so there are no additional HBA's installed in our servers.

[pluke]

I changed something and now I have a different error message - feel like I am getting closer.
Created a new HyperV VM and on the first screen of the New VM Wizard under "store virtual machine in a different location" I pointed to a folder on the LUN that also holds my VHDs. So now my SAN volume also holds the XML, BIN and VSV files that constitute the VM configuration.

Now the error I get when trying to back up is
"Unable to create snapshot (Nimble Storage VSS Provider on This server) Details: Access is denied. RPC function call failed. Function name: [HvTestConnection]. Target machine: [10.10.10.10]. --tr:Failed to check whether HvIntegration service is available on '10.10.10.10:6163'."
Where 10.10.10.10 is the (obfuscated) IP address of my offhost proxy.

Note that I have made the LUN available to my offhost proxy server as well as the HyperV server - although in Disk Management on the offhost proxy, I have the disk Offline. And when I look at the backup proxy properties, under Connected Volumes, I do see the volume with the same drive letter as it appears to the HyperV server.

So now I think I need to troubleshoot the HvIntegrationService error message. My backup proxy server does have the HyperV role installed, but since it is a physical server and not a VM, I am unable to install the HyperV Integration Services on it.

[Roendi]

Hi Guys
I have exact the same Issiue.

I have installed the rights Hardware Provider Dell Equalogic and my Eventlog is full with this:
Log Name: Application
Source: VSS
Date: 24.04.2012 14:08:40
Event ID: 8194
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: BFU103.bfu.ch
Description:
Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface. hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
Gathering Writer Data


When I select Onhost Backup Veaam will make a clean Snapshot and the Backup works fine.

But Off Host will not working for me.

[Roendi]

Hi Guys

I found for me the working Solution.

First you need a additional HyperV Proxy. So you have this Server and the new Proxy Server is on the first Position.

Then you need Manal Connectet Volumes. For me automatic not will working.

Then you need Diskpart Automount enabled.

At least Patch Nr. 3 must be installed.

We use a Dell Equalogic. So no Chap Account is defined.

For me it works now.

I hope it helps for some User.

Kind Regards

Röndi

[LinkeLoe]

I got word from HP about my VSS problem. For VSS to work with an EVA P6300, you need it connected through Fibre Channel. As we are using iSCSI, the VSS hardware provider cannot find an HBA to mount the snapshot. Transitioning to FC won't be an option as it's not supported for a Hyper-V cluster.

I guess I have to let go the off-host proxy at least until we will be migrating to VMware early 2013...

[Gostev]

Appreciate the update, Leo. You may want to check out what's coming in Windows Server 2012 in regards to Hyper-V enhancements though.

[Andreas Neufert]

Hi,

I had also a problem that I forgot to install the Hardwrae Service provider on the Hyper-V Hosts. After installation of the right hardware provider the job still uses the same wrong driver.
In my case it was the MS Software provider.

Infrastructure - Managed Servers - MS Hyper-V - MS Hyper-V Cluster right mouse Rescan
solved the problem.

CU Andy

[LinkeLoe]

Andy,

My problem had nothing to do with using the wrong VSS driver. I could see the HP VSS provider is being used, however it expects my EVA P6300 being connected through FC and not through iSCSI, so it failed mounting the snapshot.

I am making on-host backups at the moment and that works fine for the moment. As soon as we will be migrating to VMware I will look at the off-host proxy again.

[ccc1884]

Did anyone find a solution for this problem yet. Roendi.. i have the same setup as you but i do not understand your solution. can you explain in detail for me. thank you.

[rush222]

I am having the same issue. Compellent 4020 with Hyper-V 2012 R2. In manage volumes, I clearly see for each volume that I have the compellent hardware VSS provider as well as Microsoft's software VSS provider but when running the job, I keep getting this error:

5/12/2016 11:58:17 PM :: Unable to allocate processing resources. Error: There is no hardware provider for off-host backup detected. Snapshot failover is disabled.

I have notified VEEAM with support case ID: 01796653

[foggy]

Hi Rahid, if everything is configured according to the off-host backup requirements, then let's wait for what our engineers could say after reviewing the logs.

[ceez]

I am also having the issue with the error "Unable to allocate processing resources. Error: There is no hardware provider for off-host backup detected. Snapshot failover is disabled." when trying to setup an off-host proxy. I have an open ticket - 01865017.

from the link you just posted line 3 caught my attention:
"The source Hyper-V host and the off-host backup proxy must be connected (through a SAN configuration) to the shared storage".

It's either too late and I am just over thinking this, but what is considered the hyper-v host? The server that runs Veeam or the hyper-v host that makes up part of my cluster?

also can you help confirm the following
hyper-v server 2012r2 on a cluster
off-host running windows server 2012r2 (not hyper-v server).
^is this proper config for off-host or should my off-host also be running the hyper-v server version?

thanks!

[nefes]

It's either too late and I am just over thinking this, but what is considered the hyper-v host? The server that runs Veeam or the hyper-v host that makes up part of my cluster?

It is your host, that runs VMs to be backed up. To explain it simple, off-host should have read access to the storage, where VM resides, to read it's snapshot, thus backing it up.

hyper-v server 2012r2 on a cluster
off-host running windows server 2012r2 (not hyper-v server).
^is this proper config for off-host or should my off-host also be running the hyper-v server version?

Off-host need to be
a) same windows version, as your production one (2012R2 and 2012R2, your setup is OK)
b) have hyper-v role enabled in any way. off-host need to "understand" what is VM, VM config, disk, etc, to read it correctly. looks like this can be the cause of your problem
also, there is
c) have same hardware provider, as your hosts in cluster, to successfully read data directly from shared storage.

[ceez]

Thanks for clarifying the host question, I thought it was the host running the vm's but wanted to double check.

Off-host need to be
a) same windows version, as your production one (2012R2 and 2012R2, your setup is OK)
b) have hyper-v role enabled in any way. off-host need to "understand" what is VM, VM config, disk, etc, to read it correctly. looks like this can be the cause of your problem
also, there isa
c) have same hardware provider, as your hosts in cluster, to successfully read data directly from shared storage.

a
nodes= hyper-v server 2012r2
off-host= window server 2012r2
my veeam server - running on a physical server with Windows Server 2012 (non-R2) <-does this matter?

b
off-host does have the hyper-v role enabled.

c
hardware provider? as in the storage? the vm's reside in a powervault and the backups are going to a qnap. both hosts and off-host have the powervault mpio drivers. I dont have any qnap drivers. is the qnap the issue? I doubt it since the snapshot needs to be done at the powervault not the qnap, correct? this is how my setup works without the off-host in the picture; the snapshots are done on the PV and the backups saved to the qnap which does no snapshots.

thanks!

[nefes]

a) and b) are ok for you.
c) means that you need Hyper-V VSS Hardware Provider from PowerVault set up on all your cluster nodes and also on off-host. This is usually a separate installation from your primary storage vendor, that allows creating transportable shadow copies (snapshots of volumes, that are made on your production hosts and then "transported" to off-host for further reading). Some storage vendors requires separate license for using such software.

[ceez]

a) and b) are ok for you.
c) means that you need Hyper-V VSS Hardware Provider from PowerVault set up on all your cluster nodes and also on off-host. This is usually a separate installation from your primary storage vendor, that allows creating transportable shadow copies (snapshots of volumes, that are made on your production hosts and then "transported" to off-host for further reading). Some storage vendors requires separate license for using such software.

Yes, I installed the VSS from the PowerVault. It's the same one that I installed on my 3nodes.


I received an update to my Veeam support ticket stating that PowerVault is not supported when it comes to using an off-host proxy, can you confirm/deny this? After questioning that thought from level2 support I've been asked to now gather a bunch of logs so it can be escalated even further up the tier. A little disappointed right now.


Also the LUN is present on the off-host and OFFLINE




And lastly - there are three missing updates that are not applicable to the computer. All my window updates are up to date.

[nefes]

I cannot neither confirm no deny this information right now, since have had no experience with PowerVault before.
For now the only thing left to check (if not checked with support before) is whether you see that hardware provider in VBR -> Backup Infrastructure->HV Clusters in Manage Volumes on your hosts and whether it is selected there. If it is not available - you may want to rescan hosts and check it again.

Meanwhile I'll try to get a little more information about PowerVault and will come back here with it (probably, at the beginning of next week).

[ceez]

This is what that setting looks like now


My single cluster and the 3 nodes.

I am re-scanning again to see if anything changes.

And here is the Microsoft Windows section:


What should I be looking for? You say hardware provider so I am thinking that I should see the powervault listed here?

and no, support didn't even bother doing much other than asking some basic questions. many years ago veeam support seemed more proactive, now it's just wam-bam-thank you ma'm

[nefes]

Right-click on all nodes, select Manage Volumes and see, which provider is selected for your shared volumes. If not PowerVault one, check whether it appears in dropdown and try to select it. If there is no such provider at all - try to rescan hosts (also via rightclick).

If you see correct provider there, go to off-host properties, on first step hit Connected volumes - Choose, switch to Manual and hit Populate. You should also see list of your shared volumes there.

[ceez]

I see the Microsoft CSV Shadow Copy Provider on the cluster volume, the dropdown doesn't have any other options.

[nefes]

So the product don't see this PowerVault provider for your shared volume. Any chance that you have not configured credentials for your shared device after installing provider on hosts?
There should be some utility, that set up with provider, to do it.

[ceez]

An update to my situation which abruptly halts but not due to Veeam.

Spoke to higher tier support member and added a reg key on my Veeam server**:

Code: Select all

HKLM\SOFTWARE\VEEAM\VEEAM BACKUP AND REPLICATION\
RED_DWORD
UIShowAllVssProvidersForVolume
Hexadecimal=1
[i]**remember that YMMV[/i]

Then re-scanned the volumes on each of my 3 nodes.

The above key allows Veeam to see all the VSS providers and was able to choose the hardware provider vss SmVssProvider from the drop down of seen in my previous post picture which only showed Microsoft CSV Shadow Copy Provider.

We ran a test job, this time it got farther but this time it failed with a different error:

Code: Select all

[i]8/1/2016 1:59:46 PM :: Failed to create snapshot SmVssProvider on BKUP01 (mode: Crash consistent) Details: Provider [{24e4e8a3-69cb-4370-8b7b-c5276cd49765}] does not support shadow copies of volume [\\?\Volume{7dbbf4d6-370f-4e4b-868d-d332babc55d8}\].
--tr:Failed to add volumes to the snapshot set.
--tr:Failed to perform pre-backup tasks. 

8/1/2016 1:59:57 PM :: Retrying snapshot creation attempt (Provider [{24e4e8a3-69cb-4370-8b7b-c5276cd49765}] does not support shadow copies of volume [\\?\Volume{7dbbf4d6-370f-4e4b-868d-d332babc55d8}\].
--tr:Failed to add volumes to the snapshot set.
--tr:Failed to perform pre-backup tasks.) 

8/1/2016 1:59:58 PM :: Task has been rescheduled 

8/1/2016 1:59:58 PM :: Queued for processing at 8/1/2016 1:59:58 PM 

8/1/2016 2:01:03 PM :: Unable to allocate processing resources. Error: Provider [{24e4e8a3-69cb-4370-8b7b-c5276cd49765}] does not support shadow copies of volume [\\?\Volume{7dbbf4d6-370f-4e4b-868d-d332babc55d8}\].
--tr:Failed to add volumes to the snapshot set.
--tr:Failed to perform pre-backup tasks.[/i]

this time since we got farther we were able to see the following event log on the node host at the same as the test job:

Code: Select all

[i]Log Name:      Application
Source:        SmProvider
Date:          7/29/2016 1:20:35 PM
Event ID:      3020
Task Category: (2002)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      VSH01.local
Description:
The operation cannot complete because there are not enough free drives or free extents  to create the snapshot repository volume either in a new volume group or in other volume  groups. Please add more free drives and try this operation again. If the base volume is  in a diskpool, then add more free drives in the diskpool and try this operation.
[/i]

Note that this error is all over the application log, just never looked at the node log and if I did they just didn't coincide with the test jobs being ran.

My current PowerVault setup is one big raid 10 (OBR10) using all 12 drives - which sounded great at the time of implementation and I did see a performance boost on our VM's, but hindsight didn't say "hey you might need some extra space for Veeam and PowerVault snapshots *facepalm*
So since I have no extra empty hard drive space anywhere on my PV the snapshot process does not work when the off-host proxy tries to take that snapshot, I have a 10TB LUN with 5TB of free space within that LUN but the PV does not see that free space, it's just seeing that there are no empty non-configured disks for it to use for snapshot creation- and that's where my journey ends and dies, no off-host proxy for me (unless anyone has an awesome idea!!!!).

So if you have a PowerVault MD series, this is a limitation of the software/firmware of the unit. DELL support said that the Equalogic boxes have a setting somewhere where this would not happen (I can't confirm nor deny this).


One last question, so since apparently there's no snapshot taking place on my PV, does it mean that the Snaps are taking place on the node hosts themselves?

Thanks!

[Vitaliy S.]

Yes, snapshot is always created on the source host (Hyper-V) and then depending on the configuration either gets transferred to the off-host proxy or gets processed via on-host proxy.

[smartens]

This reg key worked for me - Thanks!