No offhost Hyper-V backup proxies are available

[LinkeLoe]

Hi all,

We have a three node Hyper-V cluster connected to an HP EVA P6300, that we want to backup with Veeam B&R. We also have a physical server with lots of storage, acting as the repository and another phsyical machine with Hyper-V role that I want to use as a backup proxy. I have installed the VSS hardware provider on the backup proxy and all three Hyper-V nodes. I also have selected this VSS hardware provider to use within Veeam B&R (right click on cluster nodes --> manage volumes).

In the backup job I deselected the option to failover to on-host backup if no off-host proxy is available. I want to be sure that the off-host proxy is being used. However, the job fails because "there is no offhost Hyper-V backup proxy available". What steps do I need to take to make sure the offhost backup proxy is being used?

[Gostev]

Hi Leo,

Sounds like you nailed it, and did everything absolutely right except one thing. Did you actually add your physical backup proxy server to Veeam using the Add Hyper-V Proxy wizard?

Thanks.

[LinkeLoe]

Hi Gostev,

Thanks for your reply. Yes, I have added the proxy through the add proxy wizard. In my job properties, I see an offhost backup proxy available...

[LinkeLoe]

One other question: Do I need to present the cluster shared volume LUN's to the backup proxy server? If I review the properties of the backup proxy in Veeam, I do not see any connected volume it can backup VM's from. If I need to present my CSV LUN, isn't there a chance they get corrupted?

[collinp]

In terms of your question i n presenting the cluster share volume LUN's to the backup proxy server, I am not seeing many posts about this. I don't know how many people are successfully doing hardware vss backups with hyper-v. I have the same questions as you. I have an open to EMC, and am trying to explain how all of this works and they are at a loss. They want to throw it back to the backup vendor to figure out. Apparently transportable snapshots have been around for a long time but I don't see a lot of step by step articles on this. I have gotten to the point where Veeam initiates a backup through the offhost proxy using the EMC VSS provider. I see the snapshot being created on my array. But that is about it. I am assuming that Veeam is sending instructions to the EMC array on how to present the snapshot to the offhost proxy and then copy the files over. But that part isn't happening yet.

I don't think you present the storage to both the Hyper-v host and the offhost proxy. As you said, that would cause problems. I am assuming that the offhost proxy is zoned to the storage array but no LUNS are presented to the offhost proxy 'yet'. I am assuming that the Veeam backup process creates a snapshot of the Hyper-v CSV and then temporarily presents this snapshot to the offhost proxy and when the backup is done, unmounts it. How all of this happens on the storage array automatically, I do not know.

[Gostev]

Sorry for delayed response, I have been traveling. Please check to see if you possibly have automount disabled on your offhost backup proxy server. Thanks!

[LinkeLoe]

Gostev,

I have checked the settings of the backup proxy, but I can't find an option 'automount' or something like that. Where can I find it?

Is there something I need to do on my EVA, before the VSS hardware provider will work? Is there anyone who can tell me more about doing hardware vss snapshots on an HP EVA system?

[foggy]

I have checked the settings of the backup proxy, but I can't find an option 'automount' or something like that. Where can I find it?

Open command prompt on backup proxy server and type "diskpart" and then "automount". This will display the automount status.

[LinkeLoe]

I have enabled automount on the backup proxy, but I still get the error message...

Like I wrote, are there additional steps required to enable vss hardware snapshots of my EVA system?

[Gostev]

No additional steps are required typically, but there might be device-specific requirements. I recommend you open a support case and let our technical staff take a look at this in live.

[collinp]

Hopefully I am not hijacking this thread but the subject matter seemed similar enough. I have a case open with EMC on hardware VSS snapshots. I also just opened a case with Veeam (5184531) since EMC would like to talk with Veeam. What we are seeing is interesting. From an EMC perspective, the EMC logs are blank on the offhost proxy, even though I have the Veeam job specifically set to use the offhost proxy. All of the SAN activity is happening on the Hyper-v host itself. If you look at the Hyper-v host, the coordination of the EMC SAN snapshot, etc is happening from the Hyper-v host. The snapshot is getting created on the SAN successfully but isn't getting mounted on the offhost proxy. The EMC technician said that every other backup product that they have worked with use the Backup Proxy to coordinate all of the snapshot activity. Is this how it is supposed to work?

Not that anyone would understand these EMC logs. But you can get an idea of the activity happening on the Hyper-v host. The SAN lun is PhysicalDrive3. Again, the EMC logs are blank on the offhost proxy.

Code: Select all

15:48:02 04/10/2012 UnmaskLUNS: AddAccessToDev failed ... 
15:48:02 04/10/2012 UnmaskLUNS: Failed to add access to the device <Device uid="600601607760250008d8810f5f83e111" <Array type="CLARiiON" encoding="String" value="APM00092503853" /> />. 
15:48:02 04/10/2012 UnmaskLUNS:Sym is non Tigon, proceed with StorMaskSessionEnd... 
15:48:02 04/10/2012 UnmaskLUNS: Leaving with FAILURE. 
15:48:02 04/10/2012 LocateLuns: Failed to perform pre unmasking tasks. Returning with failure! 
15:48:02 04/10/2012 OnLunEmpty: Entering. 
15:48:02 04/10/2012 IsLocalHostVM: Entering... 
15:48:02 04/10/2012 IsLocalHostVM: Leaving with success. 
15:48:02 04/10/2012 ::ExtractPairingFromLUNInfo: Entering. 
15:48:02 04/10/2012 ::ExtractPairingFromLUNInfo: Leaving with SUCCESS. 
15:48:02 04/10/2012 OnLunEmpty: Calling iArraySync for <Array type="CLARiiON" encoding="String" value="APM00092503853" />. 
15:48:02 04/10/2012 iArraySync: Entering. 
15:48:02 04/10/2012 iArraySync: StorArraySync(-1) for <Array type="CLARiiON" encoding="String" value="APM00092503853" />. 
15:48:20 04/10/2012 iArraySync: Leaving with SUCCESS. 
15:48:20 04/10/2012 OnLunEmpty: Calling StorDevShow for <Device uid="600601607760250008d8810f5f83e111" <Array type="CLARiiON" encoding="String" value="APM00092503853" /> /> on Array <Array type="CLARiiON" encoding="String" value="APM00092503853" />. 
15:48:20 04/10/2012 OnLunEmpty: Removing access to . 
15:48:20 04/10/2012 RemoveAccessToDev: Entering. 
15:48:20 04/10/2012 RemoveAccessToDev: Calling StorDevShow(2593) for <Device uid="600601607760250008d8810f5f83e111" /> on Array <Array type="CLARiiON" encoding="String" value="APM00092503853" />. 
15:48:20 04/10/2012 RemoveAccessToDev: Calling StorDeviceIdConvert to get name of device = <Device uid="600601607760250008d8810f5f83e111" />. 
15:48:20 04/10/2012 RemoveAccessToDev: Found that <Device uid="600601607760250008d8810f5f83e111" /> is not in a storage group, so returning successfully. 
15:48:20 04/10/2012 OnLunEmpty: Calling StorSnapControl for SRC=<Device name="00035" <Array type="CLARiiON" encoding="String" value="APM00092503853" /> /> and TGT=<Device name="VSS_SEHYPERVA_00035_154719_04102012" /> on <Array type="CLARiiON" encoding="String" value="APM00092503853" />. 
15:48:40 04/10/2012 OnLunEmpty: StorSnapControl failed to terminate <Device name="00035" <Array type="CLARiiON" encoding="String" value="APM00092503853" /> />-><Device name="VSS_SEHYPERVA_00035_154719_04102012" /> in Array <Array type="CLARiiON" encoding="String" value="APM00092503853" />.
ERROR: [1100007]:A device attribute makes its use invalid for the requested operation. [4580]:The device specified cannot be used as it is in a storage group. The device specified cannot be used as it is in a storage group.  
15:48:40 04/10/2012 AbortSnapshots: Entering. 
15:48:40 04/10/2012 CancelSnapshotSet: Entering. 
15:48:40 04/10/2012 ~CSnapshotSet: Entering destructor. 
15:48:40 04/10/2012 CSnapshotSet::FreeArrayData() entering. 
15:48:40 04/10/2012 CSnapshotSet::FreeArrayData() exiting. 
15:48:40 04/10/2012 CSnapshotSet::FreeLunData() entering. 
15:48:40 04/10/2012 CSnapshotSet::FreeLunData() exiting. 
15:48:40 04/10/2012 ~CSnapshotSet: Leaving destructor with success. 
15:48:40 04/10/2012 CancelSnapshotSet: Leaving with SUCCESS.

[collinp]

I was able to clean some things up on the EMC side and now I am down to the same error message that everyone is getting.

"Unable to allocate processing resources. Error: No offhost Hyper-V backup proxies are avaliable."

My offhost proxy is available and has 2008 r2 with the hyper-v roll and has been added as an offhost proxy. automount was enabled in diskpart.

How are we all getting this error across different arrays and environments? We must be missing something

[LinkeLoe]

In the meantime I have opened a support case with Veeam. I will post any updates on this matter in this thread. It's good to see that I'm not the only one with this problem. We surely must be missing here... Is there anyone who has setup an Hyper-V proxy succesfully?

[foggy]

Guys, just to confirm - can you see the LUNs in Disk Management on offhost proxy machine? You need to present the LUNs to the backup proxy server (they will not get corrupted if you won't bring them online, just get them in Disk Management snap-in).

[LinkeLoe]

LUN's are not presented at the moment, but I have tried making a backup with the LUN's presented without any success. I keep getting the same error message...

[LinkeLoe]

I just tried to make a backup again with the LUN's presented, but it fails with the same message.

When I investigate the properties of the backup proxy, I do not see any volumes Veeam can backup VM files from. Even if I want to select a volume manualy and populate the volumes, nothing happens...

[foggy]

Leo, have you tried to rescan the storage (Veeam GUI > right-click Backup Infrastructure > Rescan Storage)? If this does not help either, then please proceed investigating with support, looks like the issue is not on the surface.

[LinkeLoe]

I did a rescan of the storage as you suggested, but still no luck... I still don't see any volumes in the properties on the backup proxy server and the backup is still failing...

[collinp]

Has anyone else presented the LUNS to both the Hyper-v host(s) and the Proxy server? NTFS is not a clusterable file system. I believe I was reading where automount is supposed to be enabled on the Proxy server. I'm afraid that the Proxy server would try to mount the Hyper-v LUN and cause corruption.

[LinkeLoe]

As suggested by foggy, I presented my CSV's to both the Hyper-V nodes and the backup proxy. As long as the CSV's remain offline on the backup proxy, they don't get corrupted. I believe the automount is necessary to mount the snapshot of the CSV's.

Even after presenting the CSV's to the backup proxy, I get the message there's no proxy available.

[pluke]

I have a very similar set of symptoms to collinp - presented LUNs to both HyperV and the backup proxy, the proxy config actually shows the shared volumes (they show with the drive letters as they appear on the HyperV host), I made sure all my VHDs were on these shared volumes, but backup fails after 30 seconds with "no offhost backup proxies are available." Also tried installing the hardware VSS provider from my SAN vendor (Nimble) on both HyperV host and backup proxy, this did not help.

Stumped. Feel like there is a subtle missing requirement or an incorrect assumption we are all making. I will open a support ticket but would appreciate feedback from others who figure out how to make this work. Cheers.

[collinp]

I actually can't see the Hyper-v LUNs on the backup proxy although I'm sure I would be able to see them if I chose to present the Hyper-v LUNs to it. I found out that the LUNS that I was seeing were just initiator records. If you look at the SAN logs on the offhost proxy, the logs are completely blank. I don't believe the Veeam Backup Server is even talking to the offhost proxy, thus the message "no offhost backup proxies are available". I don't understand why the Backup Proxy would need to see the Hyper-v CSV LUN. I thought it only needs to see the snapshot lun and backup the snapshot lun. Isn't this the whole point? Hyper-v keeps running off of the CSV, while the offhost proxy backups up the snapshot LUN.

One thing I did try was creating a Hyper-v "onhost" backup proxy. I don't know if using the onhost proxy is supported with transportable snapshots. Anyhow, when trying this method, I do see the EMC VSS Logs are being written to. It is going through the steps to try to make a snpashot. So I think theoretically, this can work but I just don't think the Veeam server is able to contact an offhost proxy using hardware vss without a setting change or hotfix of some sort. Here is the error I got when trying to do a backup using the emc vss provider with the onhost proxy. Again, this is more promising then when trying with using an offhost proxy where the EMC vss and hwprov logs are completely blank.

Code: Select all

Unable to create snapshot (EMC VSS Hardware Provider on This server) Details: Access is denied RPC function call failed. Function name: [HvTestConnection]. Target machine: [<Hyperv host IP>]. --tr:Failed to check whether HvIntegration service is available on '<hyperv host IP>:6163'.
Unable to allocate processing resources. Error: Unable to allocate resources for taskId: '0d6725a3-300e-419d-9bc7-3590d221a263'

[LinkeLoe]

I have also tried making an on-host backup using the HP EVA VSS hardware provider, but my backup also fails. When I check the Windows application log on the host where the backed up VM resides, I see several HP VSS specific events, but I also see these two events:

Code: Select all

Log Name:      Application
Source:        VSS
Date:          4/13/2012 7:22:38 PM
Event ID:      8194
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AMS5-CL01-SRV01.root.local
Description:
Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process. 

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {8466de0a-0d0d-4062-bd8e-7758662fb318}
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="VSS" />
    <EventID Qualifiers="0">8194</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-04-13T17:22:38.000000000Z" />
    <EventRecordID>4020</EventRecordID>
    <Channel>Application</Channel>
    <Computer>AMS5-CL01-SRV01.root.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>0x80070005, Access is denied.
</Data>
    <Data>

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {8466de0a-0d0d-4062-bd8e-7758662fb318}</Data>
    <Binary>2D20436F64653A20575254575254494330303030313236302D2043616C6C3A20575254575254494330303030313231342D205049443A202030303030313039322D205449443A202030303030313237322D20434D443A2020433A5C57696E646F77735C73797374656D33325C737663686F73742E657865202D6B204E6574776F726B53657276696365202020202020202D20557365723A204E616D653A204E5420415554484F524954595C4E4554574F524B20534552564943452C205349443A532D312D352D3230</Binary>
  </EventData>
</Event>

Code: Select all

Log Name:      Application
Source:        Microsoft-Windows-WMI
Date:          4/13/2012 7:23:03 PM
Event ID:      5605
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AMS5-CL01-SRV01.root.local
Description:
The Root\MSCluster namespace is marked with the RequiresEncryption flag. Access to this namespace might be denied if the script or application does not have the appropriate authentication level. Change the authentication level to Pkt_Privacy and run the script or application again.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-WMI" Guid="{1edeee53-0afe-4609-b846-d8c0b2075b1f}" EventSourceName="WinMgmt" />
    <EventID Qualifiers="49152">5605</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-04-13T17:23:03.000000000Z" />
    <EventRecordID>4021</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>AMS5-CL01-SRV01.root.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Root\MSCluster</Data>
  </EventData>
</Event>

Strangely enough, I also see these two events when I try to do an off-host backup. It seems that in the case of an off-host backup, the cluster node tries to snapshot the CSV, but that fails. The backup proxy cannot mount the snapshot and therefore the backup fails...

Monday I am having a WebEx session with tech support, I hope this issue can finally be resolved then...

[collinp]

The logs that I was referring to are those from the EMC Navicli software. I'm not sure about the EVA, but there was some EMC software that I had to install on the Backup proxy to authorize it to communicate with the Storage Array. I had to install the emc navicli software and execute the symauth command to authorize the proxy host to communicate with the array. I also had to install the EMC VSS provider on the offhost proxy. I can actually go to the snapshot pane and watch the snapshot being created on my array after I execute the Veeam backup job. If I look at the EMC hwprov logs (part of the navicli software) on each host during the backup process all of the coordination for the snapshot is being done from the Hyper-v server itself. There is also some coordination happening on the Veeam Backup and Replication server when I choose "onhost". But even when I choose "offhost" proxying, the Veeam offhost proxy is not even in the mix.

[LinkeLoe]

Can you check you Windows application logs for the same error messages? I do not see a snapshot being created on the EVA... I know a separate license is needed for EVA, but we are still in trial for this license, so it should work. Perhaps there's something missing on the EVA side in my setup...

[collinp]

Yes, I get both of these errors in the application log of my hyper-v hosts. I only get the "no offhost available" error messages when under proxy settings I have "connected volumes" set to automatic. I do not see the snapshot being created on my array with connected volumes set to automatic. However, when I manually select the volume to the CSV volume under connected volumes, I then see the snapshot being created on my array and Veeam no longer gives me the "no offhost available" message. It gives me a completely different message of:

Unable to create snapshot (EMC VSS Hardware Provider on offhost proxy) Details: Unknown status of async operation The shadow copy provider had an error. Check the System and Application event logs for more information. --tr:Failed to create VSS snapshot. --tr:Failed to perform pre-backup tasks.
Unable to allocate processing resources. Error: Unable to allocate resources for taskId: 'a966b127-637d-4b26-8ac0-f8c47203548c'

I am in the process of presenting Hyper-v LUNs to offhost proxy.

[LinkeLoe]

Presenting CSV to off-host proxy won't solve the problem... This is something I've done several times with different settings. I believe we have to look for the solution in the two events I posted earlier. It must be something with te "requiresencryption" flag. It's just that to turn it off, I have to edit a file, wich is in use by the WMI service. Stopping this service will also stop Hyper-V services. I'll try it tomorrow evening by entering maintenance mode on one host. This solution isn't officially supported by Microsoft, however...

Another thing I'll try is presenting a LUN to one of the hosts, wich is not a CSV. I can put a newly created VM on it and try to snapshot that LUN, then see if I get the same error messages in the application log...

[collinp]

I am not going to put any more time into this so please let me know how your research goes. I agree that presenting the lun to the hyper-v hosts and the offhost proxy at the same time will not solve this issue. What I am seeing from a storage standpoint is that the snapshot gets created on the array but the array isn't receiving the guidance necessary in order to present the snapshot LUN to the offhost proxy. The EMC tech said that he had a lot of experience with transportable snapshots with Backup Exec, Networker, etc and that the way that it usually works is that the snapshot lun gets presented to the offhost proxy for backup.

[foggy]

Guys, apparently there is an issue with detecting offhost proxies in some situations. We are currently investigating based on your support cases, I will get back to you as soon as we have more information.

[LinkeLoe]

After investigation it turns out that I had the wrong VSS hardware providers installed. We have an EVA P6300, so I installed the providers for EVA 6x00, but I should have installed providers for EVA P6000...

I can see now that a snapshot is being created on the EVA, but it is immediately deleted and the backup fails. On the physical hosts I see the following message from HP VSS provider in the application log:
Shadowcopy import failed, Unable to find any HBA ports on local server.