No offhost Hyper-V backup proxies are available

Hyper-V specific discussions

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby LinkeLoe » Thu Apr 12, 2012 3:10 pm

I just tried to make a backup again with the LUN's presented, but it fails with the same message.

When I investigate the properties of the backup proxy, I do not see any volumes Veeam can backup VM files from. Even if I want to select a volume manualy and populate the volumes, nothing happens...
LinkeLoe
Enthusiast
 
Posts: 35
Liked: never
Joined: Fri Apr 06, 2012 8:31 pm
Full Name: Leo Schroer

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby foggy » Thu Apr 12, 2012 3:54 pm

Leo, have you tried to rescan the storage (Veeam GUI > right-click Backup Infrastructure > Rescan Storage)? If this does not help either, then please proceed investigating with support, looks like the issue is not on the surface.
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby LinkeLoe » Thu Apr 12, 2012 5:14 pm

I did a rescan of the storage as you suggested, but still no luck... I still don't see any volumes in the properties on the backup proxy server and the backup is still failing...
LinkeLoe
Enthusiast
 
Posts: 35
Liked: never
Joined: Fri Apr 06, 2012 8:31 pm
Full Name: Leo Schroer

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby collinp » Thu Apr 12, 2012 8:13 pm

Has anyone else presented the LUNS to both the Hyper-v host(s) and the Proxy server? NTFS is not a clusterable file system. I believe I was reading where automount is supposed to be enabled on the Proxy server. I'm afraid that the Proxy server would try to mount the Hyper-v LUN and cause corruption.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby LinkeLoe » Thu Apr 12, 2012 8:21 pm

As suggested by foggy, I presented my CSV's to both the Hyper-V nodes and the backup proxy. As long as the CSV's remain offline on the backup proxy, they don't get corrupted. I believe the automount is necessary to mount the snapshot of the CSV's.

Even after presenting the CSV's to the backup proxy, I get the message there's no proxy available.
LinkeLoe
Enthusiast
 
Posts: 35
Liked: never
Joined: Fri Apr 06, 2012 8:31 pm
Full Name: Leo Schroer

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby pluke » Fri Apr 13, 2012 1:15 am

I have a very similar set of symptoms to collinp - presented LUNs to both HyperV and the backup proxy, the proxy config actually shows the shared volumes (they show with the drive letters as they appear on the HyperV host), I made sure all my VHDs were on these shared volumes, but backup fails after 30 seconds with "no offhost backup proxies are available." Also tried installing the hardware VSS provider from my SAN vendor (Nimble) on both HyperV host and backup proxy, this did not help.

Stumped. Feel like there is a subtle missing requirement or an incorrect assumption we are all making. I will open a support ticket but would appreciate feedback from others who figure out how to make this work. Cheers.
pluke
Novice
 
Posts: 3
Liked: never
Joined: Wed Apr 04, 2012 11:10 pm
Full Name: Luke

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby collinp » Fri Apr 13, 2012 4:40 pm 1 person likes this post

I actually can't see the Hyper-v LUNs on the backup proxy although I'm sure I would be able to see them if I chose to present the Hyper-v LUNs to it. I found out that the LUNS that I was seeing were just initiator records. If you look at the SAN logs on the offhost proxy, the logs are completely blank. I don't believe the Veeam Backup Server is even talking to the offhost proxy, thus the message "no offhost backup proxies are available". I don't understand why the Backup Proxy would need to see the Hyper-v CSV LUN. I thought it only needs to see the snapshot lun and backup the snapshot lun. Isn't this the whole point? Hyper-v keeps running off of the CSV, while the offhost proxy backups up the snapshot LUN.

One thing I did try was creating a Hyper-v "onhost" backup proxy. I don't know if using the onhost proxy is supported with transportable snapshots. Anyhow, when trying this method, I do see the EMC VSS Logs are being written to. It is going through the steps to try to make a snpashot. So I think theoretically, this can work but I just don't think the Veeam server is able to contact an offhost proxy using hardware vss without a setting change or hotfix of some sort. Here is the error I got when trying to do a backup using the emc vss provider with the onhost proxy. Again, this is more promising then when trying with using an offhost proxy where the EMC vss and hwprov logs are completely blank.

Code: Select all
Unable to create snapshot (EMC VSS Hardware Provider on This server) Details: Access is denied RPC function call failed. Function name: [HvTestConnection]. Target machine: [<Hyperv host IP>]. --tr:Failed to check whether HvIntegration service is available on '<hyperv host IP>:6163'.
Unable to allocate processing resources. Error: Unable to allocate resources for taskId: '0d6725a3-300e-419d-9bc7-3590d221a263'
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby LinkeLoe » Fri Apr 13, 2012 5:58 pm

I have also tried making an on-host backup using the HP EVA VSS hardware provider, but my backup also fails. When I check the Windows application log on the host where the backed up VM resides, I see several HP VSS specific events, but I also see these two events:

Code: Select all
Log Name:      Application
Source:        VSS
Date:          4/13/2012 7:22:38 PM
Event ID:      8194
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AMS5-CL01-SRV01.root.local
Description:
Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {8466de0a-0d0d-4062-bd8e-7758662fb318}
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="VSS" />
    <EventID Qualifiers="0">8194</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-04-13T17:22:38.000000000Z" />
    <EventRecordID>4020</EventRecordID>
    <Channel>Application</Channel>
    <Computer>AMS5-CL01-SRV01.root.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>0x80070005, Access is denied.
</Data>
    <Data>

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {8466de0a-0d0d-4062-bd8e-7758662fb318}</Data>
    <Binary>2D20436F64653A20575254575254494330303030313236302D2043616C6C3A20575254575254494330303030313231342D205049443A202030303030313039322D205449443A202030303030313237322D20434D443A2020433A5C57696E646F77735C73797374656D33325C737663686F73742E657865202D6B204E6574776F726B53657276696365202020202020202D20557365723A204E616D653A204E5420415554484F524954595C4E4554574F524B20534552564943452C205349443A532D312D352D3230</Binary>
  </EventData>
</Event>


Code: Select all
Log Name:      Application
Source:        Microsoft-Windows-WMI
Date:          4/13/2012 7:23:03 PM
Event ID:      5605
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AMS5-CL01-SRV01.root.local
Description:
The Root\MSCluster namespace is marked with the RequiresEncryption flag. Access to this namespace might be denied if the script or application does not have the appropriate authentication level. Change the authentication level to Pkt_Privacy and run the script or application again.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-WMI" Guid="{1edeee53-0afe-4609-b846-d8c0b2075b1f}" EventSourceName="WinMgmt" />
    <EventID Qualifiers="49152">5605</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-04-13T17:23:03.000000000Z" />
    <EventRecordID>4021</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>AMS5-CL01-SRV01.root.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Root\MSCluster</Data>
  </EventData>
</Event>


Strangely enough, I also see these two events when I try to do an off-host backup. It seems that in the case of an off-host backup, the cluster node tries to snapshot the CSV, but that fails. The backup proxy cannot mount the snapshot and therefore the backup fails...

Monday I am having a WebEx session with tech support, I hope this issue can finally be resolved then...
LinkeLoe
Enthusiast
 
Posts: 35
Liked: never
Joined: Fri Apr 06, 2012 8:31 pm
Full Name: Leo Schroer

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby collinp » Fri Apr 13, 2012 6:34 pm

The logs that I was referring to are those from the EMC Navicli software. I'm not sure about the EVA, but there was some EMC software that I had to install on the Backup proxy to authorize it to communicate with the Storage Array. I had to install the emc navicli software and execute the symauth command to authorize the proxy host to communicate with the array. I also had to install the EMC VSS provider on the offhost proxy. I can actually go to the snapshot pane and watch the snapshot being created on my array after I execute the Veeam backup job. If I look at the EMC hwprov logs (part of the navicli software) on each host during the backup process all of the coordination for the snapshot is being done from the Hyper-v server itself. There is also some coordination happening on the Veeam Backup and Replication server when I choose "onhost". But even when I choose "offhost" proxying, the Veeam offhost proxy is not even in the mix.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby LinkeLoe » Fri Apr 13, 2012 6:39 pm

Can you check you Windows application logs for the same error messages? I do not see a snapshot being created on the EVA... I know a separate license is needed for EVA, but we are still in trial for this license, so it should work. Perhaps there's something missing on the EVA side in my setup...
LinkeLoe
Enthusiast
 
Posts: 35
Liked: never
Joined: Fri Apr 06, 2012 8:31 pm
Full Name: Leo Schroer

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby collinp » Fri Apr 13, 2012 8:34 pm

Yes, I get both of these errors in the application log of my hyper-v hosts. I only get the "no offhost available" error messages when under proxy settings I have "connected volumes" set to automatic. I do not see the snapshot being created on my array with connected volumes set to automatic. However, when I manually select the volume to the CSV volume under connected volumes, I then see the snapshot being created on my array and Veeam no longer gives me the "no offhost available" message. It gives me a completely different message of:

Unable to create snapshot (EMC VSS Hardware Provider on offhost proxy) Details: Unknown status of async operation The shadow copy provider had an error. Check the System and Application event logs for more information. --tr:Failed to create VSS snapshot. --tr:Failed to perform pre-backup tasks.
Unable to allocate processing resources. Error: Unable to allocate resources for taskId: 'a966b127-637d-4b26-8ac0-f8c47203548c'

I am in the process of presenting Hyper-v LUNs to offhost proxy.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby LinkeLoe » Fri Apr 13, 2012 8:52 pm

Presenting CSV to off-host proxy won't solve the problem... This is something I've done several times with different settings. I believe we have to look for the solution in the two events I posted earlier. It must be something with te "requiresencryption" flag. It's just that to turn it off, I have to edit a file, wich is in use by the WMI service. Stopping this service will also stop Hyper-V services. I'll try it tomorrow evening by entering maintenance mode on one host. This solution isn't officially supported by Microsoft, however...

Another thing I'll try is presenting a LUN to one of the hosts, wich is not a CSV. I can put a newly created VM on it and try to snapshot that LUN, then see if I get the same error messages in the application log...
LinkeLoe
Enthusiast
 
Posts: 35
Liked: never
Joined: Fri Apr 06, 2012 8:31 pm
Full Name: Leo Schroer

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby collinp » Fri Apr 13, 2012 9:35 pm

I am not going to put any more time into this so please let me know how your research goes. I agree that presenting the lun to the hyper-v hosts and the offhost proxy at the same time will not solve this issue. What I am seeing from a storage standpoint is that the snapshot gets created on the array but the array isn't receiving the guidance necessary in order to present the snapshot LUN to the offhost proxy. The EMC tech said that he had a lot of experience with transportable snapshots with Backup Exec, Networker, etc and that the way that it usually works is that the snapshot lun gets presented to the offhost proxy for backup.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby foggy » Sun Apr 15, 2012 8:48 pm

Guys, apparently there is an issue with detecting offhost proxies in some situations. We are currently investigating based on your support cases, I will get back to you as soon as we have more information.
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: No offhost Hyper-V backup proxies are available

Veeam Logoby LinkeLoe » Wed Apr 18, 2012 8:51 am

After investigation it turns out that I had the wrong VSS hardware providers installed. We have an EVA P6300, so I installed the providers for EVA 6x00, but I should have installed providers for EVA P6000...

I can see now that a snapshot is being created on the EVA, but it is immediately deleted and the backup fails. On the physical hosts I see the following message from HP VSS provider in the application log:
Shadowcopy import failed, Unable to find any HBA ports on local server.
LinkeLoe
Enthusiast
 
Posts: 35
Liked: never
Joined: Fri Apr 06, 2012 8:31 pm
Full Name: Leo Schroer

PreviousNext

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 7 guests