Failed to Deploy AHV proxy VM v2.0.558 ...

[Didi7]

Hi everybody,

tried my first attempts to install and deploy the AHV Plug-in 2.0.558 and got the following errors while deploying the Proxy Appliance ...

Code: Select all

Failed to Deploy AHV proxy VM: Failed to deploy Proxy - Backup REST error: HTTP StatusCode:BadRequest', Description:'"Unable to connect to VBR Server 'VBRSRV:10006'"'

Creating proxy protection policy : Backup REST error: HTTP StatusCode:BadRequest', Description:'"Unable to connect to VBR Server 'VBRSRV:10006'"'

Deleting proxy appliance : Backup REST error: HTTP StatusCode:BadRequest', Description:'"Unable to connect to VBR Server 'VBRSRV:10006'"'

I completely uninstalled anti-virus software and deactivated the firewall an re-checked DNS-configuration. The proxy is on the same subnet, as the VBR-server and from the same subnet on a different VM, I can successfully do the following in PowerShell ...

Code: Select all

Test-NetConnection VBRSRV -port 10006

I get a 'True' result for TcpTestSucceeded, so port 10006 from the VBR-Server is working from another machine.

Hints like ...

https://vinception.fr/veeam-how-to-reso ... appliance/
http://www.ms4u.info/2020/03/failed-to- ... m-vbr.html

... didn't work either.

The Proxy Appliance is deployed and a boot-screen and login-screen is presented. As soon, as the proxy protection policy creation fails, the Proxy Appliance is powered off and deleted.

Anybody else with these problems? Currently I have no clue at all.

Code: Select all

2020-06-05 14:15:42.379 00066 INFO     | [NutanixCluster]: Getting cluster information [DnsIp:'ntxcluster.colhub.org', Port:'9440']... 
2020-06-05 14:15:42.515 00066 INFO     | [NutanixCluster]: Information from cluster [DnsIp:'ntxcluster.colhub.org', Port:'9440'] has been updated. [ClusterUUID:'00059696-ade2-5d4a-0000-000000029dee', Version:'5.15']  
2020-06-05 14:17:08.837 00090 WARN     | [ProxyManager]: No registered proxies found. 
2020-06-05 14:17:08.883 00090 INFO     | [ApplianceManager]: Appliance has been registered [UID:1f8e4f77-473b-4882-8683-1a104a768f8d]. 
2020-06-05 14:17:08.899 00090 INFO     | [Appliance]: Start of processing image files. 
2020-06-05 14:17:08.899 00090 INFO     | [Appliance]: Start computing the MD5 hash for the image file 'C:\Program Files\Veeam\Plugins\Nutanix AHV\Service\PlatformPlugins\AhvProxyManagement\Images\NutanixAHVPluginProxy_2.0.558.vmdk'... 
2020-06-05 14:17:10.528 00090 INFO     | [Appliance]: Computing the MD5 hash was finished, hash value: '7f66fadb3fccb2d72a5fe8e67a8171a6' 
2020-06-05 14:17:10.528 00090 INFO     | [Appliance]: Check if the VM image file with Name = 'VAN_7f66fadb3fccb2d72a5fe8e67a8171a6' already exists. 
2020-06-05 14:17:10.828 00090 INFO     | [Appliance]: Uploading the VM image file with Name = 'VAN_7f66fadb3fccb2d72a5fe8e67a8171a6'... 
2020-06-05 14:17:26.039 00090 INFO     | [Client]: Start disk image [Name: 'VAN_7f66fadb3fccb2d72a5fe8e67a8171a6'] uploading... 
2020-06-05 14:18:18.635 00090 INFO     | [Appliance]: VM image [UUID:'446424ec-33e5-4ddf-ad36-3553ade53ea8', DeviceIndex:'0'] was marked as bootable. 
2020-06-05 14:18:18.635 00090 INFO     | [Appliance]: VM image UUID is '446424ec-33e5-4ddf-ad36-3553ade53ea8'. 
2020-06-05 14:18:33.954 00090 INFO     | [Appliance]: VM has been created [Name:'VBRPROXY', UUID:'80aa8e7c-6842-401e-aa6a-292464252b90', PowerState: 'OFF'] 
2020-06-05 14:18:49.137 00090 INFO     | [Appliance]: VM [UID:'80aa8e7c-6842-401e-aa6a-292464252b90'] disk was successfully resized [Size: 53687091200]. 
2020-06-05 14:18:49.137 00090 INFO     | [Appliance]: Powering ON VM [ID:'80aa8e7c-6842-401e-aa6a-292464252b90']... 
2020-06-05 14:19:04.323 00090 INFO     | [Appliance]: VM [ID:'80aa8e7c-6842-401e-aa6a-292464252b90'] was powered ON. 
2020-06-05 14:19:04.323 00090 INFO     | [Appliance]: Getting an IP of the VM [ID:'80aa8e7c-6842-401e-aa6a-292464252b90']... 
2020-06-05 14:25:59.455 00090 INFO     | [Appliance]: New IP address ['10.10.10.10'] has been detected for the VM [ID:'80aa8e7c-6842-401e-aa6a-292464252b90']. 
2020-06-05 14:25:59.471 00090 INFO     | [Appliance]: Appliance has been deployed [ID:'1f8e4f77-473b-4882-8683-1a104a768f8d'] 
2020-06-05 14:25:59.471 00090 INFO     | [AhvProxy]: Validate proxy [ID:'', ValidateForStarted:'yes', ValidateCert:'no'] 
2020-06-05 14:25:59.471 00090 INFO     | [AhvProxy]: Update credentials data for proxy [ID:''] 
2020-06-05 14:26:25.801 00090 INFO     | [Appliance]: Image [ID:'446424ec-33e5-4ddf-ad36-3553ade53ea8'] has been removed. 
2020-06-05 14:26:25.849 00090 INFO     | [Appliance]: Apply new configuration for proxy VM [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36'] 
2020-06-05 14:26:25.849 00090 INFO     | [AhvProxy]: Changing default configuration for proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36'] 
2020-06-05 14:26:25.878 00090 INFO     | [AhvProxy]: Applying new configuration to proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36'] 
2020-06-05 14:26:26.385 00090 INFO     | [AhvProxy]: Waiting for the proxy vm to appear [Proxy:'550f97c7-b60d-4656-80e3-39e1f493ec36', Cluster:'00059696-ade2-5d4a-0000-000000029dee', VM:'80aa8e7c-6842-401e-aa6a-292464252b90'] after new configuration has been applied 
2020-06-05 14:26:31.505 00090 INFO     | [AhvProxy]: Update credentials data for proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36'] 
2020-06-05 14:26:37.584 00090 ERROR    | [Client]: PROXY ==> Status:0, Url:https://10.10.10.10:8100/api/v1/ {"content": , "error": An error occurred while sending the request. Error 12029 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'A connection with the server could not be established'.
Request method: GET, parameters: { [sensitive data], type = HttpHeader } 
2020-06-05 14:26:43.984 00090 ERROR    | [Client]: PROXY ==> Status:Unauthorized, Url:https://10.10.10.10:8100/api/v1/Account/whoami {"content": , "error": ""}
Request method: GET, parameters: { [sensitive data], type = HttpHeader } 
2020-06-05 14:26:43.984 00090 INFO     | [AhvProxy]: No active session for proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36'] found. Create new session to proxy ... 
2020-06-05 14:26:44.206 00090 INFO     | [AhvProxy]: New session for proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36'] created 
2020-06-05 14:26:44.412 00090 INFO     | [AhvProxy]: Adding cluster [UUID:'00059696-ade2-5d4a-0000-000000029dee']... 
2020-06-05 14:26:47.679 00090 INFO     | [AhvProxy]: Cluster [UUID:'00059696-ade2-5d4a-0000-000000029dee'] has been registered in proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36']. 
2020-06-05 14:26:47.915 00090 INFO     | [NutanixCluster]: Entities was added to protection domain [Name: 'VeeamBackupProtection']. 
2020-06-05 14:26:47.915 00090 INFO     | [AhvProxy]: AHV Proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36'] VM [ID:'80aa8e7c-6842-401e-aa6a-292464252b90'] was added to PD [Name: 'VeeamBackupProtection'] 
2020-06-05 14:26:48.327 00090 INFO     | [AhvProxy]: PD policy for proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36'] was created. 
2020-06-05 14:26:48.327 00090 INFO     | [AhvProxy]: Ip|DnsName changed for proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36', OldIpDns:'10.10.10.10', NewIpDns:'VBRPROXY'] 
2020-06-05 14:26:48.756 00090 INFO     | [AhvProxy]: Configuration for proxy [DnsIp:'VBRPROXY', Port:'8100'] has been saved. 
2020-06-05 14:26:48.756 00090 INFO     | [AhvProxy]: Proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36', Build_version:'558'] is supported. 
2020-06-05 14:26:48.844 00105 INFO     | [MQ]: Register new message: [Channel:'event/ConnectionPlatformSvcToManagedProxyBecameAvailable', Producer Type:'proxy', Producer Id:'550f97c7-b60d-4656-80e3-39e1f493ec36', Action:'ConnectionPlatformSvcToManagedProxyBecameAvailable', Message Data:'null'] 
2020-06-05 14:26:48.844 00105 INFO     | [MQ]: Register new message: [Channel:'proxy', Producer Type:'proxy', Producer Id:'550f97c7-b60d-4656-80e3-39e1f493ec36', Action:'update', Message Data:'{"clusterUid":"00059696-ade2-5d4a-0000-000000029dee","vmUid":"80aa8e7c-6842-401e-aa6a-292464252b90","maxConcurrentTasks":4,"description":"Created by VBRSRV\\Administrator (Veeam Backup & Replication) at 6/5/2020 2:15 PM.","Uid":"550f97c7-b60d-4656-80e3-39e1f493ec36"}'] 
2020-06-05 14:26:48.859 00090 INFO     | [AhvProxy]: Register backup server [IP: 'VBRSRV'] on proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36']... 
2020-06-05 14:26:48.964 00090 ERROR    | [Client]: PROXY ==> Status:BadRequest, Url:https://VBRPROXY:8100/api/v1/BackupServers {"content": "Unable to connect to VBR Server 'VBRSRV:10006'", "error": ""}
Request method: POST, parameters: { [sensitive data], type = HttpHeader } { name = application/json, value = {"Ip":"VBRSRV","Port":"10006",[sensitive data],[sensitive data],[sensitive data],"@odata.id":null,"Description":"VBRSRV","@Copyright":"Copyright 2015-2018 Veeam Software","@odata.type":"BackupServerSettings"}, type = RequestBody } 
2020-06-05 14:26:48.964 00090 ERROR    | [AhvProxy]: Failed to add backup server to proxy [UUID:'550f97c7-b60d-4656-80e3-39e1f493ec36']. Error ('Backup REST error: HTTP StatusCode:BadRequest', Description:'"Unable to connect to VBR Server 'VBRSRV:10006'"'') Backup REST error: HTTP StatusCode:BadRequest', Description:'"Unable to connect to VBR Server 'VBRSRV:10006'"'
2020-06-05 14:26:48.964 00090 INFO     | [ProxyManager]: Remove proxy from db storage [Id:'550f97c7-b60d-4656-80e3-39e1f493ec36'] 
2020-06-05 14:26:48.989 00090 INFO     | [ProxyManager]: Proxy removed from db storage [Id:'550f97c7-b60d-4656-80e3-39e1f493ec36'] 
2020-06-05 14:26:48.990 00090 INFO     | [InfrastructureManager]: Remove infrastructure [Id:'00059696-ade2-5d4a-0000-000000029dee'] 
2020-06-05 14:26:48.990 00085 INFO     | [AhvProxy]: Detach backup server from proxy [Id:'550f97c7-b60d-4656-80e3-39e1f493ec36'] ... 
2020-06-05 14:26:48.990 00090 INFO     | [InfrastructureManager]: Infrastructure removed [Id:'00059696-ade2-5d4a-0000-000000029dee'] 
2020-06-05 14:26:48.990 00090 ERROR    | [Appliance]: Exception has occurred while deploying appliance. Error ('Backup REST error: HTTP StatusCode:BadRequest', Description:'"Unable to connect to VBR Server 'VBRSRV:10006'"'') 
2020-06-05 14:26:49.033 00085 INFO     | [AhvProxy]: No backup servers were found on proxy [ID:'550f97c7-b60d-4656-80e3-39e1f493ec36']. 
2020-06-05 14:26:49.037 00085 INFO     | [ProxyManager]: Proxy has been removed [Id:'550f97c7-b60d-4656-80e3-39e1f493ec36'] 
2020-06-05 14:27:19.357 00090 INFO     | [Appliance]: VM [ID:'80aa8e7c-6842-401e-aa6a-292464252b90'] has been removed. 
2020-06-05 14:27:19.357 00090 ERROR    | [Plugin:AHV]: Failed to deploy Proxy - Backup REST error: HTTP StatusCode:BadRequest', Description:'"Unable to connect to VBR Server 'VBRSRV:10006'"'
   at Veeam.NutanixBackup.Client.VeeamAhvBackupRestClient.ThrowResultException(IRestResponse response)
   at Veeam.NutanixBackup.Client.VeeamAhvBackupRestClient.TryGetResultFromRequest[T](IRestResponse response)
   at Veeam.NutanixBackup.Client.VeeamAhvBackupRestClient.AddBackupServer(BackupServerSettings serverSettings)
   at StarStack.Platform.Plugin.AhvProxy.ProxyManagement.AhvProxy.RegisterBackupServer(BackupServerSettings serverSettings)
   at StarStack.Platform.Plugin.AhvProxy.ApplianceManagement.ApplianceAhvProxy.SetupService(DtoDeployAhvProxyRequest serviceSettings, IOperationProgress progress)
   at StarStack.Platform.Plugin.AhvProxy.AhvProxyInstance.DeployProxy(DtoDeployAhvProxyRequest proxySettings, IOperationProgress progress)
   at StarStack.Platform.Plugin.AhvProxy.AhvProxyPlugin.DeployProxy(DtoDeployAhvProxyRequest proxySettings, ICommandExecutionContext context)

[Didi7]

https://imgur.com/gyyGaDp

https://imgur.com/NByj3LB

https://imgur.com/MHFeYum

[PTide]

Hi,

Could you open a support case and provide your case ID, please?

Thanks!

[ronnmartin61]

@@Didi7 in 100% of the cases where I've seen this error it's always come back to a name resolution problem. On your VBR server edit "C:\Program Files\Veeam\Plugins\Nutanix AHV\Service\appsettings.json" change "RemoveVmOnFailedDeploy" to false. That will allow the Proxy VM to persist on deploy failure. You can then boot and log onto the proxy and explicitly check to see if the "VBRSRV" hostname is successfully resolve-able.

[Didi7]

@ronnmartin61:
Will check that immediately and post results!

Proxy Deploying in progress!

Thanks

[Didi7]

Ok, VBR-server needs to be rebooted in order to activate ...

"RemoveVMOnFailedDeploy": false

Now, the Proxy Appliance is still there, I can login and ping the ip address of my VBR-server but DNS isn't working, as if he doesn't accept my manual configuration of DNS servers.

VBR successfully accepts the ip-address, subnet mask and gateway, but manual DNS-IPs seem to be overlooked!

[Didi7]

No, DNS or name servers are in /etc/resolv.conf

I can ping the DNS servers, but I can only ping the VBR-server, when adding the DNS-suffix, not without adding the DNS-suffix.

And I suspect this behavior from the beginning.

Creating proxy protection policy (in proxy deployment screen) might fail, because the VBR-server name cannot be reached without the DNS-suffix.

How can this be solved?

[ronnmartin61]

Yes I saw this identical behavior for another customer recently. Since you can resolve the FQDN from the deployed proxy the DNS server is likely set correctly but we're not picking up a correct default search domain. You can of course add the VBR hostname and FQDN to /etc/hosts or alternately edit /etc/networks/interfaces.d/[adapter] and add your default domain e.g. "dns-search my.domain"

[Didi7]

Hi, ok next thing I will try to do. I suppose there is only vi as editor available. Time to regain experience with this wonderful editor

Is there a possibility to supply the Proxy Appliance with a search domain before deploying the Appliance?

Otherwise I suppose, I need to cancel the Proxy Appliance deployment in Veeam B&R and reconnect the appliance once I have done the manual corrections for the search domain via Launch Console (AHV hypervisor) and command line?

Regards
Didi7

[Didi7]

If it's not possible to deliver a search domain in advance before deploying a Proxy Appliance, maybe Veeam need to adjust the wizard and add a field to manually add a search domain, before deploying the Proxy Appliance for Nutanix AHV?

[Didi7]

Hi ronnmartin61,

ok, I got the Proxy Appliance to only fully work and register with the VBR-server by adding the FQDN to the /etc/hosts file and by re-connecting the Proxy Appliance.

Adding ...

dns-search my.domain

to /etc/network/interfaces.d/[adapter] (in this case eth0) doesn't work.

Adding ...

Domains=my.domain

to /etc/systemd/resolved.conf doesn't work either.

(Of course, my.domain was replaced with the real domain names)

For those having the same problem, please note that it is not /etc/networks/interfaces.d/[adapter], it is /etc/network/interfaces.d/[adapter] !

The Nutanix AHV Proxy Appliance v2.0.558 is based on Ubuntu 16.04.6 LTS and offers nano and vi as editors.

Regards,
Didi7

[Didi7]

Feature request for the Nutanix AHV Proxy Appliance ...

Please add a field DNS-Suffices in the Proxy Appliance Deployment wizard beneath the IP address, network mask, gateway and DNS-servers, as otherwise the Proxy Appliance might fail to register with the VBR server and get removed again.

[ericschott]

thanks for the request - noted. In your case having the suffixes would have resolved your issue?

[Didi7]

Hi ericschott,

the Proxy Appliance is working now, thanks to some infos from ronnmartin61 and settings in ...

"C:\Program Files\Veeam\Plugins\Nutanix AHV\Service\appsettings.json"

... and changing ...

"RemoveVmOnFailedDeploy" to false

I always have to cancel the Proxy Appliance Deployment wizard, because the creation of proxy protection policy fails, while trying to contact the VBR-server with the hostname but no DNS-suffix (i.e. no FQDN). Then, the Proxy Appliance wasn't removed and I could login to the Appliance.

Unfortunately, I had no luck adding a DNS-suffix, as I explained above. I could ping the VBR-server using the ip-address and I could ping the VBR-server by using the FQDN.

So, Veeam might need to add a DNS-suffix field in the Proxy Appliance wizard and integrate it somehow into the Ubuntu 16.04.6 LTS, so that the Proxy Appliance can successfully register with the VBR-server and does not fail while 'creating proxy protection policy'.

Btw, I noticed that the Veeam Nutanix AHV Proxy Appliance had one IDE CDROM to which something was still attached but was not ejected. I suppose it is the image to deploy the AHV Proxy Appliance, but was already remove from Nutanix Image Configuration. I also suppose that it's save to eject, what's in the CDROM, as per 'appsettings.json' the "RemoveVmdkImage": is true?

Regards,
Didi7

[Kochkin]

Btw, I noticed that the Veeam Nutanix AHV Proxy Appliance had one IDE CDROM to which something was still attached but was not ejected. I suppose it is the image to deploy the AHV Proxy Appliance, but was already remove from Nutanix Image Configuration. I also suppose that it's save to eject, what's in the CDROM, as per 'appsettings.json' the "RemoveVmdkImage": is true?

The CD-ROM is created by AHV when a CloudInit script is specified (Custom Script checkbox in VM creation window). The drive is not used after installation finish. "RemoveVmdkImage" is not related to it.

[Didi7]

Yes, that's what I supposed, that the CD-ROM is no longer used after the Proxy Appliance installation is finished and why I decided to eject the contents of the CD-ROM.

As the standard Proxy Appliance deployment doesn't work here, because DNS-suffixes are missing and are not part of the deployment wizard and the proxy cannot contact the VBR-server, I need to cancel the wizard, because of the error with registering 'proxy protection policy'. I suppose that, because of cancelling the wizard, the image of the CD-ROM is not automatically ejected as well.

Updates of the Proxy Appliance are also failing, because there is also no proxy configuration available, not in the wizard and not after the Proxy Appliance is deployed via the web console.

DNS-suffix configuration in the wizard is a must. Proxy configuration for updates of the Proxy appliance can also be added in the web console, but without proxy configuration abilities, proxy updates will always fail in environments with only internet access via proxies.

Regards,
Didi7

[radagast]

I am unable to login as root or sudo to root
Tried the default veeam/veeam - no go

[ronnmartin61]

If veeam/veeam doesn't work use the credentials you specified during proxy deployment.

[erushlow]

Hey I was able to correct this by adding the AHV proxy address to the VBR windows server host file C:\Windows\System32\drivers\etc and adding the VBR address to the AHV Proxy host file /etc/hosts

[Didi7]

Hi,

will try that next time.

Regards,
Didi7