VMware VIX app aware backups failing after 6.5 to 6.7 upgrade

[svallance]

We're a cloud provider and provide backups for our hosted customers. We use vCloud and NSX to segment and isolate our customer environments, so our backup appliance/proxies don't have any direct network connection to the customer VMs.

So we're using VIX to get app aware vCloud backups.

We've got the registry keys set to make the appliance default to VIX first, and we've had our customers disable UAC in their environments so we can use non-named-'administrator' service accounts.

These backups were running successfully up until we upgraded our hosts and vCenters from 6.5 to 6.7 this past month.

Now all of our hosted customer backups are failing to do app aware processing with the following error:

Failed to prepare guest for hot backup. Details: Cannot connect to host [10.18.8.43] over web services. Login: [vsphere.local\veeam_backup_service]. Guest Login: [veeamserviceaccount@customer.domain]. Could not copy host file [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\VeeamGuestHelpers\VeeamVixProxy.exe] to guest [C:\Users\CENTER~1.COM\AppData\Local\Temp\{509178d5-5528-40c3-a9a5-4eb2c96f58ee}] Could not copy host file [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\V
Failed to index guest file system. Veeam Guest Agent is not started


I've had a support case, Case # 04102720, open for a little over a week now and besides asking for logs and suggesting reboots I'm not really getting any feedback or indication the issue's being looked into. It feels like pulling teeth requesting updates, which is a far cry from what I'm used to getting with Veeam support.

In the VMware logs I can see the logins are succesful and I'm even able to connect to the vCenters through VMware powercli and upload the same file to the same directory from the same server.

I'd open a support case with VMware too, but I don't see any errors or indication the issue on the VMware side.

We actually added a host to the cluster in the last couple days and some of the VMs were moved over by DRS. It seems to be the same version, network, and configuration, but all of those VMs on the new host are successfully doing app aware backups. The only difference seems to be that the other hosts were upgraded and this was a fresh install.

Based on the documentation, older support cases, and forum posts, plus the fact that it was working before the upgrades, we should be meeting all the requirements for VIX to work.

Has anyone had similar issues or know of gotchas upgrading to 6.7 causing VIX app aware backups to fail?


TL;DR: Upgraded hosts and vCenter from 6.5 to 6.7 and now VIX backups won't work, except for VMs on one host that was a fresh install of 6.7.
Case # 04102720

[PetrM]

Hi Shane,

I believe that there are many possible reasons of this error, may be even not related to recent upgrade, for example connectivity issues between Veeam server and VMware hosts.

I would recommend to escalate the case, we should define the exact reason of the issue. As a temporal workaround you may take a look at guest interaction proxy to avoid usage of VIX.

Thanks!

[svallance]

Thanks Petr, looks like maybe you were able to reach out to get someone to reply. Or we lucked out and finally got a response 24+ hours later on our sev 2 case.
Support suggested a few more things, including moving VMs from the one host that is working to one's where it isn't to see if by not making any changes the issue is somehow fixed, and then restarting management agents on the ESXI hosts, unfortunately it's still not working. I guess I’ll need to request an escalation to hopefully avoid another day long wait for a response.

Do you have more information on what you mean about the guest interaction proxy? From the article you linked it says:
"Guest interaction proxy must have either a LAN or VIX connection to the VM that will be processed. You do not have to set up both connections — only one connection is required."
Which is what we're doing, but you mention that a guest interaction proxy should avoid usage of VIX ?

Like I mentioned in the OP, these customer environments are segmented and isolated through vCloud and VIX. So there's intentionally no path from the network our hosts and backup appliance are on to the customers VM network. Which is why we use the VIX connection. Am I wrong in thinking this is a supported configuration?

The issues doesn't seem to be with connectivity between Veeam and Vmware either, because I'm able to migrate the VM to the newly built 6.7 with what should be the same configuration and guest processing is successful. Like I mentioned I can also connect to these hosts and upload files to the VMs using powerCli from the Veeam server.

Are there specific connectivity requirements for VIX that I can check on the hosts where this isn't working and compare to the host where it is working?

[svallance]

Another thing thats strange is that the VIX tests in the Job settings > Guest Processing > Guest Credential Testing section for these VMs is successful. It only fails during the job actually running.

[soncscy]

Heya Shane,

Will be curious what support finds, but I can say a few of my clients had similar experiences going up to 6.7, and for them it was some SSL issue on the proxies. Since it's not really VIX but web services in 6.5 and higher, I believe, are you getting any bad responses from the VMware Web Services?

Also, I'm just gonna throw this out there as I was pulling my hair out on another client's environment until I stumbled across a forum thread here about it, but for the Guest Processing admin accounts, are you dot-slashing the domain/computer name? (e.g., .\some-admin-acct) I cannot for the life of me figure out why it matters, but apparently either Veeam or VMware or Windows freaks out when this is passed for Guest Processing lately (I'm trying to find the topic), but as long as you're waiting on a response, might as well spin up a test job and see

[svallance]

Great info soncscy,

It seems like you might be onto something with the web services. That's the specific error message that we're getting: "cannot connect to host [vcenter-IP] over web services"
In the Veeam logs for the backups of VMs on the upgraded host seems like it's connecting, and I can see the authentications on the VMWare side, but then fails out at uploading files, or starting vss services.

Do you happen to remember where they ended up looking for the SSL issues on the proxies?

As for the admin accounts, it's a mixture. The production account credentials I have currently are in 'username@domain.com' format, I've tried a few different combinations including the '.\user' but haven't seen a change.

[soncscy]

Interesting.

I'm digging through some old email exchanges and billables here, and I have notes that the clients had actually been doing ssl3 without realizing it and this got disabled on the vcenter, so we just had to adjust the proxies. But this was early last year, so not sure what the defaults are now for fresh upgrades. But it's just a thought.

[svallance]

I think I owe you a coffee! You definitely set me on the right track looking at certificates.

I tried to browse to the hosts from the backup server, and only the new build host was giving me the option to ignore the self signed cert. The other hosts were only giving me the option to close the window.

I didn't notice this before because Chome and Firefox both gave me the normal option to continue despite the warning. But in IE only Host 8 had that option.

I checked the browser certificate for the new host and it was showing me the CA, but when I checked the other hosts in Chrome, I didn't see the CA.

I viewed the CA cert and installed it manually into the Trusted CA/intermediate CA folders for the local machine on the Veeam server, and Voilà, I was able to browse to the other hosts in IE and the browser certs showed the CA again.

When I re-ran the my test job again guest processing was successful! We’ll have to figure out what the issue is with these host certificates on the upgraded hosts, but at least it's a step forward.

[soncscy]

Wonderful!

Legit, I go pretty much full Stallmanite on information sharing (it's why I love forums) so I'm just happy it helped, and thanks for the tips. I'll be adding these to my checks also.

Hope it continues working.

[PetrM]

Hello,

@svallance

"Guest interaction proxy must have either a LAN or VIX connection to the VM that will be processed. You do not have to set up both connections — only one connection is required."
Which is what we're doing, but you mention that a guest interaction proxy should avoid usage of VIX ?

There are 2 methods to copy binaries of guest runtime component from Veeam server to a VM: via admin share or using Web Services/VIX if the admin share is not available.
For example, guest interaction proxy might be useful if you have no network access from backup server to a VM. You do not have to configure both admin share and Web Services/VIX connection from proxy to a VM, it's enough to have just one of them.

@soncscy

can say a few of my clients had similar experiences going up to 6.7, and for them it was some SSL issue on the proxies. Since it's not really VIX but web services in 6.5 and higher,

Thanks for sharing this idea with us!

Thanks!