Discussions specific to the VMware vSphere hypervisor
Post Reply
tonyb
Service Provider
Posts: 5
Liked: 1 time
Joined: Oct 28, 2010 6:07 pm
Full Name: Tony B
Contact:

VIX with VMWare Tools newer than 10.1.15

Post by tonyb » Sep 05, 2018 5:55 pm

Is anyone else having issues with guest OS processing using VIX on VMWare tools newer than 10.1.15?

We have found that guest processing breaks on nearly all of our VMs that use VIX if we upgrade VMWare tools to a version newer than 10.1.15. We are using the built-in domain admin account and the windows firewall is turned off so it shouldn't be a UAC or firewall issue. It looks like with the newer VMWare tools when VIX logs it there is an event in the event viewer about loading a temporary profile for the VIX user then Veeam fails to install the guest agent. If I login to the console then run the backup while logged in then guest processing works.

So far the only solution we have found is to downgrade back to 10.1.15.

I have a support case open #03170972 but we haven't found a solution. I was hoping someone else has ran into this issue before I contact VMWare support.

kkitzul_SE
Novice
Posts: 7
Liked: never
Joined: Jan 22, 2013 9:01 pm
Full Name: Kyle
Contact:

Re: VIX with VMWare Tools newer than 10.1.15

Post by kkitzul_SE » Sep 13, 2018 4:38 pm

Any update on your issue? We are experiencing the same thing. I have not created a support case yet but will be doing so shortly.

v.eremin
Product Manager
Posts: 16130
Liked: 1314 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VIX with VMWare Tools newer than 10.1.15

Post by v.eremin » Sep 13, 2018 4:59 pm

The case is still under investigation. Kindly, open your own ticket. Thanks.

tonyb
Service Provider
Posts: 5
Liked: 1 time
Joined: Oct 28, 2010 6:07 pm
Full Name: Tony B
Contact:

Re: VIX with VMWare Tools newer than 10.1.15

Post by tonyb » Sep 13, 2018 5:30 pm

We got at least one server working by using username@domain instead of domain\username for the credentials.

I've also found that if i use PowerCLI I can sometimes reproduce the issue.

Here is the command I test with:
Invoke-VMScript -VM BrokenVM -ScriptType bat -GuestCredential $creds -ScriptText "echo %temp%"

On VM Tools 10.1.15 that returns "C:\Windows\Temp". On newer VM Tools that returns a path inside the user's profile. Sometimes when I run that command with the new VM Tools i get the same error veeam gets (File not found, where the file is inside the user's temp directory which is inside a temporary profile).

The issue happens on most of our servers across multiple AD domains and environments, but not all of them. So far no pattern to which break and which don't other than RDS servers (which use user profile disk) always break.

kkitzul_SE
Novice
Posts: 7
Liked: never
Joined: Jan 22, 2013 9:01 pm
Full Name: Kyle
Contact:

Re: VIX with VMWare Tools newer than 10.1.15

Post by kkitzul_SE » Sep 13, 2018 6:59 pm

Thanks Tony,

using username@domain fixed it up for us.

foggy
Veeam Software
Posts: 17708
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VIX with VMWare Tools newer than 10.1.15

Post by foggy » Sep 14, 2018 3:58 pm

We've seen similar issues caused by the fact that there's no temp folder (used during backup over VIX) in case of non-interactive user logon. This should be addressed in the next VMware Tools version, as far as I know.

tonyb
Service Provider
Posts: 5
Liked: 1 time
Joined: Oct 28, 2010 6:07 pm
Full Name: Tony B
Contact:

Re: VIX with VMWare Tools newer than 10.1.15

Post by tonyb » Sep 18, 2018 3:01 am 1 person likes this post

I've done some additional testing and it appears the username@domain workaround does NOT work for RDS servers that use user profile disk. The only solution we have found is to downgrade vmware tools.

tonyb
Service Provider
Posts: 5
Liked: 1 time
Joined: Oct 28, 2010 6:07 pm
Full Name: Tony B
Contact:

Re: VIX with VMWare Tools newer than 10.1.15

Post by tonyb » Nov 26, 2018 4:07 pm

Has anyone gotten this to work with vmware tools 10.3.5 that is supposed to include the fix?

We have upgraded a few servers and it does NOT appear to fix the issue. I'm just starting to investigate now.

tonyb
Service Provider
Posts: 5
Liked: 1 time
Joined: Oct 28, 2010 6:07 pm
Full Name: Tony B
Contact:

Re: VIX with VMWare Tools newer than 10.1.15

Post by tonyb » Nov 26, 2018 4:44 pm

I can confirm we are still seeing the issue with VMWare tools version 10.3.5. The "Invoke-VMScript" test still shows "%temp%" to be inside the user's profile and an extended test using the CreateTemporaryFileInGuest method shows the temp file still inside the user's profile.

Can Veeam or anyone else confirm if 10.3.5 fixed the issue for them?

Gostev
SVP, Product Management
Posts: 24016
Liked: 3252 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VIX with VMWare Tools newer than 10.1.15

Post by Gostev » Nov 27, 2018 2:48 pm

Confirming that 10.3.5 fixes the issue, we're now seeing C:\windows\TEMP leveraged again during networkless guest processing via VIX in our labs.

mats.jansson
Service Provider
Posts: 17
Liked: 1 time
Joined: Mar 18, 2014 9:13 am
Full Name: Mats

Re: VIX with VMWare Tools newer than 10.1.15

Post by mats.jansson » Apr 11, 2019 8:08 pm

We are also having this problem.
We changed to username@domain
VMware tools 10.2.5 working, 10.3.x not working.
VMware tools 10.1.5 on rds servers working, 10.2.x and 10.3.x not working.

Today we installed the new 10.3.10 on two servers and it's working (server 2019)
Next week we are going to install 10.3.10 on two rds servers.
I will update the post after the test.
Mats

mats.jansson
Service Provider
Posts: 17
Liked: 1 time
Joined: Mar 18, 2014 9:13 am
Full Name: Mats

Re: VIX with VMWare Tools newer than 10.1.15

Post by mats.jansson » Apr 15, 2019 1:42 pm

I have installed 10.3.10 on a rds server and it´s not working.
In total:
VMware tools 10.3.10 working, tested on server 2019.
VMware tools 10.1.15 working on rds servers but not newer versions.
Mats

tony.grilley
Veeam Software
Posts: 38
Liked: 12 times
Joined: Apr 21, 2014 2:40 pm
Full Name: Tony Grilley
Contact:

Re: VIX with VMWare Tools newer than 10.1.15

Post by tony.grilley » Apr 17, 2019 4:49 pm 1 person likes this post

I'm a support engineer, and was working on a case with a customer (03384744) Regarding this issue.

There are 2 different issues being experienced.

One is the one Anton mentions, where 10.3.5 will now write to the correct location (C:\Windows\Temp instead of the temp directory in the user profile)

The other is a HTTP Response Timeout when processing a VM with AAIP over VIX (VMware Tools). Support has a fix available to enable the use of a registry key to extend the timeout for the HTTP response, however I also received the following information from VMware:

"[t]he problem only happens when you use a domain administrator account to authenticate to the guest VM. If you use a local administrator the problem does not happen."

This is what that particular issue looks like:

Code: Select all

Info Writing file [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\VeeamGuestHelpers\VeeamVixProxy.exe] of size [942008] to the HTTP server 
Info Sending HTTP request with Content-Length [942008] to the HTTP server 
Info Sending HTTP request with Content-Length [942008] to the HTTP server. Ok. 
Info Uploaded 100% 
Info Receiving response from the HTTP server 
Info Receiving response from the HTTP server. Failed.
If anyone is experiencing issues with VIX Processing and is using VMware Tools 10.1.15 or higher, check what account is being used. If possible, use a local machine administrator. Credentials can be set on a per-vm basis: https://helpcenter.veeam.com/docs/backu ... l?ver=95u4

As always, a support case is always recommended to verify which issue is being encountered.

I'll see if a KB can be created with this information to further help as well, I just need to collect the necessary information from my case and write it up.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 10 guests