VIX with VMWare Tools newer than 10.1.15

[tonyb]

Is anyone else having issues with guest OS processing using VIX on VMWare tools newer than 10.1.15?

We have found that guest processing breaks on nearly all of our VMs that use VIX if we upgrade VMWare tools to a version newer than 10.1.15. We are using the built-in domain admin account and the windows firewall is turned off so it shouldn't be a UAC or firewall issue. It looks like with the newer VMWare tools when VIX logs it there is an event in the event viewer about loading a temporary profile for the VIX user then Veeam fails to install the guest agent. If I login to the console then run the backup while logged in then guest processing works.

So far the only solution we have found is to downgrade back to 10.1.15.

I have a support case open #03170972 but we haven't found a solution. I was hoping someone else has ran into this issue before I contact VMWare support.

[kkitzul_SE]

Any update on your issue? We are experiencing the same thing. I have not created a support case yet but will be doing so shortly.

[veremin]

The case is still under investigation. Kindly, open your own ticket. Thanks.

[tonyb]

We got at least one server working by using username@domain instead of domain\username for the credentials.

I've also found that if i use PowerCLI I can sometimes reproduce the issue.

Here is the command I test with:
Invoke-VMScript -VM BrokenVM -ScriptType bat -GuestCredential $creds -ScriptText "echo %temp%"

On VM Tools 10.1.15 that returns "C:\Windows\Temp". On newer VM Tools that returns a path inside the user's profile. Sometimes when I run that command with the new VM Tools i get the same error veeam gets (File not found, where the file is inside the user's temp directory which is inside a temporary profile).

The issue happens on most of our servers across multiple AD domains and environments, but not all of them. So far no pattern to which break and which don't other than RDS servers (which use user profile disk) always break.

[kkitzul_SE]

Thanks Tony,

using username@domain fixed it up for us.

[foggy]

We've seen similar issues caused by the fact that there's no temp folder (used during backup over VIX) in case of non-interactive user logon. This should be addressed in the next VMware Tools version, as far as I know.

[tonyb]

I've done some additional testing and it appears the username@domain workaround does NOT work for RDS servers that use user profile disk. The only solution we have found is to downgrade vmware tools.

[tonyb]

Has anyone gotten this to work with vmware tools 10.3.5 that is supposed to include the fix?

We have upgraded a few servers and it does NOT appear to fix the issue. I'm just starting to investigate now.

[tonyb]

I can confirm we are still seeing the issue with VMWare tools version 10.3.5. The "Invoke-VMScript" test still shows "%temp%" to be inside the user's profile and an extended test using the CreateTemporaryFileInGuest method shows the temp file still inside the user's profile.

Can Veeam or anyone else confirm if 10.3.5 fixed the issue for them?

[Gostev]

Confirming that 10.3.5 fixes the issue, we're now seeing C:\windows\TEMP leveraged again during networkless guest processing via VIX in our labs.

[mats.jansson]

We are also having this problem.
We changed to username@domain
VMware tools 10.2.5 working, 10.3.x not working.
VMware tools 10.1.5 on rds servers working, 10.2.x and 10.3.x not working.

Today we installed the new 10.3.10 on two servers and it's working (server 2019)
Next week we are going to install 10.3.10 on two rds servers.
I will update the post after the test.

[mats.jansson]

I have installed 10.3.10 on a rds server and it´s not working.
In total:
VMware tools 10.3.10 working, tested on server 2019.
VMware tools 10.1.15 working on rds servers but not newer versions.

[tony.grilley]

I'm a support engineer, and was working on a case with a customer (03384744) Regarding this issue.

There are 2 different issues being experienced.

One is the one Anton mentions, where 10.3.5 will now write to the correct location (C:\Windows\Temp instead of the temp directory in the user profile)

The other is a HTTP Response Timeout when processing a VM with AAIP over VIX (VMware Tools). Support has a fix available to enable the use of a registry key to extend the timeout for the HTTP response, however I also received the following information from VMware:

"[t]he problem only happens when you use a domain administrator account to authenticate to the guest VM. If you use a local administrator the problem does not happen."

This is what that particular issue looks like:

Code: Select all

Info Writing file [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\VeeamGuestHelpers\VeeamVixProxy.exe] of size [942008] to the HTTP server 
Info Sending HTTP request with Content-Length [942008] to the HTTP server 
Info Sending HTTP request with Content-Length [942008] to the HTTP server. Ok. 
Info Uploaded 100% 
Info Receiving response from the HTTP server 
Info Receiving response from the HTTP server. Failed.

If anyone is experiencing issues with VIX Processing and is using VMware Tools 10.1.15 or higher, check what account is being used. If possible, use a local machine administrator. Credentials can be set on a per-vm basis: https://helpcenter.veeam.com/docs/backu ... l?ver=95u4

As always, a support case is always recommended to verify which issue is being encountered.

I'll see if a KB can be created with this information to further help as well, I just need to collect the necessary information from my case and write it up.

[morrie]

Hi all,

We've just built a Windows 2016 Server for a customer and then we are just building our own Windows Server 2019 Server.
All of these are seeing the same issue with VIX failures. VG Auth eventlogs are showing that the password is succeeding.
For the 2016 Server, there are other servers in the same Customer, which work.

I can confirm that I can use;

PowerCLI C:\Scripts> copy-vmguestfile -source c:\techno\resize.ps1 -Destination c:\windows\temp\ -vm tp-dc-fp01 -LocaltoGuest -GuestUser domain\administrator -GuestPassword ###

This copies the test file to the correct location, using the same credentials that I have used in Veeam..
Does anyone know what which file(s) are being copied so that I can replicate it with the necessary Service registering commands as well. That way I can prove I can fully install it manually..

What other logging can I get to know what Veeam is sending to VMWare and then VMware to the VMTools in the client..

Need to understand end to end if we are going to get to the bottom of this.

Have got a case open with Veeam. I'm not 100% sure it's a VMware issue, if I can manually connect and do thinks via the API?
I'm probably wrong though..

Andrew Morison.
Techno Group

[wishr]

Hi Andrew,

Could you please let us know your support case ID?

Thanks in advance

[morrie]

03523465.
Rashmi is currently escalating it as we aren't going anywhere at the moment..

Also, forget about the 2016 Server, it was a red herring. That actually had incorrect credentials.
Current issue only relates to 2019 servers.

Andrew.

[wishr]

Hi Andrew,

Thanks for letting us know. Feel free to escalate the case if necessary and keep working with the support team.

Thank you

[morrie]

Hi all,

Just to update my issue. After working with support, and after a lot of trial and error, I found that the issue related to a GPO change.
As some of you will know, there is an annoying problem when using the standard network icon, which stops you from opening the Change Adapter options which logged in as the domain\administrator account. Basically it throws essentially a permission error concerning control.exe.

To work around this, there are many articles which suggest the following setting which reenables the functionality, but looks like breaks VIX.

GPO - Computer - Policies - Windows Settings - Security Settings - Local Policies/Security Options - User Account Control User Account Control: Admin Approval Mode for the Built-in Administrator Account - Enable

Disabling this, in my case, fixed my problem. Just in case this is something people are seeing..
I've tested both 10.1.15 and 10.3.10 without issues since disabling the above ( disable is the default setting)

Andrew.

[raronson@dga.org]

This doesn't work for us. Our local admin account has been renamed through GPO. I thought changing permissions on the runtime broker might help. No luck there either.

[tonyb]

For anyone that is having the same issue as the original post (after upgrading to VMWare tools 10.3.5 guest processing stops work) I have an update from VMware.

They are now able to reproduce this issue in their lab. VMware tools 10.3.5 using a domain admin account and RDS User Profile disk is 20x slower than the same exact server on VMware tools 10.1.15. They think it is a bug in VMWare tools but didn't provide any details beyond that.

[wishr]

Hi Tony,

Thanks a lot for sharing your findings! It's highly appreciated.

Regards,
Fedor

[tonyb]

Latest update from VMWare: The issue appears fixed in VMWare tools 11.0.0.

VMware support sent me an Engineering Release of 11.0.0 that I tested on one of our VMs and it is working much better. Testing using the "Copy-VMGuestFile" command in PowerCLI take ~40-50 seconds (and some times as long as 135 seconds) on 10.3.5 (~800Kb File) on 11.0.0 it took 5 seconds.

That is still a little slower than on 10.1.15 but it is at least usable.

[Sanil Port]

Hi all,
GPO - Computer - Policies - Windows Settings - Security Settings - Local Policies/Security Options - User Account Control User Account Control: Admin Approval Mode for the Built-in Administrator Account - Enable

Disabling this, in my case, fixed my problem. Just in case this is something people are seeing..
I've tested both 10.1.15 and 10.3.10 without issues since disabling the above ( disable is the default setting)

Andrew.

This worked brilliantly for me after some troubleshooting, simply leaving the GPO at Not configured did not work, I had to disable it, the Microsoft Security Baseline enables this GPO.

This is a combination of Vmware Tools 10.3.10 and the GPO mentioned above, in saying that am not sure if the tools upgrade was required, well that was a day! Thanks for the help community!

[mats.jansson]

Hi all

Today I have upgraded to the new VMware tools 11.0 on our 2019 RDS server and finally the vix backup is working again.

/Mats

[UT2015]

Hi there,

we are currently having the very same problem. We are using VMware Tools 12.1.0 on a Windows Server 2022 and because we are using the Microsoft Baseline-GPO (MSFT Windows Server 2022 - Member Server) the setting mentioned above (UAC: Admin Approval Mode for the Built-in Administrator Account) is enabled.

Veeam B+R is actually successfully backing up the VM, but the eventlog on the Windows Server 2022 vm shows many of the above mentioned entries (Source VGAuth, ID 1000 and Source User Profile Service, ID 1511).

We do not have those events on our Windows Server 2012 VMs (which are also not using the MSFT-GPO).

Edit: We are not using a domain administrator-account but a Tier-1-account which has administrative privileges on the VM to backup.

Is this a different bug or is it the same problem as mentioned in this thread?

Best regards,
UT2015

[UT2015]

Some additional information:

When executing the script mentioned above (Invoke-VMScript -VM BrokenVM -ScriptType bat -GuestCredential $creds -ScriptText "echo %temp%") the following directory is displayed:

C:\Users\TEMP\Appdata\Local\Temp

When changing the credentials from domain\username to username@domain, the correct path C:\Users\username\Appdata\Local\Temp is displayed.