Stange restore problem

[signal]

I'm working on Veeam as a "module" for a security oriented customer. We have two test environment, one for internal development testing (IT), and one for acceptance testing (AT). They are (supposed to be) configured exactly the same. Each environment has 2 ESXi hosts (6.0), one vCenter (VCSA, 6.0), one 3PAR storage, one backup proxy/repo and one StoreOnce for long term backups. Everyting is set up with maximum security in mind. For Windows that means UAC, AppLocker and GPOs that allow a minimum of interaction. Access to the ADMIN$ share is allowed for the Veeam backup/restore user, and all signed Veeam software is allowed to run. Firewalls are opened up as they need to be.
All servers, including the backup server, is set up using SCCM task sequence. The same task sequences are used on both environments. The vSphere environments are deployed and hardened using scripts.
In the IT environment I can perform VSS backups and restore files as expected.
In the AT environment the backups work as they should, but I am unable to restore guest files directly to the VMs. I have checked network traffic (using Wireshark), and there is no attempt at accessing ADMIN$ or IPC$ share. In the eventlogs I see only logins using vmtoolsd. A new, plain, Windows 2012 R2 server was deployed to test further, placed in the same network zone as the backup server (where we run the console), but still there seems to be no attempt at connecting to the shares, only vmtoolsd access.

I've spent a whole workday trying to figure this out, comparing the environments, disabling firewalls, removing IPv6, etc. I can't figure it out. What am I missing? Is there some fundamental difference that would force the use of VIX only during restore?

[foggy]

What do you mean by unable to restore - do you get an error and, if yes, what kind of error? What if the target VM is located in the same network as the repository server? Have you contacted support already?

[signal]

Unable to restore as in restore of single file back to it's original location.

In Veeam Console the generic error of "Unable to establish connection with the VM. Check credentials and try again." is shown.

In the logs I see lots of stack traces, but the first error (in FLRSessions\Windows\date_time.log) is "Failed to validate credentials on backup service".
On the target guest system I see lots of logons, all with vmtoolsd as process. On the functioning systems I see no logons using vmtoolsd, only direct network/ADMIN$/IPC$ logons.
Later in the FLR log there are messages like "Failed to connect to guest agent. Errors: 'Cannot connect to the host's administrative share'...". I believe these are blanket error messages since there is no network traffic showing any attempted connection over the network.
Using VIX for restore is not going to work, since UAC is in place, and the default local Administrator account is disabled.

On the target guest OS I see the VeeamVixProxy_date.log file has 4 lines per attempt:

Code: Select all

INFO ===
INFO VeeamVixProxy started.
     Command line: ["C:WINDOWS\TEMP\{<numbers here>}" -func GetTargetAdminFolder -out "C:\WINDOWS\TEMP\{<different numbers here>}"]
     Getting target admin folder.

I have not opened a support case sinceI have no way to export logs as this is part of a closed network.

[foggy]

One thing you could do is to compare settings between repository mount server and backup server, since credentials are validated on the mount server. Also, try to open administrative share on the guest VM from the mount server.

[signal]

The environments are set up using the same procedures, and the VBR Console is being used on the backup server in both environments. The user account running the operations have the same AD group memberships, and therefore the same access rights.

[signal]

The next step was of course to set up a new server and test with that.
Using local storage on the new backup server works, but when the backup proxy is added (and data is stored there) it fails to restore.
Which differences is there in a remote and local repository when single file restore is performed?
Is there any communication between the backup proxy and the esxi and/or guest system? From what I gather there is no mention of this in your network documentation.

[foggy]

Also, try to open administrative share on the guest VM from the mount server.

In case where backup is stored on the proxy server, it also holds the mount server role, while mount server should have access to the guest VM.

[signal]

In case where backup is stored on the proxy server, it also holds the mount server role, while mount server should have access to the guest VM.

So the mount server is run where the repository is located, as long as it is a Windows server? I was under the impression that the mount server was run on the machine where the console is located.
As this is a high security environment where we want to limit the exposure of all servers, especially the backup proxy/repositories.
Is there a way to have a separate machine be the mount server to act as a "restore proxy"?

[Andreas Neufert]

Mount Server need to have access to the VM for File Level Recovery (over Network)

[foggy]

You can assign the role of a mount server to any Windows server when creating/editing backup repository. Console also plays this role during initial FLR mount (with the purpose of browsing files), in case the restore is initiated there. Actual restore to original location uses mount to the mount server, if it is configured on different server (or to the backup server otherwise).

[signal]

Great. Then I can have the backup server be the mount server for all backups.

How will this affect backups?

[Andreas Neufert]

Mount Server are used at restore only.