Isolated Network VM backupable?

[B.F.]

Greetings,

Here is the scenario:
We have 2 sites where each backs up or replicates to the other site for disaster recovery.

Site1 <----Backup / Replicates ----> Site2

We are going to setup a VM on Site2 that is isolated from any of the other networks on Site2. No PC will be able to access this isolation VM from either Site1 or Site2 via the network.

My question:
Will IsolatedVM still be able to be backed up / replicated back to Site1 with Veeam? We are using vSphere 6.0 U1b and each site does have a Veeam v8 presence with Veeam proxies.

Thanks!

[foggy]

Yes, you can still backup VMs without having direct network connection to them. Just make sure you have VMware Tools up and running.

[prolix21]

If you want application aware jobs you may have some issues, however the new guest interaction proxy option may provide you some options for that. Everything we backup is isolated from our Veeam Backup systems, so we assign one VM in each network and give it a nic attached to our 'backup network' and assign it as a guest interaction proxy for the job. Gives us full application aware backups in the same type of scenario you describe.

[larry]

If you want application aware jobs .

create a local user on that VM with rights

[alanbolte]

VIX requires either UAC is disabled (not an option in all versions of Windows) or the built-in Administrator account.

[B.F.]

Application aware did not work as you folks pointed out.

so we assign one VM in each network and give it a nic attached to our 'backup network' and assign it as a guest interaction proxy for the job.

Unfortunately this would not be an option for us unless we are willing to go through layers of approval only to most likely get denied.

create a local user on that VM with rights

Tried that and it would fail

VIX requires either UAC is disabled (not an option in all versions of Windows) or the built-in Administrator account.

Disabled UAC per this Veeam KB, rebooted, now it works!

Thanks all!

[hyvokar]

Hi!

I ran into a similar problem.
I'm trying to backup a vm from an unreachable network.

I have set a local admin account on the Win2012 VM.

However, when I try to do aaip backup, I get an error:

17.6.2016 0:40:51 :: Processing VM Error: Cannot upload guest agent's files to the administrative share [C:\Windows].
Cannot create folder in guest: [C:\Windows\VeeamVssSupport].
VIX Error: You do not have access rights to this file Code: 13

I have no anti-virus software running on that server right now.

Another question, I a DC on that same network. Would I be able to back that up, if I installed a guest interaction proxy on some VM on that network, and could I possibly even use domain-accounts to backup rest of the VMs on that network?

[B.F.]

So our scenario has been working great for months...until now.

We are no longer able to backup the VM that is on the isolated network. Here is the error it throws (IP, server name, and account name has been changed):

Failed to prepare guest for hot backup. Error: Failed to connect to guest agent. Errors: 'Cannot connect to the host's administrative share. Host: [10.X.Y.Z]. Account: [servername\accountname]. Win32 error:The network path was not found. Code: 53 '
Error: Failed to connect to guest agent. Errors: 'Cannot connect to the host's administrative share. Host: [10.X.Y.Z]. Account: [servername\accountname]. Win32 error:The network path was not found. Code: 53 '

This all seemed to start just after we applied MS security patches on the Veeam box and the isolated VM. I have not made any changes to the Veeam job since we got it working.

Isolated VM had the following installed
KB3197875 November 2016 Preview of Monthly Quality Roll-up for Windows Server 2012 R2
KB3197874 November 2016 Security Monthly Quality Roll-up for Windows Server 2012 R2

Veeam Server had the following installed
KB3205401 December 2016 Monthly Quality Roll-up for Windows Server 2012 R2
KB890830 Windows Malicious Softare Removal Tool for Server 2012 R2
KB3205404 December 2016 Security and Quality Roll-up for .NET Framework...

Unless anyone has suggestions, I may have to systematically remove the updates to see if that remedies the issue.

Thanks!

[foggy]

We'd appreciate you coming back with the particular update causing this behavior.

[B.F.]

Foggy,

Funny, I just opened a ticket with Support in hopes there would be another option than to remove updates from a production environment. Wonder if I should start removing from the isolated VM or from the Veeam server first.... hmmmmm

[foggy]

If no other changes were performed, one of these updates seem to change something that resulted in proxy not having access to VM anymore (firewall settings, UAC, etc. depending on whether it worked over network or VIX). You could check those prior to removing updates.

[B.F.]

Is there some equivalent to Ping to verify that Veeam is able to touch this VM? We don't manage the FW so it could very well be possible there was some changes done on that which we are not aware of.

[foggy]

Try to open ADMIN$ share on the VM under account specified in the job.

[B.F.]

Not sure if I'm following. The network this VM is on is isolated. Can't hit anything from the Veeam server to this VM via normal network means. Are you referring to trying to open the ADMIN$ while logged into the isolated VM with the Veeam account? I was able to log into the VM via the vSphere console with the account used by Veeam but never tried ADMIN$.

[foggy]

Not sure, but opening vShpere console under the same account and trying to open the administrative share should be the same that the proxy does when connecting over VIX.

[B.F.]

Got it figured out! Somehow UAC was re-enabled which caused VIX to stop working. Perhaps one of those MS Patches re-enabled it on us? We are happily backing that isolated VM again!

Thanks!

[Vitaliy S.]

For future readers: even if UAC is enabled, you can still use built-in administrator account for VSS quiescence.

[B.F.]

Vitaliy, are you referring to this?

[jmmarton]

No, that's a setting used when you aren't using AAIP and then typically just with Linux VMs. Vitaliy is talking about the local administrator account within the Windows VMs in order to do networkless AAIP via VIX. Here's some info.

https://www.veeam.com/kb1788

Joe

[Vitaliy S.]

Yes, Joe is correct. I'm saying that in order to make VIX work, you need to use built-in administrator account in AAP settings over here > Application-Aware Processing

[B.F.]

Per KB1788:

If the account being used is not named “Administrator”, you must disable UAC on the Guest OS of the VM to be backed up.

Guess that's why we had to disable UAC, we don't have an account named "Administrator" on that VM. Good to know though, thanks!

[Mike Resseler]

B.F.

Any news on this? Error 53 as you showed above is in most cases indeed related to this ADMIN$ not being accessible anymore. Can you use the console and check if all necessary services are running on that VM? There might be some services that went gaga after the reboot...

[B.F.]

Mike,

In an isolated scenario, no shares (including ADMIN$) are accessible since that range is unreachable via the network. We had been using VIX with Veeam but one has to disable UAC on the VM in order to make it work. One of the MS updates must have reset this bit and we were no longer able to backup the VM. Reset the UAC in Regedit and we are now back in business.

[Mike Resseler]

B.F.,

Sorry, 2 things went wrong:

1. There were 2 pages to this thread which I missed so I didn't noticed that you already solved it
2. I actually wanted to know if you already figured out which update caused the problem and failed to ask the right question . Many times after an update and reboot services hang and it is always good to know which one is the problematic one for other people

Again, sorry

[B.F.]

No worries Mike,

I never attempted to figure out which update could have been the culprit. There were only 2 patches applied that seemed to occur right before the backups stopped failing.

KB3197875 November 2016 Preview of Monthly Quality Roll-up for Windows Server 2012 R2
KB3197874 November 2016 Security Monthly Quality Roll-up for Windows Server 2012 R2

I guess I can't be 100% certain that one of those was the cause since this VM is also available to a 3rd party vendor as well. It's possible they could have made the change but it just seemed a bit too coincidental that the backups stopped right after these patches were applied.

[Vitaliy S.]

B.F., is there any reason not to use built-in local administrator with UAC enabled in your scenario?

[B.F.]

It is a practice (before I got here) to always rename the built in Admin account. I'm assuming the thought is it would be a security measure?

[Vitaliy S.]

Yes, that could be the case. Thanks for the info.

[RubinCompServ]

FYI, I believe that, if you've renamed the built-in Administrator account, you can still use that account to bypass the UAC issues. It's not about the name of the account but the GUID of the account, and that doesn't change when the account is renamed.

[B.F.]

That would make sense. Looking deeper at the backup configuration in Credential Manager, I see that the local admin is not added but has a Backup account. That would explain why it started to fail after UAC got flipped on.

Thanks