Host-based backup of VMware vSphere VMs.
Post Reply
Jay.N
Veeam Software
Posts: 123
Liked: 50 times
Joined: Jun 02, 2021 9:15 pm
Full Name: Jay Newell
Contact:

SureBackup and NSX-T Workaround

Post by Jay.N » 21 people like this post

EDIT 20.06.22: Upon further testing I've found that the SureBackup job script tests are also skipped when the mapping error mentioned below occurs. This means the workaround in the post won't work. I am currently doing some further investigation on this and will update the post with what I find. For now, you could use the PowerShell script on a schedule to automatically restore networking to the SureBackup restored virtual machines for manual testing, but automated ping testing as part of the SureBackup job won't work.

EDIT 23.06.22: I have put another reply post further down which may be helpful to anyone struggling with this issue. The solution is not perfect and will require your own testing, but it could be useful.

I recently encountered an issue for a customer who wanted to use SureBackup for testing the recovery of their virtual machines, but as they were using VMware NSX-T this was not working. SureBackup with VMware NSX-T is not officially supported by Veeam yet, but by working with Andreas Neufert in the product management team and getting some help with some script code from my colleagues, Solutions Architect Ed Howard, and Technical Account Manager Roland Kiper, I was able to create a workable solution for the customer.

The problem is that when using an NSX-T port group in the virtual lab configuration, the SureBackup job won't be able to automatically map the network to the restored virtual machine. This means the virtual machine is left without a virtual network adapter and therefore can't communicate on the network. Below is a screenshot of the error.

Image

Now, this can be solved by putting the virtual network adapter into the virtual machine and connecting it to the virtual lab isolated network (I'll show how to do this with a script below), but the second problem for this customer was that the SureBackup job ping test was skipped (which happens automatically because of the mapping error), and they wanted to see that this was working.

To get around these two issues we can add two additional roles into the Application Group verification options. One role can run the script to put the virtual network adapter into the SureBackup restored virtual machine(s), the second can perform a ping test. You will likely want to tweak the below details to fit your needs.

1. Firstly on the VBR server you'll need to install VMware PowerCLI and, if SSL certificates aren't used for the vSphere Web Interface, you'll want to suppress the certificate warning prompts in the PowerCLI configuration using something similar to the below.

Code: Select all

Set-PowerCLIConfiguration -Scope AllUsers -InvalidCertificateAction Ignore 
2. Secondly, on the VBR server I created a PowerShell script to add the network adapter to the SureBackup restored virtual machine(s) and saved this as C:\SB_NetworkAdapter.ps1. The code for this is below (I am no coding expert and there are other, probably better, ways to do this, but it gives you an idea). You'll need to replace the information in square brackets.

Code: Select all

#Connect to vCenter server or ESXi host using PowerCLI.  You'll want to use an XML so credentials aren't in plain text.
Connect-viserver -Server [IP address] -User [username] -Password [password]

#Get virtual machine names with an underscore in (which is what a SureBackup provisioned VM will have in the name)
#You can also add other naming rules with -Like and -NotLike as shown here.
$vmNames = get-vm | Where-Object {$_.name -Like "*_*" -And $_.name -NotLike "*_Restore" -And $_.name -NotLike "*_Lab"}

#Takes the virtual machine names found in the command above and checks if any of those don't have a virtual network adapter.
#These virtual machines are stored in a new variable.
$vmNamesNoNetwork = $vmNames | where { $Null -match $_.extensiondata.network }

#Adds the network adapter to the virtual machines and connects it to the virtual lab network.
$vmNamesNoNetwork | foreach {Get-VM -name $_ | New-NetworkAdapter -NetworkName [Virtual Lab Network] -StartConnected}

exit
3. Third, on the VBR server you can find the roles used in the Application Group here C:\Program Files\Veeam\Backup and Replication\Backup\SbRoles. I added two XML files to this folder shown below. You can use https://helpcenter.veeam.com/docs/backu ... ml?ver=110 to help with the XML file creation. You'll need to make sure each XML has a unique ID otherwise you'll receive an error when trying to edit the Application Group.

Image

The code for each of these is below.

AddNetworkAdapter.xml

Code: Select all

<?xml version="1.0" encoding="utf-8" ?>
<SbRoleOptions>
  <Role>
    <SbRole>
      <Id>4CDC7CC4-A906-4de2-979B-E5F74C44832A</Id>
      <Name>Add Virtual Network Adapter</Name>
    </SbRole>
  </Role>
  <Options>
    <SbVerificationOptions>
	<TestScripts>
       	 <TestScripts>
          <TestScript>
	    <Name>Add Virtual Network Adapter</Name>
            <Type>Custom</Type>
		<TestScriptFilePath>C:\SB_NetworkAdapter.ps1</TestScriptFilePath>
		</TestScript>
	</TestScripts>
      </TestScripts>
      <PingEnabled>True</PingEnabled>
    </SbVerificationOptions>
  </Options>
</SbRoleOptions>
PingTest.xml

Code: Select all

<?xml version="1.0" encoding="utf-8" ?>
<SbRoleOptions>
  <Role>
    <SbRole>
      <Id>4CDC7CC4-A906-4de2-979B-E5F74C44832B</Id>
      <Name>Ping Test</Name>
    </SbRole>
  </Role>
  <Options>
    <SbVerificationOptions>
	<TestScripts>
       	 <TestScripts>
          <TestScript>
	    <Name>Ping Test</Name>
            <Type>Custom</Type>
		<TestScriptFilePath>C:\Windows\System32\ping.exe</TestScriptFilePath>
		<Arguments>%vm_ip%</Arguments>
            </TestScript>
        </TestScripts>
      </TestScripts>
      <PingEnabled>True</PingEnabled>
    </SbVerificationOptions>
  </Options>
</SbRoleOptions>
You'll now be able to select these application tests in the Application Group as shown below.

Image

Now when you run the SureBackup job you'll see the below in the UI when clicking on the virtual machine being restored. If you watch the tasks in the vSphere Web Interface you'll notice the virtual machine being reconfigured where the first script is adding the virtual network adapter.

Image

You'll want to disable the ping check in the Application Group as shown below as it's being done by the script.

Image

I hope this helps anyone struggling with this until SureBackup and NSX-T is officially supported.

Jay Newell
Technical Account Manager
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: SureBackup and NSX-T Workaround

Post by HannesK » 1 person likes this post

Hello,
nice one, thanks for sharing! 👍

Best regards,
Hannes
laurentG
Veeam Software
Posts: 24
Liked: 5 times
Joined: Jan 09, 2020 2:17 pm
Full Name: Laurent Godet
Contact:

Re: SureBackup and NSX-T Workaround

Post by laurentG » 1 person likes this post

Nice One Jay, Ed and Roland !!!
Congrats!
Saleh Saad
Veeam Software
Posts: 28
Liked: 3 times
Joined: Sep 02, 2021 2:17 pm
Full Name: Saleh Saad
Contact:

Re: SureBackup and NSX-T Workaround

Post by Saleh Saad » 1 person likes this post

Very well done Jay!
marcodalli
Novice
Posts: 4
Liked: 1 time
Joined: Sep 02, 2016 8:04 am
Full Name: Marco Dalli
Contact:

Re: SureBackup and NSX-T Workaround

Post by marcodalli » 1 person likes this post

Hi and thank you for the scripts!
I have a similar issue when trying to run a SureBackup job of a VM originally attached to an NSX-T segment, restoring to a different cluster for testing purposes.
I tried the above scripts, but the job seems to skip scripts and roles due to the fact that the VM has no network interface:

Code: Select all

[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] Begin 'Running test scripts'
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] Dump operation
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > NoVmNetworks = True
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > ScriptCount = 2
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] Dump test script(s)
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] #1
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > Type = Custom
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > Name = Add Virtual Network Adapter
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > FileName = C:\Scripts\Add_Network_01.ps1
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > Arguments = 
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > WorkingDirectory = 
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] #2
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > Type = Predefined
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > Name = Web Server
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > FileName = Veeam.Backup.ConnectionTester.exe
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > Arguments = %vm_ip% 80
[17.06.2022 09:29:32] <01> Info         [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] > WorkingDirectory = 
[17.06.2022 09:29:32] <01> Warning      [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] One or more networks cannot be mapped, check virtual lab settings
[17.06.2022 09:29:32] <01> Warning      [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] Summary: operation is skipped
[17.06.2022 09:29:32] <01> Warning      [SureBackup] [W2K19-TEST_MEDA_D4] [ScriptTests] End 'Running test scripts'
Am I missing something? Is there a way to tell SureBackup job to run scripts even if no network is attached?
Jay.N
Veeam Software
Posts: 123
Liked: 50 times
Joined: Jun 02, 2021 9:15 pm
Full Name: Jay Newell
Contact:

Re: SureBackup and NSX-T Workaround

Post by Jay.N »

Hi marcodalli,

I wasn't aware that the script tests were skipped due to the mapping error but I did try the scripts in a different environment separate from NSX-T as I don't have constant access to an NSX-T environment, so I may have missed this. Thanks for sharing. Obviously, this isn't properly supported by Veeam currently, but I tried some other options during testing which we can try and I'd be happy to help you do some further testing.

Please feel free to reach out to me separately via email and we can arrange a screen sharing session. If we can do some testing together it would be good to provide an update in this forum post based on the findings.

Thanks
kevin.mcdonnell
Influencer
Posts: 17
Liked: 3 times
Joined: Sep 23, 2014 12:58 pm
Full Name: Kevin McDonnell
Contact:

Re: SureBackup and NSX-T Workaround

Post by kevin.mcdonnell » 1 person likes this post

Hi,

Great post Jay, however if you are looking to avoid putting the username & password in you VMware PowerShell script then make used of the stored credential feature. See this (rather old) link:

https://blogs.vmware.com/PowerCLI/2011/ ... ature.html

Cheers,

Kev
Kevin McDonnell
Jay.N
Veeam Software
Posts: 123
Liked: 50 times
Joined: Jun 02, 2021 9:15 pm
Full Name: Jay Newell
Contact:

Re: SureBackup and NSX-T Workaround

Post by Jay.N » 2 people like this post

As per my first edit in my original post, shortly after my original post I was made aware of an issue by marcodalli where the SureBackup script test is also skipped because of the network mapping error mentioned previously. After testing myself I received the same outcome and this dashed my excitement and the thought I'd created a really helpful workaround, but it'll teach me to do more testing before posting about a fully working solution. Since discovering that issue I have worked closely with my colleague Roland Kiper to create something else that could be useful.

I have tested the below in an NSX-T environment with a single SureBackup job with a single virtual machine in and it works as expected, however, you will want to perform your own testing of this as the code will likely need adjusting, especially as it hasn't been tested when multiple SureBackup jobs are running at the same time and in multiple different environments. I wanted to post the code for now though as it might create further ideas to help you with the issue, but it's not meant to be a fully working solution. If even just a snippet of the code is helpful to anyone then it's worth it.

The code below does the following:

- Get any SureBackup provisioned virtual machines that are running and that don't have a network adapter attached
- Adds a virtual network adapter to them and connects it to the virtual lab network
- Gets the IP address from the virtual machines
- Converts the IP address to the masqueraded version (which you have to set in the code to match the virtual lab settings)
- Pings the masqueraded IP address - If the ping fails it stops the SureBackup job associated with the pinged virtual machine early, causing it to fail - If the ping is successful it lets the SureBackup job complete.

You can maybe run this script on a schedule every 1 minute in task scheduler, which means it will keep checking for any SureBackup provisioned virtual machines ready for when a SureBackup job runs.

Code: Select all

$logPath = [log path location]

#Connect to vCenter server or ESXi host using PowerCLI.  You'll want to use stored credentials or similar so credentials aren't in plain text.
Connect-viserver -Server [IP address or hostname] -User [username] -Password [password]

#Get virtual machine names with an underscore in (which is what a SureBackup provisioned VM will have in the name)
#You can also add other naming rules with -Like and -NotLike as shown here.
$vmNames = get-vm | Where-Object {$_.name -Like "*_*" -And $_.name -NotLike "*_Restore" -And $_.name -NotLike "*_Lab"}

#Takes the virtual machine names found in the command above and checks if any of those don't have a virtual network adapter.
#These virtual machines are stored in a new variable.
$vmNamesNoNetwork = $vmNames | where { $Null -match $_.extensiondata.network }

#Adds the network adapter to the virtual machines and connects it to the virtual lab network.
$vmNamesNoNetwork | foreach {Get-VM -name $_ | New-NetworkAdapter -NetworkName [Virtual Lab Network Name] -StartConnected}

#Wait for 20 seconds for vCenter to see VM IP address after adding virtual network adapter.
Start-Sleep -Seconds 20

#Gets SureBackup virtual machine IP address and store it in a variable.
$ipAddress = (get-vm $vmnamesNoNetwork).extensiondata.guest.IPAddress.Summary.Guest.IPAddress

#Change IP address in variable to match masqueraded version that is set in the virtual lab.
#This code changes the value in the second octet of an IP address to "137".  You'll need to change this to work in your environment. 
$masqueradedIPAddress = $ipaddress -replace $ipaddress.split(".")[1], "137"
Write-Output "I am going to ping the following IP address $($masqueradedIPaddress)" | out-file -FilePath $logPath -Append

#Change variable $vmNamesNoNetwork to a string and store in a new variable.
$vmNamesNoNetworkString = $vmNamesNoNetwork.Name

#Remove SureBackup ID from VM name (everything including and after the last underscore in the VM name)
$vmNamesOriginal = $vmNamesNoNetworkString.Substring(0, $vmNamesNoNetworkString.LastIndexOf("_"))

#Get SureBackup job that VM is part of
$sureBackupJob = get-vbrsurebackupjob | where-object {$_.get_ApplicationGroup().get_VM().Name -eq "$($vmNamesOriginal)"}
Write-Output "$(Get-Date -Format dd-MM-yyyy_HHmmss) - SureBackup Job = $($sureBackupJob)" | out-file -FilePath $logPath -Append

#Ping command against masqueraded IP address and if ping fails stop the SureBackup job
if (Test-Connection $masqueradedIPAddress -quiet) {write-output "$(Get-Date -Format dd-MM-yyyy_HHmmss) - Ping Successful for $($masqueradedIPaddress)" | out-file -filepath $logPath -Append}else{Get-VBRSureBackupJob -Name $sureBackupJob | Stop-VBRSureBackupJob
write-output "$(Get-Date -Format dd-MM-yyyy_HHmmss) - Ping Not Successful for $($masqueradedIPaddress) stopping $($sureBackupJob)" | out-file -filepath $logPath -Append
}

exit
Please feel free to reach out if you have any questions.

Thanks

Jay Newell
Technical Account Manager
JanT
Novice
Posts: 3
Liked: never
Joined: Feb 18, 2021 11:58 am
Full Name: Jan Tománek
Contact:

Re: SureBackup and NSX-T Workaround

Post by JanT »

When will this issue be resolved in the production version? Are there so few customers using NSX-T? Surebackup is a significant feature of Veeam and we are paying unnecessarily for something we can't fully use.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: SureBackup and NSX-T Workaround

Post by HannesK »

Hello,
and welcome to the forums.

The plan is to fix it after V12 as soon as possible. Unfortunately, we cannot provide a timeline.

Question: do you use VDS or N-VDS?

Sorry for the inconvenience,
Hannes
JanT
Novice
Posts: 3
Liked: never
Joined: Feb 18, 2021 11:58 am
Full Name: Jan Tománek
Contact:

Re: SureBackup and NSX-T Workaround

Post by JanT »

Thank you Hannes for your response. Answer to your question: we use VDS.
Jan
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: SureBackup and NSX-T Workaround

Post by HannesK »

thanks!
DontPanic
Lurker
Posts: 1
Liked: never
Joined: Mar 01, 2023 10:06 am
Full Name: Andreas Störlein
Contact:

Re: SureBackup and NSX-T Workaround

Post by DontPanic »

Hello Forum!

i am new here so this is my first entry ;-)

we tried the workaround above without success in our enviroment ,but the basic idea is great and we adapted it in our "solution" which works fine for us under certain circumstances.

These are:
- NSX-T Portgroup Name matches to VLab Portgroup Name with Prefix for Virtual Lab uses
- VMXNet3 Network Adapters only
- HotAdd enabled on all VMs

First we created an Alarm in VSphere on the SureBackup ESXi Cluster. This Alarm is triggered when a VM is powered on and the name is not like "SureBackup" (the Lab Appliance itself). So this Alarm triggers a Python Script for every restored VM when it is powered on. The script is residing on our vCenter Appliance and connects to out PowerShell Scripting host via paramiko and executes a PowerShell Script called "surebackup_network.ps1".

The Python Script looks like this:

Code: Select all

##################################################################################################
#!/usr/bin/python

import paramiko
import time

# declare credentials
host = 'OurPowerShellHostFQDN'
username = '****'
password = '******'

# define command

cmd = "powershell -file P:\\scripts\\veeam\\surebackup_network.ps1"

# connect to server
con = paramiko.SSHClient()
con.load_system_host_keys()
con.connect(host, username=username, password=password)

# run the command
# use the -1 argument so we can split the files by line

stdin, stdout, stderr = con.exec_command(cmd)

time.sleep(3)

# process the output
#if stderr.read() == b'':
#    for line in stdout.readlines():
#        print(line.strip()) # strip the trailing line breaks
#else:
#        print(stderr.read())
################################################################################################

The PowerShell Script on the scripting host looks like this:

################################################################################################

$date = Get-Date
$logfile = c:\work\surebackup.log

write-output "$date - start script" | out-file -append -filepath $logfile;

Connect-viserver -Server #VCenterFQDN# -User 'XXX' -Password 'XXX'

$vmNames = get-cluster #SureBackupClusterName# | get-vm | Where-Object {$_.name -Like "*_*" -And $_.name -NotLike "*_Restore" -And $_.name -NotLike "*_Lab" -And $_.name -NotLike "*_Virtual*"}
$vmNamesNoNetwork = $vmNames | where { $Null -match $_.extensiondata.network }
$vmNamesNoNetwork | foreach {
    # get Folder to identify SureLab
    $surelab = (($_.Folder).ToString()).length
    $surelab = (($_.Folder).ToString()).Remove(0,$surelab-1)

    # get original VM name and split at "_"
    $oriname = $_.name.Split("_")

    write-output "$date - VM: $oriname" | out-file -append -filepath $logfile;
    
    # get original Network 
    $orinetwork = (Get-NetworkAdapter -VM $oriname[0]).NetworkName
    write-output "$date - OriNetwork: $orinetwork" | out-file -append -filepath $logfile;
    
    # set SureLab Network Name
    $sbnetwork = "SureBackup_VL" + $surelab + "_" + $orinetwork
    write-output "$date - SBNetwork: $sbnetwork" | out-file -append -filepath $logfile;
    
    # connect SB Network to SB VM
    try
    {
        Get-VM -name $_ | New-NetworkAdapter -NetworkName $sbnetwork -Type Vmxnet3 -StartConnected -ErrorAction stop;
    }
    catch   
    {
        $ErrorMessage = $_.Exception;
        write-output "$ErrorMessage" | out-file -append -filepath $logfile;
    }

}

Disconnect-VIServer -force -Confirm:$false -ErrorAction SilentlyContinue
write-output "$date - disconnect vcenter" | out-file -append -filepath $logfile;

exit 
############################################################################################
So please excuse, i am not a scripting guy. It might be not the most elegant version but it works quite well for us.

Unfortunately the result window of the surebackup job still complains about not being able to remap networking which results in a skipped ping test, but this do not reduce the usability of the restored VMs itself.

We haven´t finished testing this. Feedback welcome.

Kind regads

Andi
mamosorre84
Veeam Legend
Posts: 336
Liked: 34 times
Joined: Oct 24, 2016 3:56 pm
Full Name: Marco Sorrentino
Location: Ancona - Italy
Contact:

Re: SureBackup and NSX-T Workaround

Post by mamosorre84 »

HannesK wrote: Dec 01, 2022 1:04 pm Hello,
and welcome to the forums.

The plan is to fix it after V12 as soon as possible. Unfortunately, we cannot provide a timeline.

Question: do you use VDS or N-VDS?

Sorry for the inconvenience,
Hannes
Hi Hannes, I think it has not been fixed..right? :(
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: SureBackup and NSX-T Workaround

Post by HannesK » 1 person likes this post

Hello,
correct. "After V12" means "a version after V12" (could be 12a earliest, but not guaranteed)

Best regards,
Hannes
mv@cloudio.dk
Service Provider
Posts: 3
Liked: 1 time
Joined: Sep 06, 2019 11:30 am
Full Name: Martin Veng
Contact:

Re: SureBackup and NSX-T Workaround

Post by mv@cloudio.dk »

Hi,
Will update 12.1 have any support for NSX-T with surebackup?
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: SureBackup and NSX-T Workaround

Post by HannesK »

Hello,
yes, 12.1 is the solution.

Best regards,
Hannes
VinMeisterT
Lurker
Posts: 2
Liked: never
Joined: Feb 14, 2018 6:04 am
Full Name: Vinu Thomas
Contact:

Re: SureBackup and NSX-T Workaround

Post by VinMeisterT »

Confirmed this has been resolved in 12.1 - tested successfully.
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 113 guests