SureBackup v9 Error

[JaxIsland7575]

I upgraded a site to v9 yesterday and everything went smoothly. Backups, backup copy, and tape jobs all worked without issue. I tried to run the SureBackup job this morning and it fails with:

3/17/2016 2:12:06 PM Error [SERVER1]: Error: An item with the same key has already been added.
3/17/2016 2:12:12 PM Error Failed to stabilize virtual machine 'SERVER1'
An item with the same key has already been added.

The issue seems to stem from multiple network adapters in the machine. The SERVER1 I am testing has 3 adapters on 3 separate networks. If I only add the primary the job goes through but warns about not mapping the second adapter. If I add the second adapter to the VLAB and add the Vnic then it fails with the above error.

I opened a case #01733603 and they wanted me to install updates and reboot the guest, re-run the backup job, then try the SureBackup job again. I did those steps and I still see the same error.

The SureBackup job is running on the physical Veeam server itself, not in the Hyper-V cluster. So on the Veeam server I went to Hyper-V Manager and added the 2 other networks into the vswitch configuration but when adding them in the virtual lab it still shows that it cannot resolve network settings.

My guest servers that have jobs linked to the SureBackup job have a combination of 1, 2 or 3 of these adapters in them so I need to be able to run a SureBackup job(s) without getting warnings that it cannot map a network or failures that is described above. I updated the case via email and am waiting to hear back. I will put this on the forums in case someone has encountered this before.

Thank you for any help!

[nefes]

There is no logs in your case, unfortunately, so I can't check them to find out the reason.
Described error can be caused by misconfiguration of Virtual Lab.

[JaxIsland7575]

Thank you for the reply. I have uploaded the latest logs to the case.

I have moved past the first error, which turned out that I needed to manually create a vSwitch for each adapter subnet in Hyper-V manager. Once I have a unique vSwitch name for the different subnets it moved past the error and started the SureBackup job.

The issue I have now is I added the AD and GC scripts to the job with this being a domain controller and the first time it ran the job failed on the script. The second time I ran it, the job ran for over 16 hours stuck on the script step. I had to manually kill the job and reboot the server to get it cleared.

Can you take a look and see why the scripts are failing?

Thanks!

[nefes]

I've checked your logs and looks like you have the same problem like we already researching in another case, number 01724637.
Could you please create similar job, but don't select credentials at Verification options?
It should help you to avoid hangs and fails, meanwhile we will continue researching this problem.

[JaxIsland7575]

I went through and recreated the App Group, VLabs, and SureBackup jobs without modifying the verification option and it worked on the first pass. I am going to enable verification of the backup job and also link jobs to the SureBackup job. If I have any issues I will let you know. Thank you for taking a look at the logs.

As a side note, did I miss any documentation on having to manually create External Network Switches in Hyper-V vSwitch Manager to get multiple subnets to work on SureBackup jobs? I could not find any and if it doesn't exist hopefully this post will help someone else!

Thanks!

Edited to fix my terrible spelling

[nefes]

Glad you have temporary solution, hope will reach you back soon with permanent one.
As for External Switches - not sure that I've understood your point. Could you please describe it with more details - which configuration have you had and what was solution for your problem?

[nefes]

Ryan, please take a look at this topic: veeam-backup-replication-f2/surebackup- ... 33744.html
Looks like your problem with script tests are caused by MS update https://support.microsoft.com/en-us/kb/3139914
We are working on solution for this issue, meanwhile removing this KB or having script credentials blank are temporary workarounds.

[JaxIsland7575]

I will try and explain the switches configuration in more depth:

The configuration is to use the physical Veeam server to run the SureBackup jobs. During the setup I installed the Hyper-V role onto the Veeam server which gave me access to Hyper-V manager on the Veeam physical server. When I was troubleshooting and we determined that the issue was from the network selections in the Virtual Lab setup I opened Hyper-V manager and opened Virtual Switch Manager.
In there I saw 4 items under Virtual Switches:
3 networks attached to physical NIC ports:
LAN1 port#1
LAN2 port#2
LAN3 port#3

Also there was one Private Virtual Switch named: "Lab Isolated Network ()"

When I opened the properties of each item the 3 LAN ports were configured as External networks with a NIC port selected in the dropdown. The Lab Isolated Network () was configured as a Private Network.



When I opened the properties of the Virtual Lab I noticed that each of the subnets was using the same Lab Isolated Network () and assumed that that might be causing the error.



I removed the Virtual Lab and reconfigured except this time I manually typed in a unique name for each Isolated Network. Now each unique adapter, has a unique subnet, and a unique isolated network name is when the job ran without issue. I did not think to take screenshots of the bad configuration but these screenshots are of the current working configuration.



I hope that helps clear it up a bit, I can add more screenshots or try and go into more detail if you need me to.

Cheers!

[JaxIsland7575]

Ryan, please take a look at this topic: veeam-backup-replication-f2/surebackup- ... 33744.html
Looks like your problem with script tests are caused by MS update https://support.microsoft.com/en-us/kb/3139914
We are working on solution for this issue, meanwhile removing this KB or having script credentials blank are temporary workarounds.

Blanking out the credentials did work to get it going but I received 3 SQL failures on the last job run. I uploaded fresh logs this morning.

Cheers!

[nefes]

Let me rephrase your configuration. Correct me if I got you wrong.
You have 3 prod networks, say Prod1, Prod2, Prod3.
When you create virtual lab mappings for that networks, all of them mapped by default to Lab Isolated Network ().
After that on next step (Network Settings) you've created 3 NICs with different mapping settings.
When you renamed isolated networks to Isolated1, Isolated2, Isolated3 manually all works good.

Looks like we have an issue in wizard that allows you to create several NICs with same network name and different mapping options that leads to the errors you seen. We will fix it in one of upcoming patches/releases.

As for your new failures with SQL script - checked your logs and found access denied error. So we have chicken and egg problem: with creds all test scripts fails, without them sql check fails. Now it is up to you to decide, what to do: wait for the credentials problem fix from our side, give your current Veeam service account rights to sql servers in question (I believe, minimal permissions are enough, details may be found here, looks like you've had an experience with this problem before), or remove KB3139914 on backup server (not the best idea, cause it is security fix).

Sorry for inconvenience, that MS KB was a surprise to us.

[JaxIsland7575]

Let me rephrase your configuration. Correct me if I got you wrong.
You have 3 prod networks, say Prod1, Prod2, Prod3.
When you create virtual lab mappings for that networks, all of them mapped by default to Lab Isolated Network ().
After that on next step (Network Settings) you've created 3 NICs with different mapping settings.
When you renamed isolated networks to Isolated1, Isolated2, Isolated3 manually all works good.

That is a 100% accurate description of the issue.

As for your new failures with SQL script - checked your logs and found access denied error. So we have chicken and egg problem: with creds all test scripts fails, without them sql check fails. Now it is up to you to decide, what to do: wait for the credentials problem fix from our side, give your current Veeam service account rights to sql servers in question (I believe, minimal permissions are enough, details may be found here, looks like you've had an experience with this problem before), or remove KB3139914 on backup server (not the best idea, cause it is security fix).

This one was on me, I was looking in the wrong directory when I posted the new logs. I went through and found the correct log which showed me it was the access denied issue, and you are correct, I have had to resolve this myself in the past. What I will end up doing is going through each one individually and either removing the script from the SureBackup job or modifying permissions on the DB side for the Veeam service account.

I really appreciate all the time and help working through these, I only have until Friday at this site to get them all resolved and you were a big help!

Cheers!

[JaxIsland7575]

Also I want to add one thing, I was trying to recreate this fix on a second VBR server but kept continually getting warnings on one or two of the networks not mapping during the job. I removed all configuration and set it up multiple times. What happened is when I was in Hyper-V Manager configuring the network adapters I did not name it exactly the same as the Cluster settings on the production network. So if someone else has to use this workaround make sure that the networks from the production cluster to the SureBackup server match exactly, and that means spaces too

Cheers and thanks again for the help!