V12 - SureBackups failing since upgrade

[SnakeSK]

Case 05877943, after upgrade the appliance cannot configure IPs on the isolated sides of adapters, after 10 minutes it fails, Hyper-V environment with Windows Server 2019 and latest GA build, is anyone seeing the same? We tried recreating labs, no avail.

15.02.2023 13:49:55 Error Failed to start Hyper-V Virtual Lab 'SureBackup'

Failed to power on virtual lab proxy appliance

Proxy appliance VM network initialization failed. Check proxy appliance IP settings.

[Dima P.]

Hello,

Thank you for your post, investigating!

[SnakeSK]

QA team suggested enabling IPv6 communication in Network rules, dont know why this is required and it may be a bug, however after checking it, we successfully ran the SB tests.

[Egor Yakovlev]

We are trying to get to the root of the problem, and it does seems like a bug.
/Thanks!

[afocf]

Enabling IPv6 communication work for us as well. Thank you for posting the workaround, SnakeSK.

[StoopidMonkey]

What is IPv6 even used for in an all-IPv4 network?

[SnakeSK]

Dont know yet, maybe after adding support they forgot this edge case of IPv4 comms.

[Dima P.]

Thank you for the heads up folks! The case is still under investigation, we will update this thread with the results as soon as possible.

[JPMS]

QA team suggested enabling IPv6 communication in Network rules, dont know why this is required and it may be a bug, however after checking it, we successfully ran the SB tests.

'Network rules', where?!

Can we have a complete answer please? There are network settings on the Hyper-V host, the individual VMs, the Surebackup Lab. What exactly should we be changing and where?

[Mildur]

I believe it’s the Veeam network traffic rules within the Veeam console. You need to enable the „Enable IPv6 communication“ checkbox.

https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Best,
Fabian

[JPMS]

Many thanks for the quick response Fabian. I don't think I have looked at Network Rules since I first setup B&R about 5 years ago. I had forgot it existed and would never have found it!

I can confirm that Surebackup is now working after making that change.

[tvveen]

Same problem here. Enabling IPv6 communication within the Preferred Networks was not enough. We also had to enable IPv6 in the Proxy appliance settings.

[tvveen]

I've got one SureBackup which is still failing. This job is using Virtual Lab Proxy which has 6 isolated networks. Any updates on this issue?

[Dima P.]

Hello Tim,

If the mentioned workaround does not help then likely it's a different issue. Please raise a support case and share the case ID with us. Thank you!

[tvveen]

I've raised a support case: #05896536

[Dima P.]

Passed to QA, thank you for the case Tim!

[tvveen]

For anyone wondering or having the same problem: I've received a hotfix from the supportteam and this resolved our issue with the vLab in question. After installing the fix we also don't require applying the IPv6 workaround anymore.

[SnakeSK]

Latest patch undid the hotfix, what the hell?

[Egor Yakovlev]

Latest patch code was finished before SureBackup private fix was created, thus the rollback is expected.
Feel free to open the ticket and request updated version of the private fix if enabling-IPv6-workaround is not an option(private fix will be re-compiled soon).

/Thanks!

[RexfordHaugen_COLT]

Egor,

Any updates on the private fix being rolled out to a broader update? I'm seeing this error in our environment and we are not yet ready to enable IPv6.

[Egor Yakovlev]

Hi.

Fix for that issue is included in latest cumulative patch.

/Cheers!

[RexfordHaugen_COLT]

That patch resolved the issue.

Thank you!

[Sparkie]

Case 05964441

Hi,

I'm new to Veeam, and am having the same or similar issue as the OP.
We upgraded from V10 to V12 two months ago and since then all SureBackups have failed, as there is a connectivity issue with the virtual lab proxy.
Looking at Hyper-V when running a SureBackup job, it starts the proxy, before shutting it down, saying it failed to start.
We've applied the above patch (running Build: 12.0.0.1420 P20230412), but the problem persists.

From the proxy, I cannot even ping its default gateway.

Opened Case 05964441, but got nowhere with support and it has since been closed by Veeam.

Any assistance would be appreciated.

31/05/2023 3:02:30 PM Job started at 31/05/2023 3:02:28 PM
31/05/2023 3:02:31 PM Getting virtual lab configuration
31/05/2023 3:02:31 PM Using virtual lab Veeam V12 Virtual Lab
31/05/2023 3:02:40 PM Checking virtual lab
31/05/2023 3:02:45 PM Preparing proxy appliance network specification
31/05/2023 3:02:42 PM Network traffic throttling is enabled
31/05/2023 3:08:10 PM Error Failed to start virtual lab proxy appliance VM
31/05/2023 3:08:12 PM Powering off proxy appliance VM
31/05/2023 3:08:15 PM Error Failed to start Hyper-V Virtual Lab 'Veeam V12 Virtual Lab'
Failed to power on virtual lab proxy appliance
Failed to connect to proxy appliance VM using IPv4 address.
31/05/2023 3:08:15 PM Job finished
31/05/2023 3:08:15 PM Sending e-mail report

[Egor Yakovlev]

Hi Sparkie,

As far as I can see the case progress, it surely does seem like a networking issue.

1. Try redeploying the lab from the scratch. Just in case.
2. When it comes to networks, best way is to launch virtual lab appliance manually(HV manager), wait for it to load, and log in locally via KB1447.
At this point you have 2 machines on same network that must communicate with each other. Troubleshoot that part.

/Hope that helps!

[Sparkie]

Hi Egor,

Thanks for your response.

I’ve tried redeploying the lab a couple of times but keep getting the same result. I can log on to the proxy VM but I cannot even ping the default gateway from the VM deployed by Veeam. The settings (all IPs) are the same as those used in V10, but stopped working as soon as I upgraded to V12.

I guess we’ll just have to live without the lab.

[Egor Yakovlev]

Based on your description, I take it as virtual lab appliance VM cannot ping default gateway from prod-net side?

Do you have a static IP settings here or retrieve them from DHCP?
- once appliance is up, login and verify it gets correct IP settings (ifconfig)
- double check virtual network to which it has prod-NIC connected to - after all, this is your link to VBR server

P.S. you can always bump your case up with "Escalate" button and request troubleshooting with a support engineer in an online session

[CharlesK]

The workround that fixed it for us was to apply the IPv6 settings and then remove them.

IPv6 was set and then removed in both the Preferred Networks and the Proxy Appliance settings. The Proxy appliance were left as using DHCP rather than assigning a fixed IP. Did this for each of our 3 Virtual Labs and all are now fine.

Circumstances were:
3 July 23 - Upgraded from V11 to V12 build number 12.0.0.1420 P20230412 (which included the hotfix referred to above)
Set IPv6 on Preferred Networks
Set IPv6 on Proxy Appliance of each lab - DHCP and saved the settings
Removed IPv6 from Proxy appliance and saved the settings
Removed IPv6 on Preferred Networks

[AlexHeylin]

Apparently we just got hit by this too on v12.0.0 P20230718
Workaround from CharlesK fixed it.
Veeam, please address before 12.1 release.

[Egor Yakovlev]

QA is positive it was fixed in 12.1 code.
/Cheers!

[AlexHeylin]

Thanks Egor