vSphere 7.0 U2 Support

[Andreas Neufert]

Veeam provided support for latest VMware/Microsoft versions only with the latest version and most actual patch level. It will be the same here.

[lolbebis]

Hi,
after upgrading vCenter to 7.0 U2 I am having problems with linux hot add proxies.

I am receiving "vmdk has been skipped due to an unsupported type (independent disk).
All of the servers above only have dependent disks.
Then the disk get stuck on the hot add proxy and I have to remove the mapping manually.

And yes I know that U2 is not supported so no need to point that out

This has only occurred on linux proxies and with different virtual machines.
Never on a Windows proxy, and we have way more Windows proxies.

[soncscy]

Heya Mattias,

Well, since it is unsupported, is this mostly just to let others know about the issue or hoping to get a resolution?

I've peeked at logs a bit in the past and you should be able to see on your proxy where it tries to hotadd the device and how it's detecting the disks.

I suppose something must somehow be sending an identifier that Veeam picks up as either independent or maybe there's some catchall where if it cannot identify the disks normally, it just considers it unsupported and returns such a message.

Out of curiosity, you're on v11 or v10? If it's v11, does the same thing happen with NBD?

[Andreas Neufert]

Please see above. Merged it into the official thread.

[lolbebis]

Heya Mattias,

Well, since it is unsupported, is this mostly just to let others know about the issue or hoping to get a resolution?

I've peeked at logs a bit in the past and you should be able to see on your proxy where it tries to hotadd the device and how it's detecting the disks.

I suppose something must somehow be sending an identifier that Veeam picks up as either independent or maybe there's some catchall where if it cannot identify the disks normally, it just considers it unsupported and returns such a message.

Out of curiosity, you're on v11 or v10? If it's v11, does the same thing happen with NBD?

If someone has a resolution I would be happy
But otherwise I just use NBD on our Windows proxies no biggie ( I have not tried NBD on the Linux proxy).
I am on v10 will upgrade as soon first patch arrives.

[Gostev]

Please note that using NBD transport mode with Linux-based backup proxies requires V11.

[Gostev]

Just a quick update: our autotests did not show any issues with the 7.0 U2 GA code. But they verify core functionality only, while full regression testing still continues!

[al40inn]

no issues with vCenter 7.0 U2, confirm. but not working when updating esxi to 7.0 U2

Code: Select all

Error: Failed to open VDDK disk [...] ( is read-only mode - [true] )
...
Transports: [nbd];Read Only: [true]] failed because of the following errors:
Failed to open disk for read.
Failed to upload disk. Skipped arguments: [vddkConnSpec>];
Agent failed to process method {DataTransfer.SyncDisk}.

seems to be same old story

[Gostev]

We're not seeing this particular issue in our own labs.

[Gostev]

UPDATE: I just heard from QC that 7.0 U2 breaks V11 CDP functionality.

[jorgedlcruz]

I am facing the same issue with the transport modes, randomly needing NBD when never needed that before.

[tsightler]

@jorgedlcruz What transport modes are failing for you? The note above was about NBD failing, which just doesn't seem right to me as I've seen no failure with NBD transport mode in my own lab since upgrading to 7.0 U2 earlier in the week. Also, so far no issues with Direct NFS mode, which is the primary transport mode I use in my lab setup. I haven't tried other transport modes however, so curious about your experience so far with others.

[jorgedlcruz]

Yes, of course.
I have

• ESXi 7.0U2
• A normal Windows VM to backup inside this host
• Linux Proxy on this host that was working fine using HotAdd before VMware upgrade

Config looks looks like this:

Error looks the usual:

And log

Code: Select all

[26.03.2021 17:39:45] <01> Error        Failed to dispatch pending task. SessId: [85b4aeaf-35ec-4465-874f-8be6f26d7e36], Task: [AD-DNS-001 from job "VMware - Backup of AD Domain Controller (Incremental)" (task id:ae4eb0d9-a5f8-4f72-874a-6e9a96a3458e, task sess id:76021dd3-0e35-4982-88f3-570cc57e83e1, job sess id:85b4aeaf-35ec-4465-874f-8be6f26d7e36)] (System.Exception)
[26.03.2021 17:39:45] <01> Error        No backup proxy is able to process this VM due to proxy processing mode restrictions. (System.Exception)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CProxyAquireHelper.FindBestProxy(CResDispatchIterationContext dispatchContext, CResourceTask task, CViProxyResourceRequest request, CResourceUsages resUsages, CViProxyBackupTaskLocks viProxyLocks, CViProxyResourceDesc[] firstPriorityProxies, CViProxyResourceDesc[] unsuitableProxies, String& customNotReadyMessage)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CViProxyResDispatcher.CanAcquireResource(CResDispatchIterationContext dispatchContext, CResourceTask task, IResourceRequest resourceRequest, Boolean sessionLogEnabled, IResourceResponse[] acquiredResources)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CViProxyRepositoryPairResourceDispatcher.CanAcquireResource(CResDispatchIterationContext dispatchContext, CResourceTask task, IResourceRequest resRequest, Boolean sessionLogEnabled, IResourceResponse[] acquiredResources)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CMultiResDispatcher.CanAcquireResource(CResDispatchIterationContext dispatchContext, CResourceTask task, IResourceRequest resRequest, Boolean sessionLogEnabled, IResourceResponse[] acquiredResources)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CNumerousResDispatcher.CanAcquireResource(CResDispatchIterationContext dispatchContext, CResourceTask task, IResourceRequest resRequest, Boolean sessionLogEnabled, IResourceResponse[] acquiredResources)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CResourceTaskScheduler.IsAllResourcesAvaliable(CResDispatchIterationContext dispatchContext, CResourceTask task)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CResourceTaskScheduler.DispatchPendingTask(CResDispatchIterationContext dispContext, CResourceTask task)
[26.03.2021 17:39:45] <01> Error        Unable to allocate processing resources.

If I enable the small checkbox to failover to Network mode, works. So, at least the good news are the Linux Proxy in v11 is using correctly the new NBD mode:


It might be my environment, just odd as it happened after upgrade the ESXi to 7.0 U2.

CDP is broken for me with the next error:

Code: Select all

3/25/2021 4:35:43 PM :: VM configuration for the initial sync completed with errors Error: The underlying connection was closed: The connection was closed unexpectedly.
The remote server returned an error: (503) Server Unavailable.
Unable to connect to the remote server

EDIT
More randomness, that was working before, all Linux Proxies, failing with the next error.


These were working fine a few days ago, before update the ESXi to 7.0 U2:

[tsightler]

Yeah, definitely know CDP doesn't work. With hotadd failure, that's an interesting one that I haven't seen. Almost wonder if it's just some kind of infrastructure scan failure as I just tested hotadd mode (not a transport mode I normally use) with two Linux proxies and it worked perfectly and super fast!

[jorgedlcruz]

Deploying one new Linux Proxy in Ubuntu 20.04, to see if that is the issue. If I force the same job to a Windows Proxy, then it works, if I put the 2 Linux Proxies it fails. Same for the first job, only working in NBD and failing in hotadd. All of this was working before upgrade the ESXi.

Truly frustrating couple of days, and not knowing what is really happening.

[tsightler]

Truly frustrating couple of days, and not knowing what is really happening.

Shouldn't be too frustrating since you chose to upgrade knowing it's not supported yet!
That being said, I'm definitely not able to reproduce the issue you are having, hope you get it sorted.

[jorgedlcruz]

Yes, I know. Still bummed to see strange uncoherent errors, like Windows Proxies working good and not Linux.
So, I've deployed 2 new Linux Proxies (set to Automatic with Network failover), Ubuntu 20.04, from scratch, added to VBR and exactly the same issue, be careful as it says hotadd but then it fails to network:


Do you have full hot-add on ESXi 7.0U2 with Linux Proxies? Does not jump into that nbd for you?

I guess I will wait until a patch is released then, and try it again so do not waste anyone else time.

[tsightler]

Yes, full hotadd support works for me, it's easy to tell by the performance alone!

[jorgedlcruz]

Small update:
* All my Windows Proxies were working as expected, HotAdd, and all
* The Linux Proxies that started to to do nbd, updated them to 20.04, removed them and re-added them to the VBR, and all good.
* CDP is still broken for me

At least it seems there is no real issue with the HotAdd, it might be some broken stuff on my VBR, will monitor the thread to read more about the CDP issue.

[jamerson]

We are testing CDP for one of our customer.
this error at CDP keeps showing up
VM configuration for the initial sync completed with errors Error: The remote server returned an error: (503) Server Unavailable.
is this because the hosts are at VMware ESXi, 7.0.2, 17630552?

[HannesK]

@jamerson: yep, please see above... U2 breaks CDP

[Frosty]

Having read through the above, it looks like the key issues are (1) Linux proxies; and (2) CDP issue.
I don't use either of those, I have a very small setup, single Windows Server acting as backup server and proxy.
I just updated VBR from v10 to v11 a couple of days ago and it seems to be fine.
I'd already upgraded vCenter to v7.0 U2 and got a one-off warning, but no errors on any backups.
Haven't upgraded any hosts to ESXi 7.0 U2 yet and I have a couple of good reasons why I want to do that sooner rather than later.
Wondering if it is worth taking the risk on one of my 3 hosts, just to see if it is okay.

[HannesK]

If it's working now, what would be the advantage of upgrading to something unsupported?

[Regnor]

Wondering if it is worth taking the risk on one of my 3 hosts, just to see if it is okay.

What is the advantage for you in upgrading to 7.0U2?

[juanmachado]

I think we are still NO Go on 7.0U2:
Issues:
- Linux proxies
- CDP issue

[Gostev]

You can add reduced NBDSSL transport performance to the list... about 30% drop, we are opening a case with VMware.

[popjls]

So I've updated to 7.0U2 and have just seen this - I actually thought 7U2 was a completely different build Hmm.... Have had a few backups work fine but should I be rolling esxi back now? Will that affect the backups?

[Shaokat]

I think it would no affect the backup. I suggest for take full backup after rolling back.

[b.gahr]

@Gostev

is there any ETA for the next cumulative patch as mentioned in your newsletter from April 19th? Did I understand this correctly, that this patch will bring support for vSphere 7 U2?

[Gostev]

Between a week and a few weeks, depending on how testing goes (it is a big patch). Yes, your understanding is correct.