Discussions specific to the VMware vSphere hypervisor
Andreas Neufert
VP, Product Management
Posts: 5063
Liked: 1032 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: vSphere 7.0 U2 Support

Post by Andreas Neufert »

Veeam provided support for latest VMware/Microsoft versions only with the latest version and most actual patch level. It will be the same here.

lolbebis
Influencer
Posts: 23
Liked: 4 times
Joined: Feb 26, 2020 9:33 am
Full Name: Mattias Jacobsson
Contact:

[MERGED] Problem with backups after upgrade to 7.0 U2

Post by lolbebis »

Hi,
after upgrading vCenter to 7.0 U2 I am having problems with linux hot add proxies.
Image
I am receiving "vmdk has been skipped due to an unsupported type (independent disk).
All of the servers above only have dependent disks.
Then the disk get stuck on the hot add proxy and I have to remove the mapping manually.

And yes I know that U2 is not supported so no need to point that out :)

This has only occurred on linux proxies and with different virtual machines.
Never on a Windows proxy, and we have way more Windows proxies.

soncscy
Veeam Legend
Posts: 361
Liked: 176 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey Carel
Contact:

[MERGED] Re: Problem with backups after upgrade to 7.0 U2

Post by soncscy » 1 person likes this post

Heya Mattias,

Well, since it is unsupported, is this mostly just to let others know about the issue or hoping to get a resolution?

I've peeked at logs a bit in the past and you should be able to see on your proxy where it tries to hotadd the device and how it's detecting the disks.

I suppose something must somehow be sending an identifier that Veeam picks up as either independent or maybe there's some catchall where if it cannot identify the disks normally, it just considers it unsupported and returns such a message.

Out of curiosity, you're on v11 or v10? If it's v11, does the same thing happen with NBD?

Andreas Neufert
VP, Product Management
Posts: 5063
Liked: 1032 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: vSphere 7.0 U2 Support

Post by Andreas Neufert »

Please see above. Merged it into the official thread.

lolbebis
Influencer
Posts: 23
Liked: 4 times
Joined: Feb 26, 2020 9:33 am
Full Name: Mattias Jacobsson
Contact:

Re: [MERGED] Re: Problem with backups after upgrade to 7.0 U2

Post by lolbebis »

soncscy wrote: Mar 17, 2021 12:34 pm Heya Mattias,

Well, since it is unsupported, is this mostly just to let others know about the issue or hoping to get a resolution?

I've peeked at logs a bit in the past and you should be able to see on your proxy where it tries to hotadd the device and how it's detecting the disks.

I suppose something must somehow be sending an identifier that Veeam picks up as either independent or maybe there's some catchall where if it cannot identify the disks normally, it just considers it unsupported and returns such a message.

Out of curiosity, you're on v11 or v10? If it's v11, does the same thing happen with NBD?
If someone has a resolution I would be happy :)
But otherwise I just use NBD on our Windows proxies no biggie ( I have not tried NBD on the Linux proxy).
I am on v10 will upgrade as soon first patch arrives.

Gostev
SVP, Product Management
Posts: 28165
Liked: 4983 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 7.0 U2 Support

Post by Gostev »

Please note that using NBD transport mode with Linux-based backup proxies requires V11.

Gostev
SVP, Product Management
Posts: 28165
Liked: 4983 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 7.0 U2 Support

Post by Gostev » 4 people like this post

Just a quick update: our autotests did not show any issues with the 7.0 U2 GA code. But they verify core functionality only, while full regression testing still continues!

al40inn
Lurker
Posts: 1
Liked: never
Joined: Mar 24, 2021 11:52 am
Full Name: Alexey Sorokin
Contact:

Re: vSphere 7.0 U2 Support

Post by al40inn »

no issues with vCenter 7.0 U2, confirm. but not working when updating esxi to 7.0 U2

Code: Select all

Error: Failed to open VDDK disk [...] ( is read-only mode - [true] )
...
Transports: [nbd];Read Only: [true]] failed because of the following errors:
Failed to open disk for read.
Failed to upload disk. Skipped arguments: [vddkConnSpec>];
Agent failed to process method {DataTransfer.SyncDisk}.

seems to be same old story

Gostev
SVP, Product Management
Posts: 28165
Liked: 4983 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 7.0 U2 Support

Post by Gostev »

We're not seeing this particular issue in our own labs.

Gostev
SVP, Product Management
Posts: 28165
Liked: 4983 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 7.0 U2 Support

Post by Gostev » 3 people like this post

UPDATE: I just heard from QC that 7.0 U2 breaks V11 CDP functionality.

jorgedlcruz
Veeam Software
Posts: 434
Liked: 222 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: vSphere 7.0 U2 Support

Post by jorgedlcruz »

I am facing the same issue with the transport modes, randomly needing NBD when never needed that before.
Jorge de la Cruz
Senior Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2021 / Nutanix Technology Champion 2018-2019 / InfluxAce

tsightler
VP, Product Management
Posts: 5837
Liked: 2662 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: vSphere 7.0 U2 Support

Post by tsightler »

@jorgedlcruz What transport modes are failing for you? The note above was about NBD failing, which just doesn't seem right to me as I've seen no failure with NBD transport mode in my own lab since upgrading to 7.0 U2 earlier in the week. Also, so far no issues with Direct NFS mode, which is the primary transport mode I use in my lab setup. I haven't tried other transport modes however, so curious about your experience so far with others.

jorgedlcruz
Veeam Software
Posts: 434
Liked: 222 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: vSphere 7.0 U2 Support

Post by jorgedlcruz »

Yes, of course.
I have
  • ESXi 7.0U2
  • A normal Windows VM to backup inside this host
  • Linux Proxy on this host that was working fine using HotAdd before VMware upgrade
Config looks looks like this:
Image
Error looks the usual:
Image
And log

Code: Select all

[26.03.2021 17:39:45] <01> Error        Failed to dispatch pending task. SessId: [85b4aeaf-35ec-4465-874f-8be6f26d7e36], Task: [AD-DNS-001 from job "VMware - Backup of AD Domain Controller (Incremental)" (task id:ae4eb0d9-a5f8-4f72-874a-6e9a96a3458e, task sess id:76021dd3-0e35-4982-88f3-570cc57e83e1, job sess id:85b4aeaf-35ec-4465-874f-8be6f26d7e36)] (System.Exception)
[26.03.2021 17:39:45] <01> Error        No backup proxy is able to process this VM due to proxy processing mode restrictions. (System.Exception)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CProxyAquireHelper.FindBestProxy(CResDispatchIterationContext dispatchContext, CResourceTask task, CViProxyResourceRequest request, CResourceUsages resUsages, CViProxyBackupTaskLocks viProxyLocks, CViProxyResourceDesc[] firstPriorityProxies, CViProxyResourceDesc[] unsuitableProxies, String& customNotReadyMessage)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CViProxyResDispatcher.CanAcquireResource(CResDispatchIterationContext dispatchContext, CResourceTask task, IResourceRequest resourceRequest, Boolean sessionLogEnabled, IResourceResponse[] acquiredResources)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CViProxyRepositoryPairResourceDispatcher.CanAcquireResource(CResDispatchIterationContext dispatchContext, CResourceTask task, IResourceRequest resRequest, Boolean sessionLogEnabled, IResourceResponse[] acquiredResources)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CMultiResDispatcher.CanAcquireResource(CResDispatchIterationContext dispatchContext, CResourceTask task, IResourceRequest resRequest, Boolean sessionLogEnabled, IResourceResponse[] acquiredResources)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CNumerousResDispatcher.CanAcquireResource(CResDispatchIterationContext dispatchContext, CResourceTask task, IResourceRequest resRequest, Boolean sessionLogEnabled, IResourceResponse[] acquiredResources)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CResourceTaskScheduler.IsAllResourcesAvaliable(CResDispatchIterationContext dispatchContext, CResourceTask task)
[26.03.2021 17:39:45] <01> Error           at Veeam.Backup.ResourceScheduler.CResourceTaskScheduler.DispatchPendingTask(CResDispatchIterationContext dispContext, CResourceTask task)
[26.03.2021 17:39:45] <01> Error        Unable to allocate processing resources.
If I enable the small checkbox to failover to Network mode, works. So, at least the good news are the Linux Proxy in v11 is using correctly the new NBD mode:
Image

It might be my environment, just odd as it happened after upgrade the ESXi to 7.0 U2.

CDP is broken for me with the next error:

Code: Select all

3/25/2021 4:35:43 PM :: VM configuration for the initial sync completed with errors Error: The underlying connection was closed: The connection was closed unexpectedly.
The remote server returned an error: (503) Server Unavailable.
Unable to connect to the remote server
EDIT
More randomness, that was working before, all Linux Proxies, failing with the next error.
Image

These were working fine a few days ago, before update the ESXi to 7.0 U2:
Image
Jorge de la Cruz
Senior Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2021 / Nutanix Technology Champion 2018-2019 / InfluxAce

tsightler
VP, Product Management
Posts: 5837
Liked: 2662 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: vSphere 7.0 U2 Support

Post by tsightler »

Yeah, definitely know CDP doesn't work. With hotadd failure, that's an interesting one that I haven't seen. Almost wonder if it's just some kind of infrastructure scan failure as I just tested hotadd mode (not a transport mode I normally use) with two Linux proxies and it worked perfectly and super fast!

jorgedlcruz
Veeam Software
Posts: 434
Liked: 222 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: vSphere 7.0 U2 Support

Post by jorgedlcruz »

Deploying one new Linux Proxy in Ubuntu 20.04, to see if that is the issue. If I force the same job to a Windows Proxy, then it works, if I put the 2 Linux Proxies it fails. Same for the first job, only working in NBD and failing in hotadd. All of this was working before upgrade the ESXi.

Truly frustrating couple of days, and not knowing what is really happening.
Jorge de la Cruz
Senior Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2021 / Nutanix Technology Champion 2018-2019 / InfluxAce

tsightler
VP, Product Management
Posts: 5837
Liked: 2662 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: vSphere 7.0 U2 Support

Post by tsightler » 2 people like this post

Truly frustrating couple of days, and not knowing what is really happening.
Shouldn't be too frustrating since you chose to upgrade knowing it's not supported yet! :D
That being said, I'm definitely not able to reproduce the issue you are having, hope you get it sorted.

jorgedlcruz
Veeam Software
Posts: 434
Liked: 222 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: vSphere 7.0 U2 Support

Post by jorgedlcruz »

Yes, I know. Still bummed to see strange uncoherent errors, like Windows Proxies working good and not Linux.
So, I've deployed 2 new Linux Proxies (set to Automatic with Network failover), Ubuntu 20.04, from scratch, added to VBR and exactly the same issue, be careful as it says hotadd but then it fails to network:
Image

Do you have full hot-add on ESXi 7.0U2 with Linux Proxies? Does not jump into that nbd for you?

I guess I will wait until a patch is released then, and try it again so do not waste anyone else time.
Jorge de la Cruz
Senior Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2021 / Nutanix Technology Champion 2018-2019 / InfluxAce

tsightler
VP, Product Management
Posts: 5837
Liked: 2662 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: vSphere 7.0 U2 Support

Post by tsightler »

Yes, full hotadd support works for me, it's easy to tell by the performance alone! :)

Image

jorgedlcruz
Veeam Software
Posts: 434
Liked: 222 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: vSphere 7.0 U2 Support

Post by jorgedlcruz »

Small update:
* All my Windows Proxies were working as expected, HotAdd, and all
* The Linux Proxies that started to to do nbd, updated them to 20.04, removed them and re-added them to the VBR, and all good.
* CDP is still broken for me

At least it seems there is no real issue with the HotAdd, it might be some broken stuff on my VBR, will monitor the thread to read more about the CDP issue.
Jorge de la Cruz
Senior Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2021 / Nutanix Technology Champion 2018-2019 / InfluxAce

jamerson
Expert
Posts: 324
Liked: 18 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: vSphere 7.0 U2 Support

Post by jamerson »

We are testing CDP for one of our customer.
this error at CDP keeps showing up
VM configuration for the initial sync completed with errors Error: The remote server returned an error: (503) Server Unavailable.
is this because the hosts are at VMware ESXi, 7.0.2, 17630552?

HannesK
Veeam Software
Posts: 7626
Liked: 1306 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: vSphere 7.0 U2 Support

Post by HannesK »

@jamerson: yep, please see above... U2 breaks CDP

Frosty
Expert
Posts: 190
Liked: 42 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: vSphere 7.0 U2 Support

Post by Frosty »

Having read through the above, it looks like the key issues are (1) Linux proxies; and (2) CDP issue.
I don't use either of those, I have a very small setup, single Windows Server acting as backup server and proxy.
I just updated VBR from v10 to v11 a couple of days ago and it seems to be fine.
I'd already upgraded vCenter to v7.0 U2 and got a one-off warning, but no errors on any backups.
Haven't upgraded any hosts to ESXi 7.0 U2 yet and I have a couple of good reasons why I want to do that sooner rather than later.
Wondering if it is worth taking the risk on one of my 3 hosts, just to see if it is okay.

HannesK
Veeam Software
Posts: 7626
Liked: 1306 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: vSphere 7.0 U2 Support

Post by HannesK »

If it's working now, what would be the advantage of upgrading to something unsupported?

Regnor
Veeam Legend
Posts: 439
Liked: 98 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: vSphere 7.0 U2 Support

Post by Regnor »

Frosty wrote: Apr 09, 2021 5:58 am Wondering if it is worth taking the risk on one of my 3 hosts, just to see if it is okay.
What is the advantage for you in upgrading to 7.0U2?

juanmachado
Novice
Posts: 4
Liked: never
Joined: Nov 18, 2020 3:59 pm
Full Name: Juan Machado
Contact:

Re: vSphere 7.0 U2 Support

Post by juanmachado »

I think we are still NO Go on 7.0U2:
Issues:
- Linux proxies
- CDP issue

Gostev
SVP, Product Management
Posts: 28165
Liked: 4983 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 7.0 U2 Support

Post by Gostev » 3 people like this post

You can add reduced NBDSSL transport performance to the list... about 30% drop, we are opening a case with VMware.

popjls
Influencer
Posts: 19
Liked: 1 time
Joined: Jun 25, 2018 3:41 am
Contact:

Re: vSphere 7.0 U2 Support

Post by popjls »

So I've updated to 7.0U2 and have just seen this - I actually thought 7U2 was a completely different build :( Hmm.... Have had a few backups work fine but should I be rolling esxi back now? Will that affect the backups?

Shaokat
Novice
Posts: 3
Liked: never
Joined: Jan 27, 2020 4:03 pm
Full Name: Md Shaokat Ullaha
Location: Dhaka, Bangladesh
Contact:

Re: vSphere 7.0 U2 Support

Post by Shaokat »

I think it would no affect the backup. I suggest for take full backup after rolling back.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 22 guests