Host-based backup of VMware vSphere VMs.
Post Reply
blindsidematt
Novice
Posts: 9
Liked: 1 time
Joined: Mar 06, 2018 5:55 pm
Full Name: Matt Thomas
Contact:

Replication "Resource in use" Errors

Post by blindsidematt »

Since I have had a support case open for several weeks now with no solution here I am.

Ever since (I think) about the time I went to V10 i have been plagued with random errors in my replication job. The errors occur randomly across all roughly 110 VM's in my job, with only 1 or 2 being affected each run. The error I am getting is as follows:

3/29/2021 4:02:51 PM :: Processing configuration Error: The resource '26' is in use.
3/29/2021 4:03:11 PM :: Error: The resource '26' is in use.


The resource number changes every time randomly and the VM that fails changes every time randomly. I have tracked the error into vSphere as well as follows:

The resource '26' is in use. com.vmware.vim.vpxd.dvs.portInUse.label

Now I assume that the issue is for some reason when Veeam processes the configuration and realizes I have network remapping turned on that it removes and tries to re-add the VM to the VDS but for some reason tries to assign a port that is already in use by another VM.
Why doesn't Veeam just realize that the replica VM is already in the right port group and nothing has changed and simply skip that part? That would solve the issue in the first place.

I have recreated the VDS on the VMware side, re-added the vCenter's to Veeam, created all new Veeam proxies with more resources, Forced a full replication and even deleted and recreated the replication job.

After the job fails it usually retries and succeeds but sometimes it fails again a second time. I have opened a VMware support case as well but they seemed perplexed as well.
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Replication "Resource in use" Errors

Post by veremin »

Hi, Matt,

Can you provide us with the stuck ticket number? We will check the logs and the case details before coming up with concrete answer.

Thanks!
blindsidematt
Novice
Posts: 9
Liked: 1 time
Joined: Mar 06, 2018 5:55 pm
Full Name: Matt Thomas
Contact:

Re: Replication "Resource in use" Errors

Post by blindsidematt »

Veeam Support - Case # 04691246
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Replication "Resource in use" Errors

Post by PetrM »

Hi Matt,

Don't hesitate to request an escalation of the support case if you feel that deeper analysis is required, especially if you need to simplify interaction with 3rd party support team. Also, you could probably take a look at vpxd.log to fetch some additional information regarding objects which lock a port at the moment of replica configuration update.

Thanks!
segfault
Enthusiast
Posts: 48
Liked: 21 times
Joined: Dec 14, 2017 8:07 pm
Full Name: John Garner
Contact:

Re: Replication "Resource in use" Errors

Post by segfault »

Was there a resolution to this issue?

We also see it happen about 10 times a day (out of a few thousand replications). It always clears on the next retry so it is annoying but not super critical.

It would be good to know before I open a support case if it is a known issue that will be fixed in a future patch or if it is something that we can contact support to have fixed.

Thanks,

--john
Egor Yakovlev
Veeam Software
Posts: 2536
Liked: 680 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Replication "Resource in use" Errors

Post by Egor Yakovlev »

Hi John,

From what I see we have just a few similar reported cases. The error itself pops when Veeam performs target VM configuration request(via VMware API), and receives back an error. You should be able to track the issue source in vSphere client too.
Reason why Vmware kicks back our request is unknown(no customers came back with VMware ticket results), however it might be due to other software interfering - for example monitoring system might apply tags to the same VM at the same moment when Veeam has requested configuration change, leading our request to be rejected. That would explain why error pops on random VMs, at random times and on retry it passes just well.

You can open a new ticket and refer to one above, so support will be aware. And it would be great if you share your case ID here as well.

/Thanks!
segfault
Enthusiast
Posts: 48
Liked: 21 times
Joined: Dec 14, 2017 8:07 pm
Full Name: John Garner
Contact:

Re: Replication "Resource in use" Errors

Post by segfault »

Case #04753900 created to track the issue and 4G of logs (last days worth) were uploaded. I noted the other case number in the ticket.

Based on the comments above, I suspect that the "other" application is veeam itself as the vCenter server and ESX hosts are dedicated as a DR site. There are several replication jobs that run, each with 50+ vm's. So it is possible that one job is bumping into another when it comes to the timing the network port changes.

--john
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Replication "Resource in use" Errors

Post by PetrM »

Hi John,

It might be one of the possible versions but I would let our engineers to look for some specific traces in debug logs in order to understand either this assumption is true or not.

Thanks!
Fredde7803
Novice
Posts: 4
Liked: 2 times
Joined: Jul 11, 2018 7:57 am
Full Name: Fredrik Carlsson
Contact:

Re: Replication "Resource in use" Errors

Post by Fredde7803 »

Hi

I can confirm that we see these errors as well, they appear very random and the frequency is rare.

We replicate around 500 VMs from one site to another (only one vCenter) daily and the error appears a few times per week.

What errors should be visible on the vSphere side - I've never managed to see something of interest but I've obviously not been digging deep enough?
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Replication "Resource in use" Errors

Post by PetrM »

Hi Fredrik,

Don't hesitate to contact our support if you want to dig deeper into it. You may paste a support case ID over here for our reference.

Thanks!
segfault
Enthusiast
Posts: 48
Liked: 21 times
Joined: Dec 14, 2017 8:07 pm
Full Name: John Garner
Contact:

Re: Replication "Resource in use" Errors

Post by segfault » 1 person likes this post

Update:

Support confirmed that in my particular case, the other application is Veeam itself. Logs showed two replication threads both making a call for the next available port number at the same time. At that point it is a race to see which thread gets to claim the port. The looser tosses the error. It appears to be a textbook concurrency issue that I've been guilty of making myself a few times in the threaded apps that I've developed over the years.

Now it is off to elevate the issue to the next level to verify the initial findings.
Post Reply

Who is online

Users browsing this forum: Google Feedfetcher and 73 guests