Replication "Resource in use" Errors

[blindsidematt]

Since I have had a support case open for several weeks now with no solution here I am.

Ever since (I think) about the time I went to V10 i have been plagued with random errors in my replication job. The errors occur randomly across all roughly 110 VM's in my job, with only 1 or 2 being affected each run. The error I am getting is as follows:

3/29/2021 4:02:51 PM :: Processing configuration Error: The resource '26' is in use.
3/29/2021 4:03:11 PM :: Error: The resource '26' is in use.

The resource number changes every time randomly and the VM that fails changes every time randomly. I have tracked the error into vSphere as well as follows:

The resource '26' is in use. com.vmware.vim.vpxd.dvs.portInUse.label

Now I assume that the issue is for some reason when Veeam processes the configuration and realizes I have network remapping turned on that it removes and tries to re-add the VM to the VDS but for some reason tries to assign a port that is already in use by another VM.
Why doesn't Veeam just realize that the replica VM is already in the right port group and nothing has changed and simply skip that part? That would solve the issue in the first place.

I have recreated the VDS on the VMware side, re-added the vCenter's to Veeam, created all new Veeam proxies with more resources, Forced a full replication and even deleted and recreated the replication job.

After the job fails it usually retries and succeeds but sometimes it fails again a second time. I have opened a VMware support case as well but they seemed perplexed as well.

[veremin]

Hi, Matt,

Can you provide us with the stuck ticket number? We will check the logs and the case details before coming up with concrete answer.

Thanks!

[blindsidematt]

Veeam Support - Case # 04691246

[PetrM]

Hi Matt,

Don't hesitate to request an escalation of the support case if you feel that deeper analysis is required, especially if you need to simplify interaction with 3rd party support team. Also, you could probably take a look at vpxd.log to fetch some additional information regarding objects which lock a port at the moment of replica configuration update.

Thanks!

[segfault]

Was there a resolution to this issue?

We also see it happen about 10 times a day (out of a few thousand replications). It always clears on the next retry so it is annoying but not super critical.

It would be good to know before I open a support case if it is a known issue that will be fixed in a future patch or if it is something that we can contact support to have fixed.

Thanks,

--john

[Egor Yakovlev]

Hi John,

From what I see we have just a few similar reported cases. The error itself pops when Veeam performs target VM configuration request(via VMware API), and receives back an error. You should be able to track the issue source in vSphere client too.
Reason why Vmware kicks back our request is unknown(no customers came back with VMware ticket results), however it might be due to other software interfering - for example monitoring system might apply tags to the same VM at the same moment when Veeam has requested configuration change, leading our request to be rejected. That would explain why error pops on random VMs, at random times and on retry it passes just well.

You can open a new ticket and refer to one above, so support will be aware. And it would be great if you share your case ID here as well.

/Thanks!

[segfault]

Case #04753900 created to track the issue and 4G of logs (last days worth) were uploaded. I noted the other case number in the ticket.

Based on the comments above, I suspect that the "other" application is veeam itself as the vCenter server and ESX hosts are dedicated as a DR site. There are several replication jobs that run, each with 50+ vm's. So it is possible that one job is bumping into another when it comes to the timing the network port changes.

--john

[PetrM]

Hi John,

It might be one of the possible versions but I would let our engineers to look for some specific traces in debug logs in order to understand either this assumption is true or not.

Thanks!

[Fredde7803]

Hi

I can confirm that we see these errors as well, they appear very random and the frequency is rare.

We replicate around 500 VMs from one site to another (only one vCenter) daily and the error appears a few times per week.

What errors should be visible on the vSphere side - I've never managed to see something of interest but I've obviously not been digging deep enough?

[PetrM]

Hi Fredrik,

Don't hesitate to contact our support if you want to dig deeper into it. You may paste a support case ID over here for our reference.

Thanks!

[segfault]

Update:

Support confirmed that in my particular case, the other application is Veeam itself. Logs showed two replication threads both making a call for the next available port number at the same time. At that point it is a race to see which thread gets to claim the port. The looser tosses the error. It appears to be a textbook concurrency issue that I've been guilty of making myself a few times in the threaded apps that I've developed over the years.

Now it is off to elevate the issue to the next level to verify the initial findings.