resource not ready: source vmware host

[jeremyh8]

anyone else seeing this message during replication jobs? we have pointed our jobs to the cluster which has 3 hosts but it seems that we get this message if we try to process more than 7 disks at 1 time. We have multiple jobs staggered and 7 disks whether from 1 job or many cause this message. if we point the jobs to different hosts instead of just to the cluster we see the same issue but it is per host so we could technically hit 21 disks at one time. ticket id 00440567

[foggy]

Jeremy, could you please describe your setup in a bit more detail? How many proxy servers are involved in data processing (in both cases: when the job is pointed to the entire cluster and separate hosts), what transport modes are used by the proxies?

[veremin]

I know that the error seems to be related to the source host, but I’m still wondering whether you’ve previously set the number of concurrent tasks that proxy/repository can handle. Thanks.

[jeremyh8]

we have 3 proxies on remote side 6 proxies on source. We are using Network based for transport. Yes we have the concurrent tasks set very high for the proxies and the repository for metadata.

[foggy]

Well, according to R&D, there is actually a limit of 7 simultaneous network connections to the host/cluster (caused by some VMware API limitations, increasing this value could lead to job failures). In case of reaching this number of connections jobs will wait for the connection to free up before starting processing the next disk.

[Gostev]

This was one of the issues that was caught "in the wild" during the beta program. The limit is for network processing mode (NBD) only.

[jeremyh8]

support is having us change some reg key from 7 to 10 on the replication server. we will see what it does. our work around if nothing else works will be to point each job to a different host manually so we can get over 20 going at once.

[veremin]

As mentioned, there is,indeed, a logic behind the decision of limiting the number of connections made to a host via NDB, since increasing the default number might lead to failing jobs. So, after increasing the connections number, kindly, keep a closer look at your backup environment and see whether everything works as expected or not. Thanks.

[jeremyh8]

you keep saying "host" what is a host to you? a host to me is an esx host so if i point my replication destination at a cluster it should be 7X#of hosts.

[foggy]

You never know on which host the next VM to be processed resides, so to avoid situations where all of them are on the same host, the 7 connections limit is assigned to the whole cluster (if it is specified as a source for the job.)

[lars@norstat.no]

I have also reached this limitation it seems, what is the reg key where i can change this ?

[foggy]

Lars, you can get it via technical support. Thanks.

[lars@norstat.no]

Or you could maybe PM me the key, if you don't want it to become public. Seems much to open a support case for one reg key, but i will do that if you want to.

[foggy]

Ok, but once again I must repeat that we are talking about VMware limitation here, and increasing the default number (7 connections) may result in intermittent job failures due to reaching max number of API connections, thus making your replication jobs less reliable.

Code: Select all

HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication
ViHostConcurrentNfcConnections
DWORD

Don't forget to restart the backup service after creating the key.

[lars@norstat.no]

Thanks Alexander

Yes i will be VERY careful. And i hereby take on all responsibility for any problems that might arise using this hack.

I have very strict control on my vm's and are the only person that are touching any of it, therefore i know there will never be more than 7 servers on each target host because i have manually placed them where they should be

[foggy]

Then, I'm sure, you're pretty safe. But still, I had to state that in bold.

[lars@norstat.no]

Seems that this had a HUGE impact as i cut my replication time by around 80% ....

But i would not recommend using this if you don't have absolute control over your environment and/or are not running continuous replication because what i have done with this is cut 4-5 minutes of wait time from a job that was taking 10 minutes to run and if you are running a weekly job that would take several hours then those 4 minutes is not worth potentially having jobs fail ... Better to have a stable system than a fast one if you must choose.

now i'm running 13 vm's continuously with 29 disks combined and all are running at the same time. There is however consequences by doing this:

4 core Vcenter server is running at about 70 % CPU load always
8 core Veeam Console Server is running at about 50-70 % CPU load always
16 core physical proxy is running at about 50% CPU load always

Production SAN is fine, but DR SAN is running on very high load because it's applying retention, and removing snapshots all the time now .... So if you are gonna press your system make sure the target SAN is up to it and maybe is Veeam exclusive as mine are.

[lars@norstat.no]

Can i get a comment on the following from Vsphere Documentation that states that vSphere 5 does not have a connection limit but rather a buffer limit that i don't quite understand how works yet.

http://pubs.vmware.com/vsphere-50/index ... t.5.5.html

Also it states that NBD don't support disk's over 1TB, but i have replicated disk's larger than that.

[Gostev]

This is the very reason for us limiting the connection count. The number of concurrent connections we are limiting our jobs to is obtained empirically, based on comprehensive stress testing. As such, while you a free to increase the limit with the registry key above, the bigger values may result in intermittent job failures.

[apastore]

I have the same problem - CASE [ID#00547578]
there is a limit to what I can set the value of NFC connections?

[jbarrow.viracoribt]

Ok, but once again I must repeat that we are talking about VMware limitation here, and increasing the default number (7 connections) may result in intermittent job failures due to reaching max number of API connections, thus making your replication jobs less reliable.

Code: Select all

HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication
ViHostConcurrentNfcConnections
DWORD

Don't forget to restart the backup service after creating the key.

Is this the lime still even for the newer 5.5 vmware versions? I'm asking due to the age of this thread. Thanks.

[tsightler]

The limits are documented in the vSphere VDDK documentation. Look for NFC session limits.

For vSphere 5 they list the limits as follows:

Limited by a transfer buffer for all NFC connections, enforced by the host; the sum of all NFC connection buffers to an ESXi host cannot exceed 32MB.
52 connections through vCenter Server, including the above per-host limit.

It wasn't really clear to me if that meant 52 connections on a vCenter per ESXi hosts, or 52 connections total for the vCenter as a whole, but according to other online sources, it's the latter, no matter how many hosts. I still think you could probably increase the count over 7, but I'd do so slowly until you were sure there were no issues.

[jbarrow.viracoribt]

If the registry key is already there, and you change it, say from 7 to 14, does that instantly take effect or do you need to restart the service for it to honor the increase?

[foggy]

Yes, you need to restart Veeam B&R console for the key to take effect.

[jbarrow.viracoribt]

Yes, you need to restart Veeam B&R console for the key to take effect.

Just close and reopen the console UI or completely stop and restart the system service?

[foggy]

Sorry for confusion, you need to restart Veeam Backup Service, of course.

[jbarrow.viracoribt]

Sorry for confusion, you need to restart Veeam Backup Service, of course.

Thank you, and good morning!

[jbarrow.viracoribt]

What types of job errors should we look out for when these settings are modified to see if the number we set this too is too high?

[jbarrow.viracoribt]

So if this setting is at 7 by default, and you have a physical proxy with 20 cores set to 20 concurrent tasks, only 7 would get used at a time due to this throttle?

[PTide]

only 7 would get used at a time due to this throttle?

The "cores-tasks" relationship is not "1 task = 1 core" thus all depends on how OS's (Windows) resource manager will disrtibute the load.

Thank you.