Connection Problems with Connect-VBRServer

[stephan.troxler]

In a reporting script I work with PS jobs so I can gather infos multithreaded. I basically loop through all backups and fire up a job that gathers the restorepoints and some meta information.
Inside the job i ensure that there is a connection like this.

Code: Select all

if ((Get-VBRServerSession) -eq $null)
	{
		Connect-VBRServer -Server <servername>
	}

At the end of the job I close the connection with "Disconnect-VBRServer".

While this works 90% of the time easily with up to 20 Jobs in parallel, I often reach a point were the master server seems to refuse new connections:
"PowerShell environment initialization failed: Failed to connect to Veeam Backup & Replication server:
A connection attempt failed because the connected party did not properly respond after a period of time, or established
connection failed because connected host has failed to respond"
Sometimes it also comes with this message:
"WARNING: Files are locked by another console, waiting for another console to finish update for 30 seconds"

Once I hit this status the master server is not responding at all and also connections with the Console (gui) are hanging. There is no abnormal CPU or memory usage visible. After 10-20 min. everything goes back to normal.

What can I do to prevent the master server from reaching this blocked status? Is there maybe a connection limit? Is there an option to work with connection pooling?

[david.domask]

Hi Stephan,

Just to ensure I'm understanding correctly, your script is connecting to multiple VBR servers from one server executing the script? Or you're connecting to the same VBR server multiple times within the same script?

I'm not aware of an intentional limit on such connections so would need to check internally, but if you could clarify the workflow a bit more (or share a copy of the script), we can check internally -- depending on if it's multiple or a single VBR the answer will change a bit, so any clarification would be great.

Thanks!

[stephan.troxler]

I'm connecting to the same VBR server and from the same Client Host (a script server) but from different sessions (PS jobs) in parallel.

[david.domask]

Thank you for the confirmation. I was not able to reproduce with 30x parallel sessions in my lab -- does it seem to matter if you start a child process of the existing session vs starting a new job? I would assume child-sessions wouldn't reproduce the same issue. Similarly, is it common that you need 20+ sessions running concurrently?

Again, I am not certain that there is a specific limit beyond environmental limits, and since the workflow on your script is a bit unclear for me, I'm wondering if the script processing time is negatively affected by simply capping your parallel sessions -- if so not a problem, just means I'll ask that you open a Support Case and share logs from the Veeam Server as well as a copy of your script so we can review the behavior a bit more. In my admittedly quick and lazy test, I didn't have the parallel sessions do much, but I couldn't get them to give the same behavior with a few basic scripts being run in parallel to/from the same VBR server.

[stephan.troxler]

The timeouts starting normally after 50+ processed sessions somewhere. If I execute the script for the first time, it's slower and after a couple executions it's running way faster and sometimes even without any problems. I use between 8-20 Jobs in parallel and it seems it does not make a change in the behaviour:

Code: Select all

#Job Code
$sb =  {
	Param($BackupID)
	#connect
	if ((Get-VBRServerSession) -eq $null)
	{
		#the timeouts happen here!
		Connect-VBRServer -Server <Server>
	}
	...
	...
	#disconnect
	if ((Get-VBRServerSession) -eq $null)
	{
		Disconnect-VBRServer
	}
}

$vbrbackup = Get-VBRBackup
foreach($v in $vbrbackup)
{
	...
	$BackupID = $v.ID
	..
	#here some queue management...
	while ( .. )
	{
	Start-Job -Name "$jobname" -Scriptblock $sb -ArgumentList "$BackupID"
	Start-Sleep -Seconds 5;
	}
}

[stephan.troxler]

I guess I've just found the culprit. :-/
The disconnect block had the wrong operator and therefore the sessions from the jobs were not properly closed.

Corrected version:

Code: Select all

#disconnect
	if ((Get-VBRServerSession) -ne $null)
	{
		Disconnect-VBRServer
	}

[david.domask]

Hi Stephan,

Ah, yes that could do it. So everything is working well now? No more blocking or disconnects?

[stephan.troxler]

Yes, the problem is solved. As long as connections to the server are closed as soon as they are no longer used, I can run 20 connections in parallel without problems.