Veeam 12 : Resource not ready : Gateway Server

[pmichelli]

Did someone change the title of this post? I am the one who opened this and did not say Gateway server for hours. I was only ever experiencing an up to 3 minute delay on the gateway. I have come to learn of a new patch from the support escalation engineer.

I will install this and see how I make out.

https://www.veeam.com/kb4420

[Gostev]

Apologize, fixed the title.

I totally misread "no jobs have run for over 1 hour" and expanded the title to highlight an additional incorrect symptom.

Please let us know if P20230412 helps!

[dr_garyk]

I am also seeing the same results as Asahi...the issue has not been resolved with the latest patch.

[marcseitz]

I also can confirm that the situation with P20230412 did not change.

[pmichelli]

The patch did not fix this issue.

1st backup took exactly 2 minutes to find the data domain gateway (even though it is explicitly set) 5pm run time
My other jobs at 7 and 9 pm found the gateway within 30 seconds.

Pretty much same behaviour as before

[Gostev]

The patch addresses the issue with extremely long delays of automatic gateway selection process in case when one of the gateways is offline and/or unreachable. The very workaround for that issue was to just set the gateway explicitly, so clearly you're experiencing a completely different issue then what the patch addresses.

[dr_garyk]

Since the most recent patch did not address this issue...what is the current status of the trouble ticket?

[pmichelli]

It seems to have stalled out. My ticket was escalated to L2 and they seem to feel this is normal behavior with V12. I gave the tech this forum thread and he was going to go over it and get back to me.

[dr_garyk]

Thanks for the update. It will be interesting to see what the L2 tech says...

[pmichelli]

The L2 tech looked through this thread and replied to me that he doesn't see open cases by anyone else. If we want to figure this out, more of you need to open SRs and post the case number here.

[dr_garyk]

Ok, thanks for the update.

Is there any indication in the ticket details that they are unable to recreate this issue in their lab environment?

[Asahi]

Hi,

I have created several test Veeam servers and all of them have been pending for about 2 minutes with the message "Resource not ready : Gateway Server".
It is very easy to reproduce this problem.
Just install Veeam, register a repository that uses a Gateway server such as CIFS/NFS/Object Storage, and run the backup.

I will contact Veeam support when I have more time to work.

Kind Regards,
Asahi,
Climb Inc.

[EWMarco]

I upgraded the fifth of our Veeam environments to v12 yesterday and I am seeing this issue. A whole SOBR is doing nothing at the moment, in one job we've been waiting for a gateway server for two hours now.

The bucket in the SOBR is set to direct and for the life of me, I cannot find a setting pertaining to gateways on the local repos. The local repos have iSCSI or local storage anyway.

I notice that the B&R master server is listed as the gateway used. This machine does not seem to be having resource constraints.

Do we have any new information as to what might be happening? One of our other environments is setup just like this one though it is much smaller and works fine so far.

[Gostev]

There are no known issues with the current V12 build at the moment which would match this description "one job we've been waiting for a gateway server for two hours now".

[hpadm]

There is mention of waiting "long time" in the KB4420 for the april patch for V12, related to automatic gateway selection when one of the gateways is 'unavailable'. Backup gateways are defined in Backup Infrastructure -> Backup Repositories -> Properties -> Share -> Gateway Server and are selected from the list of Managed Servers. They are used when the type of storage cannot act as a VBR repository on its own. Finally, when you define backup jobs, you just choose a backup repository object, and that will result in the job interacting with the gateway managing it.

The topic of this thread is not what you describe. The delays we are observing are small. For me, it's exactly 2min on application servers, 0min on hyper-v hosts, and 1min on hyper-v VMs.

[pmichelli]

Update from my end. The case is still open with an escalations team.

I was asked to prove that I did in fact install the April patch for Veeam 12 and provide more logs after weeks of silence. I have to admit, this is not a good look for Veeam to not trust their customers that say they've already applied the patches you asked them to over a month ago. I'll update once I hear more.

[Asahi]

Hi,

I have opened a case too.
If there are any updates I will post them here again.

Kind Regards,
Asahi,
Climb Inc.

[Asahi]

Hi,

A Veeam engineer informed me that even though I was using the latest version of Veeam, I was having problems selecting a gateway server in certain scenarios.
Apparently, the problem is caused by the scan operation.

I will dig a little deeper and update the post here.

Asahi

[pmichelli]

Hello everyone

After a long time with an escalations team, here is the reply from Veeam. I am ok with this and will just leave things alone and move on.

The logs showed that the latest patch had actually resolved the issue that I was seeing in regards to multiple rescan being performed on the same resource for as many times as there are tasks pending and waiting for resources. Due to this, I was now able to account for around 1 minute and 13 seconds of the wait for resources on the May 28th run of the "Backup Veeam" job where it waited for a solid 2 minutes. For this 1 minute and 13 seconds, I can say with confidence that this is should be expected behavior, because it is no longer running those rescans multiple times, and during this time frame, the timestamps do not show any 'holes' or sudden time jumps that would suggest that a certain thread is waiting for another thread's action or process complete.

I have discussed this internally to try and account for the rest of the 2 minutes, and the reason for this seems to be some behavior that I had over looked. Veeam performs "Resource Dispatch Iteration" cycles every minute. If a task that is waiting on resources misses the current cycle due to the time taken to prepare that task, it must wait for the next cycle to come through. Because of this minute-by-minute cycle, it explains why in some cases you see a wait on resources that is exactly 2 minutes long.

Also, from my discussion it seems that whatever fixes the developers added to the latest patch is more of a stopgap for now, with more improvements being discussed in the background, that is coming down the pipeline in a later release. So in summary, that everything seen thus far with the latest patch can be determined as behavior by design, with a decent chance that more improvements in this area will be coming in the future.

[hpadm]

That is... kinda disappointing, but it confirmed my very early suspicion that the entire backup network is now operating at a speed of 1 operation per minute. For anyone reading this, please look at https://i.imgur.com/8n6owZf.png for direct evidence that these delays did not exist in v11.

When jobs are chained, these delays multiply. For fast incremental backup jobs, these delays double the required runtime. And this is not the only delay - the "collecting recovery media" step at the end also takes several minutes now, with significant variance.

[Asahi]

Hello everyone,

My support case also informed me that it was a rescanning issue, as described in my post at @pmichelli.
However, as @hpadm mentioned, I also know that this kind of delay does not occur in v11.

Let's hope and pray that this process will be improved in the future!

[jakub.krocek]

I just had 15minut wait on Hyper-V virtual server. There is exactly one Host, so even with automatic gateway selection there are not many possibilities.

[pmichelli]

If anyone is curious, the latest Veeam 12 patch still not does fix the 2 minute (or more) gateway timeout. The wait continues....

[Gostev]

@pmichelli thank for sharing, your post got some devs curious about the issue and they went digging the code

[Asahi]

Hi,

I believe Veeam v12.1 will be released soon.
Does this update have any approach to the issues related to gateway servers?

Kind Regards,
Asahi,
Climb Inc.

[djbo]

hello,

To solve the problem of the GW being left pending, what I did was uninstall the INSTALLER SERVICE of version 12 and install the INSTALLER SERVICE of version 11 in all the proxies and in the VBR, after this I no longer have issues.
Waiting for the patch for version V12.1 to fix it.

Greetings.

[BostjanUNIJA]

any ETA on VBR 12.1?

[Gostev]

Yesterday.

[pmichelli]

12.1 has resolved this issue for me. I have run a number of backups and even rebooted the VBR server. Find the gateway server immediately.
Many thanks for finally sorting this out !