Veeam 12 : Resource not ready : Gateway Server

[pmichelli]

Hi Everyone,

I just deployed Veeam 12 from scratch and was doing testing before importing the config backup from V11. We use Dell Data Domain 6400 and 6900 as target repositories. I have 2 gateway servers that I use for each mTree that I have mounted.

In v11 the backups would just start instantly. With v12, if no jobs have run for over 1 hour, I get this message when the job starts that stalls for 1.5 - 2.5 minutes before it says ready and continues onto the next task. I cannot even find anything referencing this. Backups work fine once the Gateway is initialized. Once I do one backup, any other jobs will find the Gateway instantly. If I leave it for an hour or so and come back to kick off new jobs, it has to do the long wait again. Almost as if it unmounts or disconnects the Gateway.

Gateway is on the Veeam server (Server 2022) and another server which is also Server 2022. In v11, my Gateway is the Veeam server running 2019.

I can't seem to attach a pic that shows what is going on. Anyone have ideas, or should I open a SR?

[Gostev]

Hi, this is not a known issue at the moment so you should open a support case.

[pmichelli]

Hi Gostev

Thank you for the quick reply. I have an existing SR open that I replied to with these details. I'll see if the engineer wants me to open a new case in parallel or if he'll work this issue as well.

[hpadm]

Hello, could you check if the behavior is the same as mine in veeam-backup-replication-f2/clarificati ... 85735.html ? Specifically, no delay on virtualization hosts, exactly 1 minute on VMs, and exactly 2 minutes on bare metal hosts? (And no delay when reusing the same repo right away, like you describe). My main suspect is Veeam's custom NFS for windows component. Do your repositories use nfs shares?

You can right click on the job log to add a hidden Timestamp column for precise job timing information (the duration column is kinda unreliable). Also it doesn't seem to be possible to attach images on this forum, so I use an image hosting service and link via the img tag.

[pmichelli]

Hi everyone

Support got back to me and said this :

"Thank you for the logs, from them it looks like the 2min delay is due to Veeam determining which gateway server to use.
[15.03.2023 14:22:11.747] <31> Info [RepositoryGateHostsCollector] Getting possible gateway hosts for repository DD6400 - Secure (2254f979-fa52-4a67-ab6a-b5e91f4c4e5c).
[15.03.2023 14:22:11.762] <31> Info [ResourceScanner] Handle Veeam.Backup.ResourceScanner.Ready.d7c4ff97-b99b-4d1f-884d-283b7b6b9ee3 for backup03 created
The 2 lines above is the job getting the gateways for repository DD6400 - Secure
[15.03.2023 14:23:12.032] <31> Info [RepositoryGateHostsCollector] Getting possible gateway hosts for repository DD6400 - Secure (2254f979-fa52-4a67-ab6a-b5e91f4c4e5c).
[15.03.2023 14:23:12.047] <31> Info Validate gateway server backup03 (6745a759-2205-4cd2-b172-8ec8f7e60ef8)
After approximately 1min this job run decides on Backup03 as the gateway server
[15.03.2023 14:24:13.356] <31> Info [GateRepoAccessResourceCreator] For repository DD6400 - Secure (2254f979-fa52-4a67-ab6a-b5e91f4c4e5c) RepoAccessGroupId is 00000000-0000-0000-0000-000000000000.
[15.03.2023 14:24:13.356] <31> Info [RepositoryGateHostsCollector] Getting possible gateway hosts for repository DD6400 - Secure (2254f979-fa52-4a67-ab6a-b5e91f4c4e5c).
[15.03.2023 14:24:13.356] <31> Info Validate gateway server backup03 (6745a759-2205-4cd2-b172-8ec8f7e60ef8)
[15.03.2023 14:24:13.388] <31> Info [RTS] Capturing resource 'DD6400 - Secure', id '2254f979-fa52-4a67-ab6a-b5e91f4c4e5c', for task 'f09ca7de-6c92-45a4-bf71-17858e4d4413'
From there the job puts in a request to use the gateway and then flags it as in use, and the job continues. To remove this step from the process can we list only a single gateway server for the repository? The Gateway server can be set when editing the repository, screenshot below for reference:"

I have the Gateway set explicit on all my Data Domain mounts but it still times out for 1-2 minutes when it tries to start backup jobs.

I think there is a bug. V11 does not do this at all.

[pmichelli]

Hello, could you check if the behavior is the same as mine in veeam-backup-replication-f2/clarificati ... 85735.html ? Specifically, no delay on virtualization hosts, exactly 1 minute on VMs, and exactly 2 minutes on bare metal hosts? (And no delay when reusing the same repo right away, like you describe). My main suspect is Veeam's custom NFS for windows component. Do your repositories use nfs shares?

You can right click on the job log to add a hidden Timestamp column for precise job timing information (the duration column is kinda unreliable). Also it doesn't seem to be possible to attach images on this forum, so I use an image hosting service and link via the img tag.

Yes, the Dell Data Domain use NFS as the underlying protocol, then tack on DDBOOST on top of it.

[hpadm]

In my silly configuration, each host is its own backup repository, and its own backup gateway (explicitly set, no automatic selection). All backup activity should thus be happening on-host, aside from talking to the NAS. It also causes the worst case delays because each backup repo is used exactly once.

[dr_garyk]

Just some additional info. I am using SMB-based repositories and see this issue as well. In my case, the delay can be as long as 6-7 minutes...Resource Not Ready: Gateway Server. It finally takes off and completes normally. If I run another job within a few minutes, it starts normally. I also did not see this issue with version 11.

[Mildur]

Hi GKG

Thanks.
All. Please open a support cases.
We need logs. Without logs we cannot confirm it as a bug.

Best,
Fabian

[hpadm]

My attempt was Case #05941477 (description, observable changes, infographic, short log snippet) but since I don't have a support contract, it expired. Usually my threads get noticed by support staff given enough time and effort. Previously 4 of my 5 cases that received attention led to confirmed issues and fixes, 1 went for later consideration.

Regarding logs, if possible narrow it down so we don't dump everything on you. Also I observed that the same issue happens during 'Export content as virtual disks', which might be interesting to test against since it's much simpler, repeatable, does not cause permanent changes to production, and has its own extra logs.

[Mildur]

Hi hpadm

It may happen that your case was closed this time because support load after a new release is higher than usual.
This forum is run by product management. We are not trained on analyzing log files.
Each Job has it's own folder in %ProgramData%\Veeam\Backup. And there are also logs for each service.
You may check out those.

Best,
Fabian

[hpadm]

I know, but forum staff is able to escalate it for review by support, which is very helpful. As you mentioned, logs are often needed, however I always wait for support to instruct which logs to provide, instead of guessing preemptively. I did look through a lot of logs on my own, both on the VBR server and on the host running the job, but 9 out of 10 times, the log timestamps start when the job starts, that is, _after_ the delay during resource allocation. It definitely helps narrow down those logs that actually recorded something, but none of the ones I checked explain the minute-long timeout delays. Since it is suspected that this is a NFS driver issue, there might not even be any logs to see. I hope that support will be able to replicate this easily and save us a lot of trial & error.

[dr_garyk]

HPADM,

Just a quick note...I am seeing this same delay symptom, however, my backup repository is SMB not NFS...as such, I'm not sure about the assertion that the issue is related to NFS drivers...

[pmichelli]

Hi GKG

Thanks.
All. Please open a support cases.
We need logs. Without logs we cannot confirm it as a bug.

Best,
Fabian

Here you go. Alan has been very helpful but can't trace it down yet

Case #05949531

[dr_garyk]

As I am unable to view the case referenced by pmichelli, is there any update?

Thanks...

[Mildur]

Hi @dr_garyk

Yesterday the case was escalated to the next tier.
Do you have a case number on your own? Please share it with me.

Best,
Fabian

[pmichelli]

If you're going to open a case, I may suggest you add this registry setting first. Support will likely ask you to do it (they did for me), and it may save a day of back and forth communications.

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\

GatewayDetailedLogging
Type: REG_DWORD
Value: 1

[Asahi]

Hi,

The Veeam v12 server that exists in my Lab environment is experiencing the exact same event.

Kind Regards,
Asahi,
Climb Inc.

[dr_garyk]

It's been another week...any update on the escalated ticket?

[veremin]

The assigned support engineer is actively working on the referenced ticket, and the investigation continues.

If you want to address your issue sooner, kindly, reach our support team and create your own ticket.

Thank you for understanding.

[dr_garyk]

Veremin,

Thanks for the update. I have not created my own ticket because I am currently using the Community edition.

[veremin]

The community edition owners are eligible for best-effort support, so you can give it a shot.

Otherwise, let's wait until the investigation is over and see whether the found solution is applicable to your problem.

Thanks!

[pmichelli]

I will be sure to post anything useful, once I hear back from the escalation engineer. I have not heard anything since the case was sent up the chain.

[aprestin]

Getting the same error here too after an upgrade to Veeam 12. We have a P2 case open currently and I have uploaded the past day's worth of logs.

[aprestin]

A reboot of the VBR server worked for a while, but now all jobs are hanging again with the same message. Interestingly, killing the SOBR archiving jobs awakens all the backup jobs and they run like normal.

[dr_garyk]

I have rebooted my VBR server several times since the original issue was observed. Although I no longer see the 6-7 minute delays I saw a couple of weeks ago...I still get 1-2 minute delays if the VBR has been inactive for say, 2 hours.

[absys34]

Same problem for me...
Not sure if it usefull to open my own ticket...

[dr_garyk]

I am testing the new release of VBR just posted today. So far, the issue appears to be resolved for me.

[dr_garyk]

It looks like I spoke too soon....I am still seeing the delays after VBR has been inactivate for a couple of hours.

[Asahi]

Hi,

I applied the patch in my Lab and checked the backup job at night.
However, the problem was not resolved.
The "Resource not ready" message is output for almost 2 minutes when the gateway server is ready.

Kind Regards,
Asahi,
Climb Inc.