Clarification of 'Required backup infrastructure resources have been assigned' step

[hpadm]

After updating to VBR 12, I've noticed that some jobs take longer. Close examination revealed oddities in the 'Required backup infrastructure resources have been assigned' step that happens at the beginning of each backup job:

• hyper-v servers' OS backup reports 00:00, just like before the update.
• DB server reports 02:00, previously was blank. From history, the total job duration was always either 2:40 or 4:40.
• file server does the same thing - now 02:00, previously the total time was often one of two values exactly 2 minutes apart.
• VM backup reports 01:00, previously was blank but from the surrounding timestamps it can be determined that it was 00:00. VM backups take 1+ minute longer than before.

What is the deal with this particular step? Is it doing anything more than just opening sessions to hosts involved in the backup job?

Aside from VMs, in all cases the job is 'managed by server', each host is its own backup proxy and backup gateway. All of the jobs work the same way, so I'm not sure why the hyperv servers have no delay while the others get 2 minutes. And why vms get 1 minute. I'm also very suspicious about the delays being exact multiples of 1 minute. Maybe there is some kind of timeout involved? The fact that those two bare metal servers used to randomly jitter between two durations exactly 2min apart means sometimes the delay happened and other times it didn't - very strange. I have not yet tried digging through the logs. The only difference I can think of is that the db and file server have several times more data than the others. But then again, the VMs consist of pretty much just the OS, same as the hyper-v servers, yet they get an extra minute.

[HannesK]

Hello,

What is the deal with this particular step?

if you like a real answer instead of guesses, then I suggest asking support what you described here. Support needs the logs from V11 and V12. Please post the support case number.

If anyone has similar issues: to improve things, we need logs and support cases. Guessing about things like this on a forum makes little sense.

Best regards,
Hannes

[DanielJ]

Guessing about things? Aren't there members on this forum who are involved in the actual development of the software?

[HannesK]

yes, the forums are mainly run by Product Management. I know that message... if you like to have a solid answer, then log investigations are the way to go.

[hpadm]

This is difficult because I've only begun investigating the issue after observing it for the first time. My previous issue reports too up to tens of hours of work to resolve, so I'm trying to skip it in case it's a well-known issue, something like needing to open an extra port in the firewall.

Another complication is that it might be two separate issues being mixed together. I have discovered that the job progress log has a hidden timestamp column, which allows determining the true duration of certain steps, even if the log itself doesn't display that info. This helps me pinpoint where the additional delay is happening.

The VM issue is definitely real. The hyper-v hosts for some reason are unaffected. The db/fs servers get hit the hardest. I don't get it.


There is an additional delay that I don't understand, regarding the "Collecting recovery media" step. Sometimes it takes 8 seconds. Sometimes 2-3 minutes. It doesn't seem to be related to anything and varies a lot each day on each host. Please let me know if you have an idea what's going on here.

Case #05941477

[hpadm]

I also observed this during a 'Export content as virtual disks' operation. The first time a particular backup gateway is used (within a certain time period), there is always an exact 2 minute delay during which the job reports "Resource not ready: gateway server".

The gateway connects to backup storage using VBR's built-in third-party NFS for Windows component. I now suspect that this is yet another issue with that thing.

[hpadm]

Unfortunately my case timed out. I would really like to fix this since it's delaying my backup schedule for an extra half hour total. I can trivially reproduce it at will. It is most likely a NFS-windows issue since I only found one other report about it. vmware-vsphere-f24/veeam-12-resource-no ... 85841.html

[pmichelli]

I think we are having the exact same issue. I am the one who started the thread you quoted. Almost 100% identical issues. Has to be something with their NFS for Windows in v12

[hpadm]

I have reviewed the patch notes for P20230412, which mentions a fix for a long delay for "Backup and restore tasks using a repository with Automatic gateway selection mode if one or more gateway servers are unavailable". Unfortunately it does not address my issue, since all my gateways are manually pre-selected, and all of them should be available at all times. The logged delays are identical to the pattern shown in screenshots above.

[hpadm]

An update: In the last 2 days it stopped happening, with no configuration change to the VBR software, or the server hosting it, or the hosts being backed up. What changed was the following:

• The router and switch were replaced for current, expensive models. This shouldn't really affect VBR on the lan, but who knows.
• The NAS was offline for a few days, causing all backup repository entries to become 'Unavailable' and then to come back.
• At the end of the downtime, I updated the Synology NAS firmware from 7.1.1-u3 to 7.2-u1 (log). The minor version update lists a large pile of fixes; NFS is not mentioned but it could have been omitted from the listing.

[hpadm]

Correction - it wasn't related to the above changes. It was only related to the VBR management server uptime. And it's a bit weird. When I reboot the server 8 hours before backups, the delay becomes 0:00. If I reboot it 23 hours before backups, the delays are 2:00.

[HannesK]

When I reboot the server 8 hours before backups, the delay becomes 0:00. If I reboot it 23 hours before backups, the delays are 2:00.

the next cumulative update should bring improvements around gateway server selection. If you can reproduce the same issue after the next update, it would be great if you could open a support case (and post the case number here for reference)