RHEL On-Prem Backup Repository Offload Problem; Veeam Support Ref 08151265

[huddie]

Hi all,

Re: RHEL On-Prem Backup Repository Offload Problem; Veeam Support Ref 08151265

Is anyone else experiencing this problem ?

About 3 months ago the offload process for backup copies, which had been working flawlessly for about 6 years, developed an issue for no apparent reason. This is an on-prem repository which forms the performance extent of a Scale Out Backup Repository (SOBR), with the capacity extent being object storage in Azure. Offload jobs take backup data copies in the on-prem repository Transit extent and copy them to object storage in the dedicated Azure storage account.

The issue is that offload jobs complete successfully for a day or two, but slow down and then grind to a halt. You'll look at the actions in a frozen SOBR offload job and the file copy action will be ticked off but subsequent actions don't occur, the job never finishes and no errors are logged in job Actions. The log files show that the repository is trying to prune aged backup copies from object storage. The repository is sending these requests to prune, the Azure storage account is receiving them, sending acknowledgements, but the repository is no longer receiving the acknowledgements. This inbound control traffic, from Azure storage account to on-prem repository, is eventually being blocked. The thing is, I've been working with Veeam Support, our network MSP and Microsoft on this for a while now and we have support tickets open with all of them for it. Let's rule out the obvious: there are no Linux, Windows or Network firewall rules configured to do this. Our MSP has promised repeatedly that QoS (Quality of Service) and packet shaping of inbound / download traffic are not happening. But it works for a couple of days, then randomly stops working.

Details below.

VB&R Server: Hyper-V VM, Windows Server 2019, VB&R v13.0.2 (was also happening on v13.0.1)
SOBR: Performance extent - Red Hat Enterprise Linux v8.10 fully patched, JeOS install with no desktop, HPE ProLiant DL380 Gen10 Plus v2.
Capacity extent - Azure blob storage, storage account.