RHEL On-Prem Backup Repository Offload Problem; Veeam Support Ref 08151265

[huddie]

Hi all,

Re: RHEL On-Prem Backup Repository Offload Problem; Veeam Support Ref 08151265

Is anyone else experiencing this problem ?

About 3 months ago the offload process for backup copies, which had been working flawlessly for about 6 years, developed an issue for no apparent reason. This is an on-prem repository which forms the performance extent of a Scale Out Backup Repository (SOBR), with the capacity extent being object storage in Azure. Offload jobs take backup data copies in the on-prem repository Transit extent and copy them to object storage in the dedicated Azure storage account.

The issue is that offload jobs complete successfully for a day or two, but slow down and then grind to a halt. You'll look at the actions in a frozen SOBR offload job and the file copy action will be ticked off but subsequent actions don't occur, the job never finishes and no errors are logged in job Actions. The log files show that the repository is trying to prune aged backup copies from object storage. The repository is sending these requests to prune, the Azure storage account is receiving them, sending acknowledgements, but the repository is no longer receiving the acknowledgements. This inbound control traffic, from Azure storage account to on-prem repository, is eventually being blocked. The thing is, I've been working with Veeam Support, our network MSP and Microsoft on this for a while now and we have support tickets open with all of them for it. Let's rule out the obvious: there are no Linux, Windows or Network firewall rules configured to do this. Our MSP has promised repeatedly that QoS (Quality of Service) and packet shaping of inbound / download traffic are not happening. But it works for a couple of days, then randomly stops working.

Details below.

VB&R Server: Hyper-V VM, Windows Server 2019, VB&R v13.0.2 (was also happening on v13.0.1)
SOBR: Performance extent - Red Hat Enterprise Linux v8.10 fully patched, JeOS install with no desktop, HPE ProLiant DL380 Gen10 Plus v2.
Capacity extent - Azure blob storage, storage account.

[david.domask]

Hi Paul,

Not aware of any issue specifically regarding RHEL, so please continue with Support; topics about technical issues must only be posted after the conclusion of the case as per the forum rules.

I understand you're looking to see if there are others with the experience, but we cannot meaningfully troubleshoot such issues over the forums and a support case is required.

As I see it, your case has already been escalated and is with Advanced Support for review. I will be locking this topic for now as the investigation is on-going; the Engineer appears to have identified several investigation points and proposed an isolation test that seemed to prove useful, so please continue with Support as the investigation appears to be progressing.