Scale-out Backup Repository issue

[CitySAV]

Anybody on this forum can help me or have any suggestions

I current have a support case setup for this problem and the case # 04909092 it has been almost a month

We have setup a "Scale-out Backup Repository" that has two volumes that are 36.4 TB, both volumes are formatted as NTFS
It was working until the first volume got full. It is not going over to the second volume and job is set to use the Scale-out Backup Repository

Our Backup Server is HP DL 380 info on two volumes
1st Volume
Format: NTFS
12 Disks ,4 TB, SAS, Model # HP MB4000JVYZQ
RAID 6

2nd Volume
Format: NTFS
12 Disks, 4 TB, SATA, Model # ATA MB004000GWKGV
RAID 6

I was thinking of formatting both volumes and use Microsoft to strip RAID 0 and not use Scale-out Backup Repository

EDIT moderator: removed logs and support comments

I researched this issue further. Looking at the most recent logs I have for the 22 July run, I can see the reported issue. I can see the 500GB disk backed up successfully, and the 13TB disk processing up until ~4:06pm which is right around when the 38TB disk backup starts. I can also see ~30 min later by ~4:36pm we get the first report that the transmission pipeline between the source and target agents on BKUPSRV1.

So we are backing up to the latest PD ArbitratorD2021-07-22T150033_3F79.vib incremental backup file on F:\Backups\PD_Arbitrator without issue until the 38TB disk is processed. Furthermore, since this VM had not been backed up successfully, these disks are processed as a full backup for the W2016-SqdCar-Vid VM inside the incremental backup. Yet almost as soon as we start processing the 38TB W2016-SqdCar-Vid_1.vmdk disk, I see the pipeline timeouts.

Checking on the PerfData logging I can see that all file seek and write operations cease by 16:06:51.076227, which is <1 second before the first FIB update taking longer than 5 minutes, indicating the requested blocks have not been received by the target agent. I then see no file activity on the target agent and no read activity on the source agent past this point, as the target agent is essentially hung or not processing any further data.

At this point I am unfortunately still researching this issue as the currently available logging did not present an obvious root cause. As soon as I have my findings in order I will provide next steps to resolution ASAP.

[HannesK]

Hello,
and welcome to the forums.

How much free space do you have on the volume where you want to fail over to? As I understood, there is one disk with 38TB and the VM is probably even larger. So I assume that the placement detection logic never finds enough free space as your volumes have only 36TB size. That worked in the beginning, because at that time everything was empty.

I assume that you use forever forward incremental backup mode... If you switch to striped dynamic disks, I would think about REFS to improve merge speed: https://www.veeam.com/blog/advanced-ref ... suite.html

Best regards,
Hannes

[CitySAV]

Yes, I'm using forever forward incremental backup mode and I've considered that of changing to Stripe set

Michael

[PetrM]

Hi Michael,

I'd suggest to wait for the result of technical analysis performed by our Tier-2 support team. Once the root cause becomes clear, we'll be able to figure out the most robust way to resolve the issue.

Thanks!