Huge backups and SOBR rebalancing - can V12 help us somehow

[mkretzer]

Hello,

we have some VMs backups which are ~16 TB big. Sadly, the extend of the SOBR one of there backups was on nearly was full and thus fast clone could not be used for ~25 % our our backups. After about 46 hours nearly all "slow clones" were finished but our 16 TB backup file is still at ~42 % of "slow cloning". This also means there is no backup beeing created for more than two entire days.

The only idea how to "fix" this is to use much larger filesystems (so not 200 TB but 600 TB - 1 PB). I am not sure if this is a good idea with ReFS since about one year ago MS dev told me not to use such large FS. Maybe things changed with 2022... With XFS i know such large FS work but we don't want to use XFS for copy source & target to limit the risk.

Is there anything in V12 that can help us?

Markus

[HannesK]

Hello Markus,
the "VeeaMover" (current working title for the feature to move backups between repositories) can move REFS to REFS while preserving fastclone capabilities. Actually it can do that even from NTFS to REFS, SMB to REFS, REFS to XFS etc. So whatever kind of migration is needed, fast cloning can be preserved.

"Move" is done on "per job level" between repositories. As far as I see, you like to re-balance with one SOBR, right? That would be a "re-balance SOBR" operation then then. You add new extents to a SOBR and then the data is re-balanced between the extents.

Best regards,
Hannes

[mkretzer]

Hi Hannes,

two problems with that:

- While the move is running backups cannot be run of the backups in the moved chain, correct? (thats our main issue)
- Entire chains must be moved, correct? That means if i have a HUGE backup chain of a HUGE VM we need to move everything else on that extend - manually....

Problem is that creating only the one synthetic full already takes more than three days. The entire chain might take even more.

Markus

[micoolpaul]

Hi Markus,

Jumping in with my thoughts on here if you don’t mind:

What OS are you using atm and how much RAM has the repository got?

For such poor performance for the move it’s either likely RAM starved or IO starved. I think this VeeaMover is gonna be a handy tool, but it’s not designed to overcome repository performance limitations, instead designed to enable better lifecycle management.

Microsoft state the maximum file system size of an ReFS disk can be 35PB and I’ve seen repositories far larger than yours on ReFS without issue, it’s always been a RAM dependency for large sized ReFS volumes.

Just wondering if there’s other ways forward for you.

[mkretzer]

Server 20H2, 2,2 TB Ram.
It feels like IO goes single thread without async processing. I get ~150 MB/s at 9 ms. We might have high fragementation but still that should not be an explanation for the low data rate. We only have 64 disks in that storage, still its quite low.

Meanwhile a tape backup runs at full LTO-8 speed from the same disk.

[mkretzer]

A solution for our issue would be the possibility to do active full instead of "slow cloning" - our storage can do 20x the speed when doing active fulls (write IO) as it does now with "slow cloning" - perhaps the nearly full ReFS causes this, i really do not understand

[HannesK]

Hello Markus,

While the move is running backups cannot be run of the backups in the moved chain, correct? (thats our main issue)

correct. we cannot move and change a backup chain at the same time.

I'm still discussing some details on the VeeaMover, but it it looks like V12 will not solve the issue you try to get around.

Best regards,
Hannes

[edh]

Hi,

What RAID controller and RAID level are you using in that repo? 150 MB/s is less than a single disk throughput

[spiritie]

Hello Markus,
the "VeeaMover" (current working title for the feature to move backups between repositories) can move REFS to REFS while preserving fastclone capabilities. Actually it can do that even from NTFS to REFS, SMB to REFS, REFS to XFS etc. So whatever kind of migration is needed, fast cloning can be preserved.

"Move" is done on "per job level" between repositories. As far as I see, you like to re-balance with one SOBR, right? That would be a "re-balance SOBR" operation then then. You add new extents to a SOBR and then the data is re-balanced between the extents.

Best regards,
Hannes

This sounds really good, any plans on making this a option for a single/specific VM, instead on an entire job?

[HannesK]

there are always plans to improve things

[mkretzer]

@edh Hitachi G350 dedicated FC storage with RAID 60. My guess is that the problem is connected to very high fragmentation and a nearly full ReFS...

The thing is that storage is not highly loaded - Veeam simply does not seem to use the same read method as it would do with tape backup as an example - as i said, tape backup from the same storage can keep two LTO-8 streaming 400 - 550 MB/s.

[HannesK]

It feels like IO goes single thread without async processing.

per design, async read should be used everywhere. the only way to find out seems to be checking with support. the logs should have the details.

[mkretzer]

Already on it - case 05300691

The thing is last week we had a full tape backup of the problematic job - it took 1 1/2 days just for that part of the backup which is fine as it was not the only object backed up at that time.

And now suddenly dskspd shows a rate of < 50 MB/s without anything else running. Write speed is still >700 MB/s.

Nothing changed on the HW side, only the mass "slow cloning" on the software side.

Storage vendor is also involved but it "smells" like a ReFS issue.

[mkretzer]

Ok, it really seemed to be a Windows/ReFS issue.

Just rebooting the veeam repo leads to reading speeds 10 times of the previous speed.

We also did a restore test and read speed from the storage is now between 500 and 600 MB/s.

We will upgrade to 2022 asap, we hope the updated ReFS driver will help.

[fractalenergy]

We had problem with ReFS with slow block cloning in the past (a lot of data with approx. 110TB of data without the cloned block with 45 days retention) and is now resolved with a XFS repository, although I have not made data move since we prefered make fresh chain by sealing the ReFS repo into the SOBR, it still goes very well since we have made that choice (1 year ago). If I can suggest you one things is to keep away from ReFS since they keep fix and break things up since 2016 server version with updates.

Best chance.

[mkretzer]

Yes, we did the same for our copy target. Best thing is the added flexibility with LVM under the XFS...

Still, we don't want to put "all our eggs in one basket" - with ~2 PB of backup data we want to use two different kind of systems...