Scale-out backup repository

[stewsie]

Hi

I have an extent in an SOBR that is running low on free space. Do I need to put this into maintenance and then evacuate the backups? I thought the who point of using SOBR is to let Veeam manage the repositories. The other extents have plenty of free space

Thanks

[Shestakov]

Hi Paul,
You don't really need to evacuate backups. If you want backups to spread more equally among the extents, just change the SOBR policy.

[stewsie]

Not sure what I can change on the policy?

It is already configured as follows

We only use the performance tier made up of 10 x 15.6 TB extents
We are using per-VM backup files
Placement policy = Data locality

Without using Capacity Tier which we cannot use at the moment what else can I change to resolve this issue with one extent almost running out of free space?

[Shestakov]

If you change the policy to Performance the backups will be placed on different extents.
With "Data locality" VBR tries to store the whole backup chain on single extent.

[veremin]

Also, you can re-balance SOBR manually to spread backup files equally among all extents. Thanks!

[stewsie]

Thanks

I noticed overnight when one of the jobs ran that uses the extent that is low on space that it was unable to meet the data placement policy as it stored the incremental on a different extent. I will investigate manually moving the files as you suggest. I delayed using SOBR as I read about some issues when it was first released and it appears some of these are still evident.

Regarding using the Performance option, that is not ideal as if an extent is lost then the chain is broken. As it is Veeam is already splitting some jobs over multiple extents rather than using the same one for all the VMs in a single job.

[Shestakov]

I noticed overnight when one of the jobs ran that uses the extent that is low on space that it was unable to meet the data placement policy as it stored the incremental on a different extent. I will investigate manually moving the files as you suggest.

That's the expected behavior.
With Data Locality policy you will naturally have the backups spread not equally, however you can leverage the workaround mentioned above by Vladimir.
Thanks

[garypigott]

Data locality is required if you want to get maximum benefit from ReFS dedupe.

[rossthorne]

We currently use SOBR on a pretty large scale and it can be tough to get a handle on at first. It also matters what kind of backup chains you are utilizing as well. If you are not running weekly full backups, your backup chains are going to stay on the extent that the wrote to the first time. We have this "issue" with our backup copy jobs because we are set up with forever forward at the moment and because a new full isn't written each week it continues to write the data to that specific location. This is can be troublesome in our case because some of our jobs are quite large and can take up the space of one of our 130TB SOBR extents. In the process now of converting things to ReFS to limit some of this.

In our case, we are running weekly full backups, where Veeam should be looking for the SOBR extent that has the most free space and write the next full and that weeks incremental backups in that location. At least this is the behavior that I have seen and it seems to do a pretty decent job of that.

[stewsie]

I managed to move some of the files to an extent with space and then imported the files with no problems. How would I move backup files to an extent that already contains backup files for the job?

Example.

If I have a job called File servers and the job has used all extents, do I just copy the backup files to the target extent but do not copy the .vbm file? Would a rescan then import the backup files correctly?

At the moment when I have had to move files to an extent, the relevant backup/copy job had not written anything to the extent so I was able to copy the complete folder without any issues.

Thanks

[foggy]

Yes, the subsequent rescan will pick up the changes. VBM is already stored on every extent, so no need to copy it.

[sosborne]

Now I can't provide any revelations but I can relate my similar experience with SOBR's and the Locality Policy which may help. Plus I would also appreciate advice if there is anything further as I have been ignoring the Warning emails about limited space on an extent.

I have a SOBR made up of an 80TB array and a 96TB. Logically I bought the 96TB array in good time before the 80TB completely filled. And I made the SOBR with a Locality Policy.

The jobs that are running on it are monthly Active Fulls with 24 RP's. Oddly it still continued to use the 80TB array albeit for decreasingly small portions (a VM or two) rather than just use the 96TB (which is now less than a 1TB).

I do get the Warning messages but like I say I have been ignoring them as I thought this month it will only use the 96TB array (but its still going).

I too was thinking of manually moving the RP's but given I am not really using incrementals or chains I thought it could be a time consuming task to simply stop receiving an email.

[foggy]

Hi Simon, what are the storages involved in this setup? Please note that for dedupe devices there's an additional logic that places fulls within the previous ones for better dedupe rate.

[sosborne]

The storage is Dell Direct Attach Storage. So no HW dedupe. Although the box (Storage Server/Backup Proxy/Backup Repository) is Win 2016 with ReFS disks.

[Ollo]

Performance Policy and ReFS makes no sense, because ReFS merge processing will only works, if the extend is on the same LUN.
We had the problem, that the maximum LUN size of Netapp Storge is 16TB. When an extend if placed on another LUN the merge processing runs hours instead of minutes. With Partial Merge processing, a lot of blocks have to be copied, so there are a lot of reads on disk parallel to write operations.
Disk Util is 100% for hours

[foggy]

The storage is Dell Direct Attach Storage. So no HW dedupe. Although the box (Storage Server/Backup Proxy/Backup Repository) is Win 2016 with ReFS disks.

If I'm not mistaken, the same logic works for ReFS as well (provided fastclone is enabled).

[stewsie]

Yes, the subsequent rescan will pick up the changes. VBM is already stored on every extent, so no need to copy it.

Worked perfectly. Thanks

[sosborne]

Hi Simon, what are the storages involved in this setup? Please note that for dedupe devices there's an additional logic that places fulls within the previous ones for better dedupe rate.

So if I were to manually move a complete set of RP's (or even a few RP's worth) for all jobs then it should just use the extent with the most storage?

[foggy]

Could you please clarify the question? Do you mean manual move or evacuate?

[sosborne]

Move the files in Windows explorer and Rescan the repositories.

[foggy]

You can move them to any other extent, selecting the one with more free space available looks reasonable.

[wane]

Hi,
we have an SOBR with 4 extents. We have set "Data locality" within Placement Policy. Within the advanced settings of the Performance Tier "Use per VM-backup files" has been checked and "Perform full backup when required extent ist offline" unchecked.

Now one extent is getting low on free space, but there is more than enough space on the remaining 3 extents.

• How does it behave if the space of one extent is too small, what happens if there is no more space available on an extent?

• Does veeam B&R write a new full on the remaining ones and start a new backup chain or do I have to do something by myself (e.g. copying backup files from one to another extent)?

[Maxim Karganov]

Hello, Walter

In such a situation when one of the extents is getting low on free space, B&R will start placing the backup files in on the extent that has enough free space remaining. This also applies to a Full Backup file. All the operations are performed automatically, so there is no need for manual intervention.

Please, refer to Backup Files Placement section of the user guide for more information.

Also, I'll merge your topic with an existing discussion - worth checking.

Do not hesitate to ask if you have additional questions. Thanks.