Comprehensive data protection for all workloads
Post Reply
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Scale-out backup repository

Post by stewsie »

Hi

I have an extent in an SOBR that is running low on free space. Do I need to put this into maintenance and then evacuate the backups? I thought the who point of using SOBR is to let Veeam manage the repositories. The other extents have plenty of free space

Thanks
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Scale-out backup repository

Post by Shestakov »

Hi Paul,
You don't really need to evacuate backups. If you want backups to spread more equally among the extents, just change the SOBR policy.
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Scale-out backup repository

Post by stewsie »

Not sure what I can change on the policy?

It is already configured as follows

We only use the performance tier made up of 10 x 15.6 TB extents
We are using per-VM backup files
Placement policy = Data locality

Without using Capacity Tier which we cannot use at the moment what else can I change to resolve this issue with one extent almost running out of free space?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Scale-out backup repository

Post by Shestakov »

If you change the policy to Performance the backups will be placed on different extents.
With "Data locality" VBR tries to store the whole backup chain on single extent.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scale-out backup repository

Post by veremin »

Also, you can re-balance SOBR manually to spread backup files equally among all extents. Thanks!
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Scale-out backup repository

Post by stewsie »

Thanks

I noticed overnight when one of the jobs ran that uses the extent that is low on space that it was unable to meet the data placement policy as it stored the incremental on a different extent. I will investigate manually moving the files as you suggest. I delayed using SOBR as I read about some issues when it was first released and it appears some of these are still evident.

Regarding using the Performance option, that is not ideal as if an extent is lost then the chain is broken. As it is Veeam is already splitting some jobs over multiple extents rather than using the same one for all the VMs in a single job.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Scale-out backup repository

Post by Shestakov » 1 person likes this post

I noticed overnight when one of the jobs ran that uses the extent that is low on space that it was unable to meet the data placement policy as it stored the incremental on a different extent. I will investigate manually moving the files as you suggest.
That's the expected behavior.
With Data Locality policy you will naturally have the backups spread not equally, however you can leverage the workaround mentioned above by Vladimir.
Thanks
garypigott
Service Provider
Posts: 7
Liked: 2 times
Joined: Sep 17, 2018 4:45 pm
Full Name: Gary Pigott
Contact:

Re: Scale-out backup repository

Post by garypigott »

Data locality is required if you want to get maximum benefit from ReFS dedupe.
rossthorne
Novice
Posts: 6
Liked: 4 times
Joined: May 03, 2018 12:30 pm
Full Name: Ross Thorne
Contact:

Re: Scale-out backup repository

Post by rossthorne » 1 person likes this post

We currently use SOBR on a pretty large scale and it can be tough to get a handle on at first. It also matters what kind of backup chains you are utilizing as well. If you are not running weekly full backups, your backup chains are going to stay on the extent that the wrote to the first time. We have this "issue" with our backup copy jobs because we are set up with forever forward at the moment and because a new full isn't written each week it continues to write the data to that specific location. This is can be troublesome in our case because some of our jobs are quite large and can take up the space of one of our 130TB SOBR extents. In the process now of converting things to ReFS to limit some of this.

In our case, we are running weekly full backups, where Veeam should be looking for the SOBR extent that has the most free space and write the next full and that weeks incremental backups in that location. At least this is the behavior that I have seen and it seems to do a pretty decent job of that.
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Scale-out backup repository

Post by stewsie »

I managed to move some of the files to an extent with space and then imported the files with no problems. How would I move backup files to an extent that already contains backup files for the job?

Example.

If I have a job called File servers and the job has used all extents, do I just copy the backup files to the target extent but do not copy the .vbm file? Would a rescan then import the backup files correctly?

At the moment when I have had to move files to an extent, the relevant backup/copy job had not written anything to the extent so I was able to copy the complete folder without any issues.

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out backup repository

Post by foggy » 1 person likes this post

Yes, the subsequent rescan will pick up the changes. VBM is already stored on every extent, so no need to copy it.
sosborne
Enthusiast
Posts: 66
Liked: 5 times
Joined: Jan 30, 2018 12:06 pm
Full Name: Simon Osborne
Contact:

Re: Scale-out backup repository

Post by sosborne »

Now I can't provide any revelations but I can relate my similar experience with SOBR's and the Locality Policy which may help. Plus I would also appreciate advice if there is anything further as I have been ignoring the Warning emails about limited space on an extent.

I have a SOBR made up of an 80TB array and a 96TB. Logically I bought the 96TB array in good time before the 80TB completely filled. And I made the SOBR with a Locality Policy.

The jobs that are running on it are monthly Active Fulls with 24 RP's. Oddly it still continued to use the 80TB array albeit for decreasingly small portions (a VM or two) rather than just use the 96TB (which is now less than a 1TB).

I do get the Warning messages but like I say I have been ignoring them as I thought this month it will only use the 96TB array (but its still going).

I too was thinking of manually moving the RP's but given I am not really using incrementals or chains I thought it could be a time consuming task to simply stop receiving an email.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out backup repository

Post by foggy »

Hi Simon, what are the storages involved in this setup? Please note that for dedupe devices there's an additional logic that places fulls within the previous ones for better dedupe rate.
sosborne
Enthusiast
Posts: 66
Liked: 5 times
Joined: Jan 30, 2018 12:06 pm
Full Name: Simon Osborne
Contact:

Re: Scale-out backup repository

Post by sosborne »

The storage is Dell Direct Attach Storage. So no HW dedupe. Although the box (Storage Server/Backup Proxy/Backup Repository) is Win 2016 with ReFS disks.
Ollo
Enthusiast
Posts: 32
Liked: 3 times
Joined: Jan 19, 2018 6:32 am
Contact:

Re: Scale-out backup repository

Post by Ollo »

Performance Policy and ReFS makes no sense, because ReFS merge processing will only works, if the extend is on the same LUN.
We had the problem, that the maximum LUN size of Netapp Storge is 16TB. When an extend if placed on another LUN the merge processing runs hours instead of minutes. With Partial Merge processing, a lot of blocks have to be copied, so there are a lot of reads on disk parallel to write operations.
Disk Util is 100% for hours
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out backup repository

Post by foggy »

sosborne wrote: Aug 06, 2019 7:45 am The storage is Dell Direct Attach Storage. So no HW dedupe. Although the box (Storage Server/Backup Proxy/Backup Repository) is Win 2016 with ReFS disks.
If I'm not mistaken, the same logic works for ReFS as well (provided fastclone is enabled).
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Scale-out backup repository

Post by stewsie »

foggy wrote: Aug 05, 2019 1:24 pm Yes, the subsequent rescan will pick up the changes. VBM is already stored on every extent, so no need to copy it.
Worked perfectly. Thanks
sosborne
Enthusiast
Posts: 66
Liked: 5 times
Joined: Jan 30, 2018 12:06 pm
Full Name: Simon Osborne
Contact:

Re: Scale-out backup repository

Post by sosborne »

foggy wrote: Aug 05, 2019 4:42 pm Hi Simon, what are the storages involved in this setup? Please note that for dedupe devices there's an additional logic that places fulls within the previous ones for better dedupe rate.
So if I were to manually move a complete set of RP's (or even a few RP's worth) for all jobs then it should just use the extent with the most storage?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out backup repository

Post by foggy »

Could you please clarify the question? Do you mean manual move or evacuate?
sosborne
Enthusiast
Posts: 66
Liked: 5 times
Joined: Jan 30, 2018 12:06 pm
Full Name: Simon Osborne
Contact:

Re: Scale-out backup repository

Post by sosborne »

Move the files in Windows explorer and Rescan the repositories.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out backup repository

Post by foggy »

You can move them to any other extent, selecting the one with more free space available looks reasonable.
wane
Enthusiast
Posts: 39
Liked: 5 times
Joined: Jun 26, 2020 9:47 pm
Full Name: Walter Neu
Contact:

[MERGED] SOBR - extent is getting low on free space

Post by wane »

Hi,
we have an SOBR with 4 extents. We have set "Data locality" within Placement Policy. Within the advanced settings of the Performance Tier "Use per VM-backup files" has been checked and "Perform full backup when required extent ist offline" unchecked.

Now one extent is getting low on free space, but there is more than enough space on the remaining 3 extents.
  • How does it behave if the space of one extent is too small, what happens if there is no more space available on an extent?
  • Does veeam B&R write a new full on the remaining ones and start a new backup chain or do I have to do something by myself (e.g. copying backup files from one to another extent)?
Maxim Karganov
Influencer
Posts: 22
Liked: 6 times
Joined: Jun 08, 2020 9:18 am
Contact:

Re: SOBR - extent is getting low on free space

Post by Maxim Karganov »

Hello, Walter

In such a situation when one of the extents is getting low on free space, B&R will start placing the backup files in on the extent that has enough free space remaining. This also applies to a Full Backup file. All the operations are performed automatically, so there is no need for manual intervention.

Please, refer to Backup Files Placement section of the user guide for more information.

Also, I'll merge your topic with an existing discussion - worth checking.

Do not hesitate to ask if you have additional questions. Thanks.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 222 guests