Discussions specific to the VMware vSphere hypervisor
Post Reply
stewsie
Expert
Posts: 128
Liked: 7 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Scale-out backup repository

Post by stewsie » Jul 22, 2019 11:06 am

Hi

I have an extent in an SOBR that is running low on free space. Do I need to put this into maintenance and then evacuate the backups? I thought the who point of using SOBR is to let Veeam manage the repositories. The other extents have plenty of free space

Thanks

Shestakov
Veeam Software
Posts: 6974
Liked: 710 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Scale-out backup repository

Post by Shestakov » Jul 22, 2019 11:19 am

Hi Paul,
You don't really need to evacuate backups. If you want backups to spread more equally among the extents, just change the SOBR policy.

stewsie
Expert
Posts: 128
Liked: 7 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Scale-out backup repository

Post by stewsie » Jul 22, 2019 1:58 pm

Not sure what I can change on the policy?

It is already configured as follows

We only use the performance tier made up of 10 x 15.6 TB extents
We are using per-VM backup files
Placement policy = Data locality

Without using Capacity Tier which we cannot use at the moment what else can I change to resolve this issue with one extent almost running out of free space?

Shestakov
Veeam Software
Posts: 6974
Liked: 710 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Scale-out backup repository

Post by Shestakov » Jul 22, 2019 7:35 pm

If you change the policy to Performance the backups will be placed on different extents.
With "Data locality" VBR tries to store the whole backup chain on single extent.

veremin
Product Manager
Posts: 16880
Liked: 1430 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Scale-out backup repository

Post by veremin » Jul 22, 2019 7:39 pm

Also, you can re-balance SOBR manually to spread backup files equally among all extents. Thanks!

stewsie
Expert
Posts: 128
Liked: 7 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Scale-out backup repository

Post by stewsie » Jul 23, 2019 7:45 am

Thanks

I noticed overnight when one of the jobs ran that uses the extent that is low on space that it was unable to meet the data placement policy as it stored the incremental on a different extent. I will investigate manually moving the files as you suggest. I delayed using SOBR as I read about some issues when it was first released and it appears some of these are still evident.

Regarding using the Performance option, that is not ideal as if an extent is lost then the chain is broken. As it is Veeam is already splitting some jobs over multiple extents rather than using the same one for all the VMs in a single job.

Shestakov
Veeam Software
Posts: 6974
Liked: 710 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Scale-out backup repository

Post by Shestakov » Jul 23, 2019 8:39 am 1 person likes this post

I noticed overnight when one of the jobs ran that uses the extent that is low on space that it was unable to meet the data placement policy as it stored the incremental on a different extent. I will investigate manually moving the files as you suggest.
That's the expected behavior.
With Data Locality policy you will naturally have the backups spread not equally, however you can leverage the workaround mentioned above by Vladimir.
Thanks

garypigott
Service Provider
Posts: 3
Liked: 1 time
Joined: Sep 17, 2018 4:45 pm
Full Name: Gary Pigott
Contact:

Re: Scale-out backup repository

Post by garypigott » Jul 29, 2019 8:57 am

Data locality is required if you want to get maximum benefit from ReFS dedupe.

rossthorne
Novice
Posts: 5
Liked: 3 times
Joined: May 03, 2018 12:30 pm
Full Name: Ross Thorne
Contact:

Re: Scale-out backup repository

Post by rossthorne » Jul 29, 2019 11:43 am 1 person likes this post

We currently use SOBR on a pretty large scale and it can be tough to get a handle on at first. It also matters what kind of backup chains you are utilizing as well. If you are not running weekly full backups, your backup chains are going to stay on the extent that the wrote to the first time. We have this "issue" with our backup copy jobs because we are set up with forever forward at the moment and because a new full isn't written each week it continues to write the data to that specific location. This is can be troublesome in our case because some of our jobs are quite large and can take up the space of one of our 130TB SOBR extents. In the process now of converting things to ReFS to limit some of this.

In our case, we are running weekly full backups, where Veeam should be looking for the SOBR extent that has the most free space and write the next full and that weeks incremental backups in that location. At least this is the behavior that I have seen and it seems to do a pretty decent job of that.

stewsie
Expert
Posts: 128
Liked: 7 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Scale-out backup repository

Post by stewsie » Aug 05, 2019 7:16 am

I managed to move some of the files to an extent with space and then imported the files with no problems. How would I move backup files to an extent that already contains backup files for the job?

Example.

If I have a job called File servers and the job has used all extents, do I just copy the backup files to the target extent but do not copy the .vbm file? Would a rescan then import the backup files correctly?

At the moment when I have had to move files to an extent, the relevant backup/copy job had not written anything to the extent so I was able to copy the complete folder without any issues.

Thanks

foggy
Veeam Software
Posts: 18244
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out backup repository

Post by foggy » Aug 05, 2019 1:24 pm 1 person likes this post

Yes, the subsequent rescan will pick up the changes. VBM is already stored on every extent, so no need to copy it.

sosborne
Enthusiast
Posts: 53
Liked: 3 times
Joined: Jan 30, 2018 12:06 pm
Full Name: Simon Osborne
Contact:

Re: Scale-out backup repository

Post by sosborne » Aug 05, 2019 3:02 pm

Now I can't provide any revelations but I can relate my similar experience with SOBR's and the Locality Policy which may help. Plus I would also appreciate advice if there is anything further as I have been ignoring the Warning emails about limited space on an extent.

I have a SOBR made up of an 80TB array and a 96TB. Logically I bought the 96TB array in good time before the 80TB completely filled. And I made the SOBR with a Locality Policy.

The jobs that are running on it are monthly Active Fulls with 24 RP's. Oddly it still continued to use the 80TB array albeit for decreasingly small portions (a VM or two) rather than just use the 96TB (which is now less than a 1TB).

I do get the Warning messages but like I say I have been ignoring them as I thought this month it will only use the 96TB array (but its still going).

I too was thinking of manually moving the RP's but given I am not really using incrementals or chains I thought it could be a time consuming task to simply stop receiving an email.

foggy
Veeam Software
Posts: 18244
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out backup repository

Post by foggy » Aug 05, 2019 4:42 pm

Hi Simon, what are the storages involved in this setup? Please note that for dedupe devices there's an additional logic that places fulls within the previous ones for better dedupe rate.

sosborne
Enthusiast
Posts: 53
Liked: 3 times
Joined: Jan 30, 2018 12:06 pm
Full Name: Simon Osborne
Contact:

Re: Scale-out backup repository

Post by sosborne » Aug 06, 2019 7:45 am

The storage is Dell Direct Attach Storage. So no HW dedupe. Although the box (Storage Server/Backup Proxy/Backup Repository) is Win 2016 with ReFS disks.

Ollo
Influencer
Posts: 21
Liked: 2 times
Joined: Jan 19, 2018 6:32 am
Contact:

Re: Scale-out backup repository

Post by Ollo » Aug 06, 2019 8:46 am

Performance Policy and ReFS makes no sense, because ReFS merge processing will only works, if the extend is on the same LUN.
We had the problem, that the maximum LUN size of Netapp Storge is 16TB. When an extend if placed on another LUN the merge processing runs hours instead of minutes. With Partial Merge processing, a lot of blocks have to be copied, so there are a lot of reads on disk parallel to write operations.
Disk Util is 100% for hours

foggy
Veeam Software
Posts: 18244
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out backup repository

Post by foggy » Aug 06, 2019 9:57 am

sosborne wrote:
Aug 06, 2019 7:45 am
The storage is Dell Direct Attach Storage. So no HW dedupe. Although the box (Storage Server/Backup Proxy/Backup Repository) is Win 2016 with ReFS disks.
If I'm not mistaken, the same logic works for ReFS as well (provided fastclone is enabled).

stewsie
Expert
Posts: 128
Liked: 7 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Scale-out backup repository

Post by stewsie » Aug 07, 2019 7:44 am

foggy wrote:
Aug 05, 2019 1:24 pm
Yes, the subsequent rescan will pick up the changes. VBM is already stored on every extent, so no need to copy it.
Worked perfectly. Thanks

sosborne
Enthusiast
Posts: 53
Liked: 3 times
Joined: Jan 30, 2018 12:06 pm
Full Name: Simon Osborne
Contact:

Re: Scale-out backup repository

Post by sosborne » Aug 07, 2019 9:07 am

foggy wrote:
Aug 05, 2019 4:42 pm
Hi Simon, what are the storages involved in this setup? Please note that for dedupe devices there's an additional logic that places fulls within the previous ones for better dedupe rate.
So if I were to manually move a complete set of RP's (or even a few RP's worth) for all jobs then it should just use the extent with the most storage?

foggy
Veeam Software
Posts: 18244
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out backup repository

Post by foggy » Aug 07, 2019 4:40 pm

Could you please clarify the question? Do you mean manual move or evacuate?

sosborne
Enthusiast
Posts: 53
Liked: 3 times
Joined: Jan 30, 2018 12:06 pm
Full Name: Simon Osborne
Contact:

Re: Scale-out backup repository

Post by sosborne » Aug 08, 2019 8:28 am

Move the files in Windows explorer and Rescan the repositories.

foggy
Veeam Software
Posts: 18244
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out backup repository

Post by foggy » Aug 08, 2019 11:06 am

You can move them to any other extent, selecting the one with more free space available looks reasonable.

Post Reply

Who is online

Users browsing this forum: No registered users and 28 guests