Feature request: Make scale out repo what I hoped it was!

Availability for the Always-On Enterprise

Feature request: Make scale out repo what I hoped it was!

Veeam Logoby kkuszek » Wed Jan 25, 2017 1:48 am

so today I ran out of space. I migrated machines and reset CBT so some machines were duplicated.
however I have a scale out repository configured. I have blob storage for just this scenario so if I for some reason fill my primary repository a second one comes to the rescue...

right?

no!

my jobs keep failing. I have to delete restore points. Instead of allowing me to accommodate temporary changes in space needs due to unusual circumstances it seems to be particular about just what files can go where and still trying to group by job or job type.
I should be able to set a minimum free space for a repo and when it drops below that the jobs switch to the scale out target. Then I can just let it age out and go back to normal over time naturally. I don't want to manually need to manipulate my backups or remember to change things around. I added failover space for that.

I envision SOR to work like the top button of my pants at a holiday feast. If required I can make a little temporary room for a short period of time till things settle down.

So why can't I use it that way?
kkuszek
Enthusiast
 
Posts: 92
Liked: 5 times
Joined: Fri Mar 13, 2015 3:12 pm
Full Name: Kurt Kuszek

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby Shestakov » Wed Jan 25, 2017 9:24 am

Hello Kurt,
Could you elaborate on your use case?
As far as I understood you have a Scale-our repository with "Data locality" policy and because of CBT reset the SOBR got filled without notifications and you would like to have a rule to forward backups to another repository if the primary one fills. Is that correct?
By the way, do you use Veeam ONE? It has an option to monitor repository free space and start custom activity when threshold is crossed.
Thanks!
Shestakov
Veeam Software
 
Posts: 4754
Liked: 385 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby kkuszek » Wed Jan 25, 2017 12:06 pm

Hello,
yes what happened this weekend is I migrated all of my VM's from one datastore to another with quick migration. Several of them VM's had to be removed and re-added to the backup jobs as they were detected as new machines.
I also have been prepping several thin disks for a shrink reclaim by a defrag/zero fill operation.
This blew up my backup size. This isn't something I do every day and it's not a problem I experience often. I expected scale out repo to take over and allow temporary space expansion by backups saving there instead for the meantime.
Another example would be a server migration where you stand up a new one and move a lot of data from one to another in another job. These are temporary changes that might mandate more space but not change company retention policy.

I do have ONE. although I don't want to just manually move files and mess up the database.
kkuszek
Enthusiast
 
Posts: 92
Liked: 5 times
Joined: Fri Mar 13, 2015 3:12 pm
Full Name: Kurt Kuszek

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby Shestakov » Wed Jan 25, 2017 4:49 pm

Well, as you said some operations require additional repository space and for the repository it`s just a set of new backups. If it is a deduplication appliance the backups of same VMs can be deduplicated on the repository though.
Veeam ONE alarms are to notify you when you have less free space than expected and you can disable some jobs until you get enough free space.
Shestakov
Veeam Software
 
Posts: 4754
Liked: 385 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby foggy » Wed Jan 25, 2017 5:05 pm

Kurt, what Veeam B&R version are you using? Basically, it should work as you expect, in case there's not enough space on the current extent, the backup should be placed on another one.
foggy
Veeam Software
 
Posts: 14560
Liked: 1060 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby kkuszek » Wed Jan 25, 2017 5:11 pm

Shestakov:
Thank you, yes. I know I can always manually move files, delete points, disable jobs, Run a deduplication garbage collect, or otherwise make space that way.

That doesn't though go into the spirit of what my expectation was of scale out repo and how I would like it to work for me (as well as what my expectations were for it).

I feel like there should be a third policy. There is data locality, there is performance, and there should be one for failover. This would become a repository failover for primary repository having insufficient space or otherwise being inaccessible. Log a warning in the job that there was insufficient space so the Scale-out was utilized so I am aware.

The idea is I don't want my backups to be compromised in completion or retention points when there is an anomaly in operations. I want to set my repository to have a free space 100gb reserve untouched by veeam, and I want backups to start using the SOR if the backup job has the potential to exceed the space available on my repository. I never want my repository to hit 0 bytes free.

On that same note it would be handy to choose a SOR which has been utilized and have a button to evacuate repository which then dumps the contents back on the primary repository.


Foggy:
I am on 9.5. My jobs for the last few days have failed for insufficient space and never saved a thing on the empty scale out repository (whether set to data locality or performance)
1/25/2017 1:39:27 AM :: Error: There is not enough space on the disk.
kkuszek
Enthusiast
 
Posts: 92
Liked: 5 times
Joined: Fri Mar 13, 2015 3:12 pm
Full Name: Kurt Kuszek

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby foggy » Wed Jan 25, 2017 5:29 pm

kkuszek wrote:I am on 9.5. My jobs for the last few days have failed for insufficient space and never saved a thing on the empty scale out repository (whether set to data locality or performance)
1/25/2017 1:39:27 AM :: Error: There is not enough space on the disk.

I'd let our engineers to verify whether this behavior is expected.
foggy
Veeam Software
 
Posts: 14560
Liked: 1060 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby Seve CH » Mon Jan 30, 2017 1:07 pm 2 people like this post

kkuszek wrote:Foggy:
I am on 9.5. My jobs for the last few days have failed for insufficient space and never saved a thing on the empty scale out repository (whether set to data locality or performance)
1/25/2017 1:39:27 AM :: Error: There is not enough space on the disk.


I think that happened to me on New Year's copyjobs (VB&R v9). Scaleout with several 8TB extents, Veeam placed two big VMs on the same extent (one approx. 7TB .VBK the other 3.5TB, plus several small ones) = not enough space to complete while other extents were empty. As a side-effect of implementing this:
https://forums.veeam.com/veeam-backup-replication-f2/process-big-vms-first-t40084.html with that information Veeam B&R would be able to (not yet implemented) spread out big VMs so they do not fill-up the same extent. I had to move the 3.5TB file and many smaller ones by hand as soon as they completed to make space for the 7TB one. The logic used I think was "Veeam chooses a specific extent for some reason and then starts as much jobs as possible on it until it reaches the maximum concurrent I/O configured". I have another repo (2x60TB) and one extent is full of VBK (no limit on concurrent I/O) and the other has .VIB and several VBK (VB&R complained having to put them there and I do not remember if the job failed before placing them there).

Other option would be this one so as soon as the extent is full, the VBK will be split and continued in another extent like good old .RAR files :) :
https://forums.veeam.com/veeam-backup-replication-f2/feature-request-split-vkb-t35406-30.html#p223067

One of the questions for your Engineers would be one: How does VB&R manage large .VBK files bigger than the biggest extent in a scaleout repository?

Regards.
Seve CH
Influencer
 
Posts: 10
Liked: 6 times
Joined: Mon May 09, 2016 2:34 pm
Location: Switzerland
Full Name: JM Severino

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby glamic26 » Mon Jan 30, 2017 1:25 pm

Our experience is also the same in that SOBRs don't act the way we expected. We had a SOBR with 50TB of free space across 11 repository extents. However, one of these extents filled up and the backups kept trying to write to it and failing due to insufficient free space. It made no attempt to just switch to another repository extent within the SOBR even though there were no policies set to prevent this. We were told the SOBR was working by design (I don't therefore understand the point of them) and that we would have to continue to monitor free space remaining on each extent and manually move files when required: just like we did previously with individual repositories.
glamic26
Service Provider
 
Posts: 15
Liked: 9 times
Joined: Tue Apr 21, 2015 12:10 pm

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby A.J. » Mon Feb 20, 2017 2:45 pm 1 person likes this post

I fully agree with your statements. The veeam SOBR is not working as we veeam users would expect.
Today one of our SOBR for copy jobs run out of space (per vm jobs, policy data locallity). The copy job already used a second extent because of low disk space on the first starting extent. So you think the sobr works as expected.
But that helps only for a few days. The first extent continues to fill up more and more maybe because of 3 reasons. First of all the copy jobs seems to use another logic for merging the incremental into the full file (see KB2207).
The second reason is that there is no space limit while filling a extent. Often we were told that there is a spare space of 10%, another supportler said there exists a reg key that responsable for the spare size... Nothing of that seems to be correct.
The third reason seems to be the biggest problem in my opinion. There is no space reclamation logic inside veeam that can detect such situations and can fix it. One solution would be that veeam tries to move some full files with their corresponding incremental files from the nearly full extent to another extent with enough space and keep the backup chain and the job intact. At the moment the jobs always retry to merge the oldest file into the full file though there is no space left on the extent. The new incremental files will be placed on another extent with enough space but the merge could never be done anymore.
The only chance at the moment is to start from scratch and create a new copy job and a new backup chain.

We opend support cases many times but had never luck with the support to solve the problems or to place a feature request. At the moment, their is no real benefit of using SOBR. You still have to monitor the repository space and take manually action when the repository is running out of space. We though the introduction o SOBR would solve all of these problems but it was a fake ;-)
A.J.
Service Provider
 
Posts: 3
Liked: 1 time
Joined: Tue Jul 26, 2016 6:19 am

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby kkuszek » Mon Feb 20, 2017 8:27 pm

Veeam engineers, are you reading these testimonies?

I feel like this is quite a large recurring issue both on the forums and just in general with the product. We shouldn't have to design with unreasonable underprovisioning or manual intervention around a flexible environment.

If the motto is that it just works, it should just work.
kkuszek
Enthusiast
 
Posts: 92
Liked: 5 times
Joined: Fri Mar 13, 2015 3:12 pm
Full Name: Kurt Kuszek

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby Gostev » Mon Feb 20, 2017 8:52 pm

Veeam engineers are not following these forums. We could have brought this up with them, but not a single post above included the support case ID (as required per forum rules) and support case ID is the first thing engineers will ask us because they need to review the debug logs to identify the issue in the specific scenario and act upon it.

Spare space is 1% by default and there's indeed a registry value to control it, this space reservation is a 9.5 feature.
Gostev
Veeam Software
 
Posts: 21354
Liked: 2333 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby kkuszek » Tue Feb 21, 2017 1:40 pm

My understanding is that the forums are where we are to make requests and feature suggestions while support is where we go when there is a problem. Since others have mentioned they have spoken to support and it sounds like this is functioning as designed I would believe this to be a design issue. Would that not be a feature suggestion to change design and function?

It sounds less like it's broken as in support and more like it's just broken as designed.

Could you please share that registry key for changing that setting?
I believe that would be helpful to me (also would be great as a repository setting right in the program instead of a registry tweak)
kkuszek
Enthusiast
 
Posts: 92
Liked: 5 times
Joined: Fri Mar 13, 2015 3:12 pm
Full Name: Kurt Kuszek

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby foggy » Tue Feb 21, 2017 1:59 pm

Kurt, as mentioned above, the behavior you're describing is not inline with the expected behavior based on the current SOBR design, so contacting support for deeper investigation is recommended to identify the actual reason for the observed behavior.
foggy
Veeam Software
 
Posts: 14560
Liked: 1060 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Feature request: Make scale out repo what I hoped it was

Veeam Logoby Delo123 » Tue Feb 21, 2017 3:40 pm

We opend support cases many times but had never luck with the support to solve the problems or to place a feature request. At the moment, their is no real benefit of using SOBR. You still have to monitor the repository space and take manually action when the repository is running out of space. We though the introduction o SOBR would solve all of these problems but it was a fake ;-)


There actually are a lot of good reasons to use SOBR, but not this one...
Delo123
Expert
 
Posts: 340
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot] and 16 guests