Feature request: Make scale out repo what I hoped it was!

[kkuszek]

so today I ran out of space. I migrated machines and reset CBT so some machines were duplicated.
however I have a scale out repository configured. I have blob storage for just this scenario so if I for some reason fill my primary repository a second one comes to the rescue...

right?

no!

my jobs keep failing. I have to delete restore points. Instead of allowing me to accommodate temporary changes in space needs due to unusual circumstances it seems to be particular about just what files can go where and still trying to group by job or job type.
I should be able to set a minimum free space for a repo and when it drops below that the jobs switch to the scale out target. Then I can just let it age out and go back to normal over time naturally. I don't want to manually need to manipulate my backups or remember to change things around. I added failover space for that.

I envision SOR to work like the top button of my pants at a holiday feast. If required I can make a little temporary room for a short period of time till things settle down.

So why can't I use it that way?

[Shestakov]

Hello Kurt,
Could you elaborate on your use case?
As far as I understood you have a Scale-our repository with "Data locality" policy and because of CBT reset the SOBR got filled without notifications and you would like to have a rule to forward backups to another repository if the primary one fills. Is that correct?
By the way, do you use Veeam ONE? It has an option to monitor repository free space and start custom activity when threshold is crossed.
Thanks!

[kkuszek]

Hello,
yes what happened this weekend is I migrated all of my VM's from one datastore to another with quick migration. Several of them VM's had to be removed and re-added to the backup jobs as they were detected as new machines.
I also have been prepping several thin disks for a shrink reclaim by a defrag/zero fill operation.
This blew up my backup size. This isn't something I do every day and it's not a problem I experience often. I expected scale out repo to take over and allow temporary space expansion by backups saving there instead for the meantime.
Another example would be a server migration where you stand up a new one and move a lot of data from one to another in another job. These are temporary changes that might mandate more space but not change company retention policy.

I do have ONE. although I don't want to just manually move files and mess up the database.

[Shestakov]

Well, as you said some operations require additional repository space and for the repository it`s just a set of new backups. If it is a deduplication appliance the backups of same VMs can be deduplicated on the repository though.
Veeam ONE alarms are to notify you when you have less free space than expected and you can disable some jobs until you get enough free space.

[foggy]

Kurt, what Veeam B&R version are you using? Basically, it should work as you expect, in case there's not enough space on the current extent, the backup should be placed on another one.

[kkuszek]

Shestakov:
Thank you, yes. I know I can always manually move files, delete points, disable jobs, Run a deduplication garbage collect, or otherwise make space that way.

That doesn't though go into the spirit of what my expectation was of scale out repo and how I would like it to work for me (as well as what my expectations were for it).

I feel like there should be a third policy. There is data locality, there is performance, and there should be one for failover. This would become a repository failover for primary repository having insufficient space or otherwise being inaccessible. Log a warning in the job that there was insufficient space so the Scale-out was utilized so I am aware.

The idea is I don't want my backups to be compromised in completion or retention points when there is an anomaly in operations. I want to set my repository to have a free space 100gb reserve untouched by veeam, and I want backups to start using the SOR if the backup job has the potential to exceed the space available on my repository. I never want my repository to hit 0 bytes free.

On that same note it would be handy to choose a SOR which has been utilized and have a button to evacuate repository which then dumps the contents back on the primary repository.


Foggy:
I am on 9.5. My jobs for the last few days have failed for insufficient space and never saved a thing on the empty scale out repository (whether set to data locality or performance)
1/25/2017 1:39:27 AM :: Error: There is not enough space on the disk.

[foggy]

I am on 9.5. My jobs for the last few days have failed for insufficient space and never saved a thing on the empty scale out repository (whether set to data locality or performance)
1/25/2017 1:39:27 AM :: Error: There is not enough space on the disk.

I'd let our engineers to verify whether this behavior is expected.

[Seve CH]

Foggy:
I am on 9.5. My jobs for the last few days have failed for insufficient space and never saved a thing on the empty scale out repository (whether set to data locality or performance)
1/25/2017 1:39:27 AM :: Error: There is not enough space on the disk.

I think that happened to me on New Year's copyjobs (VB&R v9). Scaleout with several 8TB extents, Veeam placed two big VMs on the same extent (one approx. 7TB .VBK the other 3.5TB, plus several small ones) = not enough space to complete while other extents were empty. As a side-effect of implementing this:
veeam-backup-replication-f2/process-big ... 40084.html with that information Veeam B&R would be able to (not yet implemented) spread out big VMs so they do not fill-up the same extent. I had to move the 3.5TB file and many smaller ones by hand as soon as they completed to make space for the 7TB one. The logic used I think was "Veeam chooses a specific extent for some reason and then starts as much jobs as possible on it until it reaches the maximum concurrent I/O configured". I have another repo (2x60TB) and one extent is full of VBK (no limit on concurrent I/O) and the other has .VIB and several VBK (VB&R complained having to put them there and I do not remember if the job failed before placing them there).

Other option would be this one so as soon as the extent is full, the VBK will be split and continued in another extent like good old .RAR files :
veeam-backup-replication-f2/feature-req ... ml#p223067

One of the questions for your Engineers would be one: How does VB&R manage large .VBK files bigger than the biggest extent in a scaleout repository?

Regards.

[glamic26]

Our experience is also the same in that SOBRs don't act the way we expected. We had a SOBR with 50TB of free space across 11 repository extents. However, one of these extents filled up and the backups kept trying to write to it and failing due to insufficient free space. It made no attempt to just switch to another repository extent within the SOBR even though there were no policies set to prevent this. We were told the SOBR was working by design (I don't therefore understand the point of them) and that we would have to continue to monitor free space remaining on each extent and manually move files when required: just like we did previously with individual repositories.

[A.J.]

I fully agree with your statements. The veeam SOBR is not working as we veeam users would expect.
Today one of our SOBR for copy jobs run out of space (per vm jobs, policy data locallity). The copy job already used a second extent because of low disk space on the first starting extent. So you think the sobr works as expected.
But that helps only for a few days. The first extent continues to fill up more and more maybe because of 3 reasons. First of all the copy jobs seems to use another logic for merging the incremental into the full file (see KB2207).
The second reason is that there is no space limit while filling a extent. Often we were told that there is a spare space of 10%, another supportler said there exists a reg key that responsable for the spare size... Nothing of that seems to be correct.
The third reason seems to be the biggest problem in my opinion. There is no space reclamation logic inside veeam that can detect such situations and can fix it. One solution would be that veeam tries to move some full files with their corresponding incremental files from the nearly full extent to another extent with enough space and keep the backup chain and the job intact. At the moment the jobs always retry to merge the oldest file into the full file though there is no space left on the extent. The new incremental files will be placed on another extent with enough space but the merge could never be done anymore.
The only chance at the moment is to start from scratch and create a new copy job and a new backup chain.

We opend support cases many times but had never luck with the support to solve the problems or to place a feature request. At the moment, their is no real benefit of using SOBR. You still have to monitor the repository space and take manually action when the repository is running out of space. We though the introduction o SOBR would solve all of these problems but it was a fake

[kkuszek]

Veeam engineers, are you reading these testimonies?

I feel like this is quite a large recurring issue both on the forums and just in general with the product. We shouldn't have to design with unreasonable underprovisioning or manual intervention around a flexible environment.

If the motto is that it just works, it should just work.

[Gostev]

Veeam engineers are not following these forums. We could have brought this up with them, but not a single post above included the support case ID (as required per forum rules) and support case ID is the first thing engineers will ask us because they need to review the debug logs to identify the issue in the specific scenario and act upon it.

Spare space is 1% by default and there's indeed a registry value to control it, this space reservation is a 9.5 feature.

[kkuszek]

My understanding is that the forums are where we are to make requests and feature suggestions while support is where we go when there is a problem. Since others have mentioned they have spoken to support and it sounds like this is functioning as designed I would believe this to be a design issue. Would that not be a feature suggestion to change design and function?

It sounds less like it's broken as in support and more like it's just broken as designed.

Could you please share that registry key for changing that setting?
I believe that would be helpful to me (also would be great as a repository setting right in the program instead of a registry tweak)

[foggy]

Kurt, as mentioned above, the behavior you're describing is not inline with the expected behavior based on the current SOBR design, so contacting support for deeper investigation is recommended to identify the actual reason for the observed behavior.

[Delo123]

We opend support cases many times but had never luck with the support to solve the problems or to place a feature request. At the moment, their is no real benefit of using SOBR. You still have to monitor the repository space and take manually action when the repository is running out of space. We though the introduction o SOBR would solve all of these problems but it was a fake

There actually are a lot of good reasons to use SOBR, but not this one...

[kkuszek]

That's exactly the use case SOBR should fill most and what I think the general expectation is for it to perform. If that's not a good reason, it should be.

[fraack]

It sounds less like it's broken as in support and more like it's just broken as designed.)

I can not agree with your statement. I openend a ticket to get this Reg Key today (Case # 02077476). I got a response within 5 minutes with the desired key! So thumbs up to Veeam Support.

I hope it is ok that I share the Reg Key for repository limits in this post.

There is the registry key:

Code: Select all

SobrReserveExtentSpacePercent
- Type: REG_DWORD
- Default value: 1 (percent)
- Description: defines the amount of free space reserved on the extents of SOBR, e.g. for merge (% of total extent size).

This solved my problem

[kkuszek]

Oh I am sorry if you misinterpreted my comment as there being support issues. I was saying the contrary. Support is great and responsive, but the other posts indicating discussion with support and detail on how it functioned made me believe it wasn't a support incident as something was broken but more a design limitation.
See the multiple users on page 1 with independent examples of space limitations and SOBR allowing backups to fail with free space errors.

That registry key looks like it only controls a specific sobr, but not regular repositories?
Where in the registry is that? I found nothing in a search.

[kkuszek]

posting.php?mode=quote&f=2&p=188883

data locality policy is not that strict and should place the backup on another extent if it has available space. There's an issue, however, where if there's not enough space to even update metadata file, the job will fail and chances are you're experiencing right this one. I recommend re-opening the case and escalating it to a higher tier for a closer investigation.

In this instance if the metadata is stored on the primary repository, and there is no free space percent registry key for the primary repository, then when the backups completely fill the primary repository having no space available for metadata it is guaranteed to fail.

Would that be correct?

[fraack]

You have to add this value into the Reg Key: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\
If this value is not there you have to create it by your own. If it is done you have to restart your VBR service.

It describes the free space reserved on each SOBR extent in percent.

[Delo123]

Splitting VBK's in chunks, yes!

[Gostev]

@Guido funny you should mention - the feature I thought we would [realistically] never get to any time soon, however recently it came to my attention that we've started building this functionality into the data mover for another product to use in a somewhat isolated manner, so the chances to see it in B&R have vastly improved now although this won't be a simple "drop and go", as in case of B&R it would require the complete retest of virtually all product functionality - which is a truly massive undertaking considering the amount of dependencies.

Most people want it for Windows Server dedupe though, not for SOBR.

[rnelson0]

I'd like to add my support for SOBR getting a little smarter. I had a mono-chain (the repo was updated to do per-VMs but of course when there's an existing chain it kept using it) and the drive filled up too fast to catch things in time. With the job filled all the way, I attempted to use a SOBR with a second LUN twice the size of the original. Should be plenty of room! Except it failed saying that there was no more room where the current full was. In the end, I copied the job and start a new per-vm-chain, since that was also an issue, but SOBR didn't "just work" the way I expected based on the description, regardless.

I do not know what the real fix should be, but I can say that I expected Veeam to maybe start a new chain on the new repo since the other was full, rather than just throwing its hands up in the air for every VM. While it would have used up more disk space until retention cleaned up the old fulls, at least it would have gracefully recovered from a full disk and required less human intervention.

[Delo123]

so the chances to see it in B&R have vastly improved now

Anton! Somehow i missed your post, but wow! I understand it isn't an easy task but as you say it could have quite some advantages, very cool your even looking at that Cheers!