Scale Out Repository out of space issue

Availability for the Always-On Enterprise

Re: Scale Out Repository out of space issue

Veeam Logoby Marten_med_e » Tue Apr 12, 2016 3:00 pm

I did free up some space on the disk, but B&R just kept writing to the vib/vbk files and got out of space. I had to free up enough space to let the jobs finish and then add a new repository for setting it to incremental backups only to get the jobs to start working again.

Cheers,
Marten_med_e
Enthusiast
 
Posts: 46
Liked: 4 times
Joined: Thu Sep 26, 2013 9:31 am
Full Name: Mårten Edelbrink

Re: Scale Out Repository out of space issue

Veeam Logoby nunciate » Thu Apr 14, 2016 2:10 am

I have noticed this same issue. I have 5 extents in a scale out repository. One of the extends is somewhat small (about 3Tb). The others are large (18Tb). The smaller one filled up with zero free space and no jobs which utilized that drive would run. I removed some backups from disk and reran the active fulls. Active fulls were written to a new extent but jobs kept running incremental backups to the small drive and kept filling it up to zero free space. The only way I was able to fix it was to put the extent into maintenance mode and evacuate it completely. Fortunately I had the space to do this on the other extents otherwise i would have had a real problem.

There has to be some logic to determine drive is X% full and to stop backing up to that drive. Is there anything like that because if there is it isn't working.

One other big problem I have noticed is that the jobs don't always pick the best extent that matches the size of the VM. I have a large 10Tb VM I am trying to backup to a this scale out repository for the first time. I have 12Tb free on 1 extent, 7Tb free on another and less than 2 on the others. I run the job and it tries to put the backup on the extent with 7Tb free. I kill the job, put that extent into maintenance mode and reran. Now it tries to backup to an extent with less than 2Tb free all the while ignoring the one extent it can actually be successful at using with 12Tb free. Ridiculous. I literally had to put all extents into maintenance mode to force it to the last one and then re-enabled them all.

My scale out is set to performance mode. I do not have per-vm backups enabled on any of the extents as of yet but plan on doing that soon.

I have to agree with what was already said. The scale-out repository is a great idea but I am not sure it is ready for prime time. Not sure how these issues could not have been identified in beta and RTM.
nunciate
Expert
 
Posts: 139
Liked: 23 times
Joined: Tue May 21, 2013 9:08 pm
Full Name: Alan Wells

Re: Scale Out Repository out of space issue

Veeam Logoby foggy » Thu Apr 14, 2016 1:58 pm

nunciate wrote:There has to be some logic to determine drive is X% full and to stop backing up to that drive. Is there anything like that because if there is it isn't working.

There's an heuristic mechanism that estimates the backup size and doesn't allow to use the extent if there's not enough free space.

nunciate wrote:One other big problem I have noticed is that the jobs don't always pick the best extent that matches the size of the VM. I have a large 10Tb VM I am trying to backup to a this scale out repository for the first time. I have 12Tb free on 1 extent, 7Tb free on another and less than 2 on the others. I run the job and it tries to put the backup on the extent with 7Tb free. I kill the job, put that extent into maintenance mode and reran. Now it tries to backup to an extent with less than 2Tb free all the while ignoring the one extent it can actually be successful at using with 12Tb free. Ridiculous. I literally had to put all extents into maintenance mode to force it to the last one and then re-enabled them all.

Alan, do you have a case opened for this one? Any chance the 12TB one is specified for increments only (Performance policy) or does not have free repository slots?
foggy
Veeam Software
 
Posts: 15083
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Scale Out Repository out of space issue

Veeam Logoby sdeath » Thu Apr 14, 2016 1:59 pm

I have a response from tier 2 support.

"As of now the current SOBR design still requires extents to have some space for VBM files.
VBM is written to every extent where we have backup files related to the job.

However, when backup job has backup files only on one extent, and that extent doesn't have free space at all (not even couple of MB for VBM file) then job will fail and won't switch to another extent.
The workaround is to remove a couple of MB so VBM could be written.

When job has backups files on 2 extents, ext1 and ext2, you won't experience any issue: VBM file will be written to both ext1 and ext2, and if ext1 ran out of space completely, then we will fail to write VBM to ext1 but we will be able to write VBM to ext2 and in this scenario job won't fail - it will detect that it could write at least one copy of VBM.

We have already discussed this limitation with RND, and now RND is thinking about ways to improve our workflow/logic here, probably in one of the upcoming patches for v9, but as of now - unfortunately, this is current design, limitation."


Now, we have one Backup Copy job writing to two extents so just confirming if it should be working as suggested in the third paragraph....
sdeath
Influencer
 
Posts: 20
Liked: 2 times
Joined: Thu Feb 11, 2010 12:54 pm
Full Name: Stuart Death

Re: Scale Out Repository out of space issue

Veeam Logoby nunciate » Thu Apr 14, 2016 2:18 pm

My feature request would be to add the ability to set a limitation on the extents. Have an option in there that says don't write to this extent if it is X % full. That way people can set their own value and when the volume gets to 10% free space left for example it will automatically stop backing up to that extent. That might be much harder to implement so even if you set a static value that the job checks for would be something good.
nunciate
Expert
 
Posts: 139
Liked: 23 times
Joined: Tue May 21, 2013 9:08 pm
Full Name: Alan Wells

Re: Scale Out Repository out of space issue

Veeam Logoby foggy » Thu Apr 14, 2016 3:54 pm

sdeath wrote:When job has backups files on 2 extents, ext1 and ext2, you won't experience any issue: VBM file will be written to both ext1 and ext2, and if ext1 ran out of space completely, then we will fail to write VBM to ext1 but we will be able to write VBM to ext2 and in this scenario job won't fail - it will detect that it could write at least one copy of VBM.

Now, we have one Backup Copy job writing to two extents so just confirming if it should be working as suggested in the third paragraph....

That's correct, if VBM is stored on both extents, the job shouldn't fail in case there's not enough space to update it on of them.
foggy
Veeam Software
 
Posts: 15083
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Scale Out Repository out of space issue

Veeam Logoby sdeath » Fri Apr 15, 2016 8:43 am

Our backup copy job has created a VBM on both extents so we shouldn't be seeing this issue then?
sdeath
Influencer
 
Posts: 20
Liked: 2 times
Joined: Thu Feb 11, 2010 12:54 pm
Full Name: Stuart Death

Re: Scale Out Repository out of space issue

Veeam Logoby foggy » Fri Apr 15, 2016 10:26 am

In case Veeam B&R is able to update at least one copy of metadata, the job shouldn't fail.
foggy
Veeam Software
 
Posts: 15083
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Scale Out Repository out of space issue

Veeam Logoby sdeath » Fri Apr 15, 2016 10:42 am

Support have confirmed a bug and are compiling a cumulative hotfix.
sdeath
Influencer
 
Posts: 20
Liked: 2 times
Joined: Thu Feb 11, 2010 12:54 pm
Full Name: Stuart Death

Re: Scale Out Repository out of space issue

Veeam Logoby A.J. » Thu Dec 08, 2016 10:26 am

We have a similar problem with our Scale Out Repositories when using Windows Dedup on that Repositories - especially on Backup Copy Jobs. (SOBR with per VM backup files and data locally policy)
Our latest investigations showed that the copy jobs will fill up a repository extent until 0 bytes free. The next incremental file will be placed on another extent but the merge process will fail because the primary extent of the job has 0 bytes free.
The only explanation that we currently found is that the dedup algorithm causes a "pumping" on the extent. That means that merging processes will expand the deduped files in such a way that veeam can´t recognize it or take it into consideration while placing backup files to the extents. Maybe veeam thinks that there must be enough space on the extent but there isnt because of the dedup behaviour. Maybe this happens while the jobs on the extents are running and the space isnt checked just in time during that job processing. Who knows...
As mention in the first postings there should be a possibility for the veeam Admin to control a spare space on each extent to prevent this out of space situation. And also there should be an automated process that evacuate the full backup file to an extent with enough space to keep the jobs running an the backup chain intact.
For us at the moment the only solution seems to be to disable deduplication on all repositories and clean up the whole deduped files over the time.
We are also planning to upgrade our repositories to windows 2016 with ReFS (dedup is not supported) to reduce the time for merge processes. We hope that this will solve our problems in the near future, knowing that we will need more extents on our SORBs.
A.J.
Service Provider
 
Posts: 3
Liked: 1 time
Joined: Tue Jul 26, 2016 6:19 am

Re: Scale Out Repository out of space issue

Veeam Logoby Gostev » Fri Dec 09, 2016 1:26 am

A.J. wrote:Our latest investigations showed that the copy jobs will fill up a repository extent until 0 bytes free. The next incremental file will be placed on another extent but the merge process will fail because the primary extent of the job has 0 bytes free.

Hi, I recommend you upgrade to 9.5 since it has additional new logic to prevent this from happening. Thanks!
Gostev
Veeam Software
 
Posts: 21504
Liked: 2380 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Scale Out Repository out of space issue

Veeam Logoby A.J. » Mon Dec 12, 2016 7:39 am

Hi, we are already on 9.5 because of this problem. No recognizable change in logic but that could be because of the dedup effect.
I read in another forum entry, that veeam in v9.5 reserves 1% of storage space to prevent that the the repository runs out of space. We will see if the problem occurs on our repository that had never deduped enabled.
A.J.
Service Provider
 
Posts: 3
Liked: 1 time
Joined: Tue Jul 26, 2016 6:19 am

Re: Scale Out Repository out of space issue

Veeam Logoby jeff.james@ttu.edu » Fri Aug 18, 2017 6:35 am

Veeam Support - Case # 02282755 I can confirm whatever was done to resolve this issue in Ver. 9.5 did not work. I still have all of the previous posted issues with SOBR..
Jeff M. James
Data Center Operations
Technology Operations & Systems Management
Texas Tech University System
Office: (806) 834-4407 | Mobile: (806) 535-4228
jeff.james@ttu.edu | http://www.texastech.edu/it/tosm/
jeff.james@ttu.edu
Lurker
 
Posts: 2
Liked: never
Joined: Tue Jan 13, 2015 4:31 am
Location: 8th Street & Boston Avenue Box 43051 Lubbock, TX 79409-3051
Full Name: Jeffrey Michael James

Previous

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: No registered users and 32 guests