Scale Out Repository out of space issue

Availability for the Always-On Enterprise

Scale Out Repository out of space issue

Veeam Logoby sdeath » Thu Apr 07, 2016 3:17 pm

Hi,

I am a bit confused about what I am reading and what I am being told by support.

- I have a Scale Out repository with 2 extents.
- Extent 1 capacity is 14.4TB and extent 2 is 7TB.
- I have created one Backup Copy job, containing 111 VMs and pointed it to the Scale Out repository.
- The Scale Out repository is configured as Data Locality and Per-VM backup files.

Initially the job appeared distribute the backup files evenly across the 2 extents but the 7TB extent has now run out of space and the Backup Copy job is failing.

Veeam support is telling me that is as designed, using the Data Locality policy will write the increment jobs to the same extent as the full backup file until it runs out of space and will then fail the job. I asked what would happen if I added a new extent and was told that only new VMs to the job would be written to that extent. I asked what is the benefit of Scale Out then and I received a very long pause! I believed that Veeam would handle the space issue, am I wrong?

If this is the case then Scale Out is actually wasting disk space as the other extent still had over 3TB free. Apparently there is no way to back out of a Scale Out repository so I will have to delete the data, create 2 Backup Copy jobs and point them to separate repositories. Not too happy about that.

Can anyone advise if this is correct?

Thanks.
sdeath
Influencer
 
Posts: 20
Liked: 2 times
Joined: Thu Feb 11, 2010 12:54 pm
Full Name: Stuart Death

Re: Scale Out Repository out of space issue

Veeam Logoby DaveWatkins » Thu Apr 07, 2016 9:58 pm

I've seen this manifested in a number of situations myself, so much so I don't use Scale Out anymore at all. Initially it sounded great, but it's poor detection of placement and then failing jobs if an extent gets full I've given up.

The best (and by best I mean worst) situation I saw was a scale out with a 2TB and a 7TB and an exchange backup of a server that was over 3TB and it failed because it put the backup on the 2TB extent and filled it before completing the job. Scale out seems to be missing a LOT of logic to determine the best place to put jobs, and what do do if an extent fills up.

Based on all the marketing, it was presented as a simple way to aggregate all your spare space to make a large, usable backup repo, but it's missing so much logic to catch simple and expected scenarios I've found it completely unusable so far, which is a shame because I love the idea of it, it just needs to make smarter decisions about job placement and what happens when the policy can't be fulfilled. It also doesn't seem to automatically move existing per VM chains off an extent to free up space on it so other VM chains on that extent can continue to backup, which is again, something i expected it to do.
DaveWatkins
Expert
 
Posts: 252
Liked: 61 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: Scale Out Repository out of space issue

Veeam Logoby tsightler » Fri Apr 08, 2016 3:23 pm

Can you please share your support case?
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Scale Out Repository out of space issue

Veeam Logoby sdeath » Fri Apr 08, 2016 3:27 pm

Case # 01754266

But support closed case as no way forward!

Thanks
sdeath
Influencer
 
Posts: 20
Liked: 2 times
Joined: Thu Feb 11, 2010 12:54 pm
Full Name: Stuart Death

Re: Scale Out Repository out of space issue

Veeam Logoby foggy » Fri Apr 08, 2016 3:32 pm

sdeath wrote:Veeam support is telling me that is as designed, using the Data Locality policy will write the increment jobs to the same extent as the full backup file until it runs out of space and will then fail the job.

This is not actually correct, data locality policy is not that strict and should place the backup on another extent if it has available space. There's an issue, however, where if there's not enough space to even update metadata file, the job will fail and chances are you're experiencing right this one. I recommend re-opening the case and escalating it to a higher tier for a closer investigation.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Scale Out Repository out of space issue

Veeam Logoby sdeath » Fri Apr 08, 2016 3:35 pm

Thanks foggy, I have to say that makes more sense. I will re-open and escalate.
sdeath
Influencer
 
Posts: 20
Liked: 2 times
Joined: Thu Feb 11, 2010 12:54 pm
Full Name: Stuart Death

Re: Scale Out Repository out of space issue

Veeam Logoby tsightler » Fri Apr 08, 2016 4:43 pm

Exactly why I was asking for the support case. Can you please update this post with the new case # once it is opened? I know there are several issues with data placement that were fixed in U1, but I would not expect a case of "job fails", although of course there can always be cases where en extent is completely exhausted of space, but we should break policy to do backups to other extents in most of those cases. I just want to make sure we fully understand all of the issues and failure cases.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Scale Out Repository out of space issue

Veeam Logoby foggy » Fri Apr 08, 2016 5:14 pm

DaveWatkins wrote:It also doesn't seem to automatically move existing per VM chains off an extent to free up space on it so other VM chains on that extent can continue to backup, which is again, something i expected it to do.

Dave, could you please elaborate on this scenario?
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Scale Out Repository out of space issue

Veeam Logoby DaveWatkins » Mon Apr 11, 2016 3:03 am

I've you have data locality set as the policy, and an extent is filled, my expectation would be that an entire backup chain would be moved to another extent to free up space on the full one so further incremental data can be placed on that extent for the remaining chains on that extent and so still conform to policy. That didn't seem to happen when I saw it
DaveWatkins
Expert
 
Posts: 252
Liked: 61 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: Scale Out Repository out of space issue

Veeam Logoby sdeath » Mon Apr 11, 2016 9:23 am

The case number is the same, seems it hadn't closed completely. It is now awaiting an escalation response.

By the way, I agree with Dave. My expectation was that an existing VM chain would be moved off in order to free up space.

Just to confirm, we are on Update 1.
sdeath
Influencer
 
Posts: 20
Liked: 2 times
Joined: Thu Feb 11, 2010 12:54 pm
Full Name: Stuart Death

Re: Scale Out Repository out of space issue

Veeam Logoby Marten_med_e » Mon Apr 11, 2016 10:41 am

Our copy backup jobs also stalled when space was exhausted on one disk in the scale-out repository, over 4TB free on the other three disks in the scale-out repository, is running v. 9.0.0.1491.
Code: Select all
2016-04-07 09:28:58 :: Error: There is not enough space on the disk.
Failed to write data into file [E:\Backups\definition.erm].
--tr:Error code: 0x00000070
--tr:Failed to call DoRpc. CmdName: [FcWriteFileEx] inParam: [<InputArguments><FilePath value="E:\Backups\definition.erm" /><DesiredAccess value="1073741824" /><ShareMode value="0" /><CreationDisposition value="3" /><FlagsAndAttrs value="0" /><Offset value="0" /><BytesToWrite value="5400" /></InputArguments>].
There is not enough space on the disk.
Failed to write data into

I guess that the disk got so full that the copy backup job couldn't update necessary files to continue with/move the backup to one of the disks with free space on.

Case #01759224, just opened due to excessive time-out issues when uploading log files.

Cheers
Marten_med_e
Enthusiast
 
Posts: 46
Liked: 4 times
Joined: Thu Sep 26, 2013 9:31 am
Full Name: Mårten Edelbrink

Re: Scale Out Repository out of space issue

Veeam Logoby foggy » Mon Apr 11, 2016 11:22 am

Mårten, this looks like exactly the case I was talking about. Thanks for contacting support and sharing the case ID.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Scale Out Repository out of space issue

Veeam Logoby foggy » Mon Apr 11, 2016 11:33 am

Dave, thanks for clarifying the use case, got it now.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Scale Out Repository out of space issue

Veeam Logoby Marten_med_e » Tue Apr 12, 2016 2:23 pm

foggy wrote:Mårten, this looks like exactly the case I was talking about. Thanks for contacting support and sharing the case ID.

If this is the case, I would suggest that B&R creates a "empty" *.vbm/def file that could be used for writing in case the disk gets full,if I don't remember wrong I think MS does/did this with Exchange transaction log file, haven't been messing around with Exchange for a wile so my memory can be way off. Sorry if it is.

Cheers,
Marten_med_e
Enthusiast
 
Posts: 46
Liked: 4 times
Joined: Thu Sep 26, 2013 9:31 am
Full Name: Mårten Edelbrink

Re: Scale Out Repository out of space issue

Veeam Logoby foggy » Tue Apr 12, 2016 2:25 pm

We are looking for possible ways of addressing this behavior.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot] and 23 guests