File Maintenance on ReFS

Availability for the Always-On Enterprise

File Maintenance on ReFS

Veeam Logoby DaveWatkins » Fri Mar 10, 2017 2:27 am

I've been watching a backup to tape job run slowly today and I'm wondering if it's because of fragmentation from using Fast Clone on the ReFS repository on a reverse incremental chain.

I'm assuming it is, which in itself isn't a huge problem, but that lead me to wonder what, if anything, the file maintenance operation does on a ReFS volume. On an NTFS repository it creates a new copy of the VBK in a mostly single lump removing fragmentation.

However does it do the same on a ReFS repo, or does it also just use the fast clone API and build a new copy of the file without actually moving any data?

Thanks
DaveWatkins
Expert
 
Posts: 269
Liked: 67 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: File Maintenance on ReFS

Veeam Logoby Gostev » Fri Mar 10, 2017 1:52 pm

Correct, you are not alone - we had a few similar support cases created by other reversed incremental backup mode users already. Reverse incremental is the absolute worst backup mode in terms of both fragmentation and I/O load, I would not use it with ReFS myself - there are no benefits.

Maintenance operation essentially only does "compact" operation which releases the no longer used backup file blocks to the file system, so it still makes great sense to do. As far as defragmentation, it makes no sense on ReFS volumes. Here's my explanation from another topic:

You should treat ReFS volume where block cloning was used as a sort of deduplication storage. You can never have it NOT "heavily fragmented" due to the nature of the process (different files sharing the same physical block). As such, there's absolutely no point to perform defragmentation - just like there's no point in defragmenting a deduplicating storage. In fact, the only way to actually defragment any given file would require "inflating" one by recreating the file without leveraging block cloning.
Gostev
Veeam Software
 
Posts: 21503
Liked: 2379 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: File Maintenance on ReFS

Veeam Logoby DaveWatkins » Fri Mar 10, 2017 8:51 pm

Hi Anton

We originally went reverse incremental because backing up a forward chain to tape was taking too long (for much the same reason as we're now facing). I have no issues switching back to a forward incremental chain but with ReFS I'm not sure any mode will keep the fragmentation under control other than doing regular active full backups, and we really don't have the bandwidth for that. Synthetic Full's won't actually help the fragmentation, although I guess they would held from a data management point of view as it wouldn't have to go through the whole chain working out what to put to tape, but I'm not sure that's any better than our current reverse incremental from a performance to tape perspective.

Virtual full's used to be the solution but they were I/O intensive and again will suffer the fragmentation penalty (and are they really different from a Synthetic Full at this point??).

Any suggestions?

Thanks
Dave
DaveWatkins
Expert
 
Posts: 269
Liked: 67 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: File Maintenance on ReFS

Veeam Logoby tsightler » Fri Mar 10, 2017 10:13 pm

Out of curiosity, how slow is this tape backup? Certainly using ReFS will lead to fragmentation, however, we are cloning Veeam blocks, which are relatively large overall. Assuming you are using default settings a typical Veeam block is around 350-500KB. I would think almost any reasonable file system could support enough I/O to keep a typical LTO drive busy at this block size unless perhaps there's something non-optimal about the underlying block device layout.
tsightler
Veeam Software
 
Posts: 4834
Liked: 1784 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: File Maintenance on ReFS

Veeam Logoby DaveWatkins » Fri Mar 10, 2017 10:59 pm

It sat at 20-30MB/s for the better part of 3 hours, then picked up to normal LTO6 speed. Underlying block sizes and stripe sizes are all good. It's a 20 disk RAID6 volume with NL-SATA disks attached via FC. So probably not the hardware :)
DaveWatkins
Expert
 
Posts: 269
Liked: 67 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: File Maintenance on ReFS

Veeam Logoby tsightler » Fri Mar 10, 2017 11:15 pm

By any chance did you grab the queue depth on the source repo while while you were doing this. I'm guessing we're just not keeping enough request in queue to properly keep the disk busy, but that is truly just a guess at this point. 20-30MB would imply single queue depth with about 10ms latency per read so it would be interesting to see this data during at tape job. I'm assuming the tape job shows Source as the bottleneck? What percentage?
tsightler
Veeam Software
 
Posts: 4834
Liked: 1784 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: File Maintenance on ReFS

Veeam Logoby DaveWatkins » Sat Mar 11, 2017 12:25 am

I'm not sure the source statistics would be useful as it's was a part of a single VM (a 2TB one) that was the only slow one to get to tape, the rest of the job (maybe another 2TB) was all fine, per VM chains to a TL-4000 library with two LTO6 drives. Jobs stats are 29/13/14/97 and was running slow for maybe 4 hours out of the total run time of 10 hours, the rest of the time it was running at normal speed. It also wasn't running slow for the entire time it was backing up that VM

I didn't get any volume stats unfortunately but did check to make sure nothing else was happening on the volume. There was a health check running on a different job but that backup chain is on a different volume
DaveWatkins
Expert
 
Posts: 269
Liked: 67 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: File Maintenance on ReFS

Veeam Logoby mkretzer » Sat Mar 11, 2017 8:26 am

We also have the issue that tape backup sometimes goes down to 50-60 MB/s for quite some time from the typical 160 MB/s - and still shows the target as bottleneck. We had this on NTFS, backed by 96 disks and on REFS with 24 disks as well.

Is there a good way to troubleshoot that?
mkretzer
Expert
 
Posts: 337
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: File Maintenance on ReFS

Veeam Logoby masonit » Mon Mar 13, 2017 7:38 am

Gostev wrote:Correct, you are not alone - we had a few similar support cases created by other reversed incremental backup mode users already. Reverse incremental is the absolute worst backup mode in terms of both fragmentation and I/O load, I would not use it with ReFS myself - there are no benefits..

Hi Gostev

What backup mode would you recommend with refs?

\Masonit
masonit
Service Provider
 
Posts: 141
Liked: 10 times
Joined: Tue Oct 09, 2012 2:30 pm
Full Name: Magnus Andersson

Re: File Maintenance on ReFS

Veeam Logoby ChrisGundryCEGA » Mon Mar 13, 2017 9:06 am

Also interested in the reply
ChrisGundryCEGA
Enthusiast
 
Posts: 53
Liked: 1 time
Joined: Wed Aug 26, 2015 2:56 pm
Full Name: Chris Gundry

Re: File Maintenance on ReFS

Veeam Logoby m.novelli » Mon Mar 13, 2017 9:41 am

Gostev wrote:Correct, you are not alone - we had a few similar support cases created by other reversed incremental backup mode users already. Reverse incremental is the absolute worst backup mode in terms of both fragmentation and I/O load, I would not use it with ReFS myself - there are no benefits.

Hi Gostev, I agree with you but I love the Reverse Incremental mode because it allow me to export every night a single backup file containing all VM to RDX drive, and I've standardised most of my customer with that config

Cheers,

Marco
m.novelli
Veeam ProPartner
 
Posts: 303
Liked: 36 times
Joined: Tue Dec 29, 2009 12:48 pm
Location: Asti - Italy
Full Name: Marco Novelli

Re: File Maintenance on ReFS

Veeam Logoby Gostev » Mon Mar 13, 2017 2:15 pm

Yes, certainly there are use cases for the reversed incremental mode - we've got our hands slapped quickly by sales engineers when we started discussing potentially removing one from the product to reduce the number of test scenarios ;)

@Magnus @Chris regular (forward) incremental backup, which is the default.
Gostev
Veeam Software
 
Posts: 21503
Liked: 2379 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: File Maintenance on ReFS

Veeam Logoby ITP-Stan » Tue Mar 14, 2017 8:58 am

Just a thought, but with ReFS you could actually do a synthetic full every night without much space loss because of the block cloning right?
ITP-Stan
Veeam ProPartner
 
Posts: 73
Liked: 7 times
Joined: Mon Feb 18, 2013 10:45 am
Full Name: Stan (IF-IT4U)

Re: File Maintenance on ReFS

Veeam Logoby v.Eremin » Tue Mar 14, 2017 9:35 am

Correct, but what's the idea behind such scenario? To have full backup each day and not to stress production environment in the meantime?
v.Eremin
Veeam Software
 
Posts: 13534
Liked: 1002 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: File Maintenance on ReFS

Veeam Logoby Gostev » Tue Mar 14, 2017 5:05 pm

Exactly.
Gostev
Veeam Software
 
Posts: 21503
Liked: 2379 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 38 guests