Slow file level restores from dedupe appliance

Availability for the Always-On Enterprise

Slow file level restores from dedupe appliance

Veeam Logoby cookiejc » Wed Nov 30, 2016 11:15 am

Hi, we have some large file servers which have Veeam backups stored on a dedupe appliance (NetApp Altavault). Mounting the backup for file level restores takes between 1-2 hours. Veeam support have previously advised that this is due to the time it takes to scan the entire backup file set. I noticed the following in the Svc.VeeamMount.log:

Source.DisableFullBlockRead = false

I was hoping this implied there's a way to set this value to "true" and stop the full block read prior to performing a restore?
cookiejc
Enthusiast
 
Posts: 51
Liked: never
Joined: Wed Nov 06, 2013 3:15 pm
Full Name: J Cook

Re: Slow file level restores from dedupe appliance

Veeam Logoby PTide » Wed Nov 30, 2016 2:02 pm

Hi,

Please provide some details about your setup. What are the job settings, is encryption enabled? Do you have "Decompress backup data blocks before storing" check box selected in the repository settings?

Thanks
PTide
Veeam Software
 
Posts: 3140
Liked: 262 times
Joined: Tue May 19, 2015 1:46 pm

Re: Slow file level restores from dedupe appliance

Veeam Logoby csinetops » Wed Nov 30, 2016 2:13 pm

That is odd. Are the backup sets sitting in local cache or are they out in the cloud and need to pull back down? I also have a AltaVault and just tested my biggest VM on there @ 3.8TB. A cached backup doing Windows FLR came up in under 5 minutes. I have dedupe and compression turned off and let the AltaVault handle that.
csinetops
Expert
 
Posts: 111
Liked: 15 times
Joined: Fri Jun 06, 2014 2:45 pm
Full Name: csinetops

Re: Slow file level restores from dedupe appliance

Veeam Logoby foggy » Wed Nov 30, 2016 2:16 pm

Elaborating on this, you could try to enable partial block reading (which is basically a default setting) via registry, provided backups are not encrypted and compression is not enabled.
foggy
Veeam Software
 
Posts: 15087
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow file level restores from dedupe appliance

Veeam Logoby cookiejc » Wed Nov 30, 2016 3:45 pm

Hi, thanks for your replies.

Compression, dedupe and encryption is turned off in the jobs as the Altavault does that. The "decompress backup data blocks before storing" setting is unticked in the repositories (as per the NetApp best practice they gave us)

These backups are still stored in cache as we haven't had to evict any data yet

These file servers are about 20TB each which doesn't help

foggy can you point me towards that registry setting?

thanks again
cookiejc
Enthusiast
 
Posts: 51
Liked: never
Joined: Wed Nov 06, 2013 3:15 pm
Full Name: J Cook

Re: Slow file level restores from dedupe appliance

Veeam Logoby foggy » Wed Nov 30, 2016 4:02 pm

Try setting the DisableFullBlockRead value to 3 on the Veeam B&R server and restart the Veeam Backup Service.
foggy
Veeam Software
 
Posts: 15087
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow file level restores from dedupe appliance

Veeam Logoby cookiejc » Wed Nov 30, 2016 4:06 pm

Foggy, can you advise where that registry value is, or needs to be created?
cookiejc
Enthusiast
 
Posts: 51
Liked: never
Joined: Wed Nov 06, 2013 3:15 pm
Full Name: J Cook

Re: Slow file level restores from dedupe appliance

Veeam Logoby foggy » Wed Nov 30, 2016 4:09 pm

The standard Veeam B&R hive is HKLM\SOFTWARE\Veeam\Veeam Backup and Replication. And the value type is DWORD.
foggy
Veeam Software
 
Posts: 15087
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow file level restores from dedupe appliance

Veeam Logoby cookiejc » Wed Nov 30, 2016 4:10 pm

I have looked in there and that key does not exist... does it need to be created to enforce this setting?
cookiejc
Enthusiast
 
Posts: 51
Liked: never
Joined: Wed Nov 06, 2013 3:15 pm
Full Name: J Cook

Re: Slow file level restores from dedupe appliance

Veeam Logoby foggy » Wed Nov 30, 2016 4:11 pm

Correct, you need to create it.
foggy
Veeam Software
 
Posts: 15087
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow file level restores from dedupe appliance

Veeam Logoby cookiejc » Thu Dec 01, 2016 2:23 pm

Thanks I've now implemented this change and can see it has taken effect in the log but so far it looks like mount times are still slow. The log has got to this point and just stops... I expect it to take about an hour from this point:

Code: Select all
Source.DisableFullBlockRead = true\n  (EString) Processing.Part[0].Type = DynGroup\n  (EString) Processing.Part[0].GptDiskObjectPath = DynGroup-b2528697-0783-4072-8efd-95674944f8fb\n  (EString) Processing.Part[1].Type = OracleAsm\n  (EString) Processing.Part[2].Type = DiskIds\n}\n'


Any more ideas?
cookiejc
Enthusiast
 
Posts: 51
Liked: never
Joined: Wed Nov 06, 2013 3:15 pm
Full Name: J Cook

Re: Slow file level restores from dedupe appliance

Veeam Logoby foggy » Thu Dec 01, 2016 3:11 pm

Actually partial block read doesn't affect mount process, it is effective during actual restore. While mount itself requires random reads from guest OS MFT and expected to be slow for large volumes with large amount of files and fragmented MFT, especially on a dedupe device.
foggy
Veeam Software
 
Posts: 15087
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow file level restores from dedupe appliance

Veeam Logoby rreed » Thu Dec 01, 2016 4:19 pm

What is your block size (job settings, Advanced, Storage, Storage Optimization)? In v8 we found cranking that up to Local Target 16+TB (largest block size) made a massive improvement on restores from a dedupe device. Compression off but inline dedupe on (you're only moving around the dedupe but the end device still finds some to do). I think they've made some improvements in v9 but we never went back to turn that down to confirm - it's always been working so we continue to leave well enough alone for now.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Slow file level restores from dedupe appliance

Veeam Logoby foggy » Thu Dec 01, 2016 4:27 pm

The block size doesn't really matter if blocks can be read partially.
foggy
Veeam Software
 
Posts: 15087
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 26 guests