Very slow file level restore and problem with permissions

Availability for the Always-On Enterprise

Re: Very slow file level restore and problem with permission

Veeam Logoby Vitaliy S. » Tue Nov 06, 2012 8:19 am

Yes, as far as I remember you need to have VMware Tools installed on the production VMs to determine their IP address for all operations performed by Veeam backup server.
Vitaliy S.
Veeam Software
 
Posts: 19707
Liked: 1117 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Very slow file level restore and problem with permission

Veeam Logoby masonit » Wed Nov 07, 2012 9:17 am

We run some tests with a firewall and NAT between Veeam backup server and the vm:s network. We have opened the following ports: udp 135,445
, udp 137:139, udp 1024: 137, tcp 135,139,445. I can from both Veeam backup server and proxy server connect to the target vm with unc path (\\ip-address\c$). I can send and recevice files so everthing works good in Windows. But when trying to restore files from Veeam to the target vm it does not use the network. What are we missing?

\Masonit
masonit
Service Provider
 
Posts: 141
Liked: 10 times
Joined: Tue Oct 09, 2012 2:30 pm
Full Name: Magnus Andersson

Re: Very slow file level restore and problem with permission

Veeam Logoby foggy » Wed Nov 07, 2012 9:28 am

Magnus, please review the Used Ports section of the product user guide for the full list of ports required for Veeam B&R operation.
foggy
Veeam Software
 
Posts: 14904
Liked: 1096 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Very slow file level restore and problem with permission

Veeam Logoby masonit » Wed Nov 07, 2012 11:01 am

foggy wrote:Magnus, please review the Used Ports section of the product user guide for the full list of ports required for Veeam B&R operation.


Thanks but that didn't really help. When we also opened port 50000 and up it worked. Haven't really narrowed it down yet.

\Masonit
masonit
Service Provider
 
Posts: 141
Liked: 10 times
Joined: Tue Oct 09, 2012 2:30 pm
Full Name: Magnus Andersson

Re: Very slow file level restore and problem with permission

Veeam Logoby foggy » Wed Nov 07, 2012 11:16 am

Please see similar discussion here.
foggy
Veeam Software
 
Posts: 14904
Liked: 1096 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Very slow file level restore and problem with permission

Veeam Logoby kaskm » Fri Jan 18, 2013 8:19 pm

Were you able to get this resolved? We get about 3-12k/sec on file restores with windows and linux VMs. We have a similar setup with 10GB iSCSI backup repo and all servers in VM. Backup speed is fine, it is only the restores.
kaskm
Novice
 
Posts: 3
Liked: 1 time
Joined: Fri Jan 18, 2013 8:16 pm

Re: Very slow file level restore and problem with permission

Veeam Logoby Vitaliy S. » Sun Jan 20, 2013 12:48 pm

Can you please tell me where you're trying to restore VM guest files to? Are you using Other OS restore wizard or Windows one?
Vitaliy S.
Veeam Software
 
Posts: 19707
Liked: 1117 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Very slow file level restore and problem with permission

Veeam Logoby kaskm » Mon Jan 21, 2013 3:31 pm 1 person likes this post

We have slow restores no matter what. I have done restores to the backup server itself and back to the original location. This is with the Windows wizard and with the Other OS. Once it runs for a whil (like 5+ minutes, it does increase from 3k to about 70k/sec).

I want to specify, this is only if we restore multiple files or entire directories. If it is a single file, the restore is quick.
kaskm
Novice
 
Posts: 3
Liked: 1 time
Joined: Fri Jan 18, 2013 8:16 pm

Re: Very slow file level restore and problem with permission

Veeam Logoby Vitaliy S. » Mon Jan 21, 2013 4:20 pm

Does this happen for all VMs on this repository? Can you please copy any small backup file (if you have it of course) to any other place and try to restore any file once again?
Vitaliy S.
Veeam Software
 
Posts: 19707
Liked: 1117 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Very slow file level restore and problem with permission

Veeam Logoby Fiskepudding » Wed Apr 10, 2013 10:17 am

Was this ever resolved?
If not I would also like to add that we have the same issue.

Restoring Guest files (windows) is very slow!

It starts out restoring at 220 KB/sec, but drops very fast, after 11 min the transfer rate reports 21 KB/sec.
If i cancel that job. And start another one emideatly, it reports 8 KB/sec from the start.
Witch probably was the reeal speed on the previous job when i stoped it.
(since 21KB/s was spread over the 11 min)
To mee it seem like som chace is filling up.

As kaskm says, this only happens with multiple files/folders. The more files the worse it gets.
The files size itsef does not seem to matter.


Restoing entire VMs is FAST. We also have 10G all the way. Our target host uses SSD datstore.
Preserving or not presering permission when copying seems not to make any difference.
Tested on a Inetpub folder containg 1,68 GB of data (59137 files, 6324 folders)
And Inetpub -> ziped (1,03 GB)
The IIS service was offcuase stopped during this restore test.
But anyway since the Inetpub folder still existed, the restore created its own folder"RESTORED-Inetpub_20130410_102051", so files in use should not be an issue.

Test results:


1 file "Copy to..."
- is fast (142 MB/S)

1 file "Restore"
- is fast enough (51MB/S)

Muliple files "Copy to.."
- Starts out at: 1MB/s
- After 5 min: 3MB/s
- After 9 min: 3MB/s


Multiple files "Restore" is is painfully slow
- Starts out at: 220 KB/sec
- After 5 min: 47KB/sec (Time remaining 9h 40m, transfered 14,0 MB )
- After 9 min: 25 KB/sec (Time remaining 16h 30m transfered 14,6 MB )
- After 60 min: 13 KB/sec (Time remaining 10h 11m transfered 45,8 MB )
*I guess there would be days left, but that is not visible in "Time remaining".
Fiskepudding
Expert
 
Posts: 213
Liked: 26 times
Joined: Wed Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen

Re: Very slow file level restore and problem with permission

Veeam Logoby Vitaliy S. » Wed Apr 10, 2013 4:49 pm

Espen, can you please tell me the restore job performance if you're using Other OS FLR wizard for the same files?
Vitaliy S.
Veeam Software
 
Posts: 19707
Liked: 1117 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Very slow file level restore and problem with permission

Veeam Logoby Fiskepudding » Thu Apr 11, 2013 7:40 am

Vitaliy, sure.

Restore is not available when choosing "Restore guest files (Other OS).."
Only "Copy to... " is available:

1 file "Restore" NA
Multiple files "Restore" NA

1 file "Copy to..."
- is fast enough (60 MB/S)

Muliple files "Copy to.."
- "Preparing information" lasted about 10 minutes, then the process started.
- after 22 minutes the transfer rate is 14 KB/S and it stops with and error "Failed to copy" (IMG1)
- I hit OK. and it contiues
- Then stops again similar message.(IMG2)
- I hit OK. and it contiues
- After 60 minutes, the transfer rate is 24 KB/S



IMG1
Image

IMG2
Image

Seems like the errors are related to Norwegian characters.
During the copy to, it sopped several times, only on files containing Norwegian characters.
Did not seem to be related to the path depth or file name length.
This did not happen during "Restore guest files (Windows).." witch was used in my post above.

So the copy to is just as slow when using this method, it also introduces a problem where no files or folders with Norwegian characters are copied.
Fiskepudding
Expert
 
Posts: 213
Liked: 26 times
Joined: Wed Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen

Re: Very slow file level restore and problem with permission

Veeam Logoby Vitaliy S. » Fri Apr 12, 2013 10:05 am

Fiskepudding wrote:So the copy to is just as slow when using this method, it also introduces a problem where no files or folders with Norwegian characters are copied.

Another way to rest restores of multiple files might be an instant VM recovery option and Windows Explorer, though I believe the performance will be the same, as data will be retrieved from the compressed and deduped file directly.
Vitaliy S.
Veeam Software
 
Posts: 19707
Liked: 1117 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Very slow file level restore and problem with permission

Veeam Logoby audreymann » Fri Apr 12, 2013 12:52 pm

Thanks for sharing the knowledge guys...had similar query!! solved now. :)
audreymann
Lurker
 
Posts: 2
Liked: never
Joined: Sat Apr 06, 2013 8:15 am
Full Name: Audrey Mann

Re: Very slow file level restore and problem with permission

Veeam Logoby masonit » Mon Apr 15, 2013 6:45 am

kaskm wrote:Were you able to get this resolved? We get about 3-12k/sec on file restores with windows and linux VMs. We have a similar setup with 10GB iSCSI backup repo and all servers in VM. Backup speed is fine, it is only the restores.


Not really. Restore through Vmware API injection is still slow. But we have made some changes in our setup. Now all restores send their traffic over lan. Then we get 15-20 MB/s.

\Masonit
masonit
Service Provider
 
Posts: 141
Liked: 10 times
Joined: Tue Oct 09, 2012 2:30 pm
Full Name: Magnus Andersson

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Exabot [Bot], Google Feedfetcher, Yahoo [Bot] and 24 guests