Comprehensive data protection for all workloads
masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Very slow file level restore and problem with permissions

Post by masonit » Nov 01, 2012 1:50 pm

Hi!

Setup:

VMs on ESXi 5

Veeam Backup Server (6.5) and Veeam Proxy on same dedicated ESXi5. Proxy connected to iscsi storage with write speed ~350 MB/s. Everything connected with 10 Gbps. All backups are fine with Application aware.

When running file level restore from Veeam Backup & Replication and restoring to original location I reach staggering speed of 3 KB/s!. This is when restoring a folder of 20 MB with 50 files in it. The VM is a Windows server 2008 R2. Should it be this slow!? Other things like backup and restore of entire VM is fast with speed over 50 MB/s.

If I instead choose to "copy to" and "Preserve permission and ownership" the perfomance is better with 5-10 MB/s. But the permissions isen't preserved! I restore the files locally to Veeam backup server.

\Masonit

Gostev
SVP, Product Management
Posts: 24812
Liked: 3572 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev » Nov 01, 2012 2:48 pm

Hi, I just tried the same in my lab with RTM build, and I am getting about 15 MB/s transfer speed when restoring to original location. So, this looks like to be environment-specific issue in your case. I recommend opening a support case for further troubleshooting. Thanks!

masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Nov 01, 2012 2:55 pm

Ok, thanks will do..

What about permissions. If I look in: C:\VeeamFLR\"vm" when the vm is mapped the permissions look ok. So the permissions is backed up. But when copy the files the permissions is lost.

I found this: http://forums.veeam.com/viewtopic.php?f ... ions#p4377
But it doesn't work on windows server 2008.

\Masonit

masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Nov 01, 2012 3:13 pm

Gostev wrote:Hi, I just tried the same in my lab with RTM build, and I am getting about 15 MB/s transfer speed when restoring to original location. So, this looks like to be environment-specific issue in your case. I recommend opening a support case for further troubleshooting. Thanks!
I think I have found the problem. When running a file level restore to original location. How is the files transfered to the VM?

\Masonit

Gostev
SVP, Product Management
Posts: 24812
Liked: 3572 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev » Nov 01, 2012 3:48 pm

Depends if direct network connection is available from backup server to the VM. If yes, than the file is copied normally over the network. If no, then the file is injected into the guest through ESXi host (using guest interaction API).

masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Nov 01, 2012 4:09 pm

Gostev wrote:Depends if direct network connection is available from backup server to the VM. If yes, than the file is copied normally over the network. If no, then the file is injected into the guest through ESXi host (using guest interaction API).
Ok then I haven't solved it. :( I get good performance when the backupserver and vm is on the same subnet. The slow perfomance is with guest interaction API, 3 KB/s! Is that all I can expect?

\Masonit

Gostev
SVP, Product Management
Posts: 24812
Liked: 3572 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev » Nov 01, 2012 7:45 pm

May there is some issue with the host causing this. While the guest interaction API is definitely much slower than direct network operation, you should still be seeing at least 1000 times faster speed than you do :wink: Please open a support case for investigation.

tsightler
VP, Product Management
Posts: 5424
Liked: 2244 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow file level restore and problem with permission

Post by tsightler » Nov 02, 2012 1:33 am

Yes, 3KB/sec ridiculously slow. I've typically seen speeds anywhere from 1-5MB/sec with the API, although sometimes faster.

masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Nov 05, 2012 9:25 am

Any ideas why permissions isen't retained?

\Masonit

Gostev
SVP, Product Management
Posts: 24812
Liked: 3572 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev » Nov 05, 2012 9:49 am

No idea. Please open a support case for troubleshooting.

masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Nov 05, 2012 1:34 pm

Gostev wrote:Depends if direct network connection is available from backup server to the VM. If yes, than the file is copied normally over the network. If no, then the file is injected into the guest through ESXi host (using guest interaction API).
If direct network connection is availble. How do Veeam Backup server "connect" to the vm. Using the vm:s name as DNS name? Or maybe trying to connect with ip-adress?

\Masonit

Vitaliy S.
Product Manager
Posts: 23005
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. » Nov 05, 2012 2:05 pm

Veeam backup server uses VM's DNS name resolved to IP address to connect to the VM.

masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Nov 05, 2012 2:32 pm

Vitaliy S. wrote:Veeam backup server uses VM's DNS name resolved to IP address to connect to the VM.
Is there away around this? Can't Veeam get the vm:s ip-adress through vmware tools?

We plan to backup many vm:s that belong to many different domains. It is not possible for the Veeam server to resolve all the vm:s dns names...

\Masonit

Vitaliy S.
Product Manager
Posts: 23005
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. » Nov 05, 2012 3:08 pm

I see that my last post was a bit confusing, my bad. Let me clarify it - Veeam backup server selects IP via VMware tools and then uses this information for the backup/restore jobs, so you shouldn't have problems with multiple domains.

masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Nov 06, 2012 7:15 am

Vitaliy S. wrote:I see that my last post was a bit confusing, my bad. Let me clarify it - Veeam backup server selects IP via VMware tools and then uses this information for the backup/restore jobs, so you shouldn't have problems with multiple domains.
If using u-air and I want to restore some sql items. Does it also then use vmware tools to get the ip-adress of the production vm?

\Masonit

Vitaliy S.
Product Manager
Posts: 23005
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. » Nov 06, 2012 8:19 am

Yes, as far as I remember you need to have VMware Tools installed on the production VMs to determine their IP address for all operations performed by Veeam backup server.

masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Nov 07, 2012 9:17 am

We run some tests with a firewall and NAT between Veeam backup server and the vm:s network. We have opened the following ports: udp 135,445
, udp 137:139, udp 1024: 137, tcp 135,139,445. I can from both Veeam backup server and proxy server connect to the target vm with unc path (\\ip-address\c$). I can send and recevice files so everthing works good in Windows. But when trying to restore files from Veeam to the target vm it does not use the network. What are we missing?

\Masonit

foggy
Veeam Software
Posts: 18287
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy » Nov 07, 2012 9:28 am

Magnus, please review the Used Ports section of the product user guide for the full list of ports required for Veeam B&R operation.

masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Nov 07, 2012 11:01 am

foggy wrote:Magnus, please review the Used Ports section of the product user guide for the full list of ports required for Veeam B&R operation.
Thanks but that didn't really help. When we also opened port 50000 and up it worked. Haven't really narrowed it down yet.

\Masonit

foggy
Veeam Software
Posts: 18287
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy » Nov 07, 2012 11:16 am

Please see similar discussion here.

kaskm
Novice
Posts: 3
Liked: 1 time
Joined: Jan 18, 2013 8:16 pm
Contact:

Re: Very slow file level restore and problem with permission

Post by kaskm » Jan 18, 2013 8:19 pm

Were you able to get this resolved? We get about 3-12k/sec on file restores with windows and linux VMs. We have a similar setup with 10GB iSCSI backup repo and all servers in VM. Backup speed is fine, it is only the restores.

Vitaliy S.
Product Manager
Posts: 23005
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. » Jan 20, 2013 12:48 pm

Can you please tell me where you're trying to restore VM guest files to? Are you using Other OS restore wizard or Windows one?

kaskm
Novice
Posts: 3
Liked: 1 time
Joined: Jan 18, 2013 8:16 pm
Contact:

Re: Very slow file level restore and problem with permission

Post by kaskm » Jan 21, 2013 3:31 pm 1 person likes this post

We have slow restores no matter what. I have done restores to the backup server itself and back to the original location. This is with the Windows wizard and with the Other OS. Once it runs for a whil (like 5+ minutes, it does increase from 3k to about 70k/sec).

I want to specify, this is only if we restore multiple files or entire directories. If it is a single file, the restore is quick.

Vitaliy S.
Product Manager
Posts: 23005
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. » Jan 21, 2013 4:20 pm

Does this happen for all VMs on this repository? Can you please copy any small backup file (if you have it of course) to any other place and try to restore any file once again?

Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Very slow file level restore and problem with permission

Post by Fiskepudding » Apr 10, 2013 10:17 am

Was this ever resolved?
If not I would also like to add that we have the same issue.

Restoring Guest files (windows) is very slow!

It starts out restoring at 220 KB/sec, but drops very fast, after 11 min the transfer rate reports 21 KB/sec.
If i cancel that job. And start another one emideatly, it reports 8 KB/sec from the start.
Witch probably was the reeal speed on the previous job when i stoped it.
(since 21KB/s was spread over the 11 min)
To mee it seem like som chace is filling up.

As kaskm says, this only happens with multiple files/folders. The more files the worse it gets.
The files size itsef does not seem to matter.


Restoing entire VMs is FAST. We also have 10G all the way. Our target host uses SSD datstore.
Preserving or not presering permission when copying seems not to make any difference.
Tested on a Inetpub folder containg 1,68 GB of data (59137 files, 6324 folders)
And Inetpub -> ziped (1,03 GB)
The IIS service was offcuase stopped during this restore test.
But anyway since the Inetpub folder still existed, the restore created its own folder"RESTORED-Inetpub_20130410_102051", so files in use should not be an issue.

Test results:


1 file "Copy to..."
- is fast (142 MB/S)

1 file "Restore"
- is fast enough (51MB/S)

Muliple files "Copy to.."
- Starts out at: 1MB/s
- After 5 min: 3MB/s
- After 9 min: 3MB/s


Multiple files "Restore" is is painfully slow
- Starts out at: 220 KB/sec
- After 5 min: 47KB/sec (Time remaining 9h 40m, transfered 14,0 MB )
- After 9 min: 25 KB/sec (Time remaining 16h 30m transfered 14,6 MB )
- After 60 min: 13 KB/sec (Time remaining 10h 11m transfered 45,8 MB )
*I guess there would be days left, but that is not visible in "Time remaining".

Vitaliy S.
Product Manager
Posts: 23005
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. » Apr 10, 2013 4:49 pm

Espen, can you please tell me the restore job performance if you're using Other OS FLR wizard for the same files?

Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Very slow file level restore and problem with permission

Post by Fiskepudding » Apr 11, 2013 7:40 am

Vitaliy, sure.

Restore is not available when choosing "Restore guest files (Other OS).."
Only "Copy to... " is available:

1 file "Restore" NA
Multiple files "Restore" NA

1 file "Copy to..."
- is fast enough (60 MB/S)

Muliple files "Copy to.."
- "Preparing information" lasted about 10 minutes, then the process started.
- after 22 minutes the transfer rate is 14 KB/S and it stops with and error "Failed to copy" (IMG1)
- I hit OK. and it contiues
- Then stops again similar message.(IMG2)
- I hit OK. and it contiues
- After 60 minutes, the transfer rate is 24 KB/S



IMG1
Image

IMG2
Image

Seems like the errors are related to Norwegian characters.
During the copy to, it sopped several times, only on files containing Norwegian characters.
Did not seem to be related to the path depth or file name length.
This did not happen during "Restore guest files (Windows).." witch was used in my post above.

So the copy to is just as slow when using this method, it also introduces a problem where no files or folders with Norwegian characters are copied.

Vitaliy S.
Product Manager
Posts: 23005
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. » Apr 12, 2013 10:05 am

Fiskepudding wrote:So the copy to is just as slow when using this method, it also introduces a problem where no files or folders with Norwegian characters are copied.
Another way to rest restores of multiple files might be an instant VM recovery option and Windows Explorer, though I believe the performance will be the same, as data will be retrieved from the compressed and deduped file directly.

audreymann
Lurker
Posts: 2
Liked: never
Joined: Apr 06, 2013 8:15 am
Full Name: Audrey Mann
Contact:

Re: Very slow file level restore and problem with permission

Post by audreymann » Apr 12, 2013 12:52 pm

Thanks for sharing the knowledge guys...had similar query!! solved now. :)

masonit
Service Provider
Posts: 220
Liked: 16 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Apr 15, 2013 6:45 am

kaskm wrote:Were you able to get this resolved? We get about 3-12k/sec on file restores with windows and linux VMs. We have a similar setup with 10GB iSCSI backup repo and all servers in VM. Backup speed is fine, it is only the restores.
Not really. Restore through Vmware API injection is still slow. But we have made some changes in our setup. Now all restores send their traffic over lan. Then we get 15-20 MB/s.

\Masonit

Post Reply

Who is online

Users browsing this forum: Bing [Bot], SBarrett847 and 44 guests