-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Very slow file level restore and problem with permissions
Hi!
Setup:
VMs on ESXi 5
Veeam Backup Server (6.5) and Veeam Proxy on same dedicated ESXi5. Proxy connected to iscsi storage with write speed ~350 MB/s. Everything connected with 10 Gbps. All backups are fine with Application aware.
When running file level restore from Veeam Backup & Replication and restoring to original location I reach staggering speed of 3 KB/s!. This is when restoring a folder of 20 MB with 50 files in it. The VM is a Windows server 2008 R2. Should it be this slow!? Other things like backup and restore of entire VM is fast with speed over 50 MB/s.
If I instead choose to "copy to" and "Preserve permission and ownership" the perfomance is better with 5-10 MB/s. But the permissions isen't preserved! I restore the files locally to Veeam backup server.
\Masonit
Setup:
VMs on ESXi 5
Veeam Backup Server (6.5) and Veeam Proxy on same dedicated ESXi5. Proxy connected to iscsi storage with write speed ~350 MB/s. Everything connected with 10 Gbps. All backups are fine with Application aware.
When running file level restore from Veeam Backup & Replication and restoring to original location I reach staggering speed of 3 KB/s!. This is when restoring a folder of 20 MB with 50 files in it. The VM is a Windows server 2008 R2. Should it be this slow!? Other things like backup and restore of entire VM is fast with speed over 50 MB/s.
If I instead choose to "copy to" and "Preserve permission and ownership" the perfomance is better with 5-10 MB/s. But the permissions isen't preserved! I restore the files locally to Veeam backup server.
\Masonit
-
- Chief Product Officer
- Posts: 31561
- Liked: 6725 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Very slow file level restore and problem with permission
Hi, I just tried the same in my lab with RTM build, and I am getting about 15 MB/s transfer speed when restoring to original location. So, this looks like to be environment-specific issue in your case. I recommend opening a support case for further troubleshooting. Thanks!
-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: Very slow file level restore and problem with permission
Ok, thanks will do..
What about permissions. If I look in: C:\VeeamFLR\"vm" when the vm is mapped the permissions look ok. So the permissions is backed up. But when copy the files the permissions is lost.
I found this: http://forums.veeam.com/viewtopic.php?f ... ions#p4377
But it doesn't work on windows server 2008.
\Masonit
What about permissions. If I look in: C:\VeeamFLR\"vm" when the vm is mapped the permissions look ok. So the permissions is backed up. But when copy the files the permissions is lost.
I found this: http://forums.veeam.com/viewtopic.php?f ... ions#p4377
But it doesn't work on windows server 2008.
\Masonit
-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: Very slow file level restore and problem with permission
I think I have found the problem. When running a file level restore to original location. How is the files transfered to the VM?Gostev wrote:Hi, I just tried the same in my lab with RTM build, and I am getting about 15 MB/s transfer speed when restoring to original location. So, this looks like to be environment-specific issue in your case. I recommend opening a support case for further troubleshooting. Thanks!
\Masonit
-
- Chief Product Officer
- Posts: 31561
- Liked: 6725 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Very slow file level restore and problem with permission
Depends if direct network connection is available from backup server to the VM. If yes, than the file is copied normally over the network. If no, then the file is injected into the guest through ESXi host (using guest interaction API).
-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: Very slow file level restore and problem with permission
Ok then I haven't solved it. I get good performance when the backupserver and vm is on the same subnet. The slow perfomance is with guest interaction API, 3 KB/s! Is that all I can expect?Gostev wrote:Depends if direct network connection is available from backup server to the VM. If yes, than the file is copied normally over the network. If no, then the file is injected into the guest through ESXi host (using guest interaction API).
\Masonit
-
- Chief Product Officer
- Posts: 31561
- Liked: 6725 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Very slow file level restore and problem with permission
May there is some issue with the host causing this. While the guest interaction API is definitely much slower than direct network operation, you should still be seeing at least 1000 times faster speed than you do Please open a support case for investigation.
-
- VP, Product Management
- Posts: 6013
- Liked: 2843 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Very slow file level restore and problem with permission
Yes, 3KB/sec ridiculously slow. I've typically seen speeds anywhere from 1-5MB/sec with the API, although sometimes faster.
-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: Very slow file level restore and problem with permission
Any ideas why permissions isen't retained?
\Masonit
\Masonit
-
- Chief Product Officer
- Posts: 31561
- Liked: 6725 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Very slow file level restore and problem with permission
No idea. Please open a support case for troubleshooting.
-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: Very slow file level restore and problem with permission
If direct network connection is availble. How do Veeam Backup server "connect" to the vm. Using the vm:s name as DNS name? Or maybe trying to connect with ip-adress?Gostev wrote:Depends if direct network connection is available from backup server to the VM. If yes, than the file is copied normally over the network. If no, then the file is injected into the guest through ESXi host (using guest interaction API).
\Masonit
-
- VP, Product Management
- Posts: 27120
- Liked: 2721 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Very slow file level restore and problem with permission
Veeam backup server uses VM's DNS name resolved to IP address to connect to the VM.
-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: Very slow file level restore and problem with permission
Is there away around this? Can't Veeam get the vm:s ip-adress through vmware tools?Vitaliy S. wrote:Veeam backup server uses VM's DNS name resolved to IP address to connect to the VM.
We plan to backup many vm:s that belong to many different domains. It is not possible for the Veeam server to resolve all the vm:s dns names...
\Masonit
-
- VP, Product Management
- Posts: 27120
- Liked: 2721 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Very slow file level restore and problem with permission
I see that my last post was a bit confusing, my bad. Let me clarify it - Veeam backup server selects IP via VMware tools and then uses this information for the backup/restore jobs, so you shouldn't have problems with multiple domains.
-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: Very slow file level restore and problem with permission
If using u-air and I want to restore some sql items. Does it also then use vmware tools to get the ip-adress of the production vm?Vitaliy S. wrote:I see that my last post was a bit confusing, my bad. Let me clarify it - Veeam backup server selects IP via VMware tools and then uses this information for the backup/restore jobs, so you shouldn't have problems with multiple domains.
\Masonit
-
- VP, Product Management
- Posts: 27120
- Liked: 2721 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Very slow file level restore and problem with permission
Yes, as far as I remember you need to have VMware Tools installed on the production VMs to determine their IP address for all operations performed by Veeam backup server.
-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: Very slow file level restore and problem with permission
We run some tests with a firewall and NAT between Veeam backup server and the vm:s network. We have opened the following ports: udp 135,445
, udp 137:139, udp 1024: 137, tcp 135,139,445. I can from both Veeam backup server and proxy server connect to the target vm with unc path (\\ip-address\c$). I can send and recevice files so everthing works good in Windows. But when trying to restore files from Veeam to the target vm it does not use the network. What are we missing?
\Masonit
, udp 137:139, udp 1024: 137, tcp 135,139,445. I can from both Veeam backup server and proxy server connect to the target vm with unc path (\\ip-address\c$). I can send and recevice files so everthing works good in Windows. But when trying to restore files from Veeam to the target vm it does not use the network. What are we missing?
\Masonit
-
- Veeam Software
- Posts: 21072
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Very slow file level restore and problem with permission
Magnus, please review the Used Ports section of the product user guide for the full list of ports required for Veeam B&R operation.
-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: Very slow file level restore and problem with permission
Thanks but that didn't really help. When we also opened port 50000 and up it worked. Haven't really narrowed it down yet.foggy wrote:Magnus, please review the Used Ports section of the product user guide for the full list of ports required for Veeam B&R operation.
\Masonit
-
- Veeam Software
- Posts: 21072
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Very slow file level restore and problem with permission
Please see similar discussion here.
-
- Novice
- Posts: 3
- Liked: 1 time
- Joined: Jan 18, 2013 8:16 pm
- Contact:
Re: Very slow file level restore and problem with permission
Were you able to get this resolved? We get about 3-12k/sec on file restores with windows and linux VMs. We have a similar setup with 10GB iSCSI backup repo and all servers in VM. Backup speed is fine, it is only the restores.
-
- VP, Product Management
- Posts: 27120
- Liked: 2721 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Very slow file level restore and problem with permission
Can you please tell me where you're trying to restore VM guest files to? Are you using Other OS restore wizard or Windows one?
-
- Novice
- Posts: 3
- Liked: 1 time
- Joined: Jan 18, 2013 8:16 pm
- Contact:
Re: Very slow file level restore and problem with permission
We have slow restores no matter what. I have done restores to the backup server itself and back to the original location. This is with the Windows wizard and with the Other OS. Once it runs for a whil (like 5+ minutes, it does increase from 3k to about 70k/sec).
I want to specify, this is only if we restore multiple files or entire directories. If it is a single file, the restore is quick.
I want to specify, this is only if we restore multiple files or entire directories. If it is a single file, the restore is quick.
-
- VP, Product Management
- Posts: 27120
- Liked: 2721 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Very slow file level restore and problem with permission
Does this happen for all VMs on this repository? Can you please copy any small backup file (if you have it of course) to any other place and try to restore any file once again?
-
- Expert
- Posts: 213
- Liked: 26 times
- Joined: Feb 01, 2012 7:24 am
- Full Name: Espen Dykesteen
- Contact:
Re: Very slow file level restore and problem with permission
Was this ever resolved?
If not I would also like to add that we have the same issue.
Restoring Guest files (windows) is very slow!
It starts out restoring at 220 KB/sec, but drops very fast, after 11 min the transfer rate reports 21 KB/sec.
If i cancel that job. And start another one emideatly, it reports 8 KB/sec from the start.
Witch probably was the reeal speed on the previous job when i stoped it.
(since 21KB/s was spread over the 11 min)
To mee it seem like som chace is filling up.
As kaskm says, this only happens with multiple files/folders. The more files the worse it gets.
The files size itsef does not seem to matter.
Restoing entire VMs is FAST. We also have 10G all the way. Our target host uses SSD datstore.
Preserving or not presering permission when copying seems not to make any difference.
Tested on a Inetpub folder containg 1,68 GB of data (59137 files, 6324 folders)
And Inetpub -> ziped (1,03 GB)
The IIS service was offcuase stopped during this restore test.
But anyway since the Inetpub folder still existed, the restore created its own folder"RESTORED-Inetpub_20130410_102051", so files in use should not be an issue.
Test results:
1 file "Copy to..."
- is fast (142 MB/S)
1 file "Restore"
- is fast enough (51MB/S)
Muliple files "Copy to.."
- Starts out at: 1MB/s
- After 5 min: 3MB/s
- After 9 min: 3MB/s
Multiple files "Restore" is is painfully slow
- Starts out at: 220 KB/sec
- After 5 min: 47KB/sec (Time remaining 9h 40m, transfered 14,0 MB )
- After 9 min: 25 KB/sec (Time remaining 16h 30m transfered 14,6 MB )
- After 60 min: 13 KB/sec (Time remaining 10h 11m transfered 45,8 MB )
*I guess there would be days left, but that is not visible in "Time remaining".
If not I would also like to add that we have the same issue.
Restoring Guest files (windows) is very slow!
It starts out restoring at 220 KB/sec, but drops very fast, after 11 min the transfer rate reports 21 KB/sec.
If i cancel that job. And start another one emideatly, it reports 8 KB/sec from the start.
Witch probably was the reeal speed on the previous job when i stoped it.
(since 21KB/s was spread over the 11 min)
To mee it seem like som chace is filling up.
As kaskm says, this only happens with multiple files/folders. The more files the worse it gets.
The files size itsef does not seem to matter.
Restoing entire VMs is FAST. We also have 10G all the way. Our target host uses SSD datstore.
Preserving or not presering permission when copying seems not to make any difference.
Tested on a Inetpub folder containg 1,68 GB of data (59137 files, 6324 folders)
And Inetpub -> ziped (1,03 GB)
The IIS service was offcuase stopped during this restore test.
But anyway since the Inetpub folder still existed, the restore created its own folder"RESTORED-Inetpub_20130410_102051", so files in use should not be an issue.
Test results:
1 file "Copy to..."
- is fast (142 MB/S)
1 file "Restore"
- is fast enough (51MB/S)
Muliple files "Copy to.."
- Starts out at: 1MB/s
- After 5 min: 3MB/s
- After 9 min: 3MB/s
Multiple files "Restore" is is painfully slow
- Starts out at: 220 KB/sec
- After 5 min: 47KB/sec (Time remaining 9h 40m, transfered 14,0 MB )
- After 9 min: 25 KB/sec (Time remaining 16h 30m transfered 14,6 MB )
- After 60 min: 13 KB/sec (Time remaining 10h 11m transfered 45,8 MB )
*I guess there would be days left, but that is not visible in "Time remaining".
-
- VP, Product Management
- Posts: 27120
- Liked: 2721 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Very slow file level restore and problem with permission
Espen, can you please tell me the restore job performance if you're using Other OS FLR wizard for the same files?
-
- Expert
- Posts: 213
- Liked: 26 times
- Joined: Feb 01, 2012 7:24 am
- Full Name: Espen Dykesteen
- Contact:
Re: Very slow file level restore and problem with permission
Vitaliy, sure.
Restore is not available when choosing "Restore guest files (Other OS).."
Only "Copy to... " is available:
1 file "Restore" NA
Multiple files "Restore" NA
1 file "Copy to..."
- is fast enough (60 MB/S)
Muliple files "Copy to.."
- "Preparing information" lasted about 10 minutes, then the process started.
- after 22 minutes the transfer rate is 14 KB/S and it stops with and error "Failed to copy" (IMG1)
- I hit OK. and it contiues
- Then stops again similar message.(IMG2)
- I hit OK. and it contiues
- After 60 minutes, the transfer rate is 24 KB/S
IMG1
IMG2
Seems like the errors are related to Norwegian characters.
During the copy to, it sopped several times, only on files containing Norwegian characters.
Did not seem to be related to the path depth or file name length.
This did not happen during "Restore guest files (Windows).." witch was used in my post above.
So the copy to is just as slow when using this method, it also introduces a problem where no files or folders with Norwegian characters are copied.
Restore is not available when choosing "Restore guest files (Other OS).."
Only "Copy to... " is available:
1 file "Restore" NA
Multiple files "Restore" NA
1 file "Copy to..."
- is fast enough (60 MB/S)
Muliple files "Copy to.."
- "Preparing information" lasted about 10 minutes, then the process started.
- after 22 minutes the transfer rate is 14 KB/S and it stops with and error "Failed to copy" (IMG1)
- I hit OK. and it contiues
- Then stops again similar message.(IMG2)
- I hit OK. and it contiues
- After 60 minutes, the transfer rate is 24 KB/S
IMG1
IMG2
Seems like the errors are related to Norwegian characters.
During the copy to, it sopped several times, only on files containing Norwegian characters.
Did not seem to be related to the path depth or file name length.
This did not happen during "Restore guest files (Windows).." witch was used in my post above.
So the copy to is just as slow when using this method, it also introduces a problem where no files or folders with Norwegian characters are copied.
-
- VP, Product Management
- Posts: 27120
- Liked: 2721 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Very slow file level restore and problem with permission
Another way to rest restores of multiple files might be an instant VM recovery option and Windows Explorer, though I believe the performance will be the same, as data will be retrieved from the compressed and deduped file directly.Fiskepudding wrote:So the copy to is just as slow when using this method, it also introduces a problem where no files or folders with Norwegian characters are copied.
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Apr 06, 2013 8:15 am
- Full Name: Audrey Mann
- Contact:
Re: Very slow file level restore and problem with permission
Thanks for sharing the knowledge guys...had similar query!! solved now.
-
- Service Provider
- Posts: 325
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: Very slow file level restore and problem with permission
Not really. Restore through Vmware API injection is still slow. But we have made some changes in our setup. Now all restores send their traffic over lan. Then we get 15-20 MB/s.kaskm wrote:Were you able to get this resolved? We get about 3-12k/sec on file restores with windows and linux VMs. We have a similar setup with 10GB iSCSI backup repo and all servers in VM. Backup speed is fine, it is only the restores.
\Masonit
Who is online
Users browsing this forum: Bing [Bot], Ivan239 and 154 guests