Comprehensive data protection for all workloads
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Very slow file level restore and problem with permissions

Post by masonit »

Hi!

Setup:

VMs on ESXi 5

Veeam Backup Server (6.5) and Veeam Proxy on same dedicated ESXi5. Proxy connected to iscsi storage with write speed ~350 MB/s. Everything connected with 10 Gbps. All backups are fine with Application aware.

When running file level restore from Veeam Backup & Replication and restoring to original location I reach staggering speed of 3 KB/s!. This is when restoring a folder of 20 MB with 50 files in it. The VM is a Windows server 2008 R2. Should it be this slow!? Other things like backup and restore of entire VM is fast with speed over 50 MB/s.

If I instead choose to "copy to" and "Preserve permission and ownership" the perfomance is better with 5-10 MB/s. But the permissions isen't preserved! I restore the files locally to Veeam backup server.

\Masonit
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev »

Hi, I just tried the same in my lab with RTM build, and I am getting about 15 MB/s transfer speed when restoring to original location. So, this looks like to be environment-specific issue in your case. I recommend opening a support case for further troubleshooting. Thanks!
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit »

Ok, thanks will do..

What about permissions. If I look in: C:\VeeamFLR\"vm" when the vm is mapped the permissions look ok. So the permissions is backed up. But when copy the files the permissions is lost.

I found this: http://forums.veeam.com/viewtopic.php?f ... ions#p4377
But it doesn't work on windows server 2008.

\Masonit
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit »

Gostev wrote:Hi, I just tried the same in my lab with RTM build, and I am getting about 15 MB/s transfer speed when restoring to original location. So, this looks like to be environment-specific issue in your case. I recommend opening a support case for further troubleshooting. Thanks!
I think I have found the problem. When running a file level restore to original location. How is the files transfered to the VM?

\Masonit
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev »

Depends if direct network connection is available from backup server to the VM. If yes, than the file is copied normally over the network. If no, then the file is injected into the guest through ESXi host (using guest interaction API).
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit »

Gostev wrote:Depends if direct network connection is available from backup server to the VM. If yes, than the file is copied normally over the network. If no, then the file is injected into the guest through ESXi host (using guest interaction API).
Ok then I haven't solved it. :( I get good performance when the backupserver and vm is on the same subnet. The slow perfomance is with guest interaction API, 3 KB/s! Is that all I can expect?

\Masonit
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev »

May there is some issue with the host causing this. While the guest interaction API is definitely much slower than direct network operation, you should still be seeing at least 1000 times faster speed than you do :wink: Please open a support case for investigation.
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow file level restore and problem with permission

Post by tsightler »

Yes, 3KB/sec ridiculously slow. I've typically seen speeds anywhere from 1-5MB/sec with the API, although sometimes faster.
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit »

Any ideas why permissions isen't retained?

\Masonit
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev »

No idea. Please open a support case for troubleshooting.
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit »

Gostev wrote:Depends if direct network connection is available from backup server to the VM. If yes, than the file is copied normally over the network. If no, then the file is injected into the guest through ESXi host (using guest interaction API).
If direct network connection is availble. How do Veeam Backup server "connect" to the vm. Using the vm:s name as DNS name? Or maybe trying to connect with ip-adress?

\Masonit
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. »

Veeam backup server uses VM's DNS name resolved to IP address to connect to the VM.
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit »

Vitaliy S. wrote:Veeam backup server uses VM's DNS name resolved to IP address to connect to the VM.
Is there away around this? Can't Veeam get the vm:s ip-adress through vmware tools?

We plan to backup many vm:s that belong to many different domains. It is not possible for the Veeam server to resolve all the vm:s dns names...

\Masonit
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. »

I see that my last post was a bit confusing, my bad. Let me clarify it - Veeam backup server selects IP via VMware tools and then uses this information for the backup/restore jobs, so you shouldn't have problems with multiple domains.
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit »

Vitaliy S. wrote:I see that my last post was a bit confusing, my bad. Let me clarify it - Veeam backup server selects IP via VMware tools and then uses this information for the backup/restore jobs, so you shouldn't have problems with multiple domains.
If using u-air and I want to restore some sql items. Does it also then use vmware tools to get the ip-adress of the production vm?

\Masonit
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. »

Yes, as far as I remember you need to have VMware Tools installed on the production VMs to determine their IP address for all operations performed by Veeam backup server.
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit »

We run some tests with a firewall and NAT between Veeam backup server and the vm:s network. We have opened the following ports: udp 135,445
, udp 137:139, udp 1024: 137, tcp 135,139,445. I can from both Veeam backup server and proxy server connect to the target vm with unc path (\\ip-address\c$). I can send and recevice files so everthing works good in Windows. But when trying to restore files from Veeam to the target vm it does not use the network. What are we missing?

\Masonit
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy »

Magnus, please review the Used Ports section of the product user guide for the full list of ports required for Veeam B&R operation.
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit »

foggy wrote:Magnus, please review the Used Ports section of the product user guide for the full list of ports required for Veeam B&R operation.
Thanks but that didn't really help. When we also opened port 50000 and up it worked. Haven't really narrowed it down yet.

\Masonit
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy »

Please see similar discussion here.
kaskm
Novice
Posts: 3
Liked: 1 time
Joined: Jan 18, 2013 8:16 pm
Contact:

Re: Very slow file level restore and problem with permission

Post by kaskm »

Were you able to get this resolved? We get about 3-12k/sec on file restores with windows and linux VMs. We have a similar setup with 10GB iSCSI backup repo and all servers in VM. Backup speed is fine, it is only the restores.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. »

Can you please tell me where you're trying to restore VM guest files to? Are you using Other OS restore wizard or Windows one?
kaskm
Novice
Posts: 3
Liked: 1 time
Joined: Jan 18, 2013 8:16 pm
Contact:

Re: Very slow file level restore and problem with permission

Post by kaskm » 1 person likes this post

We have slow restores no matter what. I have done restores to the backup server itself and back to the original location. This is with the Windows wizard and with the Other OS. Once it runs for a whil (like 5+ minutes, it does increase from 3k to about 70k/sec).

I want to specify, this is only if we restore multiple files or entire directories. If it is a single file, the restore is quick.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. »

Does this happen for all VMs on this repository? Can you please copy any small backup file (if you have it of course) to any other place and try to restore any file once again?
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Very slow file level restore and problem with permission

Post by Fiskepudding »

Was this ever resolved?
If not I would also like to add that we have the same issue.

Restoring Guest files (windows) is very slow!

It starts out restoring at 220 KB/sec, but drops very fast, after 11 min the transfer rate reports 21 KB/sec.
If i cancel that job. And start another one emideatly, it reports 8 KB/sec from the start.
Witch probably was the reeal speed on the previous job when i stoped it.
(since 21KB/s was spread over the 11 min)
To mee it seem like som chace is filling up.

As kaskm says, this only happens with multiple files/folders. The more files the worse it gets.
The files size itsef does not seem to matter.


Restoing entire VMs is FAST. We also have 10G all the way. Our target host uses SSD datstore.
Preserving or not presering permission when copying seems not to make any difference.
Tested on a Inetpub folder containg 1,68 GB of data (59137 files, 6324 folders)
And Inetpub -> ziped (1,03 GB)
The IIS service was offcuase stopped during this restore test.
But anyway since the Inetpub folder still existed, the restore created its own folder"RESTORED-Inetpub_20130410_102051", so files in use should not be an issue.

Test results:


1 file "Copy to..."
- is fast (142 MB/S)

1 file "Restore"
- is fast enough (51MB/S)

Muliple files "Copy to.."
- Starts out at: 1MB/s
- After 5 min: 3MB/s
- After 9 min: 3MB/s


Multiple files "Restore" is is painfully slow
- Starts out at: 220 KB/sec
- After 5 min: 47KB/sec (Time remaining 9h 40m, transfered 14,0 MB )
- After 9 min: 25 KB/sec (Time remaining 16h 30m transfered 14,6 MB )
- After 60 min: 13 KB/sec (Time remaining 10h 11m transfered 45,8 MB )
*I guess there would be days left, but that is not visible in "Time remaining".
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. »

Espen, can you please tell me the restore job performance if you're using Other OS FLR wizard for the same files?
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Very slow file level restore and problem with permission

Post by Fiskepudding »

Vitaliy, sure.

Restore is not available when choosing "Restore guest files (Other OS).."
Only "Copy to... " is available:

1 file "Restore" NA
Multiple files "Restore" NA

1 file "Copy to..."
- is fast enough (60 MB/S)

Muliple files "Copy to.."
- "Preparing information" lasted about 10 minutes, then the process started.
- after 22 minutes the transfer rate is 14 KB/S and it stops with and error "Failed to copy" (IMG1)
- I hit OK. and it contiues
- Then stops again similar message.(IMG2)
- I hit OK. and it contiues
- After 60 minutes, the transfer rate is 24 KB/S



IMG1
Image

IMG2
Image

Seems like the errors are related to Norwegian characters.
During the copy to, it sopped several times, only on files containing Norwegian characters.
Did not seem to be related to the path depth or file name length.
This did not happen during "Restore guest files (Windows).." witch was used in my post above.

So the copy to is just as slow when using this method, it also introduces a problem where no files or folders with Norwegian characters are copied.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow file level restore and problem with permission

Post by Vitaliy S. »

Fiskepudding wrote:So the copy to is just as slow when using this method, it also introduces a problem where no files or folders with Norwegian characters are copied.
Another way to rest restores of multiple files might be an instant VM recovery option and Windows Explorer, though I believe the performance will be the same, as data will be retrieved from the compressed and deduped file directly.
audreymann
Lurker
Posts: 2
Liked: never
Joined: Apr 06, 2013 8:15 am
Full Name: Audrey Mann
Contact:

Re: Very slow file level restore and problem with permission

Post by audreymann »

Thanks for sharing the knowledge guys...had similar query!! solved now. :)
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit »

kaskm wrote:Were you able to get this resolved? We get about 3-12k/sec on file restores with windows and linux VMs. We have a similar setup with 10GB iSCSI backup repo and all servers in VM. Backup speed is fine, it is only the restores.
Not really. Restore through Vmware API injection is still slow. But we have made some changes in our setup. Now all restores send their traffic over lan. Then we get 15-20 MB/s.

\Masonit
Post Reply

Who is online

Users browsing this forum: FelixW, ilisousou123, Semrush [Bot] and 196 guests