Comprehensive data protection for all workloads
Post Reply
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

slow restore speed veeam7 1GB / minute

Post by kte »

I need to restore a 16 TB file server. The full backup thakes 8h by fiberchannel backup.
But I can only restore @ 1GB/minute, is this normal seem horrible slow compared to the backup @ 500 MB/s
console of ESX is 3 GB/s and can use 10 GB/s if the bandwith is available the physical backup has 10 GB also , if I make file copies over the network I can copy 1TB/hour
any ideas suggestion how I can do a full VM recovery alot faster

K
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by Gostev » 1 person likes this post

Pick a hot add mode proxy in the restore wizard.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by dellock6 »

Also, writes to a vmfs filesystem is known to be not so fast because of metadata created by a growing file (usually vmdk are created at a certain size since their creation). Sure a hotadd proxy is a good solution, another one is to have at least one NFS storage at least for restores, NFS does not have any metadata problem like VMFS.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by kte »

if i do an instant recovery and the a storage vmotion will it go fast then?

k
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by Vitaliy S. »

Instant VM recovery will definitely bring your VM back to production much quicker, but storage vMotion performance most likely will be the same as the datastore upload speed you currently have.
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by kte »

what type of virtual scsi contoller must I use in my VM proxy for hotadd restore ? I have Vm's with paravitual disk/ serial LSI SCSI and paralel LSI SCSI or doesn't that make a difference??
Is 1 virtual scisi adaptor enough or must I create 3 additional 1MB disks to have 3 additional scsi adaptors in my proxy VM?

K
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by Gostev »

I recommend going with the default iSCSI controller type (whichever is the default when you create a new VM), I believe I recall some issues with non-default one, but it was long ago. One adaptor is usually enough for restore, unless your proxy VM has lots of own virtual disks. But, it is a good practice to have at least two iSCSI controllers for each proxy.
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by kte »

And is there not a way to have a delicated NFS network adaptor with a proper vmkernel IP, so that the storage vmotion goes faster?
Because I can't imagine that all NFS storage is that slow to migrate?
Or is there another issue?

K
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by Vitaliy S. »

Hi Koen,

Not sure I fully understand the question, but it is possible to use a separate NIC for vMotion traffic.

BTW, here is an interesting article, that you might find useful: http://frankdenneman.nl/2013/02/07/why- ... n-network/

Thanks!
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by kte »

hotadd is a s slow as as storage vmotion and quick migration 1/GB a minute is the max

case 00455736
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by kte »

I'm calculating veeam restored in hotadd mode 100 GB in 2h 40 minutes so I'm processing @ 7MB/s any way to get the performance up correctly x10 - x20
My data is backupped on a physical server via direct san mode @ 500 MB/s but I can't restore correctly I need 20 days to restore my VM of 16 TB
I create a proxy VM for hotadd and selected in the VM restore to use this proxy to restore, but still @ 7 MB/s
Instant recovery and storage vmotion where almost the same in performance. Any idea

K
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by tsightler »

There's really not that much involved in restores other than reading from the source repository and writing to the target datastore so those are really the only to things you should look at. Typically restores will be limited by target datastore performance, however, 7MB/s is quite a bit slower than what I would expect even from relatively poor storage. What type of storage are datastores and your repository. Does the storage backing your datastores have write-backup caching enabled? This is important for achieving good restore performance to VMFS.

Typical restore speeds I've seen from Veeam in the field are anywhere from 50-100MB/s for VMFS, and about 2x that speed for NFS datastores assuming reasonable hardware so something is definitely unusual in your environment.
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by kte »

i have a 3 par stireserv with 80 sas disk
i have 2 smart arrays with each 10 disk behind and 2 GB cache with and 50% read write cache on my physical proxy server.
when i copy from 1 server t another ik veeam copy i get 150 MB/s from my backup server

k
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by tsightler »

Then unfortunately I have no idea. Having recently tested similar equipment I can say that I would expect speeds closer to the top end (75-100MBs) rather than what you are seeing. Hopefully support will be able to find something. Just about the only other thing I can think of is if somehow there is a routed network or some other issue between you and the target that is causing a significant network bottleneck (perhaps even a network improperly configured), but the fact that you're seeing similar speed even with a virtual proxy (assuming the proxy was used and properly performed hotadd restore) would make this seem unlikely. Hopefully support can find our something for you. Overall restores with Veeam are really quite simple so there's not a lot of places to look.

One really silly thing to try, you could perhaps try restoring the VMDK using the "Restore Guest VM Files" method and see what that gives you.
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by yizhar »

Hi.

I have commented to a similar post, please read and try the following:

compare restore performance of very large VM vs a small one. Any difference?
For the first test - make sure that the small VM is restored from a relatively small VBK file.

try with and without the following setting:
DataMover.HardwareAcceleratedInit

See following post for details:
Slow Restore Speed - 27MB-s - Tips-Ideas view topic
http://forums.veeam.com/viewtopic.php?f=24&t=12892

Yizhar
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by kte »

tried it but no direct change, i only changed the setting on 1 esx
how does the hotadd work,also via vpower nfs , because i don't see any network traffic from the physival proxy to the virtual proxy.
or an issue in veeam 7 patch1

k
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by Vitaliy S. »

Koen,
kte wrote: because i don't see any network traffic from the physival proxy to the virtual proxy.
There shouldn't be any traffic between proxy servers when you're restoring full VM via HotAdd mode. If virtual proxy server has the direct access to the datastore you want to restore VMs to, it will use ESXi I/O stack to write VM data to the datastore.

Instant VM recovery (vPower NFS) does not use hotadd capabilities.

Thanks!
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by kte »

but the veeam repository is on my physical proxy and iwant to restor to my virtual proxy so between both proxies must be some network data traffic, no??
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by Vitaliy S. »

Yes, you need to have connection between repository and the proxy server used for hotadd restores. In your case data will go from the physical server to virtual proxy and then will be written to the datastore via ESXi I/O stack.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by veremin »

Then, I’m wondering between what components you do see the traffic flow. And what backup proxy is shown in the statistics of full VM Restore session? Thanks.
Mory
Novice
Posts: 8
Liked: 4 times
Joined: Aug 10, 2013 5:39 pm
Full Name: Chris
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by Mory » 2 people like this post

Are you using thin provisioned disk on your 3Par? Doing restore testing for a datacenter migration I was running into issues with restore speeds being awfully slow as well. I too have a 3Par SAN and was seeing around the same restore speeds you mention. Restoring to a fresh newly created thin provisioned disks was OK the 1st time through. However, subsequent restore attempts were terribly slow. With fully provisioned ('thick') virtual volumes, there were no issues on the first or any other restore.
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by kte »

I'm using thin volumes on the 3 par and also thin vmdk's
Mory
Novice
Posts: 8
Liked: 4 times
Joined: Aug 10, 2013 5:39 pm
Full Name: Chris
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by Mory » 2 people like this post

If you haven't tried yet, I'd suggest creating a thick volume and trying your restore again to see if that resolves the issue you are having. I'm not sure if there is a bug somewhere with the 3Par that is causing this or somewhere/something else.

However, when using thick volumes I've been able to restore around 4.5TB worth of .VBK's (default dedupe and compression settings) in about 3.5 - 4 hours (60 VM’s). For reference, I have 1 x physical Veeam server were the Veeam backup files reside and I'm using 8 x virtual Veeam proxies using hotadd (one on each host server). I haven’t really done single restore speed testing, but when running between 6 – 8 restores simultaneously, the Veeam statistics of the jobs range between 80 mb/s and 160 mb/s. Mostly in the 80 – 100 range though.
chjones
Expert
Posts: 117
Liked: 31 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by chjones » 1 person likes this post

After pulling my hair out for weeks and through so many restore tests I could click through the Veeam wizards blindfolded we came to the exact same conclusion today with our 3par 7400.

We just stuck to network mode restores for all tests, just wanted to keep things simple and make one change at a time during the 50+ restore attempts.

When restoring to a 3par volume we kept hitting a ceiling of 11MB/sec. Restoring to esx local disks or an older EVA volume we could get 80-140MB/sec. We finally found through dumb luck this same issue of the first restore to a thin 3par volume being fast and subsequent restores hitting the 11MB/sec limit.

3par thick volumes are consistently over 140MB/sec no matter how many times we restore to them.

Converting a thin volume to thick also then restores very fast. Then converting back to thin the restores slow right down again.

Our only possible workaround for now is have a thick volume on standby for restores (large enough to receive our largest single vm) and then storage vmotion back to the thin volumes.

Strange that a storage vmotion isn't affected by this slow down.

Did anyone ever find a proper fix for this?
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: slow restore speed veeam7 1GB / minute

Post by kte » 1 person likes this post

3PAR FW 3.3.1 solves this issue or create an new thin lun to restore to or restore to a tick lun
Post Reply

Who is online

Users browsing this forum: No registered users and 233 guests