slow restore speed veeam7 1GB / minute

[kte]

I need to restore a 16 TB file server. The full backup thakes 8h by fiberchannel backup.
But I can only restore @ 1GB/minute, is this normal seem horrible slow compared to the backup @ 500 MB/s
console of ESX is 3 GB/s and can use 10 GB/s if the bandwith is available the physical backup has 10 GB also , if I make file copies over the network I can copy 1TB/hour
any ideas suggestion how I can do a full VM recovery alot faster

K

[Gostev]

Pick a hot add mode proxy in the restore wizard.

[dellock6]

Also, writes to a vmfs filesystem is known to be not so fast because of metadata created by a growing file (usually vmdk are created at a certain size since their creation). Sure a hotadd proxy is a good solution, another one is to have at least one NFS storage at least for restores, NFS does not have any metadata problem like VMFS.

Luca.

[kte]

if i do an instant recovery and the a storage vmotion will it go fast then?

k

[Vitaliy S.]

Instant VM recovery will definitely bring your VM back to production much quicker, but storage vMotion performance most likely will be the same as the datastore upload speed you currently have.

[kte]

what type of virtual scsi contoller must I use in my VM proxy for hotadd restore ? I have Vm's with paravitual disk/ serial LSI SCSI and paralel LSI SCSI or doesn't that make a difference??
Is 1 virtual scisi adaptor enough or must I create 3 additional 1MB disks to have 3 additional scsi adaptors in my proxy VM?

K

[Gostev]

I recommend going with the default iSCSI controller type (whichever is the default when you create a new VM), I believe I recall some issues with non-default one, but it was long ago. One adaptor is usually enough for restore, unless your proxy VM has lots of own virtual disks. But, it is a good practice to have at least two iSCSI controllers for each proxy.

[kte]

And is there not a way to have a delicated NFS network adaptor with a proper vmkernel IP, so that the storage vmotion goes faster?
Because I can't imagine that all NFS storage is that slow to migrate?
Or is there another issue?

K

[Vitaliy S.]

Hi Koen,

Not sure I fully understand the question, but it is possible to use a separate NIC for vMotion traffic.

BTW, here is an interesting article, that you might find useful: http://frankdenneman.nl/2013/02/07/why- ... n-network/

Thanks!

[kte]

hotadd is a s slow as as storage vmotion and quick migration 1/GB a minute is the max

case 00455736

[kte]

I'm calculating veeam restored in hotadd mode 100 GB in 2h 40 minutes so I'm processing @ 7MB/s any way to get the performance up correctly x10 - x20
My data is backupped on a physical server via direct san mode @ 500 MB/s but I can't restore correctly I need 20 days to restore my VM of 16 TB
I create a proxy VM for hotadd and selected in the VM restore to use this proxy to restore, but still @ 7 MB/s
Instant recovery and storage vmotion where almost the same in performance. Any idea

K

[tsightler]

There's really not that much involved in restores other than reading from the source repository and writing to the target datastore so those are really the only to things you should look at. Typically restores will be limited by target datastore performance, however, 7MB/s is quite a bit slower than what I would expect even from relatively poor storage. What type of storage are datastores and your repository. Does the storage backing your datastores have write-backup caching enabled? This is important for achieving good restore performance to VMFS.

Typical restore speeds I've seen from Veeam in the field are anywhere from 50-100MB/s for VMFS, and about 2x that speed for NFS datastores assuming reasonable hardware so something is definitely unusual in your environment.

[kte]

i have a 3 par stireserv with 80 sas disk
i have 2 smart arrays with each 10 disk behind and 2 GB cache with and 50% read write cache on my physical proxy server.
when i copy from 1 server t another ik veeam copy i get 150 MB/s from my backup server

k

[tsightler]

Then unfortunately I have no idea. Having recently tested similar equipment I can say that I would expect speeds closer to the top end (75-100MBs) rather than what you are seeing. Hopefully support will be able to find something. Just about the only other thing I can think of is if somehow there is a routed network or some other issue between you and the target that is causing a significant network bottleneck (perhaps even a network improperly configured), but the fact that you're seeing similar speed even with a virtual proxy (assuming the proxy was used and properly performed hotadd restore) would make this seem unlikely. Hopefully support can find our something for you. Overall restores with Veeam are really quite simple so there's not a lot of places to look.

One really silly thing to try, you could perhaps try restoring the VMDK using the "Restore Guest VM Files" method and see what that gives you.

[yizhar]

Hi.

I have commented to a similar post, please read and try the following:

compare restore performance of very large VM vs a small one. Any difference?
For the first test - make sure that the small VM is restored from a relatively small VBK file.

try with and without the following setting:
DataMover.HardwareAcceleratedInit

See following post for details:
Slow Restore Speed - 27MB-s - Tips-Ideas view topic
http://forums.veeam.com/viewtopic.php?f=24&t=12892

Yizhar

[kte]

tried it but no direct change, i only changed the setting on 1 esx
how does the hotadd work,also via vpower nfs , because i don't see any network traffic from the physival proxy to the virtual proxy.
or an issue in veeam 7 patch1

k

[Vitaliy S.]

Koen,

because i don't see any network traffic from the physival proxy to the virtual proxy.

There shouldn't be any traffic between proxy servers when you're restoring full VM via HotAdd mode. If virtual proxy server has the direct access to the datastore you want to restore VMs to, it will use ESXi I/O stack to write VM data to the datastore.

Instant VM recovery (vPower NFS) does not use hotadd capabilities.

Thanks!

[kte]

but the veeam repository is on my physical proxy and iwant to restor to my virtual proxy so between both proxies must be some network data traffic, no??

[Vitaliy S.]

Yes, you need to have connection between repository and the proxy server used for hotadd restores. In your case data will go from the physical server to virtual proxy and then will be written to the datastore via ESXi I/O stack.

[veremin]

Then, I’m wondering between what components you do see the traffic flow. And what backup proxy is shown in the statistics of full VM Restore session? Thanks.

[Mory]

Are you using thin provisioned disk on your 3Par? Doing restore testing for a datacenter migration I was running into issues with restore speeds being awfully slow as well. I too have a 3Par SAN and was seeing around the same restore speeds you mention. Restoring to a fresh newly created thin provisioned disks was OK the 1st time through. However, subsequent restore attempts were terribly slow. With fully provisioned ('thick') virtual volumes, there were no issues on the first or any other restore.

[kte]

I'm using thin volumes on the 3 par and also thin vmdk's

[Mory]

If you haven't tried yet, I'd suggest creating a thick volume and trying your restore again to see if that resolves the issue you are having. I'm not sure if there is a bug somewhere with the 3Par that is causing this or somewhere/something else.

However, when using thick volumes I've been able to restore around 4.5TB worth of .VBK's (default dedupe and compression settings) in about 3.5 - 4 hours (60 VM’s). For reference, I have 1 x physical Veeam server were the Veeam backup files reside and I'm using 8 x virtual Veeam proxies using hotadd (one on each host server). I haven’t really done single restore speed testing, but when running between 6 – 8 restores simultaneously, the Veeam statistics of the jobs range between 80 mb/s and 160 mb/s. Mostly in the 80 – 100 range though.

[chjones]

After pulling my hair out for weeks and through so many restore tests I could click through the Veeam wizards blindfolded we came to the exact same conclusion today with our 3par 7400.

We just stuck to network mode restores for all tests, just wanted to keep things simple and make one change at a time during the 50+ restore attempts.

When restoring to a 3par volume we kept hitting a ceiling of 11MB/sec. Restoring to esx local disks or an older EVA volume we could get 80-140MB/sec. We finally found through dumb luck this same issue of the first restore to a thin 3par volume being fast and subsequent restores hitting the 11MB/sec limit.

3par thick volumes are consistently over 140MB/sec no matter how many times we restore to them.

Converting a thin volume to thick also then restores very fast. Then converting back to thin the restores slow right down again.

Our only possible workaround for now is have a thick volume on standby for restores (large enough to receive our largest single vm) and then storage vmotion back to the thin volumes.

Strange that a storage vmotion isn't affected by this slow down.

Did anyone ever find a proper fix for this?

[kte]

3PAR FW 3.3.1 solves this issue or create an new thin lun to restore to or restore to a tick lun