Restore speed entire VM?

[briggs]

Hi,
I am very new to Veeam, and we're just doing some test restores to ensure we're getting good backups.
I've done a restore on a VM that we've backed up. It's 60GB, but Veeam got the backup files down to 22GB.
When I restore this, across a 1Gbps network, it seems to take a long time. The speed shows 12MB/s.
Should I expect much faster than this? I can't help thinking if I lost a critical server (or all the servers if the SAN failed!) then I'd be doing restores for days on end at that speed, as some of our VM's are 600GB+

TIA, Alan

[briggs]

OK, just tried restoring the same VM to another host, and it's almost 3 times the speed.

Basically, we have ESXi 5.1 essentials plus, we have a 2 host cluster (2 x DL380 G8, with 2 x 6 core CPU@2Ghz) and a 3rd host that isn't in the cluster (DL380 G7, 2 x 4 core CPU@2.6Ghz). The 2 hosts are on an ISCSI storage network to a P2000 SAN. The 3rd host is a standalone server, with 8 x 300GB SAS disks in RAID5. The backups are on a QNAP NAS box (1 Gbps NIC).

Is the restore speed being limited by the network? Or the local storage? We're using the same Veeam server to run the restore, and that Veeam is located on the ISCSI storage.

Just trying to establish some benchmarks, if I know that it's way too slow, then I need to sort it now. If it's normal (hope not!) then I guess I accept that full server restores are going to take a long time!

Thanks, Alan

[foggy]

Alan, it most likely depends on whether you have a backup proxy installed on the host to which you are restoring. In this case hotadd transport mode is used to populate VM disks, otherwise the VM is restored over network.

[chrisdearden]

Where is your backup server ? is it a VM on the cluster or a standalone. When you are doing the restore , whats the bottleneck stats showing ? The QNAP is probably going to be the rate limiting step , either in terms of raw performance or the single NIC.

[briggs]

Unless I'm missing something, there is no bottleneck shown on the stats?

[chrisdearden]

I'm probably thinking of backup - though bottleneck stats for restores would be handy. If your Veeam server is a VM , then it will be using hotadd for the VM restore on the host of cluster it is running on. Restore to the other host/cluster would be via network mode. This goes over the management NIC of the hosts.

[briggs]

OK, thanks for the replies people. I don't suppose anybody can tell me what is 'slow'? I mean, I have nothing to go on here, it's not as fast as I'd like even to the live cluster than runs the Veeam VM. Cheers, Alan

[chrisdearden]

Have you tried doing an instant restore ?

[pcrebe]

Why don't use the third host for replicate the vm's getting a best RTO? I do that with the veeam vm on the third host to perform failover and failback with it.
I backup the vm's only for another assurance and for backup the third host.

[tsightler]

By far the biggest impact on restore performance is VMFS synchronous write semantics. As far as I can tell all writes performed via the VDDK for restore are synchronous which means that all writes must be committed to persistent storage by the storage system before being acknowledged. Synchronous writes are historically very slow and many arrays attempt to buffer them with BBWC or redundant NVRAM. This works quite well as long as syncronous traffic comes in bursts (typcal for normal workloads) but falls over when there's an abundance of writes that overrun the buffer, such as during a full VM restore.

The best way to monitor this is to look at the datastore latency from the host that is actually performing the restores. For "hotadd" this would be the ESX hosts on which the proxy that is performing the restore is running. For "network" (NBD) mode, it should be the host to which you are restoring the VM. I've seen cases where the latency spikes to 100's or even 1000's of milliseconds during the restore because of these synchronous writes.

If you need to restore a large VM quickly, instant restore is a great feature that can really help, and then you can use sVmotion to get VMs back to their "real" datastores. It's exactly the issue IR was created to solve.

I would personally consider 12MB/s to be very slow for restore, although without knowing more about the storage, and the latency you are seeing on the datastore, it's hard to know. In my home lab I get about 10MB/s with my ZFS based iSCSI SAN when I don't use an SSD for the ZIL, and datastore latency spikes to 700ms. On the other hand, if I do enable an SSD for ZIL to cache the synchronous writes then I get about 60-65MB/s for restores on the same datastore and latencies stay sub-20ms.

[DesRDD]

Alan,
Did you find a resolution to this? We are seeing the exact same behavior. Two hosts in 4.1 cluster with restore rates of 12MB/s regardless if we restore to the SAN or local storage on the hosts. Datastore latency during restores is 15 max and averages 1.7 so I don't think that is the issue. However, if we restore to a 3rd host (standalone with local storage), we get 84MB/s every time. Our Veeam server is physical so it's the same Veeam server through the same network switches. The only difference is pointing the restore to the cluster (getting 12MB/s) or the standalone host (84MB/s). I understand the comment about better RTO via replication, but we would like to get this resolved if we can.

[briggs]

I have not found a solution no, sorry. I'm trying to, and have spent £1000 on disks for the backup target (QNAP) so I can do RAID10, but to be honest, I'm regretting that already as the backup speed is pretty much the same as it was with RAID5 disks, which suggests either the QNAP itself has a difficulty processing the data, or the bottleneck of "target" is a lie!
Will obviously post any findings, hopefully there's more suggestions coming?

[briggs]

Can anyone shed any light on this? Am I to assume that 30MB/s is the best sort of performance I can expect when doing a restore? To re-itterate, this is restoring from a QNAP 459 ProII (RAID10) over a 1Gbps network connection (same switch) to a standalone VMWare ESXI host, an HP DL380 G7 with 8 x 10K disks (RAID5). Tried with a local proxy too (ie a VM located on this host, running as a Veeam proxy) and it makes no difference.

[Vitaliy S.]

To re-itterate, this is restoring from a QNAP 459 ProII (RAID10) over a 1Gbps network connection (same switch) to a standalone VMWare ESXI host, an HP DL380 G7 with 8 x 10K disks (RAID5).

Btw, can you please tell me the upload file speed if you try to upload any big file from the backup storage via vSphere Client to the target host?

[briggs]

Vitaliy, you mean from the QNAP to the VMWare host server's datastore? Or by target host, do you mean a VM that's being backed up? Or the Veeam VM? Sorry, just needed clarification on what you mean by that. Thanks, Alan

[Vitaliy S.]

I was referring to VMware host datastore. Can you upload any file from the place where you store your backups, the QNAP device, to the VMware datastore via vSphere Client?

[briggs]

OK, shall I add the QNAP as a datastore first, and copy from that datastore to the other datastore? Or just upload something via my windows client, using UNC path?

[Vitaliy S.]

The second variant.

[briggs]

Hi Vitaliy, I will do this today, I've since opened a support case on this issue. Do you want me to time this, or are you expecting it to not work or something? I'm not sure what the purpose of the test is?

[foggy]

The purpose of the test is to measure the upload speed and compare it with the one you get during restore so that we could probably exclude Veeam B&R from the equation.

[briggs]

OK, I guessed this was the test, though I can't figure out why you'd do this from the vSpehre client on a desktop PC? Surely that would be the "go-between" and slow the copy down? Does the PC not request the PC from the QNAP, then upload it to the datastore? ie 2 operations?

[briggs]

OK, I've added the QNAP as a datastore, and I've copied/pasted a file from the QNAP to the datastore, and it takes a long long time to complete. But then we've always had really slow progress when browsing a datastore through vCenter? As I say, if you have a VM running on the datastore, then you use that client OS to copy the files, we can get over 100MB/s?

[Vitaliy S.]

OK, I guessed this was the test, though I can't figure out why you'd do this from the vSpehre client on a desktop PC?

Foggy is correct, I just wanted to see what is the upload speed of your ESXi server datastore. Since all restores are performed via network stack of the ESXi server (unless you have a virtual proxy on the host you're restore VMs to), the performance of these two operations should be the same.

[infused]

It will either be the backup proxy or qnap.

We use a qnap. What model do you have? Don't use iSCSI on a qnap unless you have i3 or i5 processor in the qnap. It uses too much cpu and will slow down the transfer speed. We just use a network share on the qnap. Much faster.

[abradley]

We too use QNAP units as our backup points. We have TS-859U-RP+ units that have Atom processors.
We see similar results in file transfers and we are using a normal file share and not iSCSI.

Similar setup(s) on some of our sites, using the Essentials Plus licenses at our 5 remote sites and a standard VMWare license at our Corporate Datacenter and DR facility.
My thoughts on restores are to use the instant restore feature and once it is up it will be very, very slow but it will run.
Then we can use the "shared nothing vmotion" to move the VM's back to whatever device we have available:
http://blogs.vmware.com/vsphere/2012/09 ... -name.html

I have not had a chance to test this as we are just getting moving with Veeam as our backup/DR solution. Has anyone else done/tested this with QNAP units?

I am really waiting on the new WAN accelerated backup copy jobs in v7 - we will be using that to replicate our VM's between the QNAP devices at all locations for off-site DR purposes.

[Rumple]

Have you created a vm on each host that's 4vcpu and added that vm into veeam under the proxy settings.
If bot then you are missing 90% of the performance capabilities of veeam

[veremin]

Then we can use the "shared nothing vmotion" to move the VM's back to whatever device we have available:

In fact, you can always use VB&R in order to finalize instantly recovered VMs and let it decide what migrating mechanism should be used in your case: be it vMotion (VMware) or SmartSwitch one (Veeam).

All you need to do is to right-click instantly recovered VM and select “Migrate to production”.

Thanks.

[gingerdazza]

I think it's a real shame that there's no OS -filtering in the GUI

[PTide]

I think it's a real shame that there's no OS -filtering in the GUI

If you go to "Virtual Machines" tab you'll see "Guest OS" column which can be sorted if you click on its top.

Thank you.