Veeam Backup slow in network mode (99% source bottleneck)

[stefanpeters]

Hello,

I am a new Veeam Backup & Replication user. I have used v5 for a week or so and then there was v6.
I have a new Vmware esx5 server, with RAID10 over 8x 600GB 15.000rpm harddisks. The server has two XEON X5650 CPUs with 6 Cores each and 72GB of RAM and an LSI Controller with Cache. All hardware is on the Vmware HCL.
I have a single GBIT VMKernel connection to a HP Gbit switch. This link show as 1Gbit on server and switch.

The Veeam Backup server is running with 6x 2TB 7200rpm in RAID5 with caching enabled. This server is running a fresh copy of Windows 2008R2 standard and has 8GB RAM. It has two Xeons E5345 @ 2.33Ghz. Taskmanager shows 8 Logical CPUs.

When I run sequential benchmark on the Veeam Backup server on the RAID5 volume it gives me a read speed of over 300MBytes/Sec and almost 250Mbytes/Sec write speed. The server is connected with gbit on the same switch even in the same module. This link show as 1Gbit on server and switch.

I have installed several Windows VMs on the ESX5 host. One of them is a new fileserver with dummy data, it is not operational yet in production. When I benchmark in the VM the data partition which is located on the RAID10 (8x600gb 15.000rpm) it gives me Disk read speed of almost 500Mbytes/Sec and write speed of over 350Mbytes/sec.

The disks in both servers seem very fast.

When I run a backup of a single VM with Veeam Backup and enable Application Aware and Guest File system indexing the Veeam server gives me very poor speed. I am crying when I see the figures. The backup speed is under 1 MB/sec. And v6 is giving a 99% source bottleneck. Taskmanager shows little CPU usage.

When I disabled the Application Aware and Guest file system indexing the speed increases a little to 6MB/sec. This is still far less then I would expect. Here the source is also 99% bottleneck.

I have read the v6 FAQ and understand that there is always a bottleneck but backupping under 1MB/sec is just ridiculous.
People on this forum are complaining with 50Mbytes/sec but I would wish I'd get that speed.

To be clear I have no SANs just two servers, one running VMware esx5 and one running Windows 2008r2 sp1 with Veeam v6 as of yesterday.

Because Veeam blames the source for the bottleneck, what can I pimp in esx5 to increase the backup speed?

Please help me !!!

Thanks
Stefan

[foggy]

Stefan, have you considered different transport modes available in Veeam B&R for source data retrieval? Your backup is performed using the slowest method possible (Network mode) while if you install Veeam B&R on a virtual machine, you will be able to use Virtual Appliance mode getting direct access to VMs data through ESX server I/O stack.

[stefanpeters]

Thanks Foggy for your reply. I can install VBR in a VM but the storage is in a seperate physical server. Then the files will go over shared folder?
Also what if my esx host crashes and I need to restore from the backup, the VM running VBR is also crashed.
But do you mean that with Network Mode I can expect Backup transfers less then 1MB/sec and consider that normal or should I be able to get higher speeds?

That aside. I am now testing file transfer speed with VMware vSphere client to copy files of the Datastore back to the Windows Server where the VMWare client is installed. This failes with an I/O error. Perhaps that is a source of problem.

[Gostev]

Uhm, yes perhaps

v6 is giving a 99% source bottleneck

Network mode may give you up to 25MB/s performance, usually less (but always at least 5 MB/s in my experience). So even in ideal circumstances, it is still quite slow. By all means you should leverage virtual proxy for hot add processing, this should bump your speed signficantly.

On top of that, you clearly have some networking infrastructure issues (1MB/s and network copy error are pretty obvious pointers).

[stefanpeters]

I have found several issues with the vmware client and datastore browser. I have tried to transfer files using the Web Access to the ESXi file system:
https://192.168.0.2/folder?dsName=datastore1 (with this url)

This works perfectly and I can download files from the datastore at 30MB/Sec. I think it is limited on the 30, because it is nice and steady. I have downloaded several files from 600MB - 3GB and they all download perfectly.

Also network transfers from the Veeam backup server to other server go perfectly on speed.

Can anyone give me some figures for Network Mode backup of let's say a 50GB static VMDK server?
With 30MB/Sec it should not take very long because the 50GB are not full.
The Veeam Backup statistics keep saying about 3-6MB/sec or less.

Is there anyway to do a benchmark other then downloading files from the datastore. Some benchmark tool that measures the same protocol as VBR uses?

[Gostev]

Veeam B&R uses vStorage API for Data Protection. For network processing mode, vStorage API uses NBD (Network Block Device) protocol. NBD, in turn, leverages internal ESX(i) host protocol called NFC (this one is not public).

This NFC protocol is what vSphere Client's datastore browser relies on as well, so vSphere Client is your best benchmark for Veeam B&R "Network" processing mode.

If you have performance issues and errors using vSphere Client's datastore browser, I would recommend opening the case directly with VMware.

As far as expected performance figures for NBD, see my previous post.

By any chance, are you using ESX 4.0?

[stefanpeters]

Okay thanks for the info. I am going to do a test in a VM, see what happens.
I am using esx5.
The VSphere client only failes on the Veeam Backup server, running it from my laptop, over wifi work perfectly. The BU server has a gbit connection over a gbit switch, and I have also tried crosslinking the gbit, no difference there, but that eliminated the switch. So I think the problem is on the veeambu server. Going to investigate further.

[Gostev]

For best results, I would highly recommend leveraging virtual proxy server in the virtual appliance (hot add) processing mode instead. This would beat the "ideal" performance of network mode in a few times. Thanks.

[stefanpeters]

Who now I am back in being impressed. I have done a test this morning on my server of backupping 1 VM, it took in Network mode over 30 minutes. Now with a virtual proxy it took 6 minutes. Now we're talking.

Question that I have left and could not find in the manual very quickly. Do I need a virtual proxy on every ESX host. All my ESX storage is local?

I have ESX5.

[Gostev]

Yes - with local storage, you need at least 1 virtual proxy per host. The good thing is that you can re-use any existing Windows server to be your proxy.
With shared storage, you need at least 1 virtual proxy per vSphere cluster (or more - for load-balancing and fault tolerance).

[henno]

I can install VBR in a VM but the storage is in a seperate physical server. Then the files will go over shared folder?
Also what if my esx host crashes and I need to restore from the backup, the VM running VBR is also crashed.

I would also like to get an aswer to these two questions.

[Vitaliy S.]

Yes, all the files will be written over your network if target storage is a separate physical server.

As to possible restore options, then take a look at the existing topic which covers them all:
Restoring Backup files from another server