Slow processing rate?

[theviking28]

I have 2 esxi 5.5 hosts in a cluster. I have them both directly connected to a SAN via SAS cables. I have 2 vdisks running in this SAN. All disks are 600gb 10k SAS disks. One vdisk is made of 3 disks and runs in raid 5. One vdisk is 16 drives and runs in raid 6.

I have 2 backup jobs that run nightly and have been going fine other than speed. I do have a ticket in with VEEAM support, but they have currently advised me to talk with VMware which I am doing about the issue.

The stance at the moment is that all VMs are backing up in network mode, and not in hot add mode. VEEAM says this issue is on VMwares end, and that's fine with me.

My questions are mainly the following...

1. What is a typical speed range for incremental and full backups for veeam in an environment like mine? I just feel like 35-45MB/sec seems pretty slow. (I have a 2tb vm that is taking like 6 hours to backup each night.) When I add more larger vms to my environment this is going to get out of hand it seems like.

2. Veeam always reports to me that the "source" is the bottleneck, yet the source is 16 10k drives in raid 6... I understand raid 6 is slower than other types, but 16 spindles 10k sas drives should be ok??

3. What are the "conditions" that must exist for "hot add" backups to take place? What sort of speed improvement could someone expect by going from network mode to hot add mode? I'd be happy with 80MB/sec which would be double my current rate... Even 70MB would be good.. 35-40 is so slow.

[Gostev]

1. The speed you are getting is quite normal for network processing mode on 1 Gbps Ethernet. Using more concurrent tasks may improve overall processing time dramatically though.
2. Source bottleneck means data retrieval speed, not storage speed. The real bottleneck here is Network processing mode, not storage. VMware throttles management interface on ESXi.
3. Please review the sticky FAQ topic regarding the conditions required for "hot add" backup to take place. The speed increase for switching to hot add more will be very significant. Hard to predict, but certainly at least double the speed you are currently getting.

[theviking28]

"Hot Add transport is not supported for backup server VM with SAS adapters."

Does that mean I can't use hot add then? My esxi servers are directly attached to the SAN with sas cables... I don't know if that means exactly that, or is talking about something different...

[Gostev]

VM with SAS adapter ≠ ESXi with SAS adapter
ESXi storage connectivity does not matter...

[theviking28]

http://www.veeam.com/kb1184

I spoke with VMware and they followed this document and showed me that doing a hot add manually worked just fine. When I was on the phone with Veeam we tried and it wasn't working, but the key was that you have to select "Independent and nonpersistent" when you add the disk to the vm. The veeam rep wasn't doing that.

Anyway, the veeam backups still fail over to network mode. The last sentence in the above kb article says that if you are able to add disks to the veeam server manually, then virtual appliance mode should work... I read through the conditions for hot add as well and don't see anything that would not meet the requirements aside from the size of one of my vms... That wouldn't effect other vms or at least not other jobs entirely I wouldn't think?

[Gostev]

Please troubleshoot the failover to network with support. Debug logs should definitely shed some light on the issue. Thanks!