Hotadd slow some of the time (source bottleneck)

[doggatas]

We are in the process of building a new environment (IAAS: we control of hypervisor+VMs) consisting of 5 esx (ESXi 5.5 update 3) hosts and a Gen2 V7000 Storage array, which has a mirror at an alternate site using dark fibre. The LUNS configured on the SAN (and in turn formatted to VMFS 5 Datastores) are backed either by RAID10 with SSD EasyTier (32 HDD 10K SAS + 4 SSD) and a DRAID 6 with 46 disks. Connectivity between the hosts and storage is FC. At this stage, we do not have a physical backup server and we're looking to utilise Hotadd mode.

The Veeam Backup and Replication server has 12 vCPU and 24Gb Ram and is both repository and proxy (I have tested with a separate proxy, which made no difference). The problem we are having is that most of the time the backup jobs run very slow, at less than 70MB/sec. However, on occasion, our test jobs will run at > 200MB/sec. I have disabled NDB failover to eliminate the possibility the jobs might be using NDB. I have also forced NDB mode to test, but the jobs ran even slower.

The source is indicated as bottleneck with 99%. I must stress, this is new infrastructure and has ZERO load at the moment. The only VMs deployed are vCenter(Windows, not appliance), VeeamOne, Veeam B&R and a SQL server.

I have also ran the VDDK tool which is able to attain 200-600MB/sec read benchmark, but yes, sometimes the speed is slow here too, but it's the opposite of a backup test. I.e fast most of the time. I have also performed a diskspd benchmark and again, 600MB/Sec read speed.

Anyone have any suggestions/tweaks? Aside from using DirectSAN, which had been suggested by Veeam support.

[DavoudTeimouri]

How many jobs are running at same time?

[dellock6]

As the Veeam server is at the same time proxy and repository, and is a virtual machine, am I guessing right that you are using its virtual disks to store the backups? If yes, are they running over the same disk pools of the protected virtual machines?

If yes to both, this may impact performance, as the same IOPS are consumed to read data from vm and to write data into Veeam vmdk. Source is the bottleneck probably because in reality the load on the storage is doubled by the local writes.