Hotadd slow some of the time (source bottleneck)

VMware specific discussions

Hotadd slow some of the time (source bottleneck)

Veeam Logoby doggatas » Mon Aug 22, 2016 11:14 pm

We are in the process of building a new environment (IAAS: we control of hypervisor+VMs) consisting of 5 esx (ESXi 5.5 update 3) hosts and a Gen2 V7000 Storage array, which has a mirror at an alternate site using dark fibre. The LUNS configured on the SAN (and in turn formatted to VMFS 5 Datastores) are backed either by RAID10 with SSD EasyTier (32 HDD 10K SAS + 4 SSD) and a DRAID 6 with 46 disks. Connectivity between the hosts and storage is FC. At this stage, we do not have a physical backup server and we're looking to utilise Hotadd mode.

The Veeam Backup and Replication server has 12 vCPU and 24Gb Ram and is both repository and proxy (I have tested with a separate proxy, which made no difference). The problem we are having is that most of the time the backup jobs run very slow, at less than 70MB/sec. However, on occasion, our test jobs will run at > 200MB/sec. I have disabled NDB failover to eliminate the possibility the jobs might be using NDB. I have also forced NDB mode to test, but the jobs ran even slower.

The source is indicated as bottleneck with 99%. I must stress, this is new infrastructure and has ZERO load at the moment. The only VMs deployed are vCenter(Windows, not appliance), VeeamOne, Veeam B&R and a SQL server.

I have also ran the VDDK tool which is able to attain 200-600MB/sec read benchmark, but yes, sometimes the speed is slow here too, but it's the opposite of a backup test. I.e fast most of the time. I have also performed a diskspd benchmark and again, 600MB/Sec read speed.

Anyone have any suggestions/tweaks? Aside from using DirectSAN, which had been suggested by Veeam support.
doggatas
Enthusiast
 
Posts: 30
Liked: 2 times
Joined: Tue Jul 24, 2012 1:15 am
Full Name: David O

Re: Hotadd slow some of the time (source bottleneck)

Veeam Logoby DavoudTeimouri » Sun Aug 28, 2016 7:27 am

How many jobs are running at same time?
Best Regards
DavoudTeimouri
Influencer
 
Posts: 24
Liked: 3 times
Joined: Wed Nov 27, 2013 7:58 am
Full Name: Davoud Teimouri

Re: Hotadd slow some of the time (source bottleneck)

Veeam Logoby dellock6 » Mon Aug 29, 2016 12:10 pm

As the Veeam server is at the same time proxy and repository, and is a virtual machine, am I guessing right that you are using its virtual disks to store the backups? If yes, are they running over the same disk pools of the protected virtual machines?

If yes to both, this may impact performance, as the same IOPS are consumed to read data from vm and to write data into Veeam vmdk. Source is the bottleneck probably because in reality the load on the storage is doubled by the local writes.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5055
Liked: 1336 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 21 guests