Discussions specific to the VMware vSphere hypervisor
Post Reply
doggatas
Enthusiast
Posts: 38
Liked: 2 times
Joined: Jul 24, 2012 1:15 am
Full Name: David O
Contact:

Hotadd slow some of the time (source bottleneck)

Post by doggatas » Aug 22, 2016 11:14 pm

We are in the process of building a new environment (IAAS: we control of hypervisor+VMs) consisting of 5 esx (ESXi 5.5 update 3) hosts and a Gen2 V7000 Storage array, which has a mirror at an alternate site using dark fibre. The LUNS configured on the SAN (and in turn formatted to VMFS 5 Datastores) are backed either by RAID10 with SSD EasyTier (32 HDD 10K SAS + 4 SSD) and a DRAID 6 with 46 disks. Connectivity between the hosts and storage is FC. At this stage, we do not have a physical backup server and we're looking to utilise Hotadd mode.

The Veeam Backup and Replication server has 12 vCPU and 24Gb Ram and is both repository and proxy (I have tested with a separate proxy, which made no difference). The problem we are having is that most of the time the backup jobs run very slow, at less than 70MB/sec. However, on occasion, our test jobs will run at > 200MB/sec. I have disabled NDB failover to eliminate the possibility the jobs might be using NDB. I have also forced NDB mode to test, but the jobs ran even slower.

The source is indicated as bottleneck with 99%. I must stress, this is new infrastructure and has ZERO load at the moment. The only VMs deployed are vCenter(Windows, not appliance), VeeamOne, Veeam B&R and a SQL server.

I have also ran the VDDK tool which is able to attain 200-600MB/sec read benchmark, but yes, sometimes the speed is slow here too, but it's the opposite of a backup test. I.e fast most of the time. I have also performed a diskspd benchmark and again, 600MB/Sec read speed.

Anyone have any suggestions/tweaks? Aside from using DirectSAN, which had been suggested by Veeam support.

DavoudTeimouri
Enthusiast
Posts: 50
Liked: 6 times
Joined: Nov 27, 2013 7:58 am
Full Name: Davoud Teimouri
Contact:

Re: Hotadd slow some of the time (source bottleneck)

Post by DavoudTeimouri » Aug 28, 2016 7:27 am

How many jobs are running at same time?
Best Regards

dellock6
Veeam Software
Posts: 5754
Liked: 1644 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Hotadd slow some of the time (source bottleneck)

Post by dellock6 » Aug 29, 2016 12:10 pm

As the Veeam server is at the same time proxy and repository, and is a virtual machine, am I guessing right that you are using its virtual disks to store the backups? If yes, are they running over the same disk pools of the protected virtual machines?

If yes to both, this may impact performance, as the same IOPS are consumed to read data from vm and to write data into Veeam vmdk. Source is the bottleneck probably because in reality the load on the storage is doubled by the local writes.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Post Reply

Who is online

Users browsing this forum: No registered users and 37 guests