Discussions specific to the VMware vSphere hypervisor
Post Reply
Posts: 17
Liked: 1 time
Joined: Mar 03, 2014 5:45 pm
Full Name: James Golden

Diagnosing performance

Post by agoldenlife » Sep 14, 2018 2:41 pm

Hey everyone,

I have read through a few posts on diagnosing performance, but wondering if I can get some help myself as I am having trouble.

Most of our job are backing up using NBD mode, and I can't seem to figure out why.

The majority of our backups report like this: 9/13/2018 8:34:31 PM :: Load: Source 98% > Proxy 15% > Network 4% > Target 0%

I would say on average we are seeing a processing rate of 20-30MB/s

So I would like to tweak these as best I can to see if I can get better backup performance. Here is a little about our setup.

we are running 3 VMs, each on its own host. One server is a Veeam server the other two setup as proxies. Each server has 8 vcores setup and 24GB of RAM. The datastore is a Nimble, and they back up to a synology but that does not appear to be the performance issue. I have seen the source as low as 75% on one job, but most are in the 90%+ range. Each server is setup with iSCSI connections to the SAN, but that doesn't appear to be used. I think we need the enterprise plus license to take advantage of that.

I also can't seem to figure out why they don't hotadd. 1 or 2 jobs will. I saw only one backup that had an error about hot add. The rest just showed in the console as NBD. Let me know if this should be a ticket. I will likely enter one anyway, just in case, but thought I would reach out to the community. I am trying to figure out if it is a configure issue, hardware issue, me issue :).

I would like to make sure I understand what is going on here and understand Veeam at a more fundamental level. Like the Load/Performance and what it should look like. I have been pouring over the docs a lot!


Veeam Software
Posts: 18551
Liked: 1606 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Diagnosing performance

Post by foggy » Sep 14, 2018 4:05 pm

Hi James, the first step is to check if any of the hotadd limitations apply. Thanks.

Posts: 17
Liked: 1 time
Joined: Mar 03, 2014 5:45 pm
Full Name: James Golden

Re: Diagnosing performance

Post by agoldenlife » Sep 14, 2018 5:31 pm

Thanks Foggy!

I looked through all that, and everything is as expected. Here are my checks/responses to the KB:

1. No, all datastores are formatted vmfs6 except the vvol of course (Vvol is the storage for all VMs)
2. Check... everything resides on the same datastore
3. NA
4. NP all under 62TB VMDK files
5. Check all VM's use SCSI HDs
6. Vmtools is outdated as of today. :) So I don't think this would be an issue, but let me know.
7. Check, enough space for a VM snapshot.
8. Check we are running vSphere 6.7

Known Issue check:
• All files are on the Vvol so should not be an issue.
• Check all HDs for hardware version will be difficult. I would have a hard time thinking that the majority of our servers are running different hardware versions between disks. Maybe a couple.
• I added 2 scsci controllers on all veeam servers so we can get more disks. But I don't believe this necessary as the jobs are not processing that many disks. The very first jobs run in NBD mode.
• NA - no standalone host connection. All though vcenter
• NA
• Not using NFS datastore for VMs disks
• Have not cloned or restored our VBR server
• NA, as we don't template systems.

• I would hesitate to go through 100+ servers and attempt to hotadd each disk. Something is more fundamentally wrong.
• No hot-added disks that have not been released.
• Already disabled auto-mount on all 3 servers.

What can I check next? :)

Veeam Software
Posts: 360
Liked: 76 times
Joined: Aug 02, 2011 1:06 pm
Full Name: Joe Gremillion
Location: Dallas, TX USA

Re: Diagnosing performance

Post by vmJoe » Sep 15, 2018 3:43 pm

If you want to to use the HotAdd data transport method, then each proxy has to see the datastore that the VM's vmdk(s) are sitting in. If you have Multiple clusters in your environment (and the clusters segregate your Nimble LUNS into different datastores on different clusters) then you will need a proxy (at least) in each cluster.

one way to troubleshoot this is to go into the properties of the proxy and select the Virtual Appliance (hotadd) mode and uncheck the fail over to network mode selection. this will cause teh job to fail f the proxy that is being selected to be used is not in the same cluster as the VMDKs that are getting backed up.
Joe Gremillion
NA Core Solutions Architect - Central region

Posts: 193
Liked: 47 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel

Re: Diagnosing performance

Post by csydas » Sep 16, 2018 8:35 am

Hey James,

I'd open a ticket. From my experience in our shop, usually there is a very clear reason why HotAdd doesn't get used, and it usually means time pouring over the logs. I got lucky our first time with this and had a very helpful support person explain to me exactly what she saw in the logs, and we've been able to use that in the past to figure out where the decision point came to not use hotadd.

Posts: 446
Liked: 93 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause

Re: Diagnosing performance

Post by skrause » Sep 18, 2018 3:12 pm

One thing to remember about the bottleneck "percentages" is that they are where Veeam is seeing the 'slowest' operation. Every job will have some part that is the "primary bottleneck" but that may not be the cause of your performance issues. In my experience, most properly functioning jobs will have Source as the primary bottleneck (usually by a large margin.)
Steve Krause
Veeam Certified Architect

Post Reply

Who is online

Users browsing this forum: No registered users and 36 guests