Diagnosing performance

[agoldenlife]

Hey everyone,

I have read through a few posts on diagnosing performance, but wondering if I can get some help myself as I am having trouble.

Most of our job are backing up using NBD mode, and I can't seem to figure out why.

The majority of our backups report like this: 9/13/2018 8:34:31 PM :: Load: Source 98% > Proxy 15% > Network 4% > Target 0%

I would say on average we are seeing a processing rate of 20-30MB/s

So I would like to tweak these as best I can to see if I can get better backup performance. Here is a little about our setup.

we are running 3 VMs, each on its own host. One server is a Veeam server the other two setup as proxies. Each server has 8 vcores setup and 24GB of RAM. The datastore is a Nimble, and they back up to a synology but that does not appear to be the performance issue. I have seen the source as low as 75% on one job, but most are in the 90%+ range. Each server is setup with iSCSI connections to the SAN, but that doesn't appear to be used. I think we need the enterprise plus license to take advantage of that.

I also can't seem to figure out why they don't hotadd. 1 or 2 jobs will. I saw only one backup that had an error about hot add. The rest just showed in the console as NBD. Let me know if this should be a ticket. I will likely enter one anyway, just in case, but thought I would reach out to the community. I am trying to figure out if it is a configure issue, hardware issue, me issue .

I would like to make sure I understand what is going on here and understand Veeam at a more fundamental level. Like the Load/Performance and what it should look like. I have been pouring over the docs a lot!

Thanks,

[foggy]

Hi James, the first step is to check if any of the hotadd limitations apply. Thanks.

[agoldenlife]

Thanks Foggy!

I looked through all that, and everything is as expected. Here are my checks/responses to the KB:

1. No, all datastores are formatted vmfs6 except the vvol of course (Vvol is the storage for all VMs)
2. Check... everything resides on the same datastore
3. NA
4. NP all under 62TB VMDK files
5. Check all VM's use SCSI HDs
6. Vmtools is outdated as of today. So I don't think this would be an issue, but let me know.
7. Check, enough space for a VM snapshot.
8. Check we are running vSphere 6.7

Known Issue check:
• All files are on the Vvol so should not be an issue.
• Check all HDs for hardware version will be difficult. I would have a hard time thinking that the majority of our servers are running different hardware versions between disks. Maybe a couple.
• I added 2 scsci controllers on all veeam servers so we can get more disks. But I don't believe this necessary as the jobs are not processing that many disks. The very first jobs run in NBD mode.
• NA - no standalone host connection. All though vcenter
• NA
• Not using NFS datastore for VMs disks
• Have not cloned or restored our VBR server
• NA, as we don't template systems.

• I would hesitate to go through 100+ servers and attempt to hotadd each disk. Something is more fundamentally wrong.
• No hot-added disks that have not been released.
• Already disabled auto-mount on all 3 servers.

What can I check next?

[vmJoe]

If you want to to use the HotAdd data transport method, then each proxy has to see the datastore that the VM's vmdk(s) are sitting in. If you have Multiple clusters in your environment (and the clusters segregate your Nimble LUNS into different datastores on different clusters) then you will need a proxy (at least) in each cluster.

one way to troubleshoot this is to go into the properties of the proxy and select the Virtual Appliance (hotadd) mode and uncheck the fail over to network mode selection. this will cause teh job to fail f the proxy that is being selected to be used is not in the same cluster as the VMDKs that are getting backed up.

[csydas]

Hey James,

I'd open a ticket. From my experience in our shop, usually there is a very clear reason why HotAdd doesn't get used, and it usually means time pouring over the logs. I got lucky our first time with this and had a very helpful support person explain to me exactly what she saw in the logs, and we've been able to use that in the past to figure out where the decision point came to not use hotadd.

[skrause]

One thing to remember about the bottleneck "percentages" is that they are where Veeam is seeing the 'slowest' operation. Every job will have some part that is the "primary bottleneck" but that may not be the cause of your performance issues. In my experience, most properly functioning jobs will have Source as the primary bottleneck (usually by a large margin.)