Slow intermittent iSCSI performance

[wigglez]

Hi all,

Current setup is as follows:
2 x VMWare ESXi 5.5 hosts in a HA cluster. 7 VMs
1 x Physical HP ML350p w/ Windows Server 2008 R2 & VEEAM 9.5 installed
1 x QNAP TS-653A Backup Target
HP Core switching
Support ID 00547589

QNAP is iSCSI attached to ML350p via a single 1GB NIC currently. When running backup or backup copy jobs we are seeing intermittent throughput between the server and the NAS as per below pic.


VEEAM is reporting the souce is the bottleneck but the QNAP is also reporting very low load.


I have applied the autotuning tweak to the server with no change in performance. Switches are not reporting any issues, and have other services and the VMWare cluster which have no issues. No workstations are reporting speed issues, and I can generally copy data to and from the NAS at a good speed from within the Windows server (and without the drops of transfer rate).


I would be happy with slow but constant performance, as I'm not expecting the world but the constant drops to 0KB speed are making life extremely difficult.

Can anyone point me in the right direction?

[PTide]

Hi,

What are the proxies that you use - VM appliances or VBR default proxy? What proxy selection settings do you have? Is it a new setup or it's been working OK until now?

Also, have you tried to expose QNAP as a share instead?

Thanks

[wigglez]

Hi PTide,

Have got 2 proxies, one local and one VMWare proxy located on the vCenter server.

- Have selected the VMWare proxy for the Virtual Machines to QNAP backup job with Storage Optimization set to LAN Target and compression set to Optimal.
- The storage copy job I have manually selected the QNAP as the source, and it copying to a local USB hard disk plugged into the backup server for a seed for offsite cloud connect backup.

The issue has been ongoing, but trying to take the seed has prompted me to create the post as the seed copy job was taking an exceptionally long time to create a full.


I had another client with same issues but different model of QNAP and we had to move them across to a CIFS share eventually due to this same apparent issue. I would prefer to leave this setup as iSCSI if possible.

[wigglez]

I thought I would do a test, and copy the vull VBK through Windows from the iSCSI drive to a SMB/CIFS share on the QNAP. Found that I do not experience the same performance, and actuall get solid 75MB/s transfer rate in Windows as per below.


So it seems the issue is with the way VEEAM is handling the traffic, almost like it is buffering and then spitting out some traffic.

[PTide]

So, you have one proxy on a VM located on the same machine where vCenter is and the other one is VBR itself. If I got your config right then the data flow is something like this:

Source disk ----> Source Proxy (VMware VM) ----> target proxy (Veeam Server) ---iscsi---> QNAP

I would check NIC stats on both proxies during backup operations to see if those spikes also show up somewhere. I guess there might be a situation that one of the proxies just sits there waiting for another portion of data to send over the network.

P.S. Also please keep in mind that if such a situation occurs that all you're left with is the QNAP it would be easier to access it via CIFS than iSCSI.

Thanks

[nmdange]

Those gaps would indicate Veeam isn't actually transferring any data. So it might not a performance issue with the data transfer. One thing I've discovered in regards to the bottleneck stats, it does not take into account time spent waiting on a resource that has some kind of concurrency limit (such as a backup proxy). If possible, can you watch the backup while it is happening to see if any of the VMs have a status like "Resource not ready: Backup Proxy"?

I really wish the backup log or bottleneck stats could show this after the fact. The most you can see is looking at the timestamp between these events:
7/6/2017 3:26:25 AM :: Queued for processing at 7/6/2017 3:26:25 AM
7/6/2017 3:28:51 AM :: Required backup infrastructure resources have been assigned
7/6/2017 3:34:53 AM :: VM processing started at 7/6/2017 3:34:53 AM

However there's no way to see what Veeam was waiting on for the step "Required backup infrastructure resources have been assigned", at least that I am aware of.

[Pat490]

I have applied the autotuning tweak to the server with no change in performance. Switches are not reporting any issues, and have other services and the VMWare cluster which have no issues. No workstations are reporting speed issues, and I can generally copy data to and from the NAS at a good speed from within the Windows server (and without the drops of transfer rate).

what "autotuning tweak" are you referring too?