Slow intermittent iSCSI performance

Availability for the Always-On Enterprise

Slow intermittent iSCSI performance

Veeam Logoby wigglez » Wed Jul 05, 2017 6:37 am

Hi all,

Current setup is as follows:
2 x VMWare ESXi 5.5 hosts in a HA cluster. 7 VMs
1 x Physical HP ML350p w/ Windows Server 2008 R2 & VEEAM 9.5 installed
1 x QNAP TS-653A Backup Target
HP Core switching
Support ID 00547589

QNAP is iSCSI attached to ML350p via a single 1GB NIC currently. When running backup or backup copy jobs we are seeing intermittent throughput between the server and the NAS as per below pic.
Image

VEEAM is reporting the souce is the bottleneck but the QNAP is also reporting very low load.
Image

I have applied the autotuning tweak to the server with no change in performance. Switches are not reporting any issues, and have other services and the VMWare cluster which have no issues. No workstations are reporting speed issues, and I can generally copy data to and from the NAS at a good speed from within the Windows server (and without the drops of transfer rate).
Image

I would be happy with slow but constant performance, as I'm not expecting the world but the constant drops to 0KB speed are making life extremely difficult.

Can anyone point me in the right direction?
wigglez
Novice
 
Posts: 3
Liked: never
Joined: Wed Mar 09, 2016 1:50 am
Location: Hobart, Australia
Full Name: Lee

Re: Slow intermittent iSCSI performance

Veeam Logoby PTide » Wed Jul 05, 2017 9:31 am

Hi,

What are the proxies that you use - VM appliances or VBR default proxy? What proxy selection settings do you have? Is it a new setup or it's been working OK until now?

Also, have you tried to expose QNAP as a share instead?

Thanks
PTide
Veeam Software
 
Posts: 3078
Liked: 251 times
Joined: Tue May 19, 2015 1:46 pm

Re: Slow intermittent iSCSI performance

Veeam Logoby wigglez » Wed Jul 05, 2017 10:30 pm

Hi PTide,

Have got 2 proxies, one local and one VMWare proxy located on the vCenter server.

- Have selected the VMWare proxy for the Virtual Machines to QNAP backup job with Storage Optimization set to LAN Target and compression set to Optimal.
- The storage copy job I have manually selected the QNAP as the source, and it copying to a local USB hard disk plugged into the backup server for a seed for offsite cloud connect backup.

The issue has been ongoing, but trying to take the seed has prompted me to create the post as the seed copy job was taking an exceptionally long time to create a full.
Image

I had another client with same issues but different model of QNAP and we had to move them across to a CIFS share eventually due to this same apparent issue. I would prefer to leave this setup as iSCSI if possible.
wigglez
Novice
 
Posts: 3
Liked: never
Joined: Wed Mar 09, 2016 1:50 am
Location: Hobart, Australia
Full Name: Lee

Re: Slow intermittent iSCSI performance

Veeam Logoby wigglez » Thu Jul 06, 2017 6:11 am

I thought I would do a test, and copy the vull VBK through Windows from the iSCSI drive to a SMB/CIFS share on the QNAP. Found that I do not experience the same performance, and actuall get solid 75MB/s transfer rate in Windows as per below.
ImageImage

So it seems the issue is with the way VEEAM is handling the traffic, almost like it is buffering and then spitting out some traffic.
wigglez
Novice
 
Posts: 3
Liked: never
Joined: Wed Mar 09, 2016 1:50 am
Location: Hobart, Australia
Full Name: Lee

Re: Slow intermittent iSCSI performance

Veeam Logoby PTide » Thu Jul 06, 2017 11:33 am

So, you have one proxy on a VM located on the same machine where vCenter is and the other one is VBR itself. If I got your config right then the data flow is something like this:

Source disk ----> Source Proxy (VMware VM) ----> target proxy (Veeam Server) ---iscsi---> QNAP

I would check NIC stats on both proxies during backup operations to see if those spikes also show up somewhere. I guess there might be a situation that one of the proxies just sits there waiting for another portion of data to send over the network.

P.S. Also please keep in mind that if such a situation occurs that all you're left with is the QNAP it would be easier to access it via CIFS than iSCSI.

Thanks
PTide
Veeam Software
 
Posts: 3078
Liked: 251 times
Joined: Tue May 19, 2015 1:46 pm

Re: Slow intermittent iSCSI performance

Veeam Logoby nmdange » Thu Jul 06, 2017 3:53 pm

Those gaps would indicate Veeam isn't actually transferring any data. So it might not a performance issue with the data transfer. One thing I've discovered in regards to the bottleneck stats, it does not take into account time spent waiting on a resource that has some kind of concurrency limit (such as a backup proxy). If possible, can you watch the backup while it is happening to see if any of the VMs have a status like "Resource not ready: Backup Proxy"?

I really wish the backup log or bottleneck stats could show this after the fact. The most you can see is looking at the timestamp between these events:
7/6/2017 3:26:25 AM :: Queued for processing at 7/6/2017 3:26:25 AM
7/6/2017 3:28:51 AM :: Required backup infrastructure resources have been assigned
7/6/2017 3:34:53 AM :: VM processing started at 7/6/2017 3:34:53 AM

However there's no way to see what Veeam was waiting on for the step "Required backup infrastructure resources have been assigned", at least that I am aware of.
nmdange
Expert
 
Posts: 203
Liked: 58 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: Slow intermittent iSCSI performance

Veeam Logoby Pat490 » Fri Jul 07, 2017 8:29 am

wigglez wrote:I have applied the autotuning tweak to the server with no change in performance. Switches are not reporting any issues, and have other services and the VMWare cluster which have no issues. No workstations are reporting speed issues, and I can generally copy data to and from the NAS at a good speed from within the Windows server (and without the drops of transfer rate).

what "autotuning tweak" are you referring too?
Pat490
Expert
 
Posts: 136
Liked: 24 times
Joined: Tue Apr 28, 2015 7:18 am
Location: Germany
Full Name: Patrick


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: fika, Google [Bot] and 42 guests