vPower NFS Write Latency?

[StrangeWill]

I run SureBackup jobs running testing copies of production typically for a week+, I've had some terrible issues with keeping them running consistently, but I think I fixed it (patching our iSCSI switch backbone should help and following vendor recommendations on configuration/patches for storage).

So now my performance looks like this:

Read:


Write:


Now read is pretty OK, my underlying storage is 21ms and under on average (with a high maximum due to a couple huge spikes about 12 hours ago I haven't accounted for yet). My VeeamBackup server having a slightly higher read (has to go through a virtualized Veeam box and is backed by the same array but a different and somewhat slower shelf). So that's expected. However the VM itself is showing typical latencies from it's storage drive:



Why are writes 4x more latent than the underlying storage.

[Vitaliy S.]

Could this be related to the place where you write your VM redo logs(changed data) that are generated during SB job verification?

P.S. good to see you're using Veeam ONE for monitoring backups and virtual infrastructure

[dellock6]

William, you just made the day for Vitaliy

Writes are indeed redirected to the specified datastore in the job configuration, are not done in the vPower NFS. Even without knowing which volume you used, as per your description you are using the same array for production (so source of VM backups), the virtual Veeam server (using vmdk or rdm doesn't count...) and the vPower NFS. Even if the different volumes are in different shelves, it depends on the raid configuration, and after all everything is going through the same controller (I read DS in volume naming, so I presume an IBM DSXXXX, correct?).

[Vitaliy S.]

@Luca, I have made a screenshot and put in on my walls!

[StrangeWill]

William, you just made the day for Vitaliy

Writes are indeed redirected to the specified datastore in the job configuration, are not done in the vPower NFS. Even without knowing which volume you used, as per your description you are using the same array for production (so source of VM backups), the virtual Veeam server (using vmdk or rdm doesn't count...) and the vPower NFS. Even if the different volumes are in different shelves, it depends on the raid configuration, and after all everything is going through the same controller (I read DS in volume naming, so I presume an IBM DSXXXX, correct?).

Yeah I was confused about I/O going through vPower NFS due to it being redirected by ESXi... so I looked closer...

There is like 2IOPS on the device.



Oops Back to the drawing board on why the labs are kind of slow to provision/start (takes up to an hour to do 4 VMs)!

P.S. good to see you're using Veeam ONE for monitoring backups and virtual infrastructure

I really like the product, sadly right now we're running with the free version and I have a small pile of bugs to submit that can be annoying at times, but there was only one product out of the many that I was eyeballing that would even compete with Veeam ONE (that was vKernel whatever it's called now), but Veeam integrates with all of our backup stuff so it's a hard choice when it comes to buying something.

You guys kicked the pants off of VMTurbo which is what my boss at the time wanted to buy.

And I really appreciate Veeam putting out the free version, made me really get to like it a lot and want to buy it for the reporting that I can take to upper management on projected resource needs.