High Disk Latency on Remote Proxy VM

[benwhitmore]

Hey,

We are running VeeamB&R 7.0.0.871. ESX5.5U2, VM version 10.

We have 2 sites and we replicate vms from site A to site B. We have a combination of Veeam proxies on the B end. Most are 4vcpu, 12GB, Server 2012R2 utilizing "hotadd" as the backup method.

When a replication takes place, we see massive write latency on our vm stats for the veeam proxies using hotadd. Ranging between 700 and 1000 ms. We never see any latency on our Dell Compellent SAN, or ESX host/Veeam Proxy CPU and memory. The only latency we see is on the veeam proxy disk statistics.

We have tried disabling Parallel processing but even running one job on the datastore has this high write latency.

Can anyone recommend a good place to start troubleshooting?

[m_zolkin]

Hey Ben,

We recently had a client with a similar question. There was a joint call with VMware and according to their tech representative: "it's confirmed that we have a snapshot limitations where it can have performance difference compared to base disk". They also provided us with the following link http://pubs.vmware.com/vsphere-55/index ... 88613.html

Mike

[benwhitmore]

Hey Mike,

Thanks for the link. Do you know if there are supported maximums for snapshots on a single datastore for a Veeam backup/Replication Job?

In my last test, when I disabled parallel processing, I experienced very high disk latency while processing only 1 VM in the replication job. That would mean there would only be a single snapshot on the destination datastore being processed at that time.

Thanks for the reply, I appreciate the conversation being started.

[m_zolkin]

Do you know if there are supported maximums for snapshots on a single datastore for a Veeam backup/Replication Job?

We do not define any limitations, especially on the datastore level. There is limit of maximum allowed snapshots per VM, but that would be a VMware limitation though.

[Vitaliy S.]

When a replication takes place, we see massive write latency on our vm stats for the veeam proxies using hotadd.

Have you tried forcing these proxy servers to use NBD mode to write data? What is the latency and job performance in this case?

[benwhitmore]

Thanks Mike.

Not getting anywhere fast with this. As I said, I get the problem using "hotadd" even with just one vm in the backup job. It has been suggested to use Veeam One and Windows Perf Mon to look at what is going on. Both just confirm that the "hot added" disk latency for the veeam proxy is through the roof but CPU/MEM/Network all remain practically untouched.

Case # 00642574

ps. NBD Mode works fine, no disk latency. Please someone do not suggest turning hotadd off on the proxy and forcing NBD - that is just skirting around the issue.

[benwhitmore]

Ok, read up on CBT a bit. I disabled it for my test job. The job actually completes in the expected time frame, shows the bottle neck as the source (it was always the target before). I still see large spikes in disk latency but I now see it across the datastore and storage adapter too.

Could CBT be causing me some grief? Has anyone got any more info on how CBT works and why it could be causing me vm disk latency.

[benwhitmore]

turns out that we don't have CBT enabled on our VMs so having it switched on (by default) in the Veeam job could potentially cause some issues perhaps? Or does Veeam not use CBT if the vm doesn't have that feature enabled?

[Vitaliy S.]

]Could CBT be causing me some grief? Has anyone got any more info on how CBT works and why it could be causing me vm disk latency.

CBT is used by source proxy, and thus cannot affect target datastore performance.

turns out that we don't have CBT enabled on our VMs so having it switched on (by default) in the Veeam job could potentially cause some issues perhaps? Or does Veeam not use CBT if the vm doesn't have that feature enabled?

Switching CBT on will not cause any issues, and actually it should be enabled by default in the jobs. If VMware CBT is not enabled, Veeam starts to use our proprietary change tracking engine that scans the entire VM, and transfers changed data only. This makes incremental runs take longer.

[benwhitmore]

Thanks for the reply.

What line can I check in the log to see if CBT is being used/not used when it is enabled on a job?

CBT is used by source proxy, and thus cannot affect target datastore performance.

Maybe I am missing something really obvious because I find it strange how performance increases when I disable that option. *scratches head*. That being said, I still see high latency on the destination:-

Veeam Destination Proxy Disk Latency = 1200ms
Destination Datastore Write Latency = 300ms
Destination Host ISCSI HBA Latency= 200ms
SAN ISCSI Connection Write Latency = 100ms
SAN Volume Write Latency = 100ms
SAN Disks Write Latency = 10ms

[benwhitmore]

PS. When I have CBT Enabled on my Job (and it is not enabled on my VMs), I still see CBT in the job window, is this normal and expected?

e.g

Hard disk 1 (40.0 GB) 0.0KB read at 0KB/s [CBT]

[Vitaliy S.]

Our jobs automatically enable CBT on the VMs, so according to the log snippet you have provided VMware CBT is indeed leveraged while reading source VM disk file.

[benwhitmore]

Can you offer any troubleshooting steps as to why the job is much slower when we enable CBT? I thought you had to enable CBT on the VM too? Appreciate your time.

[Vitaliy S.]

The first thing that I would do is to compare both sessions (with CBT enabled and with no CBT) in the backup console. Try to find what operation takes most of the time, usually it is read/data transfer procedure for the job with no VMware CBT. Additionally, you can involve our technical support team as they might have some thoughts after reviewing your debug logs.

[benwhitmore]

Have a call logged on this. Don't seem to be getting anywhere and I haven't been asked to send any logs yet to support.

Case # 00642574

[benwhitmore]

logs uploaded to case

[flaren]

Ben,

Just wanted to let you know that we are working closely on your issue here in support and have been discussed the issue internally so far. As far as I'm aware, the responsible engineer has arranged another remote session with you to try some more suggestions. We will also be happy to move it up to the higher tier in case the prepared steps do not reveal the cause.
Thank you for your patience.

[m_zolkin]

I haven't been asked to send any logs yet to support

We didn't ask you because you had uploaded them initially, during ticket creation process and we have received them.

[benwhitmore]

See, my memory is being affected by the stress! My bad, appologies on that one. I have since uploaded more logs for the job with CBT enabled and disabled. Unfortuntely, our Job name was so long that the bundled logs exceeded the maximum number of characers for a windows filename.

One of your guys is back on the case. Thank you. Despite how it comes across, I do value your support at Veeam.

[benwhitmore]

OK...update....CBT Smoke screen/red herring.

We ran a disk stress tool on a manual "hotadd" disk with support and still saw high disk latency but very low datastore/SAN latency. We are going to progress this with VMware. Ill add updates as I get them in case it helps others.

Veeam support went above and beyond on this again. Sterling work guys, really appreciate it.

[m_zolkin]

Ben,

Don't forget to invite us on a call with VMware. We often help our clients with explaining issues to a 3rd party support representative and answering their questions if necessary. Just to make sure there is no Ping-Pong from one support to another.

Mike