Comprehensive data protection for all workloads
Post Reply
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

High Disk Latency on Remote Proxy VM

Post by benwhitmore »

Hey,

We are running VeeamB&R 7.0.0.871. ESX5.5U2, VM version 10.

We have 2 sites and we replicate vms from site A to site B. We have a combination of Veeam proxies on the B end. Most are 4vcpu, 12GB, Server 2012R2 utilizing "hotadd" as the backup method.

When a replication takes place, we see massive write latency on our vm stats for the veeam proxies using hotadd. Ranging between 700 and 1000 ms. We never see any latency on our Dell Compellent SAN, or ESX host/Veeam Proxy CPU and memory. The only latency we see is on the veeam proxy disk statistics.

We have tried disabling Parallel processing but even running one job on the datastore has this high write latency.

Can anyone recommend a good place to start troubleshooting?
m_zolkin
Veeam Software
Posts: 37
Liked: 17 times
Joined: Aug 26, 2009 1:13 pm
Full Name: Mike Zolkin
Location: St. Petersburg, Florida
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by m_zolkin »

Hey Ben,

We recently had a client with a similar question. There was a joint call with VMware and according to their tech representative: "it's confirmed that we have a snapshot limitations where it can have performance difference compared to base disk". They also provided us with the following link http://pubs.vmware.com/vsphere-55/index ... 88613.html

Mike
VP, WW Customer Technical Support
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore »

Hey Mike,

Thanks for the link. Do you know if there are supported maximums for snapshots on a single datastore for a Veeam backup/Replication Job?

In my last test, when I disabled parallel processing, I experienced very high disk latency while processing only 1 VM in the replication job. That would mean there would only be a single snapshot on the destination datastore being processed at that time.

Thanks for the reply, I appreciate the conversation being started.
m_zolkin
Veeam Software
Posts: 37
Liked: 17 times
Joined: Aug 26, 2009 1:13 pm
Full Name: Mike Zolkin
Location: St. Petersburg, Florida
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by m_zolkin »

Do you know if there are supported maximums for snapshots on a single datastore for a Veeam backup/Replication Job?
We do not define any limitations, especially on the datastore level. There is limit of maximum allowed snapshots per VM, but that would be a VMware limitation though.
VP, WW Customer Technical Support
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by Vitaliy S. »

benwhitmore wrote:When a replication takes place, we see massive write latency on our vm stats for the veeam proxies using hotadd.
Have you tried forcing these proxy servers to use NBD mode to write data? What is the latency and job performance in this case?
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore »

Thanks Mike.

Not getting anywhere fast with this. As I said, I get the problem using "hotadd" even with just one vm in the backup job. It has been suggested to use Veeam One and Windows Perf Mon to look at what is going on. Both just confirm that the "hot added" disk latency for the veeam proxy is through the roof but CPU/MEM/Network all remain practically untouched.

Case # 00642574

ps. NBD Mode works fine, no disk latency. Please someone do not suggest turning hotadd off on the proxy and forcing NBD - that is just skirting around the issue.
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore »

Ok, read up on CBT a bit. I disabled it for my test job. The job actually completes in the expected time frame, shows the bottle neck as the source (it was always the target before). I still see large spikes in disk latency but I now see it across the datastore and storage adapter too.

Could CBT be causing me some grief? Has anyone got any more info on how CBT works and why it could be causing me vm disk latency.
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore »

turns out that we don't have CBT enabled on our VMs so having it switched on (by default) in the Veeam job could potentially cause some issues perhaps? Or does Veeam not use CBT if the vm doesn't have that feature enabled?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by Vitaliy S. »

benwhitmore wrote:]Could CBT be causing me some grief? Has anyone got any more info on how CBT works and why it could be causing me vm disk latency.
CBT is used by source proxy, and thus cannot affect target datastore performance.
benwhitmore wrote:turns out that we don't have CBT enabled on our VMs so having it switched on (by default) in the Veeam job could potentially cause some issues perhaps? Or does Veeam not use CBT if the vm doesn't have that feature enabled?
Switching CBT on will not cause any issues, and actually it should be enabled by default in the jobs. If VMware CBT is not enabled, Veeam starts to use our proprietary change tracking engine that scans the entire VM, and transfers changed data only. This makes incremental runs take longer.
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore »

Thanks for the reply.

What line can I check in the log to see if CBT is being used/not used when it is enabled on a job?
CBT is used by source proxy, and thus cannot affect target datastore performance.
Maybe I am missing something really obvious because I find it strange how performance increases when I disable that option. *scratches head*. That being said, I still see high latency on the destination:-

Veeam Destination Proxy Disk Latency = 1200ms
Destination Datastore Write Latency = 300ms
Destination Host ISCSI HBA Latency= 200ms
SAN ISCSI Connection Write Latency = 100ms
SAN Volume Write Latency = 100ms
SAN Disks Write Latency = 10ms
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore »

PS. When I have CBT Enabled on my Job (and it is not enabled on my VMs), I still see CBT in the job window, is this normal and expected?

e.g
Hard disk 1 (40.0 GB) 0.0KB read at 0KB/s [CBT]
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by Vitaliy S. »

Our jobs automatically enable CBT on the VMs, so according to the log snippet you have provided VMware CBT is indeed leveraged while reading source VM disk file.
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore »

Can you offer any troubleshooting steps as to why the job is much slower when we enable CBT? I thought you had to enable CBT on the VM too? Appreciate your time.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by Vitaliy S. »

The first thing that I would do is to compare both sessions (with CBT enabled and with no CBT) in the backup console. Try to find what operation takes most of the time, usually it is read/data transfer procedure for the job with no VMware CBT. Additionally, you can involve our technical support team as they might have some thoughts after reviewing your debug logs.
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore »

Have a call logged on this. Don't seem to be getting anywhere and I haven't been asked to send any logs yet to support. :(

Case # 00642574
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore »

logs uploaded to case
flaren
Veeam Software
Posts: 62
Liked: 9 times
Joined: Mar 18, 2013 9:00 am
Full Name: Alexander Lupashku
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by flaren »

Ben,

Just wanted to let you know that we are working closely on your issue here in support and have been discussed the issue internally so far. As far as I'm aware, the responsible engineer has arranged another remote session with you to try some more suggestions. We will also be happy to move it up to the higher tier in case the prepared steps do not reveal the cause.
Thank you for your patience.
VCSP SE Team
m_zolkin
Veeam Software
Posts: 37
Liked: 17 times
Joined: Aug 26, 2009 1:13 pm
Full Name: Mike Zolkin
Location: St. Petersburg, Florida
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by m_zolkin »

I haven't been asked to send any logs yet to support
We didn't ask you because you had uploaded them initially, during ticket creation process and we have received them. :)
VP, WW Customer Technical Support
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore »

See, my memory is being affected by the stress! My bad, appologies on that one. I have since uploaded more logs for the job with CBT enabled and disabled. Unfortuntely, our Job name was so long that the bundled logs exceeded the maximum number of characers for a windows filename.

One of your guys is back on the case. Thank you. Despite how it comes across, I do value your support at Veeam.
benwhitmore
Influencer
Posts: 12
Liked: 4 times
Joined: Sep 29, 2014 7:16 am
Full Name: Ben Whitmore
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by benwhitmore » 4 people like this post

OK...update....CBT Smoke screen/red herring.

We ran a disk stress tool on a manual "hotadd" disk with support and still saw high disk latency but very low datastore/SAN latency. We are going to progress this with VMware. Ill add updates as I get them in case it helps others.

Veeam support went above and beyond on this again. Sterling work guys, really appreciate it.
m_zolkin
Veeam Software
Posts: 37
Liked: 17 times
Joined: Aug 26, 2009 1:13 pm
Full Name: Mike Zolkin
Location: St. Petersburg, Florida
Contact:

Re: High Disk Latency on Remote Proxy VM

Post by m_zolkin » 1 person likes this post

Ben,

Don't forget to invite us on a call with VMware. We often help our clients with explaining issues to a 3rd party support representative and answering their questions if necessary. Just to make sure there is no Ping-Pong from one support to another.

Mike
VP, WW Customer Technical Support
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 197 guests