-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
High Disk Latency on Remote Proxy VM
Hey,
We are running VeeamB&R 7.0.0.871. ESX5.5U2, VM version 10.
We have 2 sites and we replicate vms from site A to site B. We have a combination of Veeam proxies on the B end. Most are 4vcpu, 12GB, Server 2012R2 utilizing "hotadd" as the backup method.
When a replication takes place, we see massive write latency on our vm stats for the veeam proxies using hotadd. Ranging between 700 and 1000 ms. We never see any latency on our Dell Compellent SAN, or ESX host/Veeam Proxy CPU and memory. The only latency we see is on the veeam proxy disk statistics.
We have tried disabling Parallel processing but even running one job on the datastore has this high write latency.
Can anyone recommend a good place to start troubleshooting?
We are running VeeamB&R 7.0.0.871. ESX5.5U2, VM version 10.
We have 2 sites and we replicate vms from site A to site B. We have a combination of Veeam proxies on the B end. Most are 4vcpu, 12GB, Server 2012R2 utilizing "hotadd" as the backup method.
When a replication takes place, we see massive write latency on our vm stats for the veeam proxies using hotadd. Ranging between 700 and 1000 ms. We never see any latency on our Dell Compellent SAN, or ESX host/Veeam Proxy CPU and memory. The only latency we see is on the veeam proxy disk statistics.
We have tried disabling Parallel processing but even running one job on the datastore has this high write latency.
Can anyone recommend a good place to start troubleshooting?
-
- Veeam Software
- Posts: 37
- Liked: 17 times
- Joined: Aug 26, 2009 1:13 pm
- Full Name: Mike Zolkin
- Location: St. Petersburg, Florida
- Contact:
Re: High Disk Latency on Remote Proxy VM
Hey Ben,
We recently had a client with a similar question. There was a joint call with VMware and according to their tech representative: "it's confirmed that we have a snapshot limitations where it can have performance difference compared to base disk". They also provided us with the following link http://pubs.vmware.com/vsphere-55/index ... 88613.html
Mike
We recently had a client with a similar question. There was a joint call with VMware and according to their tech representative: "it's confirmed that we have a snapshot limitations where it can have performance difference compared to base disk". They also provided us with the following link http://pubs.vmware.com/vsphere-55/index ... 88613.html
Mike
VP, WW Customer Technical Support
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
Hey Mike,
Thanks for the link. Do you know if there are supported maximums for snapshots on a single datastore for a Veeam backup/Replication Job?
In my last test, when I disabled parallel processing, I experienced very high disk latency while processing only 1 VM in the replication job. That would mean there would only be a single snapshot on the destination datastore being processed at that time.
Thanks for the reply, I appreciate the conversation being started.
Thanks for the link. Do you know if there are supported maximums for snapshots on a single datastore for a Veeam backup/Replication Job?
In my last test, when I disabled parallel processing, I experienced very high disk latency while processing only 1 VM in the replication job. That would mean there would only be a single snapshot on the destination datastore being processed at that time.
Thanks for the reply, I appreciate the conversation being started.
-
- Veeam Software
- Posts: 37
- Liked: 17 times
- Joined: Aug 26, 2009 1:13 pm
- Full Name: Mike Zolkin
- Location: St. Petersburg, Florida
- Contact:
Re: High Disk Latency on Remote Proxy VM
We do not define any limitations, especially on the datastore level. There is limit of maximum allowed snapshots per VM, but that would be a VMware limitation though.Do you know if there are supported maximums for snapshots on a single datastore for a Veeam backup/Replication Job?
VP, WW Customer Technical Support
-
- VP, Product Management
- Posts: 27371
- Liked: 2799 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: High Disk Latency on Remote Proxy VM
Have you tried forcing these proxy servers to use NBD mode to write data? What is the latency and job performance in this case?benwhitmore wrote:When a replication takes place, we see massive write latency on our vm stats for the veeam proxies using hotadd.
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
Thanks Mike.
Not getting anywhere fast with this. As I said, I get the problem using "hotadd" even with just one vm in the backup job. It has been suggested to use Veeam One and Windows Perf Mon to look at what is going on. Both just confirm that the "hot added" disk latency for the veeam proxy is through the roof but CPU/MEM/Network all remain practically untouched.
Case # 00642574
ps. NBD Mode works fine, no disk latency. Please someone do not suggest turning hotadd off on the proxy and forcing NBD - that is just skirting around the issue.
Not getting anywhere fast with this. As I said, I get the problem using "hotadd" even with just one vm in the backup job. It has been suggested to use Veeam One and Windows Perf Mon to look at what is going on. Both just confirm that the "hot added" disk latency for the veeam proxy is through the roof but CPU/MEM/Network all remain practically untouched.
Case # 00642574
ps. NBD Mode works fine, no disk latency. Please someone do not suggest turning hotadd off on the proxy and forcing NBD - that is just skirting around the issue.
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
Ok, read up on CBT a bit. I disabled it for my test job. The job actually completes in the expected time frame, shows the bottle neck as the source (it was always the target before). I still see large spikes in disk latency but I now see it across the datastore and storage adapter too.
Could CBT be causing me some grief? Has anyone got any more info on how CBT works and why it could be causing me vm disk latency.
Could CBT be causing me some grief? Has anyone got any more info on how CBT works and why it could be causing me vm disk latency.
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
turns out that we don't have CBT enabled on our VMs so having it switched on (by default) in the Veeam job could potentially cause some issues perhaps? Or does Veeam not use CBT if the vm doesn't have that feature enabled?
-
- VP, Product Management
- Posts: 27371
- Liked: 2799 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: High Disk Latency on Remote Proxy VM
CBT is used by source proxy, and thus cannot affect target datastore performance.benwhitmore wrote:]Could CBT be causing me some grief? Has anyone got any more info on how CBT works and why it could be causing me vm disk latency.
Switching CBT on will not cause any issues, and actually it should be enabled by default in the jobs. If VMware CBT is not enabled, Veeam starts to use our proprietary change tracking engine that scans the entire VM, and transfers changed data only. This makes incremental runs take longer.benwhitmore wrote:turns out that we don't have CBT enabled on our VMs so having it switched on (by default) in the Veeam job could potentially cause some issues perhaps? Or does Veeam not use CBT if the vm doesn't have that feature enabled?
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
Thanks for the reply.
What line can I check in the log to see if CBT is being used/not used when it is enabled on a job?
Veeam Destination Proxy Disk Latency = 1200ms
Destination Datastore Write Latency = 300ms
Destination Host ISCSI HBA Latency= 200ms
SAN ISCSI Connection Write Latency = 100ms
SAN Volume Write Latency = 100ms
SAN Disks Write Latency = 10ms
What line can I check in the log to see if CBT is being used/not used when it is enabled on a job?
Maybe I am missing something really obvious because I find it strange how performance increases when I disable that option. *scratches head*. That being said, I still see high latency on the destination:-CBT is used by source proxy, and thus cannot affect target datastore performance.
Veeam Destination Proxy Disk Latency = 1200ms
Destination Datastore Write Latency = 300ms
Destination Host ISCSI HBA Latency= 200ms
SAN ISCSI Connection Write Latency = 100ms
SAN Volume Write Latency = 100ms
SAN Disks Write Latency = 10ms
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
PS. When I have CBT Enabled on my Job (and it is not enabled on my VMs), I still see CBT in the job window, is this normal and expected?
e.g
e.g
Hard disk 1 (40.0 GB) 0.0KB read at 0KB/s [CBT]
-
- VP, Product Management
- Posts: 27371
- Liked: 2799 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: High Disk Latency on Remote Proxy VM
Our jobs automatically enable CBT on the VMs, so according to the log snippet you have provided VMware CBT is indeed leveraged while reading source VM disk file.
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
Can you offer any troubleshooting steps as to why the job is much slower when we enable CBT? I thought you had to enable CBT on the VM too? Appreciate your time.
-
- VP, Product Management
- Posts: 27371
- Liked: 2799 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: High Disk Latency on Remote Proxy VM
The first thing that I would do is to compare both sessions (with CBT enabled and with no CBT) in the backup console. Try to find what operation takes most of the time, usually it is read/data transfer procedure for the job with no VMware CBT. Additionally, you can involve our technical support team as they might have some thoughts after reviewing your debug logs.
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
Have a call logged on this. Don't seem to be getting anywhere and I haven't been asked to send any logs yet to support.
Case # 00642574
Case # 00642574
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
logs uploaded to case
-
- Veeam Software
- Posts: 71
- Liked: 10 times
- Joined: Mar 18, 2013 9:00 am
- Full Name: Alexander Lupashku
- Contact:
Re: High Disk Latency on Remote Proxy VM
Ben,
Just wanted to let you know that we are working closely on your issue here in support and have been discussed the issue internally so far. As far as I'm aware, the responsible engineer has arranged another remote session with you to try some more suggestions. We will also be happy to move it up to the higher tier in case the prepared steps do not reveal the cause.
Thank you for your patience.
Just wanted to let you know that we are working closely on your issue here in support and have been discussed the issue internally so far. As far as I'm aware, the responsible engineer has arranged another remote session with you to try some more suggestions. We will also be happy to move it up to the higher tier in case the prepared steps do not reveal the cause.
Thank you for your patience.
VCSP SE Team
-
- Veeam Software
- Posts: 37
- Liked: 17 times
- Joined: Aug 26, 2009 1:13 pm
- Full Name: Mike Zolkin
- Location: St. Petersburg, Florida
- Contact:
Re: High Disk Latency on Remote Proxy VM
We didn't ask you because you had uploaded them initially, during ticket creation process and we have received them.I haven't been asked to send any logs yet to support
VP, WW Customer Technical Support
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
See, my memory is being affected by the stress! My bad, appologies on that one. I have since uploaded more logs for the job with CBT enabled and disabled. Unfortuntely, our Job name was so long that the bundled logs exceeded the maximum number of characers for a windows filename.
One of your guys is back on the case. Thank you. Despite how it comes across, I do value your support at Veeam.
One of your guys is back on the case. Thank you. Despite how it comes across, I do value your support at Veeam.
-
- Influencer
- Posts: 12
- Liked: 4 times
- Joined: Sep 29, 2014 7:16 am
- Full Name: Ben Whitmore
- Contact:
Re: High Disk Latency on Remote Proxy VM
OK...update....CBT Smoke screen/red herring.
We ran a disk stress tool on a manual "hotadd" disk with support and still saw high disk latency but very low datastore/SAN latency. We are going to progress this with VMware. Ill add updates as I get them in case it helps others.
Veeam support went above and beyond on this again. Sterling work guys, really appreciate it.
We ran a disk stress tool on a manual "hotadd" disk with support and still saw high disk latency but very low datastore/SAN latency. We are going to progress this with VMware. Ill add updates as I get them in case it helps others.
Veeam support went above and beyond on this again. Sterling work guys, really appreciate it.
-
- Veeam Software
- Posts: 37
- Liked: 17 times
- Joined: Aug 26, 2009 1:13 pm
- Full Name: Mike Zolkin
- Location: St. Petersburg, Florida
- Contact:
Re: High Disk Latency on Remote Proxy VM
Ben,
Don't forget to invite us on a call with VMware. We often help our clients with explaining issues to a 3rd party support representative and answering their questions if necessary. Just to make sure there is no Ping-Pong from one support to another.
Mike
Don't forget to invite us on a call with VMware. We often help our clients with explaining issues to a 3rd party support representative and answering their questions if necessary. Just to make sure there is no Ping-Pong from one support to another.
Mike
VP, WW Customer Technical Support
Who is online
Users browsing this forum: No registered users and 119 guests