Surebackup Hyper-V Disks IO latency issue.

[SBarrett847]

I've noticed a strange issue with SureBackup affecting all running VMs on a Host. I'm thinking the Main issue is Hyper-V, but it manifests itself when Surebackup is run.

The Surebackups are run off our StoreOnce, which has truly abysmal non-sequential read performance (as low as 1-5mb/s in some cases). The Surebackup Proxies and Surebackup avhdxs go on an SSD array on the Host, this array also has about 10 VMs on it too.

When the Surebackup Jobs are Spun up, all Hyper-V VM Disk performance on the SSD array goes into the toilet for the other VMs, however, physically there are no problems with the SSDs from the host perspective. e.g. I can read and write files to it at 500MB/s + , latency is minimal etc, but VM Disk performance inside the VMs is suffering very high latency.

Has Anybody seen something like this? My guess would be that there is an overall Hyper-V Disk subsystem on that host that is being hobbled by the latency of the StoreOnce. Like it is waiting for the StoreOnce to respond before it gets back to addressing the rest of the VMs Disk IO operations.

[Rick.Vanover]

This is interesting. I have not seen it. Do you think by chance you could do a comparative test on a "regular" repository? Basically do the following steps:
-Make a VM on the Hyper-V cluster
-Give it a repository role
-Do an additional backup job of the same VMs to that repository
-Re-Do the SureBackup but source it from that repository

Curious if the same behavior exists.

[SBarrett847]

I actually have another Backup Job for File server VMs, that initially goes to an intimidate Repo on an old SAN. This SAN is able to deliver reasonable IOPS for the SureBackup jobs that run off it, and I see no such issues affecting the other host VMs.

[Rick.Vanover]

I recommend you open a case with that information - maybe there is an issue that can be discovered in the logs.

[SBarrett847]

Yep, I think I'll be opening a case

VM Disk performance on a VM on the same host as the Surebackup VMs.........

Surebackup not running......



Surebackup running.......

[SBarrett847]

#03224251

Also tried disabling the Hyper-V IO load balancer to no avail.
http://www.aidanfinn.com/?p=13232

[Mike Resseler]

Stephen,

I hope support can find something more. I am thinking in redirected IO at this point in time causing the network to be the bottleneck. But can't say for sure

[SBarrett847]

Issue is still ongoing, support are stumped as to a cause. I've tried to recreate it in a LAB environment to no avail.

Findings so far....
Instant recovery has the exact same issue, and makes it easier to test and Illustrate the issue.
New Repos, mount targets and Proxyies located on another Hyper-V host, yields the same issue of Instant Restore VMs affecting IOPS on the other Production VMs on the destination Datastore.
Instant restore to another Hyper-V host, on to its local storage, affects the VMs running on that storage.

Other things of note
With the instant restore VM running in Hyper-V – All VMs on same destination datastore have vastly reduced disk performance as before.
With the instant restore VM turned off, but not migrated to production – All VMs on same destination datastore have normal disk performance.
With the Instant Restore running and finished migrating to production - All VMs on same destination datastore have normal disk performance.

I also note the Speed of the Migration to production for the instant restore is also impacted same as above. If the VM is turned off in Hyper-V the restore is much faster.

A head scratcher, and annoying, as I now have a business need to automate backup checks.

[Mike Resseler]

Stephen,

Did I understand it correctly that you don't have this issue in a lab environment?

I also do believe somewhere the Hyper-V system is going crazy over the redirection of the data all the time. Are these fully patched?

[SBarrett847]

Yeah, the hosts are patched up. QA has recreated it in their Lab I'm told, and I had another session yesterday with Tier 3 support to re-demonstrate the issue to them in person. Hopefully they get to the bottom of it.

My Lab isn't an exact replica of my production setup, so it's difficult to recreate the scenario I have in production exactly, but the general arrangement is very close. But not close enough it seems.

[SBarrett847]

Ok, it looks like this is a Hyper-V limitation according to support. They provided a KB article, which doesn't match my issue entirely, but seems like it's something they've run into before. The odd thing is I have the IO balancer in Hyper -V disabled, which I'd imagine to be the culprit, to no effect.

https://www.veeam.com/kb2028

I'll have to work around the issue another way. Won't be touching another StoreOnce again for sure.

[Mike Resseler]

Stephen,

Thanks for the update. Let us know what your solution is going to be.

Thanks
Mike