Performance issues when backing up Hyper-V

[enoch]

Hey everyone,

I'm not sure this is a Veeam issue, but if anyone have any ideas there are very welcome to share them with me

We have just for 3 month ago converted from VMware vSphere 5.5 to Microsoft Hyper-V on Windows 2012r2.

Everything runs nice in production time before backup window, but when we're running backup it's slow and everything on the Hyper-V environment is slow/unresponsive!

Compared to when we where running on VMware backup ran with 2-300 MB/sec and we actually could run backup jobs in production time without any complains!

The hardware is still the same as before: Dell PowerEdge M620 servers (blades) connected to Dell Equallogic SAN's (1G and 10G versions) iSCSI. We are currently running on
Veeam Backup and Replication version 8.0.0.831 (current version). Every M620 server is fully firmware and driver updated (from Dell support site) and every Windows 2012r2 is
fully updated including the Veeam Server which also is Windows 2012r2.

We currently runs with the on-host feature and Microsoft software VSS allowing 2 task pr. host. Using the Off-host with Dell Equallogic hardware provider it seems to be running a little better, but sometimes snapshot on the SAN's ain't automatically removed, so we got back to the software VSS. Currently running with Storage I/O control (20/25 ms) and enabled parallel processing.

It just seems very wierd that backup performance is so bad and makes everything run very bad in the VM environment. When running a full backup copy it's really a nightmare to get the backup finished

Does everyone have ideas?

[Shestakov]

Hello Peter,
To troubleshoot slow backup processes, you need to make bottleneck analysis.
Could you provide the job statistics?

We currently runs with the on-host feature and Microsoft software VSS allowing 2 task pr. host. Using the Off-host with Dell Equallogic hardware provider it seems to be running a little better, but sometimes snapshot on the SAN's ain't automatically removed, so we got back to the software VSS. Currently running with Storage I/O control (20/25 ms) and enabled parallel processing.

Do you see any kind of error when the snapshot is not removed? Did you make any changes with your Dell devices?
Thanks.

[enoch]

Hi,

Looking at the current job: Processing rate 108 MB/sec. and Bottleneck: source - need any other informations?

Errors with SAN snapshot no always correctly removed is only when using off-host with Dell Equallogic hardware provider not a problem when using Microsoft Software VSS.

I don't recall any changes on the devices when we did go from VMware to Hyper-V

[Shestakov]

Processing rate looks good. To speed the backup process up, you can use offsite proxy.
As for snapshot removal, I would suggest to check updates for Dell VSS, and if it doesn`t help, to contact Dell`s technical support and Veeam`s technical support.
Thanks.

[enoch]

It maybe seems the issue is that the CSV goes into BlockRedirected, very like this: https://social.technet.microsoft.com/Fo ... rverhyperv

Any way to troubleshoot?

[Vitaliy S.]

As far as I remember "redirected" backup mode should no longer be used given that you're on Windows Server 2012 R2. You can troubleshoot it's performance via Veeam ONE (free edition) or troubleshoot the snapshot issue on the off-host proxy.

[enoch]

I'm sure "BlockRedirected" mode is the problem.

We're using on-host proxy, where can I see this in Veeam One?

I'm running Windows 2012R2, please se below:

Code: Select all

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : vs27
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : vs25
StateInfo                    : Direct
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : vs24
StateInfo                    : Direct
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : VS26
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : vs27
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : vs25
StateInfo                    : Direct
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : vs24
StateInfo                    : Direct
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : VS26
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity

[Vitaliy S.]

Can you check these performance counters for the CSV datastore?

[enoch]

Thanks, I found an error, so VS26 and VS27 is now in Direct mode. Will try backup Again..

[Vitaliy S.]

Can you please share what error/misconfiguration you have spotted? This would be useful for future readers.

[enoch]

In my case it was iSCSI connectivity issues on host VS26 and VS27

I'm trying a backup now with Software VSS (limited to 1 snapshot task pr. datastore)

I'll get back with more informations.

[enoch]

Hi everyone,

Reinstalled Veeam on new backup host still running with Software VSS. I think backup performance is better than before!

We still get a major performance hit when backup is running on the Hyper-V hosts which is still wierd for me. This is with I/O control enabled and Network throttling set to 100 MB/sec. I've checked and it's possible for me to get about 500 MB/sec when doing backup without Network throttling policy.

Maybe the issue is somewhere on the Hyper-V Network properties? - Running with Broadcom BCM57810 NetXtreme II 10 Gbit, newest firmware and drivers for Dell PowerEdge M620 blades.

[Vitaliy S.]

Does lowering I/O control settings and network throttling thresholds help with the performance hit on the hosts?

[enoch]

We run Enterprise not Enterprise Plus so we can only control this for all datastores at once. But yes it work a little better but still issues with "good performance"

Is there a log file where I can check what Veeam detects the I/O latency when backup is running?

[Vitaliy S.]

You can either use Veeam ONE for that or job debug logs. Cannot recall the correct file name, but our support team can assist you with this.

[enoch]

Great I've contact the support team about the debug log. About Veeam ONE currently we use Veeam ONE free, havn't seen where it's possible to see I/O latency? (Hyper-V)

[Vitaliy S.]

You can navigate to the "datastores" object or Failover Cluster to review disk I/O performance.

[enoch]

Found it - thanks!