Host-based backup of Microsoft Hyper-V VMs.
Post Reply
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Performance issues when backing up Hyper-V

Post by enoch »

Hey everyone,

I'm not sure this is a Veeam issue, but if anyone have any ideas there are very welcome to share them with me :D

We have just for 3 month ago converted from VMware vSphere 5.5 to Microsoft Hyper-V on Windows 2012r2.

Everything runs nice in production time before backup window, but when we're running backup it's slow and everything on the Hyper-V environment is slow/unresponsive!

Compared to when we where running on VMware backup ran with 2-300 MB/sec and we actually could run backup jobs in production time without any complains!

The hardware is still the same as before: Dell PowerEdge M620 servers (blades) connected to Dell Equallogic SAN's (1G and 10G versions) iSCSI. We are currently running on
Veeam Backup and Replication version 8.0.0.831 (current version). Every M620 server is fully firmware and driver updated (from Dell support site) and every Windows 2012r2 is
fully updated including the Veeam Server which also is Windows 2012r2.

We currently runs with the on-host feature and Microsoft software VSS allowing 2 task pr. host. Using the Off-host with Dell Equallogic hardware provider it seems to be running a little better, but sometimes snapshot on the SAN's ain't automatically removed, so we got back to the software VSS. Currently running with Storage I/O control (20/25 ms) and enabled parallel processing.

It just seems very wierd that backup performance is so bad and makes everything run very bad in the VM environment. When running a full backup copy it's really a nightmare to get the backup finished :twisted:

Does everyone have ideas?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Performance issues when backing up Hyper-V

Post by Shestakov »

Hello Peter,
To troubleshoot slow backup processes, you need to make bottleneck analysis.
Could you provide the job statistics?
enoch wrote:We currently runs with the on-host feature and Microsoft software VSS allowing 2 task pr. host. Using the Off-host with Dell Equallogic hardware provider it seems to be running a little better, but sometimes snapshot on the SAN's ain't automatically removed, so we got back to the software VSS. Currently running with Storage I/O control (20/25 ms) and enabled parallel processing.
Do you see any kind of error when the snapshot is not removed? Did you make any changes with your Dell devices?
Thanks.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Performance issues when backing up Hyper-V

Post by enoch »

Hi,

Looking at the current job: Processing rate 108 MB/sec. and Bottleneck: source - need any other informations?

Errors with SAN snapshot no always correctly removed is only when using off-host with Dell Equallogic hardware provider not a problem when using Microsoft Software VSS.

I don't recall any changes on the devices when we did go from VMware to Hyper-V
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Performance issues when backing up Hyper-V

Post by Shestakov »

Processing rate looks good. To speed the backup process up, you can use offsite proxy.
As for snapshot removal, I would suggest to check updates for Dell VSS, and if it doesn`t help, to contact Dell`s technical support and Veeam`s technical support.
Thanks.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Performance issues when backing up Hyper-V

Post by enoch »

It maybe seems the issue is that the CSV goes into BlockRedirected, very like this: https://social.technet.microsoft.com/Fo ... rverhyperv

Any way to troubleshoot?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Performance issues when backing up Hyper-V

Post by Vitaliy S. »

As far as I remember "redirected" backup mode should no longer be used given that you're on Windows Server 2012 R2. You can troubleshoot it's performance via Veeam ONE (free edition) or troubleshoot the snapshot issue on the off-host proxy.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Performance issues when backing up Hyper-V

Post by enoch »

I'm sure "BlockRedirected" mode is the problem.

We're using on-host proxy, where can I see this in Veeam One?

I'm running Windows 2012R2, please se below:

Code: Select all

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : vs27
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : vs25
StateInfo                    : Direct
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : vs24
StateInfo                    : Direct
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : VS26
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : vs27
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : vs25
StateInfo                    : Direct
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : vs24
StateInfo                    : Direct
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : VS26
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Performance issues when backing up Hyper-V

Post by Vitaliy S. »

Can you check these performance counters for the CSV datastore?
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Performance issues when backing up Hyper-V

Post by enoch »

Thanks, I found an error, so VS26 and VS27 is now in Direct mode. Will try backup Again..
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Performance issues when backing up Hyper-V

Post by Vitaliy S. »

Can you please share what error/misconfiguration you have spotted? This would be useful for future readers.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Performance issues when backing up Hyper-V

Post by enoch »

In my case it was iSCSI connectivity issues on host VS26 and VS27

I'm trying a backup now with Software VSS (limited to 1 snapshot task pr. datastore)

I'll get back with more informations.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Performance issues when backing up Hyper-V

Post by enoch »

Hi everyone,

Reinstalled Veeam on new backup host still running with Software VSS. I think backup performance is better than before!

We still get a major performance hit when backup is running on the Hyper-V hosts which is still wierd for me. This is with I/O control enabled and Network throttling set to 100 MB/sec. I've checked and it's possible for me to get about 500 MB/sec when doing backup without Network throttling policy.

Maybe the issue is somewhere on the Hyper-V Network properties? - Running with Broadcom BCM57810 NetXtreme II 10 Gbit, newest firmware and drivers for Dell PowerEdge M620 blades.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Performance issues when backing up Hyper-V

Post by Vitaliy S. »

Does lowering I/O control settings and network throttling thresholds help with the performance hit on the hosts?
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Performance issues when backing up Hyper-V

Post by enoch »

We run Enterprise not Enterprise Plus so we can only control this for all datastores at once. But yes it work a little better but still issues with "good performance"

Is there a log file where I can check what Veeam detects the I/O latency when backup is running?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Performance issues when backing up Hyper-V

Post by Vitaliy S. »

You can either use Veeam ONE for that or job debug logs. Cannot recall the correct file name, but our support team can assist you with this.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Performance issues when backing up Hyper-V

Post by enoch »

Great I've contact the support team about the debug log. About Veeam ONE currently we use Veeam ONE free, havn't seen where it's possible to see I/O latency? (Hyper-V)
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Performance issues when backing up Hyper-V

Post by Vitaliy S. »

You can navigate to the "datastores" object or Failover Cluster to review disk I/O performance.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Performance issues when backing up Hyper-V

Post by enoch »

Found it - thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests