Performance issues when backing up Hyper-V

Hyper-V specific discussions

Performance issues when backing up Hyper-V

Veeam Logoby enoch » Wed Dec 24, 2014 2:51 pm

Hey everyone,

I'm not sure this is a Veeam issue, but if anyone have any ideas there are very welcome to share them with me :D

We have just for 3 month ago converted from VMware vSphere 5.5 to Microsoft Hyper-V on Windows 2012r2.

Everything runs nice in production time before backup window, but when we're running backup it's slow and everything on the Hyper-V environment is slow/unresponsive!

Compared to when we where running on VMware backup ran with 2-300 MB/sec and we actually could run backup jobs in production time without any complains!

The hardware is still the same as before: Dell PowerEdge M620 servers (blades) connected to Dell Equallogic SAN's (1G and 10G versions) iSCSI. We are currently running on
Veeam Backup and Replication version 8.0.0.831 (current version). Every M620 server is fully firmware and driver updated (from Dell support site) and every Windows 2012r2 is
fully updated including the Veeam Server which also is Windows 2012r2.

We currently runs with the on-host feature and Microsoft software VSS allowing 2 task pr. host. Using the Off-host with Dell Equallogic hardware provider it seems to be running a little better, but sometimes snapshot on the SAN's ain't automatically removed, so we got back to the software VSS. Currently running with Storage I/O control (20/25 ms) and enabled parallel processing.

It just seems very wierd that backup performance is so bad and makes everything run very bad in the VM environment. When running a full backup copy it's really a nightmare to get the backup finished :twisted:

Does everyone have ideas?
enoch
Service Provider
 
Posts: 114
Liked: 8 times
Joined: Sat Jun 29, 2013 12:14 pm
Full Name: Peter Enoch

Re: Performance issues when backing up Hyper-V

Veeam Logoby Shestakov » Thu Dec 25, 2014 12:00 pm

Hello Peter,
To troubleshoot slow backup processes, you need to make bottleneck analysis.
Could you provide the job statistics?
enoch wrote:We currently runs with the on-host feature and Microsoft software VSS allowing 2 task pr. host. Using the Off-host with Dell Equallogic hardware provider it seems to be running a little better, but sometimes snapshot on the SAN's ain't automatically removed, so we got back to the software VSS. Currently running with Storage I/O control (20/25 ms) and enabled parallel processing.

Do you see any kind of error when the snapshot is not removed? Did you make any changes with your Dell devices?
Thanks.
Shestakov
Veeam Software
 
Posts: 4861
Liked: 395 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Performance issues when backing up Hyper-V

Veeam Logoby enoch » Thu Dec 25, 2014 12:45 pm

Hi,

Looking at the current job: Processing rate 108 MB/sec. and Bottleneck: source - need any other informations?

Errors with SAN snapshot no always correctly removed is only when using off-host with Dell Equallogic hardware provider not a problem when using Microsoft Software VSS.

I don't recall any changes on the devices when we did go from VMware to Hyper-V
enoch
Service Provider
 
Posts: 114
Liked: 8 times
Joined: Sat Jun 29, 2013 12:14 pm
Full Name: Peter Enoch

Re: Performance issues when backing up Hyper-V

Veeam Logoby Shestakov » Thu Dec 25, 2014 1:57 pm

Processing rate looks good. To speed the backup process up, you can use offsite proxy.
As for snapshot removal, I would suggest to check updates for Dell VSS, and if it doesn`t help, to contact Dell`s technical support and Veeam`s technical support.
Thanks.
Shestakov
Veeam Software
 
Posts: 4861
Liked: 395 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Performance issues when backing up Hyper-V

Veeam Logoby enoch » Thu Dec 25, 2014 11:56 pm

enoch
Service Provider
 
Posts: 114
Liked: 8 times
Joined: Sat Jun 29, 2013 12:14 pm
Full Name: Peter Enoch

Re: Performance issues when backing up Hyper-V

Veeam Logoby Vitaliy S. » Fri Dec 26, 2014 12:24 pm

As far as I remember "redirected" backup mode should no longer be used given that you're on Windows Server 2012 R2. You can troubleshoot it's performance via Veeam ONE (free edition) or troubleshoot the snapshot issue on the off-host proxy.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Performance issues when backing up Hyper-V

Veeam Logoby enoch » Fri Dec 26, 2014 2:42 pm

I'm sure "BlockRedirected" mode is the problem.

We're using on-host proxy, where can I see this in Veeam One?

I'm running Windows 2012R2, please se below:

Code: Select all
Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : vs27
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : vs25
StateInfo                    : Direct
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : vs24
StateInfo                    : Direct
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV1
VolumeName                   : \\?\Volume{562ca513-fa4c-43fb-9ba3-3217b41bf446}\
Node                         : VS26
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume1
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : vs27
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : vs25
StateInfo                    : Direct
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : vs24
StateInfo                    : Direct
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NotBlockRedirected

Name                         : CSV3
VolumeName                   : \\?\Volume{33e65321-2e72-4505-a258-2d0489585719}\
Node                         : VS26
StateInfo                    : BlockRedirected
VolumeFriendlyName           : Volume3
FileSystemRedirectedIOReason : NotFileSystemRedirected
BlockRedirectedIOReason      : NoDiskConnectivity
enoch
Service Provider
 
Posts: 114
Liked: 8 times
Joined: Sat Jun 29, 2013 12:14 pm
Full Name: Peter Enoch

Re: Performance issues when backing up Hyper-V

Veeam Logoby Vitaliy S. » Fri Dec 26, 2014 2:55 pm

Can you check these performance counters for the CSV datastore?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Performance issues when backing up Hyper-V

Veeam Logoby enoch » Fri Dec 26, 2014 3:58 pm

Thanks, I found an error, so VS26 and VS27 is now in Direct mode. Will try backup Again..
enoch
Service Provider
 
Posts: 114
Liked: 8 times
Joined: Sat Jun 29, 2013 12:14 pm
Full Name: Peter Enoch

Re: Performance issues when backing up Hyper-V

Veeam Logoby Vitaliy S. » Fri Dec 26, 2014 4:13 pm

Can you please share what error/misconfiguration you have spotted? This would be useful for future readers.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Performance issues when backing up Hyper-V

Veeam Logoby enoch » Fri Dec 26, 2014 4:28 pm

In my case it was iSCSI connectivity issues on host VS26 and VS27

I'm trying a backup now with Software VSS (limited to 1 snapshot task pr. datastore)

I'll get back with more informations.
enoch
Service Provider
 
Posts: 114
Liked: 8 times
Joined: Sat Jun 29, 2013 12:14 pm
Full Name: Peter Enoch

Re: Performance issues when backing up Hyper-V

Veeam Logoby enoch » Fri Jan 02, 2015 12:26 pm

Hi everyone,

Reinstalled Veeam on new backup host still running with Software VSS. I think backup performance is better than before!

We still get a major performance hit when backup is running on the Hyper-V hosts which is still wierd for me. This is with I/O control enabled and Network throttling set to 100 MB/sec. I've checked and it's possible for me to get about 500 MB/sec when doing backup without Network throttling policy.

Maybe the issue is somewhere on the Hyper-V Network properties? - Running with Broadcom BCM57810 NetXtreme II 10 Gbit, newest firmware and drivers for Dell PowerEdge M620 blades.
enoch
Service Provider
 
Posts: 114
Liked: 8 times
Joined: Sat Jun 29, 2013 12:14 pm
Full Name: Peter Enoch

Re: Performance issues when backing up Hyper-V

Veeam Logoby Vitaliy S. » Fri Jan 09, 2015 4:23 pm

Does lowering I/O control settings and network throttling thresholds help with the performance hit on the hosts?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Performance issues when backing up Hyper-V

Veeam Logoby enoch » Fri Jan 09, 2015 4:26 pm

We run Enterprise not Enterprise Plus so we can only control this for all datastores at once. But yes it work a little better but still issues with "good performance"

Is there a log file where I can check what Veeam detects the I/O latency when backup is running?
enoch
Service Provider
 
Posts: 114
Liked: 8 times
Joined: Sat Jun 29, 2013 12:14 pm
Full Name: Peter Enoch

Re: Performance issues when backing up Hyper-V

Veeam Logoby Vitaliy S. » Fri Jan 09, 2015 4:33 pm

You can either use Veeam ONE for that or job debug logs. Cannot recall the correct file name, but our support team can assist you with this.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 5 guests