Discussions specific to the VMware vSphere hypervisor
Post Reply
briggs
Enthusiast
Posts: 73
Liked: 3 times
Joined: Mar 22, 2013 11:01 am
Full Name: Alan Thompson
Contact:

MASSIVE performance hit on v8 (new VM)

Post by briggs » Nov 19, 2014 12:10 pm

Hi,
I'll start out by saying this is a brand new problem, 24 hours old max. I have not spent much time on this, but thought I'd ask now whilst working on it myself.
We had a Windows 2008 VM running Veeam 7. This worked absolutely fine, but we are migrating to Windows 2012 R2, and to support dedup, I understood I need Veeam on a Win2012 R2 server.
So I've built a new VM, with the exact same resources as the existing Veeam server. It's on the same host, same datastore. So I built the Win2012 R2 OS, latest windows updates, put AV on it, all the usual, then I installed Veeam 8. I copied the configuration off the old Veeam 7 server, and restored it to our V8 (new) server. This went really well, all the config and jobs came over. Really tidy.
So I downed and powered off the old Veeam server, and let the new server run overnight. First thing I noticed, all the existing jobs ran as a full backup job. I assume this is normal?
But the big problem I've got, is the performance. The jobs all ran as full's, at around the same time, so I could blame resources and congestion etc. But some of the jobs ran at different times, without clashing, yet the job log shows performance of as little as 22MB/s, and only a max of 76MB/s. On the old Veeam VM (win2008 remember, and Veeam v7), the same jobs, using the last full run as a benchmark, ran at 53MB/s min and 104MB/s max. If I compare on the spot the same jobs, a few of them are running at 1/3rd of the speed!

I hope this is a teething problem, but can anyone shed any light on this? Is Win2012 R2 as an OS for Veeam slower? Are there any tweaks to be done?

Many thanks in advance.
Alan

foggy
Veeam Software
Posts: 18037
Liked: 1533 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: MASSIVE performance hit on v8 (new VM)

Post by foggy » Nov 19, 2014 12:21 pm

Alan, are you Hyper-V or vSphere?

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: MASSIVE performance hit on v8 (new VM)

Post by Gostev » Nov 19, 2014 12:30 pm

Yes, we need to know your hypervisor to move this to the hypervisor-specific forum - you did not create the topic in the right one.

Also, do you have actual backup performance issue in terms of "wall clock" performance (job run time), or you just judging by statistics?

Because we keep tweaking performance display from version to version to make them more correct and useful for troubleshooting, so comparing job statistics numbers between versions is pointless. For example, specifically to full backups, in v8 we have stopped considering white space processing in the job counters (because jobs do not physically read these blocks from storage), which previously resulted in inflated performance numbers on virtual disks with lots of white space.

briggs
Enthusiast
Posts: 73
Liked: 3 times
Joined: Mar 22, 2013 11:01 am
Full Name: Alan Thompson
Contact:

Re: MASSIVE performance hit on v8 (new VM)

Post by briggs » Nov 20, 2014 9:13 am

vSpehere. ESXi 5.1. Sorry for the delay, I didn't get notification of any reply! Apologies for the location of this post, I haven't been on the forum for a while and this was a rather hurried post. Should I re-post of are you moving this?

briggs
Enthusiast
Posts: 73
Liked: 3 times
Joined: Mar 22, 2013 11:01 am
Full Name: Alan Thompson
Contact:

Re: MASSIVE performance hit on v8 (new VM)

Post by briggs » Nov 20, 2014 9:15 am

Hi Gostev, I rarely paid attention to the performance counters, as you identified, they are not always useful to give you the real picture. It was 'wall clock' performance that brought this to light, as the CPU alarms kicked in, because jobs that normally finish well before the working day were still running.

veremin
Product Manager
Posts: 16710
Liked: 1396 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: MASSIVE performance hit on v8 (new VM)

Post by veremin » Nov 20, 2014 9:25 am

Hi, Alan,

I've moved your post to the corresponding subforum. As to the your issue, it might be worth opening a ticket with our support team and letting them find the root cause of the experienced behaviour. In the meantime, can you pinpoint in the job statistics what steps considerably longer than they used to previously?

Thanks.

Vitaliy S.
Product Manager
Posts: 22777
Liked: 1526 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: MASSIVE performance hit on v8 (new VM)

Post by Vitaliy S. » Nov 20, 2014 12:00 pm

Additionally, I would suggest confirming that in both configurations v8 and v7 the bottleneck resource was the same.

briggs
Enthusiast
Posts: 73
Liked: 3 times
Joined: Mar 22, 2013 11:01 am
Full Name: Alan Thompson
Contact:

Re: MASSIVE performance hit on v8 (new VM)

Post by briggs » Nov 20, 2014 3:40 pm

Thanks guys. I am not getting consistent patterns, as is typical with this sort of thing. Some of the backups that have ran, have near identical processing times to the pre-upgrade server. Though one of the backups took nearly twice as long to run on the first night. The incrementals that are running now each day, seem fine as well. The backup that was particularly slow, shows the same bottleneck (pxy) as the previous job.
I think I will just monitor this closely for the next week or so, and log a ticket if it's happens to impact productivity.
In the mean time, if anyone has any comments on Windows 2012 R2 as an OS platform, I'm all ears.

Many thanks, Alan

veremin
Product Manager
Posts: 16710
Liked: 1396 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: MASSIVE performance hit on v8 (new VM)

Post by veremin » Nov 20, 2014 3:55 pm

Though one of the backups took nearly twice as long to run on the first night.
I'm wondering whether CBT was used during that run. Did you reset CBT after installing v8, as per upgrade procedure? Thanks.

Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests