Veeam v9 Backup Performance Slow

Availability for the Always-On Enterprise

Veeam v9 Backup Performance Slow

Veeam Logoby cbc-tgschultz » Fri May 13, 2016 2:08 pm 2 people like this post

Is anyone else having issues with performance since upgrading to v9?

For a week now we've been unable to do our daily incrementals because jobs refuse to finish in a reasonable time frame. I've taken to disabling all jobs and running them one at a time as active fulls while working with support, who thus far have had very little productive input. Case number: 01794249

Here's the kind of thing I'm seeing:
Image

This is typical of the issue I'm having. Some things will run fine, but a lot of the time veeam will just stop sending network traffic entirely, drop throughput to 0, and blame the network for being a bottleneck. Meanwhile I'm watching my storage system (Linux repository) idle.
cbc-tgschultz
Enthusiast
 
Posts: 46
Liked: 9 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Veeam v9 Backup Performance Slow

Veeam Logoby PTide » Fri May 13, 2016 3:39 pm

Hi,

Please describe you backup repository configuration, proxy configuration and backup network.

Thank you.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Veeam v9 Backup Performance Slow

Veeam Logoby cbc-tgschultz » Fri May 13, 2016 4:10 pm

Sure, it should all be in the ticket, but I'll describe it again:

The backup repository is a Docker container running Ubuntu. It is running on a Synology RS2414RP+ appliance with 12 8TB Seagate 5400RPM drives in a RAID6 connected via 1Gb/s ethernet to a Cisco Nexus 3548. It is configured for a maximum of 2 concurrent connections.

The proxy is the B&R server. It is a 4proc VM with 24GB of Ram running in a 4node vSphere 6 environment connected via 10Gb to the same Nexus switch. It is configured for a maximum of 2 concurrent tasks.

Again, since Veeam is confirmed to not be sending any data during these 0-throughput times, I don't see what it matters what the storage or network is. The problem is that Veeam is not putting any data on the network. The logs (I've sent many) straight up say the repository agent is timing out waiting for blocks. Is Veeam not able to retrieve blocks from vSphere? Maybe, but it also has the same issue with the Hyper-V jobs, and doesn't report any issue in the logs that I can see. When it does send data, it often does so at <1Mb/s

So, to reiterate: Veeam is not sending data to the backup repository for long periods of time during jobs.
cbc-tgschultz
Enthusiast
 
Posts: 46
Liked: 9 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Veeam v9 Backup Performance Slow

Veeam Logoby cbc-tgschultz » Fri May 13, 2016 7:17 pm

Based on correspondence with support, it would seem that the issue was caused by having the B&R server and the proxy be one and the same. This configuration was working for us in v8, but I'm guessing v9 adds additional load that made it all too much to handle. Adding a new 16GB 4proc 3.40Ghz VM as a dedicated proxy and disabling the B&R server as a proxy seems to have had a dramatic effect on performance. Why it took the better part of a week to get to this point with support I can only guess.
cbc-tgschultz
Enthusiast
 
Posts: 46
Liked: 9 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Veeam v9 Backup Performance Slow

Veeam Logoby cbc-tgschultz » Sun May 15, 2016 2:38 am

Sadly I must report that I am still seeing performance issues even with the new, separate, proxy.

Initial results were promising, when running a single job it would often have throughput at or over 100MB/s. Unfortunately, as soon as a second job that uses that repository kicks off, performance suffers greatly. Before all this, we could run 2 server jobs at once and, though each had only about half the throughput they normally would, they would still run smoothly. Now, whenever I attempt to run two at once, performance for both jobs drops to <1MB/s.

This on its own would almost be tolerable, however I am now seeing that longer running jobs are showing the same issue as before, even when they are the only job running:

Image

Note how the job seems to begin with promising performance, only to later become tragically slow. Veeam keeps blaming the network, which makes no sense. If Veeam would better report what the actual problem is I suspect I'd be much closer to remedying it. Given the amount of time I have spent on this issue, the impact it is having on our ability to backup production data, and the relative lack of progress, I expect I'll soon be forced to revert to v8 and hope that it's just something about v9 that is causing it. Support assures me there haven't been any changes that should be having a performance effect, but then again our B&R server/proxy combo was working just fine before v9, so I don't know if I can trust them on that.

If anyone with the power to do so could escalate this ticket, please do. You may notice this post is late at night on a saturday. I'd rather not have many future weekend hours blown dealing with this.
cbc-tgschultz
Enthusiast
 
Posts: 46
Liked: 9 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Veeam v9 Backup Performance Slow

Veeam Logoby rasmusan » Sun May 15, 2016 6:51 pm

Hi

I had similar issue at a Customer, though it was not after an upgrade, but otherwise much of the same symptoms as you describe. Also i had a support case that really did not conclude anything, until i found out the antivirus was doing some kind of network traffic scanning that somehow hit performance of veeam traffic very hard. After disabling that all was well...
rasmusan
Enthusiast
 
Posts: 26
Liked: never
Joined: Tue Jan 20, 2015 9:03 pm
Full Name: Rasmus Andersen

Re: Veeam v9 Backup Performance Slow

Veeam Logoby PTide » Mon May 16, 2016 10:03 am

Hi,

I want to check something before suggesting you to escalate the support case - you've mentioned that both your proxy and repository are configured for a maximum of 2 concurrent tasks, while in your job statistics I can see that there 4 VMs in that job in total which is twice as many as the number of concurrent tasks allowed. Also I see things like "Waiting for backup infrastructure resources availability", "Resource not ready: Backup repository", "Could not allocate processing resources within allotted timeout (82800 sec) Error: Timed out waiting for backup infrastructure resources to become available (82800 sec)" and so on. Based on this I suggest you first to raise the limit of tasks to 4 (both for proxy and repository) and see if that helps, because it's clear that you have at least 4 vmdks to be processed and only 2 proxy and repo slots for allocation. Please try to change the limit and if that gives no positive impact on the performance then kindly use "Talk to manager" button to escalate the case to the higher support Tier.

Thank you
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Veeam v9 Backup Performance Slow

Veeam Logoby cbc-tgschultz » Mon May 16, 2016 1:13 pm

Your suggestion doesn't make any sense to me. Even with only a single task running the job was super slow. How would allowing more jobs have helped? Indeed, my experience has been that even if a job is running well, as soon as I kick off a second job everything tanks hard.

Anyway, thanks for the suggestion rasmusan, but there is no antivirus running on the B&R server, proxy, or repository. I have extensively verified that no process other than Veeam is gobbling up resources.

Over the weekend we finally had some indication as to what the remaining problem might be. It would be an issue with the storage array causing high write latency under certain circumstances. If true, it would mean that Veeam is not at fault for the continuing issue, though I do wonder why neither I, nor support, could surface the issue via Veeam's logs and reporting. Perhaps there is some improvement to be had there?
cbc-tgschultz
Enthusiast
 
Posts: 46
Liked: 9 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Veeam v9 Backup Performance Slow

Veeam Logoby dellock6 » Mon May 16, 2016 1:37 pm 1 person likes this post

Tanner, looking at the writing graph, the times with no activity could be indeed waiting times for the array to be ready again to write.
I'd suggest to be honest the opposite, to verify if the array is overloaded: what about "reducing" the concurrent threads? With less threads hitting the storage it may be able to flush its cache more efficiently and it may (not guaranteed) perform better.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Veeam v9 Backup Performance Slow

Veeam Logoby cbc-tgschultz » Tue May 17, 2016 1:14 pm 1 person likes this post

While the repository is, under normal circumstances, set for 2 concurrent tasks, it has been set to 1 since this issue began.

After observing over the past few days, I am quite confident that the repository was to blame for the additional issues we experienced even after separating the proxy from the B&R server. I believe there is something wrong with one of the drive slots, though it is possibly the disk, causing high write latency. When this slot errored enough to be auto-disabled our performance returned to normal. We are now operating smoothly again.

I am still disappointed this issue didn't surface during our collective troubleshooting. Veeam kept reporting "Network" as the issue when, given the problem, one would expect it to be "Target". Additionally, neither I nor support spotted anything in the logs to indicate a high write latency.
cbc-tgschultz
Enthusiast
 
Posts: 46
Liked: 9 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Veeam v9 Backup Performance Slow

Veeam Logoby larry » Thu May 19, 2016 7:59 pm

Had same issue, graph looked the same with gaps, speeds would drop to 1 mb or zero. My issue was also a failing drive in the repository, veeam did show target was the issue. What got us was the server had two repositories on two disks but both were some what effected but one was worse. The desktop of that console would also get laggy.
larry
Expert
 
Posts: 372
Liked: 89 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: Veeam v9 Backup Performance Slow

Veeam Logoby Gostev » Sun May 22, 2016 8:02 pm 1 person likes this post

One thing we can be sure is that this performance issue is not specific to v9, because I just saw someone post a screenshots with 2GB/s processing rate for full backup with v9... that's like 200 times faster than what you are getting during most of the time, looking at the graphs above.

Performance numbers around 10 MB/s always make me suspect a network problem first, most commonly due to some port on switch or NIC failing over from 1Gb to 100Mb.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Veeam v9 Backup Performance Slow

Veeam Logoby MerlIT » Mon May 23, 2016 7:37 am

May be of no relevance, but I have regular "slow backups" under v8 using a Synology repository attached via two path MPO iSCSI, giving very similar throughputs to yours and the same bottleneck. Multiple cases with Veeam support (most recently 00984158) never got to the bottom of this. Slow backups are accompanied by low transmission rates on the iSCSI connection. If I log on to the Synology or host 2012.R2 server during the "slow backup", there is a good chance that the iSCSI performance will revert to full speed and the backup continue at the normal pace. If I start getting slow backups, then they continue until the host 2012.R2 server is rebooted. I think there is an issue with the MS lazywriter on 2012.R2 and iSCSI (either at Synology or MS end). There are no specific errors, dropped packets etc to help troubleshoot this.

All my Veeam components, bar the repository, sit on the one 2012.R2 host.

I was hoping that the recent Synology upgrade plus v9 would solve this once and for all!
MerlIT
Novice
 
Posts: 4
Liked: never
Joined: Wed Jul 23, 2014 7:06 pm
Full Name: Andrew Sims

Re: Veeam v9 Backup Performance Slow

Veeam Logoby Andreas Neufert » Mon May 23, 2016 8:43 am

Tanner, can you please have a look at the Jobs. When you see that there is no data processed. What is the job status then. What did the job statistic say?
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: Veeam v9 Backup Performance Slow

Veeam Logoby MSc » Mon May 23, 2016 9:00 am

Hi all!

Synology had major problems with their DSM 6.0 and iSCSI mapping, our system just came back to a useable status with DSM 6.0 Update 7. Please check that you're up-to-date with the DSM versions! Just a thought... as our backups are going well again.
MSc
Lurker
 
Posts: 2
Liked: never
Joined: Thu Jan 19, 2012 2:46 pm
Location: Germany
Full Name: Martin Schenker

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot] and 12 guests