v6 performance decreased compared to v5

[toabama]

Hi
When we upgraded our existing v5 installation of Veeam B&R to v6 we experience lots of performance issues.
Our installation goes to using just one server handling everything without the Veeam database itself, it is hosted in a separate MS SQL cluster, this was also the recomendation from Veeam. The server itself has direct SAN connectivity dual port HBA, running Windows server 2008 R2 enterprise, server is a HP Proliant DL 385 G7 12 core CPU 8 GB RAM, destination storage is HP MSA 60 with 2TB SAS drives in 2 volumes with RAID 5.
Is there anyone else running this kind of installation that have seen decreased performance after upgrading from v5 -> v6.
Mainly jobs take longer to finish, more errors about performance and so on. Also the GUI is stealing 400MB RAM per user which seems quite strange.
Also this error got logged in Veeam tonight:
Error: Client error: Insufficient system resources exist to complete the requested service
Failed to read data from the file

So my question is how have you solved these kind of performance issues if you have any after the upgrade?
Or is it just our installation that is corrupted.

We are running latest patched version of Veeam B&R.

[Gostev]

Hi,

Kindly please include:
1. The 4 bottleneck statistics numbers from the real-time stats window - when posting about any performance issues.
2. The support case number - when posting about any technical issue (otherwise, your post will be removed, as explained when you click New Topic).

The UI process have been using 350-400MB per process in all previous product versions.

Thanks!

[toabama]

Hi

We have no ongoing case right now, this is just a general question regarding performance for other users to answer.
Should there not be possibilities to read what other users experience before submitting case to support?
Bottleneck statistics varies so i cannot give any specific details about that.

BR
Tobias

[Gostev]

Could you please post the bottleneck stat numbers from the specific job run that had "bad" performance in your opinion? We cannot really help you with generic "I have bad performance" query without seeing those numbers. Any performance problems can only be caused by the bottleneck present in your backup infrastructure, and bottleneck stats clearly identify one. Thanks!

[toabama]

So there is no interesterd in generic problems like feel and thoughts on perfromance between versions of B&R?
Those figures in GUI is for mer just a hint but in a job i can see these figures just switching from backing up one server to another

Copied from GUI where Busy source 95%

2012-01-20 23:25:20 :: Primary bottleneck: Target

Copied from GUI where Busy source 31%

2012-01-20 23:25:20 :: Primary bottleneck: Target

[Gostev]

Yes, no real interest in "feel and thoughts", only in facts and numbers. Because in the past years, all complaints on performance reduction after upgrade were found to be environment related, and had nothing to deal with the new engine.

For example, in your case, it is obvious now that bottleneck is the target storage performance. May be you are hitting your target storage with too many concurrent jobs at once - try limiting concurrent jobs amount until your target no longer sits at 99% busy all the time. This can be done in the backup repository settings (if you have created single backup repository for this storage), or by disabling some backup proxies - if you have too many proxies.

Also, if you are using reversed incremental backup mode, then changing the mode to incremental will also take off much extra load from your target storage (at the cost of required disk space, of course).

Thanks.

[toabama]

Ofcourse this could be something else but we have not changed our hardware before upgrading to v6 but we now see a performance decrease and jobs run much longer or fail often due to performance problems. We do not run more concurrent jobs than before and the Veeam software can be configured for running more jobs at once but we have choosen not to do that because of the problems.

[rhnb]

Have you checked that you've not got the known CBT issue (CBT failing on disks other than HD1 on some VM's - usually data disks so the very ones where you want CBT used). It's certainly slowed our backups down.
Veeam know about the issue and are working on a solution - they reckoned about a month a couple of weeks ago.
Worth checking that's not your issue.

[Gostev]

We do not run more concurrent jobs than before

OK, but you are ignoring the fact that a single v6 job is able to deliver significantly better throughput than v5 job due to multiple engine optimizations. Depending on the source storage, I've seen people reporting 2x performance improvement over v5 when running a single job. So, even if you keep the same amount of concurrent jobs running as in v5, this still means potentially 2x load on the target storage than with the same amount of jobs in v5.

Again, you really should reduce the amount of concurrent jobs hitting the same storage to make sure your target does not sit at 99% per bottleneck stats. Due to the nature of hard drive based storage, trying to write data with too many concurrent concurrently makes the performance drop to the floor (hard drives will spend time seeking most of the time). You will see jobs running much faster as soon as your target stops reporting being 99% busy - your backups will actually complete much faster despite fewer jobs running at the same time (because each particular job will be completing so much faster).

[Gostev]

Allan, the issue you are talking about is not new to v6, we had similar support cases for v5 as well as the issue sits in VMware. The only difference is the fact that v6 makes heavy use of hot add, and it also made creating hot add proxies super easy. So, a lot more people started using hot add with v6, which made the amount of support cases for this issue achieve the critical volume that was enough to justify addressing the issue from our side. Thanks.

[rhnb]

I hear what you're saying Gostev, but I hope you can see this from my point of view.

We had been using Veeam 5 since it came out with not a single CBT problem.
I upgrade to v6 and 'bang' (so it would seem natural to me that I would suspect v6 is the cause).
You say v6 makes 'heavy use of hotadd'. Not sure I understand that, you're either using hotadd or you're not. We always have and pre v6, no CBT issues.
OK, some people may have seen this in v5 (obviously not many or you'd surely have done something about it by now), but it's obviously orders of magnitude worse in v6. Well from my perspective anyway.

[lloyd.mills]

Hello,

We have recently done an in place upgrade, and are also getting a decrease in performance.

The only change is that now less job's run concurrently.

The bottleneck is reporting as the target.

Is there an inefficiency in the way that Veeam 6 writes to disk? Or performs reverse incrementals?

[toabama]

Hi
Great to hear that some more users have the same experience as we have.
We added more memory to our backup server and as i can see Windows eats the memory but now it don't eats all emomry as before and the jobs are not halting that often anymore. We upgraded from 8GB -> 16GB
BR
tobias

[foggy]

Lloyd, have you considered Anton's explanations above?

So, even if you keep the same amount of concurrent jobs running as in v5, this still means potentially 2x load on the target storage than with the same amount of jobs in v5.

Again, you really should reduce the amount of concurrent jobs hitting the same storage to make sure your target does not sit at 99% per bottleneck stats. Due to the nature of hard drive based storage, trying to write data with too many concurrent concurrently makes the performance drop to the floor (hard drives will spend time seeking most of the time). You will see jobs running much faster as soon as your target stops reporting being 99% busy - your backups will actually complete much faster despite fewer jobs running at the same time (because each particular job will be completing so much faster).

[lloyd.mills]

Foggy,

We already run less concurrent jobs than previously with veeam 5.

The only change as i say is Veeam 6.

What i'm finding hard to understand is how a reduced job speed can increase the load of the target storage.

So, for example we are getting 70% of the speed we were with Veeam 5, and it's saying there is 99% usage on the target storage.

If we had 2x the load on the target storage, I would expect 2x the performance, or at least some performance increase and certainly not a decrease. Hence my question's:

"Is there an inefficiency in the way that Veeam 6 writes to disk? Or performs reverse incrementals?"

[foggy]

We already run less concurrent jobs than previously with veeam 5.

How many jobs are writing to the same target simultaneously?

What i'm finding hard to understand is how a reduced job speed can increase the load of the target storage.
So, for example we are getting 70% of the speed we were with Veeam 5, and it's saying there is 99% usage on the target storage.

Job speed is actually increased in v6. As Anton said above, v6 job optimizations allow to provide up to 2x better throughput comparing to v5 job. This obviously implies the load increase on target side as data are delivered to target much faster.

If we had 2x the load on the target storage, I would expect 2x the performance, or at least some performance increase and certainly not a decrease.

Wrong expectations. As at some point target just goes to choke with incoming amount of data and drives start to seek most of the time instead of writing the data. If you decrease the load, the data will be written much faster so the whole job performance will beat the one for v5.

[lloyd.mills]

2 Jobs are run simultaneously, the same as previously with Veeam 5. There is also not much difference when you run a single job.

So by making the Veeam Agent more efficient and increasing the throughput to the target storage, this is actually over loading the target storage and in turn decreasing performance?

Can i ask my question again:

"Is there an inefficiency in the way that Veeam 6 writes to disk? Or performs reverse incrementals?"

Unfortunately we dont have any stats from the SAN to see the actual load on the lun's.

[foggy]

2 Jobs are run simultaneously, the same as previously with Veeam 5. There is also not much difference when you run a single job.

So by making the Veeam Agent more efficient and increasing the throughput to the target storage, this is actually over loading the target storage and in turn decreasing performance?

This could be the case. But seems it is not. If your single job performance is decreased comparing to the same job in v5, then I would suggest to get support guys involved here.

[AJ83]

I also have a decreased performance after the in place upgrade. Nothing is changed just v5 to v6 upgrade. The jobs take 20 to 30% longer to complete.

Our situation is as follows:
We have 2 installations of Veeam B&R, 1 on premise, 1 in co-location. The situations are exactly the same.

On-premise:
1 physical Veeam server with FC HBA`s (which is the veeam backup server and proxy)
1 8x2TB NAS connected through 1GB Ethernet, accessed through an SMB share.
5 ESX hosts with the data on SAN

NOTMAL JOB Bottleneck = Target
24-1-2012 22:08:53 :: Load: Source 22% > Proxy 53% > Network 79% > Target 91%
VSS JOB Bottleneck = Target
25-1-2012 7:58:07 :: Load: Source 15% > Proxy 57% > Network 89% > Target 98%

Co-location:
1 physical Veeam server with FC HBA`s (which is the veeam backup server and proxy)
1 8x2TB NAS connected through 1GB Ethernet, accessed through an SMB share.
4 ESX hosts with the data on SAN

NORMAL JOB1 Bottleneck = Target
25-1-2012 4:11:03 :: Load: Source 33% > Proxy 55% > Network 74% > Target 86%
VSS JOB1 Bottleneck = Target
25-1-2012 5:42:24 :: Load: Source 47% > Proxy 58% > Network 54% > Target 81%
NORMAL JOB2 Bottleneck = Source
24-1-2012 22:14:17 :: Load: Source 99% > Proxy 70% > Network 2% > Target 24%
VSS JOB2 Bottleneck = Source
24-1-2012 23:07:20 :: Load: Source 99% > Proxy 74% > Network 2% > Target 42%

The JOB2 machines have the source bottleneck because they have a small delta, so not much is actually copy`d.

All data is transferred on LAN (so no WAN in the mix) and there are no jobs running concurrently (never have).

I`ve opened a supportcase for it, but they tell me to put more proxy`s into the mix, which is odd, since it will only add an extra station for the data to pass through.

[lloyd.mills]

how accurate are the bottleneck statitics gathered by veeam?

[foggy]

It is accurate enough to understand what the weakest link is. The general idea of this feature is described in the FAQ.

[lloyd.mills]

Little update.

I installed 16gb of additional memory into 2 of our backup servers, bringing the total in each upto 20gb. The result?

Backup's now fly. The backup servers are utilising all of the additional memory. The bottleneck now flips between the source and the destination. We are able to run concurrent jobs with seemingly no huge performance hit.

So does Veeam 6 require the additional memory?

I'm curious as to why when we had slow performance the bottleneck was identified as the target, but increasing the resources in the proxy and repository (both the same server), which showed as under utilised, has increased the performance of the backup job's. Should that not have remained the same if the target was "99% busy"?

[lloyd.mills]

Just to clarify here are the backup times from the past week of a 2.1TB job of 4 VM's set to reverse incrementals with 30 restore points.

Monday :- 10:59
Tuesday :- 10:05
Wednesday :- 10:41
Thursday :- 4:10

So the backup was almost 3 times quicker with the extra memory.

[foggy]

Ok, I may try to explain that. What is important here is that the proxy and repository are both on the same server. We get the percent busy number for target component from the agent running on the backup repository. Either in the case when target storage is slow, or the network between repository and the actual storage is slow, or there are not enough resources on repository to provide data to be written to target - all these situations are identified as bottleneck Target. I think in your case, when proxy and repository are installed on the same server, target agent just had not enough resources to process all the data reсeived from the proxy (as these were occupied by proxy performing its very resource intensive tasks) reporting itself as bottleneck.
We would like have your log files though to see what was happened.

[lloyd.mills]

I can provide logs if you want me to?

[foggy]

Thanks, much appreciated. You can create a support case attaching the logs and post the ID here.

[Jamie Pert]

Lots of my backup jobs failed last night with the error below:

Error: Client error: Insufficient system resources exist to complete the requested service
Failed to read data from the file

I noticed that when a particular ob runs it seriously hogs system RAM (2GB just for Veeam) and overall physical RAM usage sits at 94% with just 1 job running, so is that why it's failing?

[foggy]

Jamie, what is your support case number? For all technical issues, you need to open a support case and provide full logs for investigation.

[Jamie Pert]

Sorry I have been looking into a lot of Veeam issues of late as it's my job to investigate them all - I didn't want to overload you guys with jobs for just me, I have about a dozen warning messages and 4 errors to look into. I am working off a list which I want to not only tick off as "success" but fully understand the reasons for warnings and errors for future reference.

[foggy]

You could try recommendations from this thread.