Slower backups after v7 upgrade?

[roadking]

So all of my backup jobs have gone from 80 - 120MB processing rate to 5MB.

Is this something new in v7 or am I missing something in the way it's calculating the processing rates?

Thanks in advance.

[foggy]

Robert, what else has been changed in your environment along with Veeam B&R upgrade? What are the bottleneck stats for the jobs that run slower than before? Have backup times increased after the upgrade or it is just the processing rate values that have dropped, but not the overall backup performance?

There are some changes in how processing rate is calculated, however, this should not result in slower backup speeds, indeed.

[Starman]

When I moved to v7 I did a fresh build of server 2012 for it and my backup speed massively increased. To boot, the old Veeam server (server 2008R2 6.5, according to VCops) needed 6 proc's and the new one needs 2 proc's. With v7 I now have a ton of "backup copy" jobs added to the list that was running in 6.5 on the old server.

Why? 64bit and parallel processing!

[roadking]

Nothing changed that I can attest to the backups. The only thing I've done was upgrade to v7 from 6.5

I will run some more tests to see if it's in the backup jobs themselves. They have not changed either but they all read in processing rate either 5 or 6 MB/s. Prior to the upgrade I would see the same rate (and the history shows) rates of 180 and higher on some jobs. Almost like the nic has lost it's 1GB negotiation -- and I did check that too

I did note that one of the proxys was set for appliance mode. Not sure if that makes any difference -- and I can't be sure if that was set for AUTO before the upgrade or not.

Will post more info after I test some freshly created jobs.

[tsightler]

Is it actually taking that much longer? That would be really interesting. Also, have you enabled any of the new V7 features?

[roadking]

Enable what "new" v7 features?

[wocomike]

Mine is taking longer after the upgrade as well. Backups that used to finished under an hour are now taking 2+ hours. Nothing at all has changed on my end. I literally just ran the installer and performed an upgrade.

[roadking]

Mine is taking longer after the upgrade as well. Backups that used to finished under an hour are now taking 2+ hours. Nothing at all has changed on my end. I literally just ran the installer and performed an upgrade.

That's EXACTLY what I did. Ran the upgrade and let it rip.

I still don't understand what 'new v7 features' I needed to enable......

[Gostev]

Nothing at all has changed on my end. I literally just ran the installer and performed an upgrade.

Believe or not, but literally every support case on performance issues after upgrade ends with these words. And in the end, every single case so far came down to some change in the environment or product configuration along, before or after upgrade. In the past 6 years, I literally don't remember us having a single backup engine bug that caused processing performance to drop after upgrade.

Anyhow, if you did not touch any settings, job run time should remain the same. Please send all logs to our support, they will compare the log between 6.5 and 7.0, and will tell you exactly what has changed in the job's behavior. Please collect the logs soon, otherwise 6.5 logs will be cleaned up by the automated log retention. Thanks!

[Gostev]

Enable what "new" v7 features?

Parallel processing and the new compression algorithm.

[Starman]

Check your backups to verify which way they are being backed up. Maybe before you were doing direct to SAN or hot add and now its switched to network.

[roadking]

Believe or not, but literally every support case on performance issues after upgrade ends with these words. And in the end, every single case so far came down to some change in the environment or product configuration along, before or after upgrade. In the past 6 years, I literally don't remember us having a single backup engine bug that caused processing performance to drop after upgrade.

You know what - all I wanted to do was ask a question. Not bash Veeam. I've never said that Veeam was at fault. I've also not said that our network was perfect. I was wondering if anyone else had this problem and at least one of you has. Evidently the two of us have the same crappy network.

I'm still testing options and working with someone from Veeam Support now. I'll update once I find anything of value. I have gathered 800+megs of logs and I'll upload them all too soon.

Thanks for the advice Gostev.

[Gostev]

Thank you for the logs (that is a lot), and sorry for coming across defensive...

[roadking]

So working with Veeam Support on this -- they have asked me to check to ensure that the following are enabled --

Parallel processing - Checked and enabled
Max Concurrent tasks are set to maximum WITH a green check -- mine was set to 2 and I was able to bump that up to 4

After making these changes I can see that multiple VMs are being processed simultaneously where I don't recall seeing this before. The test job was from a failed job from last night and the speed of the overall job completed in 2:30 hours rather than 7:20 hours. MARKED IMPROVEMENT FOLKS!! Thanks Veeam - you really do rule -- and I knew that.

One last thing - -I'm getting this error now which never happened before -- and I did a search on the forum and found that others are seeing this too

9/23/2013 2:45:05 PM :: Error: Client error: End of file
Exception from server: The process cannot access the file because it is being used by another process.

Failed to create or open file [\\CIFSSHARE\Backup 022013-09-23T144230.vbk].
Failed to backup text locally. Backup: [veeamfs:0:54db516b-67f8-4791-bfd1-c13495af4428 (vm-1296)\summary.xml@\\<BackupShareDirHere> 022013-09-23T144230.vbkVBK: 'veeamfs:0:54db516b-67f8-4791-bfd1

Still searching and don't know what to make of this one. I see some suggestions that the CIFS share may be too slow for this?

Thanks a lot folks -- making headway and all positive

[roadking]

I also wonder if this error could be due to me re-running the jobs too soon for a Veeam Background process to complete -- like a retention period "cleanup" -- I do have them set pretty low - saving only 3 backups due to disk space constraints.

Thanks a again....

[chrisdearden]

I think there may be something support regarding that error when using a CIFS repository and parallel processing. might be worth checking back with them.

[foggy]

And while you're getting back from support, please review this thread discussing similar issue. Could indeed be caused by slow CIFS share performance.

[roadking]

So working with Veeam Support on this -- they have asked me to check to ensure that the following are enabled --
One last thing - -I'm getting this error now which never happened before -- and I did a search on the forum and found that others are seeing this too
9/23/2013 2:45:05 PM :: Error: Client error: End of file
Exception from server: The process cannot access the file because it is being used by another process.
Failed to create or open file [\\CIFSSHARE\Backup 022013-09-23T144230.vbk].
Failed to backup text locally. Backup: [veeamfs:0:54db516b-67f8-4791-bfd1-c13495af4428 (vm-1296)\summary.xml@\\<BackupShareDirHere> 022013-09-23T144230.vbkVBK: 'veeamfs:0:54db516b-67f8-4791-bfd1
Still searching and don't know what to make of this one. I see some suggestions that the CIFS share may be too slow for this?
Thanks a lot folks -- making headway and all positive

Oh and I see there's a patch now that addresses this issue - -where I was told by support that my problem was due to slow storage? Interesting.............

Under certain circumstances, backup to CIFS target may fail with the following errors: “The process cannot access the file because it is being used by another process”, “Failed to create or open file \\server\share\file.vbk”."

[Gostev]

Could have been a confusion because you posted your issue in the "Slower backups after v7 upgrade" thread?