Comprehensive data protection for all workloads
Post Reply
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Slower backups after v7 upgrade?

Post by roadking »

So all of my backup jobs have gone from 80 - 120MB processing rate to 5MB.

Is this something new in v7 or am I missing something in the way it's calculating the processing rates?

Thanks in advance.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slower backups after v7 upgrade?

Post by foggy »

Robert, what else has been changed in your environment along with Veeam B&R upgrade? What are the bottleneck stats for the jobs that run slower than before? Have backup times increased after the upgrade or it is just the processing rate values that have dropped, but not the overall backup performance?

There are some changes in how processing rate is calculated, however, this should not result in slower backup speeds, indeed.
Starman
Enthusiast
Posts: 44
Liked: 10 times
Joined: Sep 27, 2011 5:11 pm
Full Name: Todd Leavitt
Contact:

Re: Slower backups after v7 upgrade?

Post by Starman » 1 person likes this post

When I moved to v7 I did a fresh build of server 2012 for it and my backup speed massively increased. To boot, the old Veeam server (server 2008R2 6.5, according to VCops) needed 6 proc's and the new one needs 2 proc's. With v7 I now have a ton of "backup copy" jobs added to the list that was running in 6.5 on the old server.

Why? 64bit and parallel processing!
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Re: Slower backups after v7 upgrade?

Post by roadking »

Nothing changed that I can attest to the backups. The only thing I've done was upgrade to v7 from 6.5

I will run some more tests to see if it's in the backup jobs themselves. They have not changed either but they all read in processing rate either 5 or 6 MB/s. Prior to the upgrade I would see the same rate (and the history shows) rates of 180 and higher on some jobs. Almost like the nic has lost it's 1GB negotiation -- and I did check that too :)

I did note that one of the proxys was set for appliance mode. Not sure if that makes any difference -- and I can't be sure if that was set for AUTO before the upgrade or not.

Will post more info after I test some freshly created jobs.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slower backups after v7 upgrade?

Post by tsightler »

Is it actually taking that much longer? That would be really interesting. Also, have you enabled any of the new V7 features?
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Re: Slower backups after v7 upgrade?

Post by roadking »

Enable what "new" v7 features?
wocomike
Influencer
Posts: 23
Liked: 1 time
Joined: Dec 16, 2009 12:50 am
Contact:

Re: Slower backups after v7 upgrade?

Post by wocomike »

Mine is taking longer after the upgrade as well. Backups that used to finished under an hour are now taking 2+ hours. Nothing at all has changed on my end. I literally just ran the installer and performed an upgrade.
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Re: Slower backups after v7 upgrade?

Post by roadking »

wocomike wrote:Mine is taking longer after the upgrade as well. Backups that used to finished under an hour are now taking 2+ hours. Nothing at all has changed on my end. I literally just ran the installer and performed an upgrade.
That's EXACTLY what I did. Ran the upgrade and let it rip.

I still don't understand what 'new v7 features' I needed to enable......
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slower backups after v7 upgrade?

Post by Gostev »

wocomike wrote:Nothing at all has changed on my end. I literally just ran the installer and performed an upgrade.
Believe or not, but literally every support case on performance issues after upgrade ends with these words. And in the end, every single case so far came down to some change in the environment or product configuration along, before or after upgrade. In the past 6 years, I literally don't remember us having a single backup engine bug that caused processing performance to drop after upgrade.

Anyhow, if you did not touch any settings, job run time should remain the same. Please send all logs to our support, they will compare the log between 6.5 and 7.0, and will tell you exactly what has changed in the job's behavior. Please collect the logs soon, otherwise 6.5 logs will be cleaned up by the automated log retention. Thanks!
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slower backups after v7 upgrade?

Post by Gostev »

roadking wrote:Enable what "new" v7 features?
Parallel processing and the new compression algorithm.
Starman
Enthusiast
Posts: 44
Liked: 10 times
Joined: Sep 27, 2011 5:11 pm
Full Name: Todd Leavitt
Contact:

Re: Slower backups after v7 upgrade?

Post by Starman »

Check your backups to verify which way they are being backed up. Maybe before you were doing direct to SAN or hot add and now its switched to network.
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Re: Slower backups after v7 upgrade?

Post by roadking »

Gostev wrote: Believe or not, but literally every support case on performance issues after upgrade ends with these words. And in the end, every single case so far came down to some change in the environment or product configuration along, before or after upgrade. In the past 6 years, I literally don't remember us having a single backup engine bug that caused processing performance to drop after upgrade.
You know what - all I wanted to do was ask a question. Not bash Veeam. I've never said that Veeam was at fault. I've also not said that our network was perfect. I was wondering if anyone else had this problem and at least one of you has. Evidently the two of us have the same crappy network.

I'm still testing options and working with someone from Veeam Support now. I'll update once I find anything of value. I have gathered 800+megs of logs and I'll upload them all too soon.

Thanks for the advice Gostev.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slower backups after v7 upgrade?

Post by Gostev »

Thank you for the logs (that is a lot), and sorry for coming across defensive...
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Re: Slower backups after v7 upgrade?

Post by roadking »

So working with Veeam Support on this -- they have asked me to check to ensure that the following are enabled --

Parallel processing - Checked and enabled
Max Concurrent tasks are set to maximum WITH a green check -- mine was set to 2 and I was able to bump that up to 4

After making these changes I can see that multiple VMs are being processed simultaneously where I don't recall seeing this before. The test job was from a failed job from last night and the speed of the overall job completed in 2:30 hours rather than 7:20 hours. MARKED IMPROVEMENT FOLKS!! Thanks Veeam - you really do rule -- and I knew that.

One last thing - -I'm getting this error now which never happened before -- and I did a search on the forum and found that others are seeing this too

9/23/2013 2:45:05 PM :: Error: Client error: End of file
Exception from server: The process cannot access the file because it is being used by another process.

Failed to create or open file [\\CIFSSHARE\Backup 022013-09-23T144230.vbk].
Failed to backup text locally. Backup: [veeamfs:0:54db516b-67f8-4791-bfd1-c13495af4428 (vm-1296)\summary.xml@\\<BackupShareDirHere> 022013-09-23T144230.vbkVBK: 'veeamfs:0:54db516b-67f8-4791-bfd1

Still searching and don't know what to make of this one. I see some suggestions that the CIFS share may be too slow for this?

Thanks a lot folks -- making headway and all positive :)
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Re: Slower backups after v7 upgrade?

Post by roadking »

I also wonder if this error could be due to me re-running the jobs too soon for a Veeam Background process to complete -- like a retention period "cleanup" -- I do have them set pretty low - saving only 3 backups due to disk space constraints.

Thanks a again....
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Slower backups after v7 upgrade?

Post by chrisdearden »

I think there may be something support regarding that error when using a CIFS repository and parallel processing. might be worth checking back with them.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slower backups after v7 upgrade?

Post by foggy »

And while you're getting back from support, please review this thread discussing similar issue. Could indeed be caused by slow CIFS share performance.
roadking
Enthusiast
Posts: 25
Liked: 1 time
Joined: Mar 22, 2012 3:42 pm
Full Name: Robert Martin
Contact:

Re: Slower backups after v7 upgrade?

Post by roadking »

roadking wrote:So working with Veeam Support on this -- they have asked me to check to ensure that the following are enabled --
One last thing - -I'm getting this error now which never happened before -- and I did a search on the forum and found that others are seeing this too
9/23/2013 2:45:05 PM :: Error: Client error: End of file
Exception from server: The process cannot access the file because it is being used by another process.
Failed to create or open file [\\CIFSSHARE\Backup 022013-09-23T144230.vbk].
Failed to backup text locally. Backup: [veeamfs:0:54db516b-67f8-4791-bfd1-c13495af4428 (vm-1296)\summary.xml@\\<BackupShareDirHere> 022013-09-23T144230.vbkVBK: 'veeamfs:0:54db516b-67f8-4791-bfd1
Still searching and don't know what to make of this one. I see some suggestions that the CIFS share may be too slow for this?
Thanks a lot folks -- making headway and all positive :)
Oh and I see there's a patch now that addresses this issue - -where I was told by support that my problem was due to slow storage? Interesting.............

Under certain circumstances, backup to CIFS target may fail with the following errors: “The process cannot access the file because it is being used by another process”, “Failed to create or open file \\server\share\file.vbk”."
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slower backups after v7 upgrade?

Post by Gostev »

Could have been a confusion because you posted your issue in the "Slower backups after v7 upgrade" thread?
Post Reply

Who is online

Users browsing this forum: Google [Bot], Mildur and 151 guests