Comprehensive data protection for all workloads
andreasaster
Influencer
Posts: 13
Liked: 2 times
Joined: Dec 24, 2014 8:40 am
Contact:

Merging oldest incremental Backup is painfully slow with v8

Post by andreasaster »

Hi,

we are using copy jobs to transfer our backup to a remote site. The copy jobs are running a merge after the copy process to merge the oldest restore point (vib file) with the vbk file. These merge operations are painfully slow, they are running with 1,5 Mb/s. The have vib files with about 500GB, so a single merge would require more then 50 hours to complete. We did not see this issue in v7. We already applied Patch 1 because of another issue, but the behavior is the same. Is there anyone else having this issue?

Thanks
Regards,
Andreas.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by foggy »

Andreas, do you have exactly the same setup as with v7 (same storages, repository types)?
andreasaster
Influencer
Posts: 13
Liked: 2 times
Joined: Dec 24, 2014 8:40 am
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by andreasaster »

Hi Foggy,

yes, we have exactly the same setup as with v7.

Andreas
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by veremin »

Then, feel free to open a ticket with our support team and let them find out the root cause of performance decrease. Thanks.
mmonroe
Enthusiast
Posts: 75
Liked: 3 times
Joined: Jun 16, 2010 8:16 pm
Full Name: Monroe
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by mmonroe »

Our backup copy jobs are *very* slow as well. The same job on V7 took 1hr 8min at 21 incrementals - this was back on 9-4-14 with V7 and all patches updated.

Today on V8, the same job with 21 incrementals took 6hr 49min. After loading V8, I had to delete the data and start it fresh. So basically a new full was created 21 days ago and incrementals (with no consolidation) have been building since.

In summary, our backup copy jobs are taking 6x longer and that is prior to reaching the point of consolidating the incrementals back into the full VBK.

This is the same server, same storage, same network, same operating system, san SAN, etc...
munklarsen
Influencer
Posts: 23
Liked: 9 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by munklarsen »

Hi,

We have the same issue as well. A job with 350 VMs and around 8TB storage takes 8-10 hours for the backup and almost 20 hours to merge data. I initially thought this was due to the new way veeam does incrementals where they removed the need for having a "new chain" completed before removing the previous chain. Could this be the reason why?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by veremin »

Did you have a chance to open a support ticket and let engineers confirm your environment? Thanks.
andreasaster
Influencer
Posts: 13
Liked: 2 times
Joined: Dec 24, 2014 8:40 am
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by andreasaster »

Eremin, I will open a support case to do that. Thanks
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by Gostev »

Checked with support and there are no known issues or similar support cases around transform performance in v8. Which is usually the sign that the issue either has a very limited scope, or is not related to the upgrade at all (otherwise, we usually get dozens of support cases created almost instantly).

Andreas and all, please post your support case IDs whenever you get a chance.
andreasaster
Influencer
Posts: 13
Liked: 2 times
Joined: Dec 24, 2014 8:40 am
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by andreasaster »

Hi Gostev, my case ID is #00715313. Regards, Andreas
HMwkr
Enthusiast
Posts: 33
Liked: 1 time
Joined: Nov 30, 2014 9:51 pm
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by HMwkr »

Just to chime in that I am also finding several tasks running exceedingly slowly to Synology NAS boxes (all DS412+) and await feedback on this thread with interest.

I've been working with the trial; licences have been ordered so I'm waiting for them to come through before raising support tickets on this and other issues.

As I have been trialling Veeam and trying to find a configuration which does what we need it to do in acceptable timescales, I don't yet have a stable configuration.

I'm not comparing V8 to V7 but to BackupExec 2010 - and also to V8 itself, as so far I'm not getting a reliable handle on speed of operations in V8 - as just one example, full backup on Saturday spends 1hr indexing the guest filing system, same job unchanged on Sunday spends 10x as long indexing - no change to data, network completely silently, etc etc. Not confidence-inspiring!

W
mwiseley
Influencer
Posts: 13
Liked: never
Joined: Jul 01, 2011 12:33 pm
Full Name: Mark Wiseley
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by mwiseley »

We had the same issue as well with our very large 9 TB backup copy job. We finally had to start all over and abandoned the copy job and just went back to doing a regular backup across the WAN as version 8 seemed to kill the performance when doing a backup copy. It was going to take it days to rollup the old job.
HMwkr
Enthusiast
Posts: 33
Liked: 1 time
Joined: Nov 30, 2014 9:51 pm
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by HMwkr »

Gah. First post to Veeam forums, and I post to the wrong thread (sorry). Then I find users can't edit or delete their own posts?! (*) That's not very friendly ...

* do the potential exceptions in the FAQ apply to these forums?
bruceb
Novice
Posts: 4
Liked: never
Joined: Dec 29, 2014 5:36 pm
Full Name: Bruce Barnett
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by bruceb »

Similar issue here.
214GB on a copy job trying to merge into oldest full backup and has been sitting at 30% for the past 4 days.
Case # 00716429
andreasaster
Influencer
Posts: 13
Liked: 2 times
Joined: Dec 24, 2014 8:40 am
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by andreasaster »

Got a feedback from Veeam today, they said, they found the root cause of the problem. I will have a phone call with them tomorrow to discuss the problem. I will update you then.

Andreas
roms
Influencer
Posts: 15
Liked: 1 time
Joined: Nov 27, 2014 3:43 pm
Full Name: Rachman Sosounov
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by roms »

Same problem in Backup Copy Job - Creating GFS restore point:

Merging oldest restore point into full backup file (41% done) - 29:44 hours
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by Vitaliy S. »

Rachman, what's your case ID and target storage for the backup copy job?
roms
Influencer
Posts: 15
Liked: 1 time
Joined: Nov 27, 2014 3:43 pm
Full Name: Rachman Sosounov
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by roms »

Case # 00717080
Target storage is iSCSI disk with good performance, the "copy backup" job working fast (191 MB/s).
Only when Creating GFS restore point: "Merging oldest restore point into full backup file" is slow.
munklarsen
Influencer
Posts: 23
Liked: 9 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by munklarsen »

As earlier mentioned, I have the same issues. I've created case #00716289.
bganjifard
Lurker
Posts: 2
Liked: never
Joined: Dec 10, 2014 10:14 pm
Full Name: Behrod Ganjifard
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by bganjifard »

any answers?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by veremin »

For any technical issue it's recommended to open a ticket with our support team, as most of the problems are environmental specific and cannot be addressed successfully via forum correspondence.

Thanks.
andreasaster
Influencer
Posts: 13
Liked: 2 times
Joined: Dec 24, 2014 8:40 am
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by andreasaster » 2 people like this post

We had a Webex yesterday. We have a very special constellation. A restorepoint is split in two vib files, so Veeam must merge these two vib files. This action is called overbuild and is very slow. The engineer is currently searching for a solution to speedup this process.
thorpenj
Novice
Posts: 4
Liked: never
Joined: Jan 03, 2015 2:42 pm
Full Name: Nick Thorpe
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by thorpenj »

I to am having the same issues. Is there a resolution yet ?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by Gostev »

Right now, there are no known issues around this functionality in the product. As far as I know, all situations reported so far have been caused by different unique circumstances (including most basic stuff like storage lacking IOPS), which is why we advise that everyone having similar issue opens a support case for investigations.
thorpenj
Novice
Posts: 4
Liked: never
Joined: Jan 03, 2015 2:42 pm
Full Name: Nick Thorpe
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by thorpenj »

I have a ticket raised 00717911 - but am still having the same issue
coldfire3000
Novice
Posts: 4
Liked: never
Joined: Feb 11, 2014 1:03 pm
Full Name: Chris Gundry
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by coldfire3000 »

I am just posting to get notifications.

We have several clients with very large backups (4TB+) and backup copy jobs that are on the verge of upgrade to v8 and I don't want to upgrade and then suffer these issues.
garoo
Lurker
Posts: 1
Liked: never
Joined: Jan 07, 2015 8:41 pm
Full Name: Gary Morris
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by garoo »

Same issue here guys. V7 was pretty good but since V8 we've had a marked slow down in the merging file times. We recently installed a new NAS to help with the issue and since the upgrade we're worse than before

Opening a ticket now. Anyone have any success? We're on the latest patch for V8
thorpenj
Novice
Posts: 4
Liked: never
Joined: Jan 03, 2015 2:42 pm
Full Name: Nick Thorpe
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by thorpenj »

Any results on this ? I'm still getting the same issue. Have a ticket open with support but think they are struggling with it to be honest. They suggested re-creating the Backup Copy job, but we use this as our off-site GFS repository for data servers. To be honest it is making the backup copy option unusable at the moment. We were looking to roll out Backup Copies to other sites but have had to put it on hold for the moment until we resolve the current issue.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by Gostev »

@Nick please never accept "solution" to recreate. It is pointless, as this will not fix the possible issue, but rather (at best) will only make it go away temporarily.
mabeeston
Novice
Posts: 8
Liked: never
Joined: Dec 20, 2014 12:43 pm
Full Name: Mark Beeston
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by mabeeston »

Watching this thread with interest - never used Backup Copy pre v8 but it does seem slow...worryingly slow. Unfortunately I've not got anything to compare to.

I know it depends on a number of factors but does anyone have a rough benchmark as to how long it should take to merge say 10GB of VIB?
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Semrush [Bot] and 274 guests