Merging oldest incremental Backup is painfully slow with v8

[andreasaster]

Hi,

we are using copy jobs to transfer our backup to a remote site. The copy jobs are running a merge after the copy process to merge the oldest restore point (vib file) with the vbk file. These merge operations are painfully slow, they are running with 1,5 Mb/s. The have vib files with about 500GB, so a single merge would require more then 50 hours to complete. We did not see this issue in v7. We already applied Patch 1 because of another issue, but the behavior is the same. Is there anyone else having this issue?

Thanks
Regards,
Andreas.

[foggy]

Andreas, do you have exactly the same setup as with v7 (same storages, repository types)?

[andreasaster]

Hi Foggy,

yes, we have exactly the same setup as with v7.

Andreas

[veremin]

Then, feel free to open a ticket with our support team and let them find out the root cause of performance decrease. Thanks.

[mmonroe]

Our backup copy jobs are *very* slow as well. The same job on V7 took 1hr 8min at 21 incrementals - this was back on 9-4-14 with V7 and all patches updated.

Today on V8, the same job with 21 incrementals took 6hr 49min. After loading V8, I had to delete the data and start it fresh. So basically a new full was created 21 days ago and incrementals (with no consolidation) have been building since.

In summary, our backup copy jobs are taking 6x longer and that is prior to reaching the point of consolidating the incrementals back into the full VBK.

This is the same server, same storage, same network, same operating system, san SAN, etc...

[munklarsen]

Hi,

We have the same issue as well. A job with 350 VMs and around 8TB storage takes 8-10 hours for the backup and almost 20 hours to merge data. I initially thought this was due to the new way veeam does incrementals where they removed the need for having a "new chain" completed before removing the previous chain. Could this be the reason why?

[veremin]

Did you have a chance to open a support ticket and let engineers confirm your environment? Thanks.

[andreasaster]

Eremin, I will open a support case to do that. Thanks

[Gostev]

Checked with support and there are no known issues or similar support cases around transform performance in v8. Which is usually the sign that the issue either has a very limited scope, or is not related to the upgrade at all (otherwise, we usually get dozens of support cases created almost instantly).

Andreas and all, please post your support case IDs whenever you get a chance.

[andreasaster]

Hi Gostev, my case ID is #00715313. Regards, Andreas

[HMwkr]

Just to chime in that I am also finding several tasks running exceedingly slowly to Synology NAS boxes (all DS412+) and await feedback on this thread with interest.

I've been working with the trial; licences have been ordered so I'm waiting for them to come through before raising support tickets on this and other issues.

As I have been trialling Veeam and trying to find a configuration which does what we need it to do in acceptable timescales, I don't yet have a stable configuration.

I'm not comparing V8 to V7 but to BackupExec 2010 - and also to V8 itself, as so far I'm not getting a reliable handle on speed of operations in V8 - as just one example, full backup on Saturday spends 1hr indexing the guest filing system, same job unchanged on Sunday spends 10x as long indexing - no change to data, network completely silently, etc etc. Not confidence-inspiring!

W

[mwiseley]

We had the same issue as well with our very large 9 TB backup copy job. We finally had to start all over and abandoned the copy job and just went back to doing a regular backup across the WAN as version 8 seemed to kill the performance when doing a backup copy. It was going to take it days to rollup the old job.

[HMwkr]

Gah. First post to Veeam forums, and I post to the wrong thread (sorry). Then I find users can't edit or delete their own posts?! (*) That's not very friendly ...

* do the potential exceptions in the FAQ apply to these forums?

[bruceb]

Similar issue here.
214GB on a copy job trying to merge into oldest full backup and has been sitting at 30% for the past 4 days.
Case # 00716429

[andreasaster]

Got a feedback from Veeam today, they said, they found the root cause of the problem. I will have a phone call with them tomorrow to discuss the problem. I will update you then.

Andreas

[roms]

Same problem in Backup Copy Job - Creating GFS restore point:

Merging oldest restore point into full backup file (41% done) - 29:44 hours

[Vitaliy S.]

Rachman, what's your case ID and target storage for the backup copy job?

[roms]

Case # 00717080
Target storage is iSCSI disk with good performance, the "copy backup" job working fast (191 MB/s).
Only when Creating GFS restore point: "Merging oldest restore point into full backup file" is slow.

[munklarsen]

As earlier mentioned, I have the same issues. I've created case #00716289.

[bganjifard]

any answers?

[veremin]

For any technical issue it's recommended to open a ticket with our support team, as most of the problems are environmental specific and cannot be addressed successfully via forum correspondence.

Thanks.

[andreasaster]

We had a Webex yesterday. We have a very special constellation. A restorepoint is split in two vib files, so Veeam must merge these two vib files. This action is called overbuild and is very slow. The engineer is currently searching for a solution to speedup this process.

[thorpenj]

I to am having the same issues. Is there a resolution yet ?

[Gostev]

Right now, there are no known issues around this functionality in the product. As far as I know, all situations reported so far have been caused by different unique circumstances (including most basic stuff like storage lacking IOPS), which is why we advise that everyone having similar issue opens a support case for investigations.

[thorpenj]

I have a ticket raised 00717911 - but am still having the same issue

[coldfire3000]

I am just posting to get notifications.

We have several clients with very large backups (4TB+) and backup copy jobs that are on the verge of upgrade to v8 and I don't want to upgrade and then suffer these issues.

[garoo]

Same issue here guys. V7 was pretty good but since V8 we've had a marked slow down in the merging file times. We recently installed a new NAS to help with the issue and since the upgrade we're worse than before

Opening a ticket now. Anyone have any success? We're on the latest patch for V8

[thorpenj]

Any results on this ? I'm still getting the same issue. Have a ticket open with support but think they are struggling with it to be honest. They suggested re-creating the Backup Copy job, but we use this as our off-site GFS repository for data servers. To be honest it is making the backup copy option unusable at the moment. We were looking to roll out Backup Copies to other sites but have had to put it on hold for the moment until we resolve the current issue.

[Gostev]

@Nick please never accept "solution" to recreate. It is pointless, as this will not fix the possible issue, but rather (at best) will only make it go away temporarily.

[mabeeston]

Watching this thread with interest - never used Backup Copy pre v8 but it does seem slow...worryingly slow. Unfortunately I've not got anything to compare to.

I know it depends on a number of factors but does anyone have a rough benchmark as to how long it should take to merge say 10GB of VIB?