Merging oldest incremental Backup is painfully slow with v8

[dellock6]

From few minutes to several hours, as it completely depends on the I/O capabilities of the underlying storage both at source and target. reading from source is usually not an issue since it's a sequential read only activity, but at destination when it comes to merging oldest restore points into the full backup, it's a 50% mix of reads and writes, heavily random. It's not slow per se, most of the times the issue comes down to be the underlying storage not being able to run it at a decent speed, or better said the expected speed from the customer.

I have a paper in advanced review stage on repository performances, hopefully it will be published in the next weeks. There are infos on the forever incremental forever backup job, that has the same IO profile of backup copy job; hopefully it will help people to better understand and plan for storage performances.

[jlester]

Over the Christmas break, we completely revamped our Veeam infrastructure. New primary backup server and split copy jobs to two different servers in two different locations. Primary backup is very fast now, our backup window went from 5-6 hours to about 1 hour. The Backup Copy jobs were also very fast initially, but I guess as the chain got longer, they are taking more time (all in the transform part). We're still well within our window, but just curious why it would get slower as time goes on? We keep 30 points offsite in our backup copy jobs, so we are past the 30 day point now.

[dellock6]

Hi Jason,
longer backup chains can impact the initial part of the restore when Veeam mounts all the points. But the merge always involves the full backup file and its next incremental, so it should not be impacted by the chain length. Feel free to open a support ticket for further investigation if you see performances decreasing even more.

[mmonroe]

I have noticed on the backup copy jobs that as the incrementals are accumulated that each nights takes longer as well even when a transform is *not* being done. For example, after 30 days of created incrementals, the time to do the nightly backup copy job can take twice or long or longer, versus the first 7-10 days. Each nightly backup takes slight longer as a new one is added to the chain. We saw this on both V7 and V8 - both with *no* transforms being done.

[JaYaKaAzZ]

I have the same issue, i was using incremental backup with active full and noticed that my backupjob didnt delete the old restore points over 30, so i disabled the active full to use the forward incremental. Started the job yesterday and now its on "Merging oldest incremental backup into full backup file (6% done) " after 21 Hours .. case is already running.

No One with updates actually ?

[veremin]

I have the same issue, i was using incremental backup with active full and noticed that my backupjob didnt delete the old restore points over 30, so i disabled the active full to use the forward incremental.

In order to enable forward forever incremental mode, you should disable periodic full backup indeed.

No One with updates actually ?

As problem might be environmental specific, please keep working with the support team on finding the root cause of the experienced behaviour.

Thanks.

[Butha]

Greetings all,

Back after a long break over the holidays - and was surprised to find I have the same problem.

Used to run reverse incremental - but changed to forever incremental after upgrade.

Upgraded V7 early December, no change to infrastructure (Network/Storage/Proxies) and after the first few runs were ok, the "merging" process at the end started taking longer and longer. When looking a the performance view on the VM where the repository is located I see minimal IO - around 10MB/s - and the storage is capable of a lot more than that.

Just wondered if any resolutions were found yet - I see a few posts, but they seem to end up as a support request and I don't see feedback yet where somebody states it's fixed?

Some details to perhaps correlate with other users experiencing the issue:

Storage:
NFS - Netapp - 10GB

Vmware:
5.5

Backup server/Proxies:
Server 2008 R2 (All VM's)

Proxy Transport Mode:
Network (Was always network from issues with hot add in earlier versions)

Forever Incremental mode selected (no Active/periodic Fulls or Synthetic Fulls)

I do have a case open as well: # 00732976

Regards

B

[tfloor]

I also have a problem with the backup copy job merging speed.

We have a direct fiber connection 100MBIT between the 2 locations.
And the processing rate of the complete backup copy job:

Processing rate: 137MB/s
Proccessed: 313,2GB (57%)
Read: 313,2GB
Transferred: 155,2 GB (2x)

But now it's on 52% trying to do the "Merging oldest restore point into full backup file (52%)
But that takes now already 6 hours.

So the backup copy is fine, but the merge is painfull slow.
On Both sides there are New QNAP 16disk Enterprise nas.

Are there other option so we can avoid the slow merge?

Thanks

Tristan

[Gostev]

Do you use CIFS or iSCSI to QNAP? Is there a gateway or repository server (respectively) on the target repository side?

[thorpenj]

I am still experiencing issues after re-building our off-site storage. Any updates on this ?

[veremin]

What were the results of your investigation with the support team? Thanks.

[haslund]

I am seeing same behavior here with a Backup Copy Job using DD Boost over Network.
Case ID 00723103.

[Gostev]

What kind of network, Rasmus? Remember that EMC does not support DDBoost 2.6.x over WAN.

[haslund]

It is LAN. Only thing that might be unusual is that this Data Domain unit is setup to replicate to a remote (WAN) Data Domain.
In the Veeam logs I see references to this remote DD unit such as:
8516> DDBoost| [DDP][ERROR] [AC8:2144] ddpi_connect_with_user_pwd() failed, Hostname: <removed>, Err: 5075-DDBOOST Error 1 for lookup /backup/ost on <removed> (nfs: Operation not permitted)

The references hostname is of the remote DD which is not in anyway configured in Veeam.
The logs are attached to the previously mentioned case if you want to look closer yourself.

[BLWL]

Anyone got an update from Veeam Support? We have a case open on this as well, have done some workarounds to utilize proxies etc better. Except from that no news yet.

[LCD_HJ]

Hi,

We just upgraded to v8 this week and initially all the backup jobs failed straight after the upgrade with an error message "network path not found". I resolved this by going into each job and not really making any changes but just a couple of things like recalculate VM size and test authentication and that seemed to have nudged the job back to life. just though of sharing this in case anyone else has come across the issue.

Second issue is with the new forever incremental feature, its now been over 24 hours that the "Merging oldest incremental backup into full backup file" has been running. We backup to a D2d which I know is not very IO friendly but 24 hours (and counting)? Is this normal? The incremental file is 20gb in size and the full backup is about 3TB.

[foggy]

Second issue is with the new forever incremental feature, its now been over 24 hours that the "Merging oldest incremental backup into full backup file" has been running. We backup to a D2d which I know is not very IO friendly but 24 hours (and counting)? Is this normal? The incremental file is 20gb in size and the full backup is about 3TB.

What type of repository do you use for this storage?

[Andrew.L]

I fear I'm going to experience the same thing, posting here to ensure I get notifications for replies.

[b.lagace]

Interesting, please keep us posted.

I had a case myself as well last week.
We discovered that Veeamzip was slower than backupjob for some variables.

In the logs we found that the jobs were nicely using multiple threads (as configured in the global options), but Veeamzip didnt take in count the parameters and was going single threaded.

Next to that, I do have the feeling that my synthetical fulls have slowed down, but it's to hard to verify, as i'm migrating my clusters and transforming backups towards reverse incrementals.

Regards,
Bjorn

[RGijsen]

Same here. We evaluated with v7 which worked quite well, as well as with merging and very well in my backup window. However, 8 was out when we went live and I was because of timeconstrains forced to go with 8 immediately. I have a 1.2TB job, the incrementals are roughly 40-50GB per day. The merge last weekend took 3 days! That means 3 days of no backups as well for that job. I don't have the most exciting storage for our backup (HP P2000 G3 on fiberchannel but for now only 6x 1TB, a box of more spindels is underway), but this is rediculous. I've now changed to forward incrementals and one active full each week.
Haven't logged a case yet though.

[LCD_HJ]

Its a HP D2d - HP StorageWorks D2D4112 - G02

The job I mentioned in my original post took 38 hours to complete

[PNWMtnBiker]

I have a merge occurring right now reaching 7 hours.
I've been watching my repository (windows server with direct attached) and it will ramp up for a few moments and then go back down to essentially zero for quite awhile and this repeats.

[Moebius]

I've noticed a slowdown as well changing from v7 with reverse-incremental-forever mode to v8 with forward-incremental-forever mode.

So far I've changed mode on one job only (so my observations are still limited), a 3-vm job that occurs every hour.
With v7 and forward incremental mode the job used to complete in 6-7 minutes.
I upgraded to v8 and switched backup mode to forward incremental forever.
The job continued completing in 7-8 minutes until there were .vrb files to remove. When the vbk became the oldest file and merge began occurring, the total job time jumped to 15-20 minutes.Not a big deal in this case given the overall short duration, but it is something to worry about?
The repository is local disk.

I will monitor the job durations as I switch more jobs to the new forward incremental forever mode.

[PNWMtnBiker]

I have a job with 75 vm's which has been merging for 18 hours 30 minutes and is at 6% completed. I'll be changing this from the forever-incremental to a regular incremental with a scheduled full. This merging issue has made the new job type unusable.

Something I also noticed, we are using storage snapshots with NetApp and the job doesn't delete the storage snapshot until the very end after the merge. Since the merge is crawling that snapshot sits longer than what seems necessary.

[Gostev]

Late last week, I have requested support to provide me a summary for all "slow merge" support cases mentioned so far in this thread. Most of them have been closed by now with the storage performance (IOPS) issue confirmed and demonstrated to customers. I will request another update on the remaining ones at the end of this week.

Also, since this thread gets out of hand lately with too many issues dumped into it (including totally unrelated ones like VeeamZIP multithreading), and because most of them are confirmed not to be caused by an issue in Veeam v8, I will be locking it down soon. We can continue discussion of any still-outstanding support cases in the new topic.

[PNWMtnBiker]

My case # 00738508, still open.

[mmonroe]

My case Case # 00713871 is still open and verified to not be a storage subsystem issue.

[GTYAlain]

Same issue, case # 00747326, transforming 818 GB of data in one job under Veeam 8.0.0.917 has been at 6% for the past 14 hours. Veeam 7 finished the same job last week in 13 minutes.

[PA1FOX]

Same here, case # 00744276.

Upon Veeam support I have changed the Synology NAS RS812+ repository from using CIFS to native Linux.

In the original setup, the proxy server (HP DL360) would run into 25% constant CPU usage (one of four CPU's on max usage) and a data throughput to the disks of 150 KB/s. Now, with the Linux server setup, the Linux server processor is constant at 25%, using 1 thread, and data throughput to the disk is about 50 KB/s. That's painfully slow as the Synology disk array can easily handle 30MB/s. It seems that the VeeamAgent drives Windows CPU and Linux CPU into full pull but hardly giving any disk activity.

With the VM of 1TB involved, merging the oldes incremental into the full backup is now at 9% after 23 hours of processing. While I await response from Veeam support, I am making extra backups on portable disks (as backup copy).

As far as I can recall it started with the current V8 patch (8.0.0.917). It did not seem to be so slow right after the V8 update, but I'm not sure about that.

Alex

[bensh]

i'm having same issue job takes more the 30 hours to merge , it's mean i can't backup my system every day.
right now as i understand i have to rollback to synthetic full.