Comprehensive data protection for all workloads
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by dellock6 »

From few minutes to several hours, as it completely depends on the I/O capabilities of the underlying storage both at source and target. reading from source is usually not an issue since it's a sequential read only activity, but at destination when it comes to merging oldest restore points into the full backup, it's a 50% mix of reads and writes, heavily random. It's not slow per se, most of the times the issue comes down to be the underlying storage not being able to run it at a decent speed, or better said the expected speed from the customer.

I have a paper in advanced review stage on repository performances, hopefully it will be published in the next weeks. There are infos on the forever incremental forever backup job, that has the same IO profile of backup copy job; hopefully it will help people to better understand and plan for storage performances.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
jlester
Enthusiast
Posts: 56
Liked: 5 times
Joined: Mar 23, 2010 1:59 pm
Full Name: Jason Lester
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by jlester »

Over the Christmas break, we completely revamped our Veeam infrastructure. New primary backup server and split copy jobs to two different servers in two different locations. Primary backup is very fast now, our backup window went from 5-6 hours to about 1 hour. The Backup Copy jobs were also very fast initially, but I guess as the chain got longer, they are taking more time (all in the transform part). We're still well within our window, but just curious why it would get slower as time goes on? We keep 30 points offsite in our backup copy jobs, so we are past the 30 day point now.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by dellock6 »

Hi Jason,
longer backup chains can impact the initial part of the restore when Veeam mounts all the points. But the merge always involves the full backup file and its next incremental, so it should not be impacted by the chain length. Feel free to open a support ticket for further investigation if you see performances decreasing even more.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
mmonroe
Enthusiast
Posts: 75
Liked: 3 times
Joined: Jun 16, 2010 8:16 pm
Full Name: Monroe
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by mmonroe »

I have noticed on the backup copy jobs that as the incrementals are accumulated that each nights takes longer as well even when a transform is *not* being done. For example, after 30 days of created incrementals, the time to do the nightly backup copy job can take twice or long or longer, versus the first 7-10 days. Each nightly backup takes slight longer as a new one is added to the chain. We saw this on both V7 and V8 - both with *no* transforms being done.
JaYaKaAzZ
Influencer
Posts: 24
Liked: 4 times
Joined: Feb 03, 2014 8:01 am
Full Name: Artur Schneider
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by JaYaKaAzZ »

I have the same issue, i was using incremental backup with active full and noticed that my backupjob didnt delete the old restore points over 30, so i disabled the active full to use the forward incremental. Started the job yesterday and now its on "Merging oldest incremental backup into full backup file (6% done) " after 21 Hours .. case is already running.

No One with updates actually ?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by veremin »

I have the same issue, i was using incremental backup with active full and noticed that my backupjob didnt delete the old restore points over 30, so i disabled the active full to use the forward incremental.
In order to enable forward forever incremental mode, you should disable periodic full backup indeed.
No One with updates actually ?
As problem might be environmental specific, please keep working with the support team on finding the root cause of the experienced behaviour.

Thanks.
Butha
Enthusiast
Posts: 39
Liked: 20 times
Joined: Oct 03, 2012 10:59 am
Full Name: Butha van der Merwe
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by Butha »

Greetings all,

Back after a long break over the holidays - and was surprised to find I have the same problem.

Used to run reverse incremental - but changed to forever incremental after upgrade.

Upgraded V7 early December, no change to infrastructure (Network/Storage/Proxies) and after the first few runs were ok, the "merging" process at the end started taking longer and longer. When looking a the performance view on the VM where the repository is located I see minimal IO - around 10MB/s - and the storage is capable of a lot more than that.

Just wondered if any resolutions were found yet - I see a few posts, but they seem to end up as a support request and I don't see feedback yet where somebody states it's fixed?

Some details to perhaps correlate with other users experiencing the issue:

Storage:
NFS - Netapp - 10GB

Vmware:
5.5

Backup server/Proxies:
Server 2008 R2 (All VM's)

Proxy Transport Mode:
Network (Was always network from issues with hot add in earlier versions)

Forever Incremental mode selected (no Active/periodic Fulls or Synthetic Fulls)

I do have a case open as well: # 00732976

Regards

B
tfloor
Veteran
Posts: 270
Liked: 15 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by tfloor »

I also have a problem with the backup copy job merging speed.

We have a direct fiber connection 100MBIT between the 2 locations.
And the processing rate of the complete backup copy job:

Processing rate: 137MB/s
Proccessed: 313,2GB (57%)
Read: 313,2GB
Transferred: 155,2 GB (2x)

But now it's on 52% trying to do the "Merging oldest restore point into full backup file (52%)
But that takes now already 6 hours.

So the backup copy is fine, but the merge is painfull slow.
On Both sides there are New QNAP 16disk Enterprise nas.

Are there other option so we can avoid the slow merge?

Thanks

Tristan
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by Gostev »

Do you use CIFS or iSCSI to QNAP? Is there a gateway or repository server (respectively) on the target repository side?
thorpenj
Novice
Posts: 4
Liked: never
Joined: Jan 03, 2015 2:42 pm
Full Name: Nick Thorpe
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by thorpenj »

I am still experiencing issues after re-building our off-site storage. Any updates on this ?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by veremin »

What were the results of your investigation with the support team? Thanks.
haslund
VeeaMVP
Posts: 839
Liked: 149 times
Joined: Feb 16, 2012 7:35 am
Full Name: Rasmus Haslund
Location: Denmark
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by haslund »

I am seeing same behavior here with a Backup Copy Job using DD Boost over Network.
Case ID 00723103.
Rasmus Haslund | Twitter: @haslund | Blog: https://rasmushaslund.com
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by Gostev »

What kind of network, Rasmus? Remember that EMC does not support DDBoost 2.6.x over WAN.
haslund
VeeaMVP
Posts: 839
Liked: 149 times
Joined: Feb 16, 2012 7:35 am
Full Name: Rasmus Haslund
Location: Denmark
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by haslund »

It is LAN. Only thing that might be unusual is that this Data Domain unit is setup to replicate to a remote (WAN) Data Domain.
In the Veeam logs I see references to this remote DD unit such as:
8516> DDBoost| [DDP][ERROR] [AC8:2144] ddpi_connect_with_user_pwd() failed, Hostname: <removed>, Err: 5075-DDBOOST Error 1 for lookup /backup/ost on <removed> (nfs: Operation not permitted)

The references hostname is of the remote DD which is not in anyway configured in Veeam.
The logs are attached to the previously mentioned case if you want to look closer yourself.
Rasmus Haslund | Twitter: @haslund | Blog: https://rasmushaslund.com
BLWL
Enthusiast
Posts: 35
Liked: 41 times
Joined: Jan 27, 2015 7:24 am
Full Name: Bjorn L
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by BLWL »

Anyone got an update from Veeam Support? We have a case open on this as well, have done some workarounds to utilize proxies etc better. Except from that no news yet.
LCD_HJ
Lurker
Posts: 2
Liked: never
Joined: Jan 24, 2014 10:13 am
Contact:

[MERGED] V8 - Merging oldest incremental backup into full ba

Post by LCD_HJ »

Hi,

We just upgraded to v8 this week and initially all the backup jobs failed straight after the upgrade with an error message "network path not found". I resolved this by going into each job and not really making any changes but just a couple of things like recalculate VM size and test authentication and that seemed to have nudged the job back to life. just though of sharing this in case anyone else has come across the issue.

Second issue is with the new forever incremental feature, its now been over 24 hours that the "Merging oldest incremental backup into full backup file" has been running. We backup to a D2d which I know is not very IO friendly but 24 hours (and counting)? Is this normal? The incremental file is 20gb in size and the full backup is about 3TB.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: [MERGED] V8 - Merging oldest incremental backup into ful

Post by foggy »

LCD_HJ wrote:Second issue is with the new forever incremental feature, its now been over 24 hours that the "Merging oldest incremental backup into full backup file" has been running. We backup to a D2d which I know is not very IO friendly but 24 hours (and counting)? Is this normal? The incremental file is 20gb in size and the full backup is about 3TB.
What type of repository do you use for this storage?
Andrew.L
Influencer
Posts: 15
Liked: 1 time
Joined: Jan 28, 2015 1:39 pm
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by Andrew.L »

I fear I'm going to experience the same thing, posting here to ensure I get notifications for replies.
b.lagace
Enthusiast
Posts: 34
Liked: 17 times
Joined: Jul 02, 2014 1:23 pm
Full Name: Bjorn Lagace
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by b.lagace »

Interesting, please keep us posted.

I had a case myself as well last week.
We discovered that Veeamzip was slower than backupjob for some variables.

In the logs we found that the jobs were nicely using multiple threads (as configured in the global options), but Veeamzip didnt take in count the parameters and was going single threaded.

Next to that, I do have the feeling that my synthetical fulls have slowed down, but it's to hard to verify, as i'm migrating my clusters and transforming backups towards reverse incrementals.

Regards,
Bjorn
RGijsen
Expert
Posts: 124
Liked: 25 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by RGijsen »

Same here. We evaluated with v7 which worked quite well, as well as with merging and very well in my backup window. However, 8 was out when we went live and I was because of timeconstrains forced to go with 8 immediately. I have a 1.2TB job, the incrementals are roughly 40-50GB per day. The merge last weekend took 3 days! That means 3 days of no backups as well for that job. I don't have the most exciting storage for our backup (HP P2000 G3 on fiberchannel but for now only 6x 1TB, a box of more spindels is underway), but this is rediculous. I've now changed to forward incrementals and one active full each week.
Haven't logged a case yet though.
LCD_HJ
Lurker
Posts: 2
Liked: never
Joined: Jan 24, 2014 10:13 am
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by LCD_HJ »

Its a HP D2d - HP StorageWorks D2D4112 - G02

The job I mentioned in my original post took 38 hours to complete :-(
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by PNWMtnBiker »

I have a merge occurring right now reaching 7 hours.
I've been watching my repository (windows server with direct attached) and it will ramp up for a few moments and then go back down to essentially zero for quite awhile and this repeats.
Moebius
Veeam ProPartner
Posts: 206
Liked: 28 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by Moebius »

I've noticed a slowdown as well changing from v7 with reverse-incremental-forever mode to v8 with forward-incremental-forever mode.

So far I've changed mode on one job only (so my observations are still limited), a 3-vm job that occurs every hour.
With v7 and forward incremental mode the job used to complete in 6-7 minutes.
I upgraded to v8 and switched backup mode to forward incremental forever.
The job continued completing in 7-8 minutes until there were .vrb files to remove. When the vbk became the oldest file and merge began occurring, the total job time jumped to 15-20 minutes.Not a big deal in this case given the overall short duration, but it is something to worry about?
The repository is local disk.

I will monitor the job durations as I switch more jobs to the new forward incremental forever mode.
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by PNWMtnBiker »

I have a job with 75 vm's which has been merging for 18 hours 30 minutes and is at 6% completed. I'll be changing this from the forever-incremental to a regular incremental with a scheduled full. This merging issue has made the new job type unusable.

Something I also noticed, we are using storage snapshots with NetApp and the job doesn't delete the storage snapshot until the very end after the merge. Since the merge is crawling that snapshot sits longer than what seems necessary.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by Gostev »

Late last week, I have requested support to provide me a summary for all "slow merge" support cases mentioned so far in this thread. Most of them have been closed by now with the storage performance (IOPS) issue confirmed and demonstrated to customers. I will request another update on the remaining ones at the end of this week.

Also, since this thread gets out of hand lately with too many issues dumped into it (including totally unrelated ones like VeeamZIP multithreading), and because most of them are confirmed not to be caused by an issue in Veeam v8, I will be locking it down soon. We can continue discussion of any still-outstanding support cases in the new topic.
PNWMtnBiker
Enthusiast
Posts: 63
Liked: 8 times
Joined: Jan 16, 2014 11:12 pm
Full Name: Jon Dufour
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by PNWMtnBiker »

My case # 00738508, still open.
mmonroe
Enthusiast
Posts: 75
Liked: 3 times
Joined: Jun 16, 2010 8:16 pm
Full Name: Monroe
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by mmonroe »

My case Case # 00713871 is still open and verified to not be a storage subsystem issue.
GTYAlain
Lurker
Posts: 1
Liked: never
Joined: Feb 09, 2015 6:23 pm
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by GTYAlain »

Same issue, case # 00747326, transforming 818 GB of data in one job under Veeam 8.0.0.917 has been at 6% for the past 14 hours. Veeam 7 finished the same job last week in 13 minutes.
PA1FOX
Novice
Posts: 7
Liked: 4 times
Joined: Mar 31, 2014 8:32 pm
Full Name: Alex Buurlage
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by PA1FOX »

Same here, case # 00744276.

Upon Veeam support I have changed the Synology NAS RS812+ repository from using CIFS to native Linux.

In the original setup, the proxy server (HP DL360) would run into 25% constant CPU usage (one of four CPU's on max usage) and a data throughput to the disks of 150 KB/s. Now, with the Linux server setup, the Linux server processor is constant at 25%, using 1 thread, and data throughput to the disk is about 50 KB/s. That's painfully slow as the Synology disk array can easily handle 30MB/s. It seems that the VeeamAgent drives Windows CPU and Linux CPU into full pull but hardly giving any disk activity.

With the VM of 1TB involved, merging the oldes incremental into the full backup is now at 9% after 23 hours of processing. While I await response from Veeam support, I am making extra backups on portable disks (as backup copy).

As far as I can recall it started with the current V8 patch (8.0.0.917). It did not seem to be so slow right after the V8 update, but I'm not sure about that.

Alex
bensh
Novice
Posts: 9
Liked: never
Joined: Nov 12, 2013 11:40 am
Full Name: ben shukrun
Contact:

Re: Merging oldest incremental Backup is painfully slow with

Post by bensh »

i'm having same issue job takes more the 30 hours to merge , it's mean i can't backup my system every day.
right now as i understand i have to rollback to synthetic full.
Post Reply

Who is online

Users browsing this forum: Ivan239, Paul.Loewenkamp and 131 guests