Merging oldest incremental Backup is painfully slow with v8

[Butha]

Good Day Fellow Veeam forum members!

I thought I'd give some feedback on my case (#00732976 ) which is about the same issue as well.

I've been busy with Veeam Support (Thanks Anatoly!) for a couple of weeks now trying to identify the cause.

I'll give an update on steps tried - might help somebody else who is just starting down this path...

If I look at the whole troubleshooting steps takes from day one, I can only assume which the cause seems to be storage IO related - but in my case at least it should not be, as we have high end storage at the back end which performed fine on V7. All measures of speed tests/latency checks have been done to try and prove it's IO related, but whenever a merge is taking place, or a manual test with debug tools (Veeamfr) is done - normal copy requests to/from the same storage performs fine - and all latencies are idling (sub 2ms) We are at a stage now where we seemed to have narrowed it down through trial and error to the following scenario:

You create a new forward-incremental forever job (In other words NO synthetic of active full selected - which would defeat the whole purpose) and it runs a normal " full" fine. Technical support uses a debug tool to manually " merge" the files and times the speed it runs at - runs 100% (in our case roughly 60 - 100MB/sec) You set the restore point in job properties to say 2 (or leave it at the default of 14 - but you need to re-run 14 times then) and run the job a 2nd time. It then performs the first merge (or "patch") and leaves you with the full file, as well as the new increment. Run the same process of manually "patching/merging" with veeamfr - slow speed. The reading seems to be ok - 200MB+ but the writing slows down to around 8Mb/sec. The same results is found in the Lab - so the investigation is still ongoing.

It's worth mentioning the same issue is also present when attempting a file level restore (Windows) - the restore process runs at the same slow 8Mb/sec. In the test scenario - where the very first merge has not taken place yet ,the restore is fast.

We are running Netapp storage on NFS ( Not BfSS yet) - but the issue does not seem to be related to something specific on NFS as far as I can see. Proxies are also virtual - but have tried local storage, different backups modes, different OS'es on the proxies (Server 2008 R2 and Server 2012 R2) - made no difference.

I have since reverted to reverse-increment mode - which seems to be running fine - but continue to identify the cause when on forward-increment-forever as I would like to use it.

Will post any update if I have news and would of course be good if anybody else has some feedback on what they have tried.

Regards

Butha

[BLWL]

Anyone found tried going back to version 7? Did that solve the issue?

Thanks

[b.lagace]

Very interesting case and thx Butha to take the time to keep us informed.

I wish I had the time to do what BLWL proposes...unfortunatly...

[mbrinkho]

It seems we are now having this issue as well. 5 hours 18 mins "Merging oldest incremental backup into full backup file (42%)". CPU usage runs at 30% for a while and disk seems to be idle, then the disk where the repository is goes up to 100% active briefly then back to 0% and the CPU goes back up. This server has 2 quad core Xeons - if there was a way to allocate more CPU to this task it wouldn't hurt.

[Eson]

Having the same issue on v8, merges on a 20gb inc backup can take several hours and the Veeam server is hardly using any resources during this time. Most of the time only 8MB/s of disk throughput as mentioned previously in the thread.

[Gostev]

Hi all, I will have a current state meeting on this issue with support/QC on Wednesday - was out of office for a few weeks and quite frankly lost track of it.

Anyone found tried going back to version 7? Did that solve the issue?

Actually, v7 did not have a backup mode in question (forever forward incremental) at all, it was added in v8 only.
So, there cannot be a comparison to start with (which is one problem I have always had with this thread)

[PA1FOX]

Hi all, I will have a current state meeting on this issue with support/QC on Wednesday - was out of office for a few weeks and quite frankly lost track of it.

Hi Anton, great you've got rid of the flu!

Please keep us informed upon developments with this issue through this thread.

[mbrinkho]

Actually, v7 did not have a backup mode in question (forever forward incremental) at all, it was added in v8 only.
So, there cannot be a comparison to start with (which is one problem I have always had with this thread)

Our issue is with the standard Incremental backup. The merge took 15:39:44 and caused the scheduled backup to be missed last night.

[tsightler]

Our issue is with the standard Incremental backup. The merge took 15:39:44 and caused the scheduled backup to be missed last night.

If you are running "standard incremental backup" then that means you should be running also an active or synthetic full, typically once a week, but these modes don't have a "merge" process (the synthetic option would of course build a synthetic full). Only forever forward incremental as a "merge" that occurs at the end of each job. This is the mode that is used when you select incremental, but don't select an active of synthetic full option and this option did not exist until v8.

Can you clarify if you are referring to daily merge as opposed to weekly synthetic operations?

[mbrinkho]

We are using the defaults which I guess makes it "forever forward incremental." We have reached the # of restore points to keep so it's merging the oldest incremental with the full at the end of the daily backup - or at least I believe that's what it's doing.

[daniel_rogers]

I too have this issue and have logged a support case - 000814639.

[veremin]

We are using the defaults which I guess makes it "forever forward incremental."

Correct. You're using forever forward incremental mode that starts to merge the oldest restore point with the subsequent one, when retention is reached. As to merging process taking a lot of time, opening a ticket and letting our engineers see your environment would be the best approach. Thanks.

[d.lansinklesscher]

Hi Guys,

Hereby some of my statistics from a client that is Running version 7 so you people can compare. The customer is doing a copy-job which does(correct me if I’m wrong) the same merge process as a forever incremental job in version 8.

At this client merging 200GB of changes into a 3.5TB backup file takes 6 hours. The backup repository resides on a physical Windows 2008R2 server with one Xeon E5-2620 and 16GB memory.
The storage consists of 11 7,2k sata drivers of 2TB each. They are connected directly to a HP P421 controller with 512MB cache(25% read 75% write). The drives are running in one Raid6 set.
When the merge-process is running the system seems quiet:
-CPU-load is about 2 to 4 %
-Veeam agent uses about 700MB of memory
-Mem statistics are:
Total 16349
Cached 3882
Available 3669
Free 0

But there is one performance counter that is continuously running very high: The average disk read queue length. I guess that is because of the random nature of the reads that occur during the merge process.

[PA1FOX]

If you are running "standard incremental backup" then that means you should be running also an active or synthetic full, typically once a week, but these modes don't have a "merge" process (the synthetic option would of course build a synthetic full). Only forever forward incremental as a "merge" that occurs at the end of each job. This is the mode that is used when you select incremental, but don't select an active of synthetic full option and this option did not exist until v8.

Hmmm weird. As the merging took too much time for our backup copy processes I have set the original jobs to incremental with active full backup every week.

Still I get the 'oldest file merging' message, but this is working intensively on the repository and finished in reasonable time. This as opposed to the merging process with forever incremental, the merging took days and only had transfers of about 150KB/s on the same repository.

The merging for both setups is present, but one is handled far more efficiently than the other.

[mbrinkho]

Here is a graph of what we're seeing for CPU usage and repository disk usage during the merge process. You can see that it uses 100% disk for a little while and the CPU drops to almost nothing then it jumps up to around 30% CPU while the disk is pretty much idle. This is on a 2-CPU, Quad Core Xeon X5355 (8 cores running at 2.6GHz no HyperThreading) with a 8-spindle RAID-5 array of 2TB 7.2k disks on a PERC-5 for the repository.

It seems like if it could process a larger amount of data in one "chunk" and/or if it could be set somehow to use more CPU resources that the merge could be quite a bit quicker.



Our case number is 00816437.

[tsightler]

This is likely caused by a known issue in v8. Normally this pause occurs as the job switches between VMs in the job, but in v8 the pause is much longer than previous versions. Is this an all-in-one install (i.e. Veeam server and repo on same server). That's the only case where I'd expect to see the correlation of CPU usage to I/O occur on the same system.

[mbrinkho]

Yes it is all in one. We just received this from support also:

This is a known issue that R&D is aware of. They are currently working on a resolution which will be in a future patch. As a workaround some customers are able to improve performance by breaking up the jobs into a smaller number of VMs so that instead of one large merge you have have multiple smaller merges.

[PA1FOX]

Hmmm weird. As the merging took too much time for our backup copy processes I have set the original jobs to incremental with active full backup every week.

Still I get the 'oldest file merging' message, but this is working intensively on the repository and finished in reasonable time. This as opposed to the merging process with forever incremental, the merging took days and only had transfers of about 150KB/s on the same repository.

The merging for both setups is present, but one is handled far more efficiently than the other.

Well, I take back my previous comment. Last Friday our backup job has performed the active full backup (as set in advanced settings under 'Create active full backups periodically'). The backup copy is has been waiting for the new restore point to appear which was the moment the active full has finished.

The backup copy job does not 'see' the backup job as a new active full. The active full file is not transferred to the secondary location, since only about 20% of the VM size is transferred by the backup copy job.
After transferring this 20% the backup copy job has run into 'merging backup files' and has completed 5% in 56 hours. Data transfer at the repository disk array is about 100KB/sec. I fear that the merging will not be completed within a week.

It would been much faster if the backup copy would have seen that an active full has been created by the primary job and transfer this complete file.
I will create this situation by making a second primary job with the same setup as the first primary job, but with the remote repository as target location. The backup copy will be disabled until development has sorted out the issue.

[Gostev]

Data transfer at the repository disk array is about 100KB/sec.

We'd like to take a look at this live please, as I'd expect at least 10x faster performance with pretty much ANY storage. While we are talking random I/O, we do it with fairly large block sizes, so it definitely should not crawl. Could be something really simple on your side, like RAID stripe size setting being too small.

[d.lansinklesscher]

We'd like to take a look at this live please, as I'd expect at least 10x faster performance with pretty much ANY storage. While we are talking random I/O, we do it with fairly large block sizes, so it definitely should not crawl. Could be something really simple on your side, like RAID stripe size setting being too small.

Is there a document on recommended stripe sizes in different configurations?

[veremin]

This blog post might be a good starting point.

[Butha]

If you are running "standard incremental backup" then that means you should be running also an active or synthetic full, typically once a week, but these modes don't have a "merge" process (the synthetic option would of course build a synthetic full). Only forever forward incremental as a "merge" that occurs at the end of each job. This is the mode that is used when you select incremental, but don't select an active of synthetic full option and this option did not exist until v8.

Can you clarify if you are referring to daily merge as opposed to weekly synthetic operations?

Just a quick question about the above statement:

I see it pop up quite often, and it almost seems there are some difference of opinion about the from Veeam technical support as well. I'm sure i read before that the practice of doing weekly/monthly active full was with early versions of Veeam only - but it has been stated recently that you no longer require this.

Would be good if Anton could perhaps put this one to bed for us?

Butha

[mdiver]

I see the same on an installation here as well.

A copy job with ~10 VMs ran fine under V7. Now we get increased merge times under V8 (Patch1).
The interesting thing is: During the merge, which should saturate the remotes storage, we see the system in idle state for most of the time.

During the merge (currently at 3% after 3h) we get read/write rates to the disk <100kb/s and a queue of 0:



Sporadically we get spikes in the load, but never even close to saturation.

So disk should not be the limit here (neither of cause CPU and/or memory).

VBK size on the remote site is ~300GB. The VIB to be commited ~10GB.

Regards,
Mike

[Butha]

Lastest Update:

I thought I'd at least give another update on progress with this issue from my side. (A bit more time has passed since previous post)

Anatoly from technical support is continuing the investigation, and are currently looking at the differences of reverse increment times vs forward-increment-forever. At the moment all the jobs are back on reverse increment, and speed seems to be similar to before on V7. I will leave it at that for now, until perhaps a solution/patch might be made available before switching any production jobs back to forward increment-forever.

The case is still open and I'm gladly continuing trying to assist by testing and will give any feedback once I have any update.

I did have another question to add that was mentioned earlier - with regards to running scheduled active full/synthetic jobs when using reverse-increment. I am sure it was mentioned that only earlier versions of Veeam (6 -6.5) required this - and that later versions don't have any issue with chain corruptions etc - so would be good if somebody can comment about this.

Butha

[mbrinkho]

This is a known issue that R&D is aware of. They are currently working on a resolution which will be in a future patch.

This issue is causing us to only be able to back up every other day. Could someone at Veeam look into this and see if there is an ETA on the patch?

[PA1FOX]

We'd like to take a look at this live please, as I'd expect at least 10x faster performance with pretty much ANY storage. While we are talking random I/O, we do it with fairly large block sizes, so it definitely should not crawl. Could be something really simple on your side, like RAID stripe size setting being too small.

Hi Gostev, throughput at the same repository can easily get to 30MB/sec, it's not a disk issue. I have Veeam Support had a look as the Proxy server near that repository had one of the CPU cores constantly running at 100% during the merging process. Upon Support's request I have changed the repository from CIFS to Linux. Now one of the repository CPU cores is running at 100%, the Proxy uses none.

It seems a CPU / Veeam Agent issue, rather than disk.

The backup copy job I mentioned some posts ago did finally complete succesfully after 66 hours. Yesterday it has done a new backup copy. Now the merging process took only 5 hours, had a constant data throughput of 30MB/sec at the repository (IOPS around 500) and a CPU usage of a few percent. In both cases merge file size was 79GB.

This VM is about 1TB. According to the statistics, only the 1TB VM backup copy has this, the other smaller ones are fine.

So sometimes the merging is fast as expected, sometimes incredible slow. Maybe the Veeam agent has two working modes it can run into or it runs out of buffer size when processing larger merges?

Below are the report blocks of those jobs.

vrijdag 27 februari 2015 23:00:20
Success 1 Start time 23:00:20 Processed 79,5 GB Backup size 20,9 GB
Warning 0 End time 01:09:14 (+4) Read 79,5 GB Dedupe 1,0x
Error 0 Duration 67:45:37 Transferred 20,8 GB (3,8x) Compression 3,8x

dinsdag 3 maart 2015 23:00:17
Success 1 Start time 23:00:17 Processed 79,4 GB Backup size 21,2 GB
Warning 0 End time 06:52:38 (+1) Read 79,4 GB Dedupe 1,0x
Error 0 Duration 7:52:21 Transferred 21,1 GB (3,8x) Compression 3,8x


Case # 00744276

[mmonroe]

Support worked with us on loading some DDL's and a new agent last week (we already had Patch #1) with regards to our speed issues. There was an improvement in the backup copy job performance. It went from being 6-7x slower than V7 to 4x slower than version V7. An improvement over what we had been seeing with V8, however, still significantly slower than V7.

Backup copy jobs are very important to us and they need to be able to be completed in a 24 hour period, obviously. We have decided to bite the bullet and have rebuilt the Veeam server back with Version 7. We have been working on the performance issues since loading V8 back in November 2014 and we need the performance back to what is was in V7. We cant afford for that process to be 4x slower with an "upgrade" to Version 8.

I was hoping that Veeam would have been able to resolve this very critical backup copy job issue after 3+ months, however, until it is resolved, V8 is not an option for us.

[Gostev]

Very sorry for this inconvenience. This actually really helps since we will now be able to compare v7 and v8 in the same environment.

[Gostev]

Monroe, I've checked your case (00713871) and it does not even have to deal with the merge? The way you have your jobs setup, merge never happens at all? At least that's what support engineers have told me, do you confirm this?

[svharris]

Seeing the same issue with LAN targeted backups using DD Boost with weekly synthetic fulls. Only the merge is slow. The backup itself is quick. This is impacting the large server jobs much more severely (servers with disks over 1TB). I was initially thinking it was due to the Data Domain possibly rehydrating data for the merge, but this post shows that it effects many different types of repositories, so that may not have anything to do with it. Anxious to hear on a fix.