Merging oldest incremental Backup is painfully slow with v8

[Gostev]

Steve, Data Domain merge is yet another separate (and known) issue unrelated to this discussion, and the fix for this one is already available through support. Remember that DDBoost was not even present in v7.

Unfortunately, most posts in this topic are irrelevant to one another - but overall, this big thread looks like there *IS* some serious global issue with merge in v8, and as a result, people are looking forward to some magical "fix" to come. There will not be one though, as globally, we cannot confirm any differences in the merge logic between v7 and v8 (the code is basically unchanged).

So as always, I recommend opening a support case to let us investigate YOUR issue (or, we will just provide a hot fix, in case it is known).

[einhirn]

Hello all,

Forum digest found me this thread - I too have the problem that merge with forever incremental is taking too much time. I've already opened Case # 00816049 with technical support - they're only telling me to split up the backup job into smaller ones. But since I'm reluctant on doing that, I'll try either synthetic full or reverse incremental for a change.

My Veeam Server is a VM with 6 vCPUs and 12Gig RAM. This one also acts as Backup proxy for my NFS Datastores (on NetApp and Isilon). My Repository server is a dual 6-Core AMD CPU Dell Server (48Gig RAM) with 12 3TB NL-SAS drives on a PERC with at least 512MB NVRAM... We've got a 10Gb-Ethernet between the VMware Servers, the storage and our repository server - so there's enough bandwidth. Full Backup speed with four concurrent jobs is between 200 and 300MB/sec.

While Veeam is merging, I see almost no disk activity on either Veeam server or repository. All I see is mediocre CPU usage (dropped when I added more vCPUs to the VM) on the backup server and around 4.5Mb/Sec Read plus 4.5Mb/Sec Write Disk usage on a MSSQL table called "temp"-something. A MSSQL stress-test on the same server revealed that it easily does 60MB/sec I/O, peaking at almost 300Mb/sec, so the virtual disk on our backup server should really be fast enough, too.

Pretty pretty please find and fix this issue. If it helps, maybe you can come visit me via Webex, too...

bye
Christian

[mmonroe]

Monroe, I've checked your case (00713871) and it does not even have to deal with the merge? The way you have your jobs setup, merge never happens at all? At least that's what support engineers have told me, do you confirm this?

Correct - I have been making a full VBK and then building incrementals. I have the retention days set to 95 and am just letting the incrementals build. Since the retention of 95 is not being hit, there is no merge or transformation.

That all being said, the backup copy speed saw a major increase in processing time when I went to V8. I was patient and waited a month for Patch #1 and when that made no difference, I opened a support ticket. After two months of sending logs and going back and forth, support really didn't seem to have any idea what the problem is.

I added my thoughts onto this thread because if support does not understand the issue, perhaps this one is related to mine or they are the same. Basically backup copy jobs really took a turn for the worse with V8 for whatever reason and three months later we will are still discussing them. My symptoms have been similar with V8 in that the program just seems to "sit" there doing nothing with virtually no disk, cpu or network activity. It is like it is "paused" for some reason. It is not doing a merge, however, it is just sitting idle like the other complaints on this thread.

I removed V8 early last week and installed V7 with Patch #4. It takes a while to build a new full VBK and whatnot, however, it looks like the speed is back to normal. So far each incremental is taking less than 1/2 the time it did even after the new DLL's and agent that was sent to me for V8. /mm

[BLWL]

Hi all, I will have a current state meeting on this issue with support/QC on Wednesday - was out of office for a few weeks and quite frankly lost track of it.
Actually, v7 did not have a backup mode in question (forever forward incremental) at all, it was added in v8 only.
So, there cannot be a comparison to start with (which is one problem I have always had with this thread)

You are right, let me re-phrase: Anyone tried going back to the backup mode you used in version 7? Did that solve the issue?. It seem to helped for some posters in this thread.

I personally don't have much of a problem with this thread, but I do have a problem with this issue. We went all-in with Veeam 8 Enterprise Plus for our entire environment in December last year. Now it's March and we only use Veeam for 25% of our environment due to this issue.

So, if reverse incremental gives acceptable performance it would work for me. If forever forward incremental can be used in the future, then great. But for now we're in a bit of a messy situation.

EDIT: Ref 00724690

[Morgenstern72]

Unfortunately, most posts in this topic are irrelevant to one another - but overall, this big thread looks like there *IS* some serious global issue with merge in v8, and as a result, people are looking forward to some magical "fix" to come. There will not be one though, as globally, we cannot confirm any differences in the merge logic between v7 and v8 (the code is basically unchanged).

This contradicts the answer I got from your support today:
"We do have some issues with merge of increments in forever forward incremental chains that will be fixed in version 9, so let me quickly take a look at the logs and see if that's the issue. We have a hotfix for one of the bugs and I will compile the steps to apply it if this issue falls under its requirements."

[senior21]

We are having the same problem with the incredibly slow copy job merge on V8 patch 1. We are doing offsite copy jobs to a VCP and cannot complete a merge. It sits at 62% and moves 1% in 5 hours. Support suggested that we kill the job, increase our storage with our VCP, and then increase retention (so it doesn't do a merge) until the issue is resolved. This was 2 weeks ago, and we will be needing to increase storage space and retention again soon.

Support told me a patch is in the works, but now I come here and read this...

Unfortunately, most posts in this topic are irrelevant to one another - but overall, this big thread looks like there *IS* some serious global issue with merge in v8, and as a result, people are looking forward to some magical "fix" to come. There will not be one though, as globally, we cannot confirm any differences in the merge logic between v7 and v8 (the code is basically unchanged).

So as always, I recommend opening a support case to let us investigate YOUR issue (or, we will just provide a hot fix, in case it is known).

[mbrinkho]

...overall, this big thread looks like there *IS* some serious global issue with merge in v8, and as a result, people are looking forward to some magical "fix" to come. There will not be one though, as globally, we cannot confirm any differences in the merge logic between v7 and v8 (the code is basically unchanged).

At the risk of sounding snarky I have to say that based on the price of the product I would expect a little better than this. And yes, we do have a ticket open but the ticket status is "we know there is an issue, no idea when it'll be fixed."

[Gostev]

This contradicts the answer I got from your support today:
"We do have some issues with merge of increments in forever forward incremental chains that will be fixed in version 9, so let me quickly take a look at the logs and see if that's the issue. We have a hotfix for one of the bugs and I will compile the steps to apply it if this issue falls under its requirements."

There is absolutely no contradiction.

Just as I've said before, there are a some known issues in the NEW v8 functionality - functionality that did not exist in v7 (such as in Data Domain DDBoost, new guest application metadata handling). These issues may - particularly, among other functionality - impacts merge speed as well. Not because of some issue in the merge algotrihm, but because the code is hanging waiting for some other processes to complete. If you look at a couple of recent comments above, they are actually inline with this (nothing happens, the process just sits and waits). These issues are fully researched by now, and most have hot fixes available or coming.

But just as I noted, there is no some "global" issue with merge that needs fixing, and people who are simply upgrading from v7 to v8 should not see any difference in merge processing performance whatsoever. As it stands right now, 90% of merge speed cases are closed by demonstrating storage performance problem to a customer without Veeam in the picture (with other 10% of cases is people using new functionality and running into those issues around new functionality).

[Gostev]

And yes, we do have a ticket open but the ticket status is "we know there is an issue, no idea when it'll be fixed."

You are supposed to get a timeline on anything that is admitted to be a known issue. If your engineer cannot give one to you, simply ask the support case escalated until you get a timeline (or correct response).

Another possibility, and this happens quite often, is that in some cases support engineers may refer something "an issue" without putting much thought into this word. For example, when it is really an architecture peculiarity (such as requirement of backup storage with sufficient IOPS).

For them, there is no difference:
"Merge is too slow? Yes, we are aware of this issue with slow backup storage... no idea when this will be fixed" - this would be a perfectly valid response from their perspective, as they cannot know when (and if) we will change the architecture, but they did see this "issue" with other customers before.

On the other hand, if you only look at the word "issue" in this response, I agree with you expecting a fix from us in this case.

Bottom line, in Veeam there is no such timeline as "no idea" for confirmed issues. Every single confirmed bug goes into the Support Issues Tracker, and one of the mandatory columns is "To be fixed in". This is an internal resource that is used heavily even outside of our support (for example, by Systems Engineers).

[Gostev]

You are right, let me re-phrase: Anyone tried going back to the backup mode you used in version 7? Did that solve the issue?. It seem to helped for some posters in this thread.

It will certainly help (and I've seen the corresponding reports) unless again your will try to leverage some new v8 functionality in addition to that (such as DD Boost, metadata processing AD/SQL enhancements in large jobs), in which case you will have to install the corresponding hot fix as well.

Generally speaking, in the same configuration, your speed with v8 should be the same as it was in v7, as globally we did not have any changes in the backup file handling engine (we specifically avoid touching what has been polished for years). Feedback posted here only proves this, as most people report that transform is slow because "nothing happens" most of the time (no actual data movement). For example, the known issue around new application metadata handling will cause the job to sit there processing metadata over 95% of its duration, instead of performing actual data movement.

Thus, my main goal with all of my posts in this topic has been to make sure people do not assume they are experiencing some "global issue" with merge performance in v8, and just sit there waiting for this non-existent issue to get fixed - but rather assume the opposite (that they have a unique issue), and open a support case to let us investigate. For one, this will raise the priority of the corresponding issues based on the amount of open support cases.

So please, don't read my responses here as "There is no issue with merge performance in v8", but rather "There are known issues in v8 which, among other functionality, may also affect merge performance - but these are corner cases around new functionality, and not some global issue in the backup file handling engine brought by, and specific to v8".

[Morgenstern72]

So please, don't read my responses here as "There is no issue with merge performance in v8", but rather "There are known issues in v8 which, among other functionality, may also affect merge performance - but these are corner cases around new functionality, and not some global issue in the backup file handling engine brought by, and specific to v8".

We use "incremental forever", which was and is recommended by veeam. We do not feel like using "corner cases around new functionality".
We have cutting edge infrastructure: BackupRepo is BeeGFS (Fraunhofer) with ~8GB/s, connected with Infiniband 40GB/sec and a Backupproxy that is connected with 10GB Ethernet over Infiniband. Our backups look like this: 1/4 of the time is used for the backup, 3/4 of the time is used for the merges. We do not see any real impact on any of the servers or filesystems involved while merging is done, so it really looks like veeam is the problem.

Your support now tells me that they know they have a problem and it will be fixed in V9. This is not acceptable. V8 is quite new and has to be maintained bug free. However you "word it": the Backup with Forever Incremental is not working like it should be and I as your customer should not be concerned which functionality is having troubles and which code was running smooth in V7 and has not changed. So, please: come up with a solution and stop defending that your wording is correct. It's not about the wording, it's about customers having troubles!

Case # 00826552, no answer from your support since >24hours, one job hangs completely since >50hours with 49% at merging, all other jobs backup super fast and merge awfully slow.

[Gostev]

Your support now tells me that they know they have a problem and it will be fixed in V9.

I will check on this response and what did they mean by that. Currently, there are no plans to "fix" any "problems" in v9. All current bugs I've mentioned above are being hot fixed right now.

This is not acceptable. V8 is quite new and has to be maintained bug free. However you "word it": the Backup with Forever Incremental is not working like it should be and I as your customer should not be concerned which functionality is having troubles and which code was running smooth in V7 and has not changed.

Well again, forever incremental backup mode is brand new to v8 - so how possibly it could have been "running smooth" in v7? You were most definitely using some other backup mode in v7, and switching back to it will bring your v7 product experience back.

So, please: come up with a solution and stop defending that your wording is correct. It's not about the wording, it's about customers having troubles!

I feel it is important to explain my previous statements in cases when they raise questions or contradict with statements from our support, which is what you have brought to my attention earlier on this page. And, I don't think it is very polite on your part to basically tell me to shut up - without letting me answer your own comment, and explain myself.

If you do not appreciate my explanations, responses and updates in this topic, you are free to simply ignore them - but remember that these are public forums, and I have tens of thousands of other customers who needs to know where exactly the issue sits, and what is the best course of action for anyone experiencing this.

[m_zolkin]

Your support now tells me that they know they have a problem and it will be fixed in V9.

Hi, It's Mike Zolkin here, I am a Regional Support Director with Veeam. I have reviewed your case and can confirm the response was incorrect (this was coming from a new hire on T1). I will be sending a note to our Worldwide support staff list to ensure they have full understanding of this issue. Please accept my apologies for any inconvenience caused by this incident.

[mmonroe]

This how a great company starts to lose customers. We have loved Veeam all the way up to Version 7. Then with Version 8 backup copy jobs, a critical feature, is now unusable. On this thread and with support as well, we just keep hearing that this isn't a global problem, check your storage, wait for Version 9 - blah, blah, blah.

I have a datacenter run and my bosses my customers expect backups to be made and they expect me to have processes in place that work. There is clearly something wrong with V8 backup copy jobs and all of us are trying our very well best to help and offer details and logs to Veeam in a quest to get things fixed. We all just want the issue fixed and we dont want to wait months for Version 9 to come out.

I have always loved Veeam - preached Veeam - but I need these backcopy job to work in a reasonable time frame. Veeam is either going to listen to all of us trying to help and get the issues fixed or we are going to have to start looking at other products that will work to replace it. Going back to Version 7 has solved the problem for me, however, that is a temporary fix. I cant stay on Version 7 forever.

At some point I am going to have make some hard decisions if these issues do not get resolved.

[Gostev]

This how a great company starts to lose customers. We have loved Veeam all the way up to Version 7. Then with Version 8 backup copy jobs, a critical feature, is now unusable. On this thread and with support as well, we just keep hearing that this isn't a global problem, check your storage, wait for Version 9 - blah, blah, blah.

Monroe, we have already established earlier (and you have confirmed) that you have your jobs set up in the way that merge never happens. Given that, I really don't understand why you keep participating in this thread about merge performance (as the topic name clearly indicates), and applying our comments made specifically regarding merge performance issues, to a completely unrelated issue of incremental backups creation speed you alone are experiencing.

I am completely confused with your activity in this topic. Can you please clarify why are you keep referring to your issue when it has nothing to deal with what is being discussed in this topic? And what is your intention of applying our comments made about apples, to oranges - so to speak?

Finally, your quoted statement implies that all people keep hearing is Veeam rejecting the merge performance issue, and not taking any real action to address it. This is simply not true: issues are not being rejected, and moreover some of them do have hot fixes already available, which we are providing through support.

[vmexpert]

Gostev you have lots of patience LOL
I would go ballistic at some folks in this topic by now...

[Gostev]

Well, I saw this thread going nowhere back on the 4th page. Should have listened to my inner voice back then! I will fix this now, I have some ideas on what format should we use for the new topic to put this discussion back on track, and make it actually useful.

[Gostev]

Here is the new topic for anyone experiencing slow merge issues to submit their support cases > v8 slow incremental backup merge: call for support cases

Hopefully this new format will prevent a mix of unrelated issues posted, and will let everyone impacted see causes and resolutions for various issues which may affect merge process performance.

[BLWL]

So please, don't read my responses here as "There is no issue with merge performance in v8", but rather "There are known issues in v8 which, among other functionality, may also affect merge performance - but these are corner cases around new functionality, and not some global issue in the backup file handling engine brought by, and specific to v8".

Gostev, sounds fair. You mention issue on metadata processing AD/SQL, is there an article where I can read more on that issue?

I forgot to mention, Veeam support engineer Timofey gives very professional assistance on case 00724690 so far.

[Gostev]

Hi, Bjorn

I don't believe there is a public KB article about the slow metadata processing issue, we rarely create KB articles for minor issues. However, you can point your support engineer to bug ID 44864 on the support issue tracker for the description and the hot fix.

And thank you for your kind words about Timofey, I will pass your feedback on to his manager.

Thanks!

[jspeicher]

I've just stopped using sythetic fulls for our large jobs. There has to be something wrong, especially for CIFS backup repositories, the full backup jobs are tons quicker than the synthetic fulls. so on the jobs that have long times for this, i've switched them to doing weekly full backups instead. I will test again after new patch, but this seems to have been the case since 6.5 or 7.0

[foggy]

Jason, what kind of target storage do you have? Have you discussed this with technical support already? Please also note, that in case of CIFS repository, synthetic activity is performed on the gateway server (backup server, by default).

[mmonroe]

Do we know if V8 update 2 resolved the issues in this thread?

[HJAdams123]

It has not for me...I updated to version 2b this past Sunday, and I am in the same boat. Merge process painfully slow. It hasn't got worse, but it hasn't made anything better.

[foggy]

Please review this post for details.

[HJAdams123]

I am assuming that Gostev's post was about Update 2b. I just commenting that I indeed updated to Update2b this weekend and have not seen any gains in performance in backup time or merge time. I was somewhat surprised by this...

[HJAdams123]

And maybe I am doing something wrong, which is why I would like to hear from anyone who has installed Update 2b and seen either their backup time decrease and\or their merge time decrease? How much of a reduction in processing time did you get? Did it take time for this processing times to manifest themselves after the patch. Cause I am just not seeing a gain anywhere...

[Gostev]

I would like to hear from anyone who has installed Update 2b and seen either their backup time decrease and\or their merge time decrease? How much of a reduction in processing time did you get?

Here is a report of 18.5 hours to 3.8 hours reduction in backup times with slow storage after installing Update 2, for example. So, you may want to have your environment investigated by our support. But one guess I have is that you are using 32-bit backup repository OS. Thanks!

[gbrunel]

i 'd updated to 2B oe week ago but i'v got the same problem too.

[foggy]

Gilles, please contact technical support for a closer look at your setup. Thanks.