Backup Copy Job - Very Poor Performance - V8

[mmonroe]

Backup copy jobs have always been "slow", even on V7, however, with V8 they are several factors worse. The performance with backup copy jobs has become very poor.

To that vain and for testing, I created a fresh brand new forever incremental job and a backup copy job on the same repository. Both jobs have the retention set to 95 days to prevent any transformations from taking place during my testing.

Both jobs have the same 36 VM's. I have them set to run at different times to avoid any contention. The repository for these jobs is only a "target" with all "source" data coming from another array - a fiber channel high speed SAN.

The VBK for both is about 1.7tb and the incrementals for both have been about 50-70gig each day.

We are finding that the backup copy job takes about 4-6 times longer to build the initial VBK. We are also finding that the daily incrementals are taking 4-6 times longer. This is with both writing to the exact same repository with the same VM's and the same amount of data.

We like the fact that the backup copy job keeps the backups the "same" and simply moves the primary backups to a secondary location, however, it is taking so long that it is almost unusable within the backup windows. We may have to "give up" on the slow backup copy jobs and just replace them with another normal backup job even if the date/time stamp on the data will be different.

Surely others are seeing similar results? I do have an open support ticket on the situation (Case # 00713871) but I am really hoping to rally some support from others with similar issues and get a fix.

[Dima P.]

Hi Monroe, what shows the bottleneck statistics and what is average performance you got? In addition, have you installed patch 1? Thank you.

[Gostev]

I can confirm there are no known issues with Backup Copy job performance in v8, which suggests your issues might be unrelated to upgrade. Let's see what our support finds out after investigating your environment. Thanks!

[mmonroe]

We have loaded Patch 1.

The repository does not seem to issue. Let me explain. Last night the backup copy job took almost 5 hours, however, the normal forever incremental that directly mimics on the exact same repository only took 1 hour (60min). Both jobs created an incremental that was around 60gig.

I suspect that as more people move to V8 we may see others with similar performance issues.

Thanks for everyone's input...

[Gostev]

Well, we have over 30% of our user base already upgraded to v8 as of now, which makes it over 40'000 installations. But of course, anything is a possibility! Please work with our support to investigate your performance issue. And if there is a bug found on our side, we will fix it in no time for you.

[manuel.wagner]

I have the same problem at 2 installations! But the Replication job works fine. Only the Backup job and Backup Copy Job have a very poor performance since upgrading to v8.

[Gostev]

Sorry to hear that, all I am saying is that we just have not been seeing actual support calls about this issue... and this says a lot to me, because for example, most issues fixed by Patch #1 have generated at least a dozen support cases each, and some very long threads on this forum.

Also, historically majority of problems that customer reports to be "upgrade related" end up not being related to an upgrade, but some other change in the environment.

Again, if you have a support case ID where our support has confirmed the issue with Backup Copy job performance that is specific to v8, let me know and I will get it sorted ASAP.

[foggy]

The repository does not seem to issue. Let me explain. Last night the backup copy job took almost 5 hours, however, the normal forever incremental that directly mimics on the exact same repository only took 1 hour (60min). Both jobs created an incremental that was around 60gig.

What kind of repositories are involved and, again, what bottleneck stats are reported by both jobs?

[mmonroe]

One thing that has changed with V8 is that the backup copy job has to read through a forever forward incremental chain now versus a regular incremental in the past when we were on V7.

0) The target repository for both the mimic/test forever incremental and the backup copy job is a Synoloday NAS.

1) The bottleneck on the forever forward incremental backup is the "target"

2) The bottleneck on the backup copy job is the "source". This is interesting since the "source" is a high-speed fiber channel IBM SAN with 15K drives and whatnot. I cant imagine this being a bottleneck versus the Synonoly NAS which is the target - 1gig network link, slow SATA drives, etc. Plus it should only be doing "reads" from the source meaning it would be very difficult for it to the bottleneck more especially with it being a fast fiber channel SAN with fast 15k fiber drives. However, remember that the backup copy job is now readying through the new "forever forward incremental".

[foggy]

One thing that has changed with V8 is that the backup copy job has to read through a forever forward incremental chain now versus a regular incremental in the past when we were on V7.

This does not make any difference.

0) The target repository for both the mimic/test forever incremental and the backup copy job is a Synoloday NAS.

How it is added to Veeam B&R console (what type of repository)?

2) The bottleneck on the backup copy job is the "source". This is interesting since the "source" is a high-speed fiber channel IBM SAN with 15K drives and whatnot.

I'm confused here. The source for the backup copy is the repository that stores backups created by the regular backup job (so, the same Synology NAS). Am I not getting your setup right? Considering backup copy is moving data within the same low-end device, I'm not surprised with the speed it provides.

[mmonroe]

There are three storage devices at play here.

1) Purestorage FA-420 Array fiber channel SAN
2) IBM DS4700 with extra cache, 15k fiber drives
3) Synology NAS - 1gig network, 12 SATA drives.

The Pure FA-420 is where all of the production VM's are located at. The IBM DS4700 is the primary storage for the Veeam server. Basically Veeam gets the VM data directly from the Pure FA-420 and creates backups to the DS4700 repository. The DS4700 is the primary repository on the Veeam server and is mapped as a drive - the D-drive to be specific.

The backup copy job sweeps through the backed up date on the D-Drive (the DS4700) and then copies it to the Synology NAS via a CIFS share. This is the job that is "slow".

I created a "test" normal forever forward incremental job that uses the exact same Synology CIFS share repository. This job has the exact same VM's in it as the backup copy job. The main difference is that the "source" is the Pure FA-420 whereas the Backup Copy Job is reading already processed Veeam backup data from the IBM DS4700.

The backup copy job above - moving data from the DS4700 to the NAS takes 5-6 hours - 50-70gig of data.

The "test" normal forever incremental copying the same VM data from the Pure FA-420 to the NAS takes about an hour. This "test" is proof that the NAS is quick and can backup up 36 VM's with 50-70gig of data in an hour.

Both the Pure FA-420 and the IBM DS4700 (both fiber channel) are light years faster than the 1gig Synology NAS and really shouldn't ever be the bottleneck as I would guesstimate.

[mmonroe]

On a side note, I save 52 weeks of logging. I am going back several months to Aug/Sept/October and whatnot looking at the exact same backup copy job and I see 1hr:52min for a 75-80gig job and last nights took 5 hours. The exact same job going to the exact same NAS from the exact same server. I have scanned over several dozen of these logs from Aug/Sept/Oct and they are all about 1/3 the time of the same job on V8.

I am not trying to beat a dead horse here, however, something happened when we loaded V8. I have already built fresh jobs and new VBKs in hopes that that might would clean something up. I am wondering if I just need to uninstall the software and start over fresh.

[foggy]

Thanks for the explanation, got it now. Seems like further investigation is required here.

[mmonroe]

I have decided to officially give up on the backup copy jobs and will implement an alternate process in its place. I have worked with support for over two weeks now and they are basically telling me that the jobs are working as designed with no issues. Ok - fair enough - I am moving on now - the performance drop - for us at least - with the move to V8 is just to much.

[foggy]

I can see that your case is still on the first tier, I can recommend you to escalate it higher, for deeper investigation, if you find it appropriate to give it some time more.

[mmonroe]

I had already asked for the issue to be escalated to a support manager several weeks ago so I am surprised to hear that it had not been escalated and was still on Tier 1. I figured that once a support manager was involved that the issue would be routed to the proper area to be worked on. Yes - I would really like to have the backup copy jobs run like they did back in Sept/Oct so if there is more that can be done by Veeam support then by all means route my issue to them.

[foggy]

Please ask your engineer to escalate the case for further assistance once again.

[Gostev]

I had already asked for the issue to be escalated to a support manager several weeks ago so I am surprised to hear that it had not been escalated and was still on Tier 1.

Checked with the support management and they didn't find any sign of an escalation, so at least it does not look like it has been escalated through the official procedure. Can you please confirm that you have used Talk to a Manager functionality under My Support on our customer support portal?

[mmonroe]

If you read through the issue history you will see that on 12/29/2014 I requested this:

Veeam Support - Case # 00713871 [ ref:_00D30RWR._50060ivCB1:ref ]

4) I am requesting that this issue be escalated to a support supervisor.

Kevin Winks then followed up and said he was a senior engineer and would be working with me on the issue. I just assumed that this meant the issue had been escalated and that whatever resources were need on the Veeam end would then be brought into play.

Perhaps it is my bad for assuming that there was escalation when possibly there wasn't even though I had requested it.

[foggy]

Yep, this was not the escalation to the higher support tier, so please either ask your engineer or use the Talk to a Manager button on the support portal.

[mmonroe]

I looks like that "button" simply creates an email to supportmanagement@veeam.com. I pasted in my case number into the email and sent it on over.

[haslund]

Any updates on this issue?

[mmonroe]

No, Veeam did escalate my trouble ticket but I do not have any new updates. If anyone else is having this issue, it would be very helpful if they would contribute this this thread.

[rgstech]

I know this thread is a couple of months old, but I am having similar issues with Backup Copy Jobs.

I have 3 different client environments setup as follows:

Backup Job - runs nightly to local storage
Backup Copy Job 1 - runs nightly across WAN to remote location
Backup Copy Job 2 - runs each morning to local storage

The reason for the 2 backup copy jobs is they both backup to external hard drives. This allows us to easily re-seed the external backup by simply switching the 2 external HDs and re-mapping the jobs in Veeam.

Since the V8 upgrade, I am unable to finish both backup copy jobs on a regular basis at all locations. The remote backup copy job runs longer, extending into the morning when the local backup copy runs. It appears that when both copy jobs are running simultaneously, they both slow down dramatically, often reaching the 24 hour window. If I notice both the local backup copy job and remote backup copy job are running in the afternoon, I can disable the local backup copy job and the remote backup copy job finishes in it's window.

We continue to play with changing backup windows to prevent the overlap and are having some success by starting the local backup copy job later in the day.

Referring back to the original poster... I believe my issues are also related to slower backup copy performance.

[foggy]

Rob, have you opened a case for this?

[rgstech]

I have not, as I'm trying to get a clean environment where I can show and log an obvious performance difference. My Veeam logs now would be inconclusive and messy, as I am constantly stopping and disabling and manually running the Backup Copy Jobs to get them to run within their own windows.

[foggy]

Thanks for taking care about our support engineers.

[mmonroe]

We finally resolved this issue by rebuilding the Veeam server fresh with Version 7 and Patch #4. Through Dec/Jan/Feb - we tried V8 with Patch #1 and several other fixes from support with only small improvements.

We are a couple of weeks in now after reverting to V7 and I can report that backcopy job performance is back where it was in November prior to loading Version 8.

[Gostev]

Monroe, I have reviewed your support case and unfortunately looks like it was only escalated to T3 (which is when R&D gets involved) at the end of February. I totally understand that by that time you were already tired pursuing the resolution. We will investigate why this support case was stuck at lower support levels for so long, and learn on our mistakes. We are very sorry about that.

I understand that you have chosen to wait for Patch #2 before returning to v8. When you do so, and if you still have the issue, please PM me and I will make your support case on this specific issue go straight to T3.

Thanks!

[CarlVon]

Well, we have over 30% of our user base already upgraded to v8 as of now, which makes it over 40'000 installations. But of course, anything is a possibility! Please work with our support to investigate your performance issue. And if there is a bug found on our side, we will fix it in no time for you.

We have 2 customers with Copy job performance that I have shown to support for the last 3 weeks. No issues on anything but copy jobs.

You have at leaste 4 customers with copy job issues (2 of our customers, us, and the person who started this thread). Of your 40,000 installations, how many are doing copy jobs? How many check their backup reports? Some people don't report, or get the run around at support and have given up. We are considering an rsync job at this point, as we have almost given up at support. There are many variables. There is likely a memory leak in the copy job process.

I actully had a tech tell me last week that I need more RAM to do the copy job, I had at the time 10G RAM 10G Swap.

Support told us at the beginning it was going to development, with logs and RAM logs as well. Then they changed their mind and never gave it to development. Now we are back with a better tech and he says he will take it to support, but now he is waiting on some tests we are doing with RAM. So I'm not sure your seeing what the customers are seeing in terms of support -- this happens I know as Veeam has grown quite a lot.