Incremental to rollback transform: possible bottlenecks?

[trafsta]

Yep WIll be running the "custom data mover agent with detailed performance logging" tonight and will provide tech support the logs on Monday morning. Looking forward to hearing the results of the testing

[KClawson]

Yeah, I am still seeing the issue after upgrading to 6.1. I have seen some notes about tweaking 2008 R2 settings. I can't seem to find that sticky. Do you have the link so I can see if there is anything I am missing? My machine that is doing the transform (that is the VEEAM host, not the proxies, right?) is a physical box (2 2.8GHz Quad core processors and 12 GB of RAM) is showing 5% or less CPU usage and about 40% memory usage. It has 3 NICs on the iSCSI network and network utilization rarely goes above 10%. Any ideas? My synthetic full has been running for 5 hours and has been stuck at 6% for the last 2 hours... Also, it has 12 machines and about 750 GB of data.

[Gostev]

Tweaking any Windows settings will not help. After much profiling, we believe we have found the main cause of the transform slowness, just need to validate now that the changes we've made improve the performance.

[trafsta]

We're running a test Veeam Agent tonight on our weekly transform, as Gostev said support thinks it will fix the problem. I'm hoping it will! Transforms usually take us 26-32+ hours, I'm hoping that will be down to 1-2 hours tonight High expectations perhaps, but we shall see. If it works I'm hoping they can put the fix into the recently released v6.1 sooner rather than later as we're waiting for a few other important Hyper-V features from 6.1 as well (currently the test veeam agent is on a 6.0 build)

[Gostev]

Based on what I know, I believe the maximum theoretical transform speed increase is 10x (that is, with lightning fast backup storage), but on more down-to-earth storage the speed increase will be not more than about 3-4x of the current transform speed. Don't expect miracles since transform will always remain slow operation (comparing to regular backups) due to heavy random I/O requirements, and we cannot do anything about modern storage design limitations around random I/O, or laws of physics currently in place on this planet

The plan is to include this fix in Patch 1 for version 6.1, whenever we collect enough issues to justify one. Meanwhile, hotfix will be available through support, as usual.

[trafsta]

Well it's only a 400GB backup file, so perhaps it will miraculously be way faster. One can hope, right?

[Gostev]

I personally do hope, and will be waiting for your report (and good news) impatiently!

By the way, I forgot to mention that the issue is caused by how transactional updates of VBK file were orchestrated, and this has been the case since v4 (which is when we first introduced transactional backup storage). So, this issue has nothing to deal with upgrade to v6, as it was suggested by some posters in the beginning of this thread. Just wanted to make this clear.

[KClawson]

Is that hot fix now available through support? Or is it still being researched. I do have a ticket open, but it was started on 6.0 and I am seeing the issue on 6.1 now, so it sounds like I would need a new tool, right?

[Vitaliy S.]

Support team will notify you once it becomes generally available, right now we need to receive a confirmation that it resolves this issue.

[trafsta]

Transform task is only 46% done after 10 hours 25 minutes so far Looks like it'll be around 22 hours at this rate (previously it was 26+ hours). Will let the transform/job finish and forward logs on to support.

We're actually still only trialling Veeam at our company so all we've ever used is v6.x (we have used Symantec for 6+ years). The issue with Symantec is that since switching to Backup Exec De-duplicated backups our backup window is getting too large, so we are hoping to use Veeam for most of our backups. This is pretty much the last remaining issue as backups take only 1-3 hours total for all 16+ of our servers - the transforms are taking 26-32+ hours and that's a big problem before we go ahead with this purchase Replication of the Hyper-V servers is also working quite well. Hopefully the logs will shed some light for support once this current transform completes.

Gostev; any idea if there are any other Veeam users testing out this fix as well?

[bmeyer99]

Transform task is only 46% done after 10 hours 25 minutes so far Looks like it'll be around 22 hours at this rate (previously it was 26+ hours). Will let the transform/job finish and forward logs on to support.

We're actually still only trialling Veeam at our company so all we've ever used is v6.x (we have used Symantec for 6+ years). The issue with Symantec is that since switching to Backup Exec De-duplicated backups our backup window is getting too large, so we are hoping to use Veeam for most of our backups. This is pretty much the last remaining issue as backups take only 1-3 hours total for all 16+ of our servers - the transforms are taking 26-32+ hours and that's a big problem before we go ahead with this purchase Replication of the Hyper-V servers is also working quite well. Hopefully the logs will shed some light for support once this current transform completes.

Gostev; any idea if there are any other Veeam users testing out this fix as well?

What kind of storage do you have on the back end? Performing the transform is very IO intensive as it has to read\write to every file in the backup chain to create the new VBK file and the change the VIBs to VRBs. I would recommend 1 of 2 options.

1) Perform an active full instead of a synthetic full. This will remove the IO intensity off of the backup storage.
2) By default the job will do a standard incremental and then create a synthetic full on Saturday. Your best bet is to change the job to Reverse Incremental so that the IO is spread across the week and you will always have a synthetic full as your last backup. If you are performing your backups in 1-3 hours you will be able to perform a reverse incremental in 2-6 most likely.

[nreutemann]

Hello, here my new experience after upgrade.
First, we reinstall Veeam B&R server, now the server is w2008 R2. After that, we install Veeam B&R 6 and start to test. And after that, upgrade to 6.1. That´s all the changhe that we do. The LUNs, the proxys and the jobs configurations remains the same than before.
So, this friday, after a week of incremental backups, start the transformation. The two most bigger jobs (one whit 15 VMs and 400Gb of full backup and the other whit 27 VMs and 500Gb of full backup) before the upgrade need around 55hs the first one and the other needs at least 65hs. This weekend, the first take 26hs to transform and the second take 30hs.
So we got a very good performance optimization.
The test will continue the next weekend to see if this numbers remains the same.

[bmeyer99]

There were improvements made regarding transforms in 6.1. Glad to hear it is working faster for you.

[trafsta]

What kind of storage do you have on the back end? Performing the transform is very IO intensive as it has to read\write to every file in the backup chain to create the new VBK file and the change the VIBs to VRBs. I would recommend 1 of 2 options.

1) Perform an active full instead of a synthetic full. This will remove the IO intensity off of the backup storage.
2) By default the job will do a standard incremental and then create a synthetic full on Saturday. Your best bet is to change the job to Reverse Incremental so that the IO is spread across the week and you will always have a synthetic full as your last backup. If you are performing your backups in 1-3 hours you will be able to perform a reverse incremental in 2-6 most likely.

The storage system is almost completely idle during the 20-32hrs that the transform is taking place - this is nothing to do with a storage bottleneck. 12 x 2TB 3.5" disks w/ 1GB cache controller raid-6. It's a 400GB file (with another 250GB or so in the chain to transform), no way should it take this long... Support ticket # is 5188903 for more info. Latest test took 20 hour 20 minutes to transform with the test build of the Veeam 6.0 agent that support provided. Still next to no I/O during that 20 hours.

[Gostev]

Gostev; any idea if there are any other Veeam users testing out this fix as well?

As far as I know, nobody is testing the fix right now (not even you), because the fix did not exist until Friday last week (which is when the 6.1 build with the fix was assembled). I don't expect anyone to get this fix until late this week, because we need to test the new code first (as usual). Thus, I am not really sure what agent code are you running right now, may be just that debug agent version with more logging around transform? At least, your results are pretty telling (they are essentially exactly the same as before).

[trafsta]

As far as I know, nobody is testing the fix right now (not even you), because the fix did not exist until Friday last week (which is when the 6.1 build with the fix was assembled). I don't expect anyone to get this fix until late this week, because we need to test the new code first (as usual). Thus, I am not really sure what agent code are you running right now, may be just that debug agent version with more logging around transform? At least, your results are pretty telling (they are essentially exactly the same as before).

Support had said on June 5th "We've created a new version of VeeamAgent and we suspect that it should increase a transform performance. It's uploaded on FTP server and placed into "new-agent-05.06" folder...." so pretty sure it was a proposed fix. Will await a response from support since I provided them the test logs yesterday.

[Gostev]

Yes, I stand corrected - checked with support and devs, and apparently we do have some "early" version of the patched agent, that was not created as a part of normal build process - which is what I monitor. That agent was given to 4 customers including you, among them two are now "happy" with transform performance, and we have not yet heard from one other guy. Devs are looking at your transform logs as well.

[trafsta]

Yes, I stand corrected - checked with support and devs, and apparently we do have some "early" version of the patched agent, that was not created as a part of normal build process - which is what I monitor. That agent was given to 4 customers including you, among them two are now "happy" with transform performance, and we have not yet heard from one other guy. Devs are looking at your transform logs as well.

Thanks for the update. Perhaps the later versions of the patched agent will be even better, time will tell.

[AlexL]

Any idea when this patch will be generally available? I've got multiple jobs with transform times over 20 hours, with the longest clocking in at 40.5 hours.

[Vitaliy S.]

As soon as all patch verification checks are performed our support team will notify everyone who has opened a case on this issue.

[AlexL]

cool, but when will it be generally available for download? Will it be in patch1 for 6.1? Is there a release schedule for patches?

[Vitaliy S.]

when will it be generally available for download?

Cannot share any exact date, as I haven't received any confirmation from our support team yet.

[Gostev]

It will certainly be included in Patch 1, however anyone who has support case open should get a separate hotfix before Patch 1 (at least, this is our standard procedure).

[itwheel]

Same issue, Case 5200644

Requested hotfix.

[trafsta]

Latest test this past Friday night with a custom build of v6.1 from support resulted in 11 hour transform instead of the usual 22-33ish hours, so definitely a large improvement

[J1mbo]

Gostev et al, any idea on the likely GA for patch 1?

[Vitaliy S.]

Hi James, the position regarding patch release hasn't changed since this post from Anton.

[J1mbo]

OK but some indication of timeline would be really helpful for me, since if I've understood this issue properly it does prevent me from deploying 6.1.

[Vitaliy S.]

Well...the transform operation is performed in exactly the same manner in both 6.0 and 6.1 versions. If everything works fine for you now, it doesn't seem like that this might be a real showstopper for you Anyway, you can always deploy the hotfix from our support team.

[J1mbo]

This is against a v5 installation. But anyway, I'll see if I can get hold of the hotfix.