Incremental to rollback transform: possible bottlenecks?

[dellock6]

Transform can really be slow, and the main reason is the combination of IOs and randomness: for every bit written to the backup you have 4 I/O:
- 1° I/O to read the bit from the .vbk or .vib files
- 2° I/O to write the same bit into the resultant .vbk file
- 3° I/O to read again the bit from the .vbk or .vib files
- 4° I/O wo write the resultant .vrb file
and those IOs happen randomly on the storage, since Veeam has to find the bits to replace them. On a 1.8 Tb of vbk, searching for bits to be replaced can be a time-consuming effort. That's probably why you see low server usage, because the load is all on the backup storage.

[Gostev]

Let me check tomorrow if this may be the result of that performance degradation issue I have referenced in my previous post.

[KevinBeaumont]

Cheers Gostev!

Just for information, the Veeam backup repo box is Windows Server 2003 R2 64-bit. That caching problem hadn't occured to me.... That's interesting for sure.

[averylarry]

Gostev -- is this the issue I brought up in the 6th post of this thread?

[averylarry]

Also -- looking forward to the manual instructions for tweaking those cache settings.

[KevinBeaumont]

Folks

This is pure speculation at this point as I haven't had time to test it, but I think I may be getting caught with this issue: http://support.microsoft.com/kb/920739

[trafsta]

Hmmm I'm in Win2k8 R2 SP1 so I don't think that would be the problem I am experiencing Kevin Thats not to say that it isn't yours.

Eagerly awaiting Gostev's pending info on this

[averylarry]

trafsta -- Kevin's issue is the same issue still seen (but much less often) in 2008 R2 (and without the workarounds available to 2003 and 2008). I don't know if this is the issue Gostev is talking about, but it's the issue I brought up very early in this thread.

[Gostev]

OK folks, so apparently these low level cache settings cannot be modified manually (not a registry setting kind of thing), it can only be done through Windows API calls. We already have a tool for that (one we used for our own testing). Thus, we would need to do the following. Please create a support case (unless you already have support case going on that issue), and PM me the support case ID. I will have your case picked up right on the required tier, where the senior support people are already standing by, ready to schedule a webex with you to apply this change using this tool.

For future readers - I do not want or need to be getting similar requests via PM after May 3rd, 2012. If the above helps, we will create support KB, and you will be able to get this change applied by tier 1 support folks.

Also, just to remind - this is already integrated into the proxy setup of soon-to-be-released version 6.1.

[trafsta]

6.1! I'm still on the trial and yet I'm somehow excited to see what new features and fixes are coming in 6.1 lol, good stuff.

I'll be calling support first thing in the morning on this as I'd like to get it tested this Saturday night on our next full transform. Thanks a lot Gostev.

[averylarry]

Thanks Gostev. Since I'm using reverse incremental now, I will not bother until 6.1 is released.

[TimeKnight]

anyone have an update to Gostev's fix?

[Gostev]

Did not help in one case, but the backup proxy there was Windows 2003 R2 anyway.
Another case, I have not heard the results yet.

[trafsta]

It didn't work in my case, still awaiting further instructions from support.

[Gostev]

It was just a theory anyway, so don't put too much hope in this

[trafsta]

It was just a theory anyway, so don't put too much hope in this

lol. I'll keep my fingers crossed for a speedy resolution. This is the only remaining problem that we have with the software and once this is resolved I am sure we'll be purchasing it (we are currently in trial, just had it extended for another 30 days). In its current state, 27+ hours to perform a transform on a 300-400GB backup file is quite an issue for us

[nreutemann]

Any update about this theme?

My situation:

Veeam Server: IBM x3650 M2 - W2003 R2 x64 SP2 - Veeam V&R 6.0.0.181 (64 Bit) - 16Gb RAM
SAN: FC
Storage: 2 x IBM DS3400

I got 4 LUNs to store backups, all on separated RAIDs an 2 on each storage.

The two extreme cases I got:

a) One backup job, 1 VM whit 629Gb, 42% done, 62 hours
b) One job, 18 VM, 1.9 Tb, 13% done, 62 hours

This data of this two jobs are in separated raids, in separated storages.

This is a bad situation, it´s monday and I need to run incremental jobs tonight. The perspective is not good at all.

I only see one way to resolve this and it is not use rollback transform anymore.

Any idea?

Thanks in advance.

nicolas.

[Gostev]

Nicolas, what is your support case number please?

[nreutemann]

I´m not created any support case.
I´m waiting the news about on this theme.

[KClawson]

I am in the same boat. I am using a physical server with iSCSI connections to a 3 node Scale Computing setup with 6 TB of space. My transforms take days to complete, which means I am missing incrementals in between. Looking at a few posts, it sounds like its possible that this could be caused by old OS's on the backup proxies? Does that sound right? To save money and space I made proxies on Windows XP. Should I suck it up and make these Windows 2008 R2 or is this a different issue? My iSCSI connections are set to fixed path with MPIO enabled. Let me know what you think!

[Gostev]

I´m not created any support case.
I´m waiting the news about on this theme.

Please create a support case and PM me the ID. We need our support to be able to collect additional required information from all people suffering from this issue, runs some tests, etc.

Looking at a few posts, it sounds like its possible that this could be caused by old OS's on the backup proxies?

No, it was just an idea and we dismissed it after monitoring the memory usage during transform in the real environments having this issue. Please also open a support case, and let me know the ID.

[KevinBeaumont]

Just to report we're still seeing issues. At first I thought it was storage related, but I've checked the storage out and it's pretty fast. Certainly, way way faster than the transforms are running. We tried the cache fixes but that didn't help.

[Gostev]

We are doing experiments internally now, since we were able to reproduce the issue.

[pizang]

[merged]

I have a problem with long transform of backups on Saturday (from current week). Today I had a job that is running for 22 hours and 75% of completion. I understand that concurrent jobs and RAID6 as a target can slow down transform job but on the other hand I have a EQL as a source direct attached SAS storage as target, 2 processors with 4 cores each and 32GB RAM (windows 2008 R2) and now the job is the only one running on the server.

I have checked performance of the server and CPU is on 10-20% level, memory is used mainly for cache and there is very litle IO trafic (checked in procexp).

What can be a problem?

[dellock6]

You first of all need to check realtime bottleneck stats of that job, you can see them by by hovering over the overall bottleneck value in the top of real-time statistics window.

[pizang]

My data is:385GB. Last time backup took 28hr, processing rate: 4MB/s; Read: 64.5GB and transferred was 35GB. It was a backup of 12 VMs.

Backup took less then two hours - transform 26hrs!

5/28/2012 12:43:03 AM :: Load: Source 75% > Proxy 43% > Network 31% > Target 18%

As I said I was surprised by low impact of backup on the server. I would expect for example CPU to be maxed out or fairly high IO but nothing like this happened.

[pizang]

One more thing. I noticed that even though my backup proxy has a limit of 3 concurrent jobs this limit does not apply to transform jobs. I have 12 backup jobs and I have seen 10 concurrent transform jobs running - is it by design and could this cause a problem?
Nevertheless slow performance occurs only when 1 transform job is running.

[Vitaliy S.]

Yes, this is how it works because backup proxies do not take any part in the transform process.
Please find more details in this topic > Synthetic Fulls running all, while 1 conc jobs per proxy

[KevinBeaumont]

Something really interesting I have found out in last few days:

Earlier in this topic, somebody posted a graph of storage IO, showing a curved line drop off during the transform. I ran a full backup (from scratch) of our file servers (2tb), and noticed a curved line drop off in CPU usage during the backup. It started really quick, and it gradually got slower and slower on the Veeam proxy (Windows Server 2008 R2). I tried it again with a Windows Server 2003 R2 backup proxy, and saw the same thing. I'm not sure if it's the length of the time the job goes on for, or the size the VBK file becomes.

[Gostev]

It's neither - in case of full backup, it's just buggy system cache behavior in Windows. This is a known issue and there is a big dedicated thread about it here, but long story short, 6.1 addresses this by tuning the system cache settings.

For the issue discussed in this topic, we have just built a custom data mover agent with detailed performance logging, so the research continues full steam ahead.