Comprehensive data protection for all workloads
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by dellock6 » 2 people like this post

Transform can really be slow, and the main reason is the combination of IOs and randomness: for every bit written to the backup you have 4 I/O:
- 1° I/O to read the bit from the .vbk or .vib files
- 2° I/O to write the same bit into the resultant .vbk file
- 3° I/O to read again the bit from the .vbk or .vib files
- 4° I/O wo write the resultant .vrb file
and those IOs happen randomly on the storage, since Veeam has to find the bits to replace them. On a 1.8 Tb of vbk, searching for bits to be replaced can be a time-consuming effort. That's probably why you see low server usage, because the load is all on the backup storage.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by Gostev »

Let me check tomorrow if this may be the result of that performance degradation issue I have referenced in my previous post.
KevinBeaumont
Influencer
Posts: 12
Liked: never
Joined: Apr 28, 2011 3:04 pm
Full Name: Kevin Beaumont
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by KevinBeaumont »

Cheers Gostev!

Just for information, the Veeam backup repo box is Windows Server 2003 R2 64-bit. That caching problem hadn't occured to me.... That's interesting for sure.
averylarry
Veteran
Posts: 264
Liked: 30 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by averylarry »

Gostev -- is this the issue I brought up in the 6th post of this thread?
averylarry
Veteran
Posts: 264
Liked: 30 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by averylarry »

Also -- looking forward to the manual instructions for tweaking those cache settings.
KevinBeaumont
Influencer
Posts: 12
Liked: never
Joined: Apr 28, 2011 3:04 pm
Full Name: Kevin Beaumont
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by KevinBeaumont »

Folks

This is pure speculation at this point as I haven't had time to test it, but I think I may be getting caught with this issue: http://support.microsoft.com/kb/920739
trafsta
Enthusiast
Posts: 45
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by trafsta »

Hmmm I'm in Win2k8 R2 SP1 so I don't think that would be the problem I am experiencing Kevin :( Thats not to say that it isn't yours.

Eagerly awaiting Gostev's pending info on this :)
averylarry
Veteran
Posts: 264
Liked: 30 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by averylarry »

trafsta -- Kevin's issue is the same issue still seen (but much less often) in 2008 R2 (and without the workarounds available to 2003 and 2008). I don't know if this is the issue Gostev is talking about, but it's the issue I brought up very early in this thread.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by Gostev »

OK folks, so apparently these low level cache settings cannot be modified manually (not a registry setting kind of thing), it can only be done through Windows API calls. We already have a tool for that (one we used for our own testing). Thus, we would need to do the following. Please create a support case (unless you already have support case going on that issue), and PM me the support case ID. I will have your case picked up right on the required tier, where the senior support people are already standing by, ready to schedule a webex with you to apply this change using this tool.

For future readers - I do not want or need to be getting similar requests via PM after May 3rd, 2012. If the above helps, we will create support KB, and you will be able to get this change applied by tier 1 support folks.

Also, just to remind - this is already integrated into the proxy setup of soon-to-be-released version 6.1.
trafsta
Enthusiast
Posts: 45
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by trafsta »

:shock: 6.1! I'm still on the trial and yet I'm somehow excited to see what new features and fixes are coming in 6.1 lol, good stuff.

I'll be calling support first thing in the morning on this as I'd like to get it tested this Saturday night on our next full transform. Thanks a lot Gostev.
averylarry
Veteran
Posts: 264
Liked: 30 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by averylarry »

Thanks Gostev. Since I'm using reverse incremental now, I will not bother until 6.1 is released.
TimeKnight
Enthusiast
Posts: 29
Liked: never
Joined: Jan 19, 2012 3:35 pm
Full Name: Brian Weinberg
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by TimeKnight »

anyone have an update to Gostev's fix?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Gostev »

Did not help in one case, but the backup proxy there was Windows 2003 R2 anyway.
Another case, I have not heard the results yet.
trafsta
Enthusiast
Posts: 45
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta »

It didn't work in my case, still awaiting further instructions from support.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Gostev »

It was just a theory anyway, so don't put too much hope in this ;)
trafsta
Enthusiast
Posts: 45
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta »

Gostev wrote:It was just a theory anyway, so don't put too much hope in this ;)
lol. I'll keep my fingers crossed for a speedy resolution. This is the only remaining problem that we have with the software and once this is resolved I am sure we'll be purchasing it (we are currently in trial, just had it extended for another 30 days). In its current state, 27+ hours to perform a transform on a 300-400GB backup file is quite an issue for us :(
nreutemann
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by nreutemann »

Any update about this theme?

My situation:

Veeam Server: IBM x3650 M2 - W2003 R2 x64 SP2 - Veeam V&R 6.0.0.181 (64 Bit) - 16Gb RAM
SAN: FC
Storage: 2 x IBM DS3400

I got 4 LUNs to store backups, all on separated RAIDs an 2 on each storage.

The two extreme cases I got:

a) One backup job, 1 VM whit 629Gb, 42% done, 62 hours
b) One job, 18 VM, 1.9 Tb, 13% done, 62 hours

This data of this two jobs are in separated raids, in separated storages.

This is a bad situation, it´s monday and I need to run incremental jobs tonight. The perspective is not good at all.

I only see one way to resolve this and it is not use rollback transform anymore.

Any idea?

Thanks in advance.

nicolas.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Gostev »

Nicolas, what is your support case number please?
nreutemann
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by nreutemann »

I´m not created any support case.
I´m waiting the news about on this theme.
KClawson
Novice
Posts: 8
Liked: never
Joined: Apr 03, 2012 4:29 pm
Full Name: Kevin Clawson
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by KClawson »

I am in the same boat. I am using a physical server with iSCSI connections to a 3 node Scale Computing setup with 6 TB of space. My transforms take days to complete, which means I am missing incrementals in between. Looking at a few posts, it sounds like its possible that this could be caused by old OS's on the backup proxies? Does that sound right? To save money and space I made proxies on Windows XP. Should I suck it up and make these Windows 2008 R2 or is this a different issue? My iSCSI connections are set to fixed path with MPIO enabled. Let me know what you think!
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Gostev »

nreutemann wrote:I´m not created any support case.
I´m waiting the news about on this theme.
Please create a support case and PM me the ID. We need our support to be able to collect additional required information from all people suffering from this issue, runs some tests, etc.
KClawson wrote:Looking at a few posts, it sounds like its possible that this could be caused by old OS's on the backup proxies?
No, it was just an idea and we dismissed it after monitoring the memory usage during transform in the real environments having this issue. Please also open a support case, and let me know the ID.
KevinBeaumont
Influencer
Posts: 12
Liked: never
Joined: Apr 28, 2011 3:04 pm
Full Name: Kevin Beaumont
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by KevinBeaumont »

Just to report we're still seeing issues. At first I thought it was storage related, but I've checked the storage out and it's pretty fast. Certainly, way way faster than the transforms are running. We tried the cache fixes but that didn't help.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Gostev »

We are doing experiments internally now, since we were able to reproduce the issue.
pizang
Enthusiast
Posts: 61
Liked: never
Joined: Aug 02, 2009 7:33 pm
Contact:

Long transform

Post by pizang »

[merged]

I have a problem with long transform of backups on Saturday (from current week). Today I had a job that is running for 22 hours and 75% of completion. I understand that concurrent jobs and RAID6 as a target can slow down transform job but on the other hand I have a EQL as a source direct attached SAS storage as target, 2 processors with 4 cores each and 32GB RAM (windows 2008 R2) and now the job is the only one running on the server.

I have checked performance of the server and CPU is on 10-20% level, memory is used mainly for cache and there is very litle IO trafic (checked in procexp).

What can be a problem?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by dellock6 »

You first of all need to check realtime bottleneck stats of that job, you can see them by by hovering over the overall bottleneck value in the top of real-time statistics window.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
pizang
Enthusiast
Posts: 61
Liked: never
Joined: Aug 02, 2009 7:33 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by pizang »

My data is:385GB. Last time backup took 28hr, processing rate: 4MB/s; Read: 64.5GB and transferred was 35GB. It was a backup of 12 VMs.

Backup took less then two hours - transform 26hrs!

5/28/2012 12:43:03 AM :: Load: Source 75% > Proxy 43% > Network 31% > Target 18%

As I said I was surprised by low impact of backup on the server. I would expect for example CPU to be maxed out or fairly high IO but nothing like this happened.
pizang
Enthusiast
Posts: 61
Liked: never
Joined: Aug 02, 2009 7:33 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by pizang »

One more thing. I noticed that even though my backup proxy has a limit of 3 concurrent jobs this limit does not apply to transform jobs. I have 12 backup jobs and I have seen 10 concurrent transform jobs running - is it by design and could this cause a problem?
Nevertheless slow performance occurs only when 1 transform job is running.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Vitaliy S. »

Yes, this is how it works because backup proxies do not take any part in the transform process.
Please find more details in this topic > Synthetic Fulls running all, while 1 conc jobs per proxy
KevinBeaumont
Influencer
Posts: 12
Liked: never
Joined: Apr 28, 2011 3:04 pm
Full Name: Kevin Beaumont
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by KevinBeaumont »

Something really interesting I have found out in last few days:

Earlier in this topic, somebody posted a graph of storage IO, showing a curved line drop off during the transform. I ran a full backup (from scratch) of our file servers (2tb), and noticed a curved line drop off in CPU usage during the backup. It started really quick, and it gradually got slower and slower on the Veeam proxy (Windows Server 2008 R2). I tried it again with a Windows Server 2003 R2 backup proxy, and saw the same thing. I'm not sure if it's the length of the time the job goes on for, or the size the VBK file becomes.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Gostev »

It's neither - in case of full backup, it's just buggy system cache behavior in Windows. This is a known issue and there is a big dedicated thread about it here, but long story short, 6.1 addresses this by tuning the system cache settings.

For the issue discussed in this topic, we have just built a custom data mover agent with detailed performance logging, so the research continues full steam ahead.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], tyler.jurgens and 233 guests