-
- VeeaMVP
- Posts: 6155
- Liked: 1968 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: transform incremental to rollback possible bottlenecks
Transform can really be slow, and the main reason is the combination of IOs and randomness: for every bit written to the backup you have 4 I/O:
- 1° I/O to read the bit from the .vbk or .vib files
- 2° I/O to write the same bit into the resultant .vbk file
- 3° I/O to read again the bit from the .vbk or .vib files
- 4° I/O wo write the resultant .vrb file
and those IOs happen randomly on the storage, since Veeam has to find the bits to replace them. On a 1.8 Tb of vbk, searching for bits to be replaced can be a time-consuming effort. That's probably why you see low server usage, because the load is all on the backup storage.
- 1° I/O to read the bit from the .vbk or .vib files
- 2° I/O to write the same bit into the resultant .vbk file
- 3° I/O to read again the bit from the .vbk or .vib files
- 4° I/O wo write the resultant .vrb file
and those IOs happen randomly on the storage, since Veeam has to find the bits to replace them. On a 1.8 Tb of vbk, searching for bits to be replaced can be a time-consuming effort. That's probably why you see low server usage, because the load is all on the backup storage.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Chief Product Officer
- Posts: 31606
- Liked: 7102 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: transform incremental to rollback possible bottlenecks
Let me check tomorrow if this may be the result of that performance degradation issue I have referenced in my previous post.
-
- Influencer
- Posts: 12
- Liked: never
- Joined: Apr 28, 2011 3:04 pm
- Full Name: Kevin Beaumont
- Contact:
Re: transform incremental to rollback possible bottlenecks
Cheers Gostev!
Just for information, the Veeam backup repo box is Windows Server 2003 R2 64-bit. That caching problem hadn't occured to me.... That's interesting for sure.
Just for information, the Veeam backup repo box is Windows Server 2003 R2 64-bit. That caching problem hadn't occured to me.... That's interesting for sure.
-
- Veteran
- Posts: 264
- Liked: 30 times
- Joined: Mar 22, 2011 7:43 pm
- Full Name: Ted
- Contact:
Re: transform incremental to rollback possible bottlenecks
Gostev -- is this the issue I brought up in the 6th post of this thread?
-
- Veteran
- Posts: 264
- Liked: 30 times
- Joined: Mar 22, 2011 7:43 pm
- Full Name: Ted
- Contact:
Re: transform incremental to rollback possible bottlenecks
Also -- looking forward to the manual instructions for tweaking those cache settings.
-
- Influencer
- Posts: 12
- Liked: never
- Joined: Apr 28, 2011 3:04 pm
- Full Name: Kevin Beaumont
- Contact:
Re: transform incremental to rollback possible bottlenecks
Folks
This is pure speculation at this point as I haven't had time to test it, but I think I may be getting caught with this issue: http://support.microsoft.com/kb/920739
This is pure speculation at this point as I haven't had time to test it, but I think I may be getting caught with this issue: http://support.microsoft.com/kb/920739
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: transform incremental to rollback possible bottlenecks
Hmmm I'm in Win2k8 R2 SP1 so I don't think that would be the problem I am experiencing Kevin Thats not to say that it isn't yours.
Eagerly awaiting Gostev's pending info on this
Eagerly awaiting Gostev's pending info on this
-
- Veteran
- Posts: 264
- Liked: 30 times
- Joined: Mar 22, 2011 7:43 pm
- Full Name: Ted
- Contact:
Re: transform incremental to rollback possible bottlenecks
trafsta -- Kevin's issue is the same issue still seen (but much less often) in 2008 R2 (and without the workarounds available to 2003 and 2008). I don't know if this is the issue Gostev is talking about, but it's the issue I brought up very early in this thread.
-
- Chief Product Officer
- Posts: 31606
- Liked: 7102 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: transform incremental to rollback possible bottlenecks
OK folks, so apparently these low level cache settings cannot be modified manually (not a registry setting kind of thing), it can only be done through Windows API calls. We already have a tool for that (one we used for our own testing). Thus, we would need to do the following. Please create a support case (unless you already have support case going on that issue), and PM me the support case ID. I will have your case picked up right on the required tier, where the senior support people are already standing by, ready to schedule a webex with you to apply this change using this tool.
For future readers - I do not want or need to be getting similar requests via PM after May 3rd, 2012. If the above helps, we will create support KB, and you will be able to get this change applied by tier 1 support folks.
Also, just to remind - this is already integrated into the proxy setup of soon-to-be-released version 6.1.
For future readers - I do not want or need to be getting similar requests via PM after May 3rd, 2012. If the above helps, we will create support KB, and you will be able to get this change applied by tier 1 support folks.
Also, just to remind - this is already integrated into the proxy setup of soon-to-be-released version 6.1.
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: transform incremental to rollback possible bottlenecks
6.1! I'm still on the trial and yet I'm somehow excited to see what new features and fixes are coming in 6.1 lol, good stuff.
I'll be calling support first thing in the morning on this as I'd like to get it tested this Saturday night on our next full transform. Thanks a lot Gostev.
I'll be calling support first thing in the morning on this as I'd like to get it tested this Saturday night on our next full transform. Thanks a lot Gostev.
-
- Veteran
- Posts: 264
- Liked: 30 times
- Joined: Mar 22, 2011 7:43 pm
- Full Name: Ted
- Contact:
Re: transform incremental to rollback possible bottlenecks
Thanks Gostev. Since I'm using reverse incremental now, I will not bother until 6.1 is released.
-
- Enthusiast
- Posts: 29
- Liked: never
- Joined: Jan 19, 2012 3:35 pm
- Full Name: Brian Weinberg
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
anyone have an update to Gostev's fix?
-
- Chief Product Officer
- Posts: 31606
- Liked: 7102 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Did not help in one case, but the backup proxy there was Windows 2003 R2 anyway.
Another case, I have not heard the results yet.
Another case, I have not heard the results yet.
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
It didn't work in my case, still awaiting further instructions from support.
-
- Chief Product Officer
- Posts: 31606
- Liked: 7102 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
It was just a theory anyway, so don't put too much hope in this
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
lol. I'll keep my fingers crossed for a speedy resolution. This is the only remaining problem that we have with the software and once this is resolved I am sure we'll be purchasing it (we are currently in trial, just had it extended for another 30 days). In its current state, 27+ hours to perform a transform on a 300-400GB backup file is quite an issue for usGostev wrote:It was just a theory anyway, so don't put too much hope in this
-
- Enthusiast
- Posts: 47
- Liked: 6 times
- Joined: Mar 06, 2012 11:45 pm
- Full Name: Nicolas Reutemann
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Any update about this theme?
My situation:
Veeam Server: IBM x3650 M2 - W2003 R2 x64 SP2 - Veeam V&R 6.0.0.181 (64 Bit) - 16Gb RAM
SAN: FC
Storage: 2 x IBM DS3400
I got 4 LUNs to store backups, all on separated RAIDs an 2 on each storage.
The two extreme cases I got:
a) One backup job, 1 VM whit 629Gb, 42% done, 62 hours
b) One job, 18 VM, 1.9 Tb, 13% done, 62 hours
This data of this two jobs are in separated raids, in separated storages.
This is a bad situation, it´s monday and I need to run incremental jobs tonight. The perspective is not good at all.
I only see one way to resolve this and it is not use rollback transform anymore.
Any idea?
Thanks in advance.
nicolas.
My situation:
Veeam Server: IBM x3650 M2 - W2003 R2 x64 SP2 - Veeam V&R 6.0.0.181 (64 Bit) - 16Gb RAM
SAN: FC
Storage: 2 x IBM DS3400
I got 4 LUNs to store backups, all on separated RAIDs an 2 on each storage.
The two extreme cases I got:
a) One backup job, 1 VM whit 629Gb, 42% done, 62 hours
b) One job, 18 VM, 1.9 Tb, 13% done, 62 hours
This data of this two jobs are in separated raids, in separated storages.
This is a bad situation, it´s monday and I need to run incremental jobs tonight. The perspective is not good at all.
I only see one way to resolve this and it is not use rollback transform anymore.
Any idea?
Thanks in advance.
nicolas.
-
- Chief Product Officer
- Posts: 31606
- Liked: 7102 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Nicolas, what is your support case number please?
-
- Enthusiast
- Posts: 47
- Liked: 6 times
- Joined: Mar 06, 2012 11:45 pm
- Full Name: Nicolas Reutemann
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
I´m not created any support case.
I´m waiting the news about on this theme.
I´m waiting the news about on this theme.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Apr 03, 2012 4:29 pm
- Full Name: Kevin Clawson
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
I am in the same boat. I am using a physical server with iSCSI connections to a 3 node Scale Computing setup with 6 TB of space. My transforms take days to complete, which means I am missing incrementals in between. Looking at a few posts, it sounds like its possible that this could be caused by old OS's on the backup proxies? Does that sound right? To save money and space I made proxies on Windows XP. Should I suck it up and make these Windows 2008 R2 or is this a different issue? My iSCSI connections are set to fixed path with MPIO enabled. Let me know what you think!
-
- Chief Product Officer
- Posts: 31606
- Liked: 7102 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Please create a support case and PM me the ID. We need our support to be able to collect additional required information from all people suffering from this issue, runs some tests, etc.nreutemann wrote:I´m not created any support case.
I´m waiting the news about on this theme.
No, it was just an idea and we dismissed it after monitoring the memory usage during transform in the real environments having this issue. Please also open a support case, and let me know the ID.KClawson wrote:Looking at a few posts, it sounds like its possible that this could be caused by old OS's on the backup proxies?
-
- Influencer
- Posts: 12
- Liked: never
- Joined: Apr 28, 2011 3:04 pm
- Full Name: Kevin Beaumont
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Just to report we're still seeing issues. At first I thought it was storage related, but I've checked the storage out and it's pretty fast. Certainly, way way faster than the transforms are running. We tried the cache fixes but that didn't help.
-
- Chief Product Officer
- Posts: 31606
- Liked: 7102 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
We are doing experiments internally now, since we were able to reproduce the issue.
-
- Enthusiast
- Posts: 61
- Liked: never
- Joined: Aug 02, 2009 7:33 pm
- Contact:
Long transform
[merged]
I have a problem with long transform of backups on Saturday (from current week). Today I had a job that is running for 22 hours and 75% of completion. I understand that concurrent jobs and RAID6 as a target can slow down transform job but on the other hand I have a EQL as a source direct attached SAS storage as target, 2 processors with 4 cores each and 32GB RAM (windows 2008 R2) and now the job is the only one running on the server.
I have checked performance of the server and CPU is on 10-20% level, memory is used mainly for cache and there is very litle IO trafic (checked in procexp).
What can be a problem?
I have a problem with long transform of backups on Saturday (from current week). Today I had a job that is running for 22 hours and 75% of completion. I understand that concurrent jobs and RAID6 as a target can slow down transform job but on the other hand I have a EQL as a source direct attached SAS storage as target, 2 processors with 4 cores each and 32GB RAM (windows 2008 R2) and now the job is the only one running on the server.
I have checked performance of the server and CPU is on 10-20% level, memory is used mainly for cache and there is very litle IO trafic (checked in procexp).
What can be a problem?
-
- VeeaMVP
- Posts: 6155
- Liked: 1968 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
You first of all need to check realtime bottleneck stats of that job, you can see them by by hovering over the overall bottleneck value in the top of real-time statistics window.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Enthusiast
- Posts: 61
- Liked: never
- Joined: Aug 02, 2009 7:33 pm
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
My data is:385GB. Last time backup took 28hr, processing rate: 4MB/s; Read: 64.5GB and transferred was 35GB. It was a backup of 12 VMs.
Backup took less then two hours - transform 26hrs!
5/28/2012 12:43:03 AM :: Load: Source 75% > Proxy 43% > Network 31% > Target 18%
As I said I was surprised by low impact of backup on the server. I would expect for example CPU to be maxed out or fairly high IO but nothing like this happened.
Backup took less then two hours - transform 26hrs!
5/28/2012 12:43:03 AM :: Load: Source 75% > Proxy 43% > Network 31% > Target 18%
As I said I was surprised by low impact of backup on the server. I would expect for example CPU to be maxed out or fairly high IO but nothing like this happened.
-
- Enthusiast
- Posts: 61
- Liked: never
- Joined: Aug 02, 2009 7:33 pm
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
One more thing. I noticed that even though my backup proxy has a limit of 3 concurrent jobs this limit does not apply to transform jobs. I have 12 backup jobs and I have seen 10 concurrent transform jobs running - is it by design and could this cause a problem?
Nevertheless slow performance occurs only when 1 transform job is running.
Nevertheless slow performance occurs only when 1 transform job is running.
-
- VP, Product Management
- Posts: 27297
- Liked: 2774 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Yes, this is how it works because backup proxies do not take any part in the transform process.
Please find more details in this topic > Synthetic Fulls running all, while 1 conc jobs per proxy
Please find more details in this topic > Synthetic Fulls running all, while 1 conc jobs per proxy
-
- Influencer
- Posts: 12
- Liked: never
- Joined: Apr 28, 2011 3:04 pm
- Full Name: Kevin Beaumont
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Something really interesting I have found out in last few days:
Earlier in this topic, somebody posted a graph of storage IO, showing a curved line drop off during the transform. I ran a full backup (from scratch) of our file servers (2tb), and noticed a curved line drop off in CPU usage during the backup. It started really quick, and it gradually got slower and slower on the Veeam proxy (Windows Server 2008 R2). I tried it again with a Windows Server 2003 R2 backup proxy, and saw the same thing. I'm not sure if it's the length of the time the job goes on for, or the size the VBK file becomes.
Earlier in this topic, somebody posted a graph of storage IO, showing a curved line drop off during the transform. I ran a full backup (from scratch) of our file servers (2tb), and noticed a curved line drop off in CPU usage during the backup. It started really quick, and it gradually got slower and slower on the Veeam proxy (Windows Server 2008 R2). I tried it again with a Windows Server 2003 R2 backup proxy, and saw the same thing. I'm not sure if it's the length of the time the job goes on for, or the size the VBK file becomes.
-
- Chief Product Officer
- Posts: 31606
- Liked: 7102 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
It's neither - in case of full backup, it's just buggy system cache behavior in Windows. This is a known issue and there is a big dedicated thread about it here, but long story short, 6.1 addresses this by tuning the system cache settings.
For the issue discussed in this topic, we have just built a custom data mover agent with detailed performance logging, so the research continues full steam ahead.
For the issue discussed in this topic, we have just built a custom data mover agent with detailed performance logging, so the research continues full steam ahead.
Who is online
Users browsing this forum: Bing [Bot], JonDoe, ronald.ducie, w20645_Duc and 163 guests