-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
Hi, will it reach 12 hours for just 1.+ TB of data?
Look at the screen shot. Job started last night at 8:00pm still didn't finish
http://www.freeimagehosting.net/jrivf
Any help or work around?
Look at the screen shot. Job started last night at 8:00pm still didn't finish
http://www.freeimagehosting.net/jrivf
Any help or work around?
-
- VP, Product Management
- Posts: 6024
- Liked: 2853 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: transform incremental to rollback possible bottlenecks
It might, what is the random R/W I/O performance of your target storage? Synthetic fulls will take longer that actual fulls in most environments. How long does a real full backup of that data take?
-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
Hi, R/W around 100mb/ W - 75mb R. The full initial back it took around 5 hours + of 1.4 TB data of of it approximately 600GB used.
Thanks,
Hussain
Thanks,
Hussain
-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
Completed 17hrs and 35 minutes still Transform 50% Done Anybody experience this issue before? I'm afraid that it will corrupt the Mail Server.
Thanks,
Thanks,
-
- Chief Product Officer
- Posts: 31606
- Liked: 7097 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: transform incremental to rollback possible bottlenecks
Transform is a post-processing operation, there is no way how it can affect your production mail server, so don't worry.
-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
22hrs and 20minuts with same status
-
- VP, Product Management
- Posts: 6024
- Liked: 2853 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: transform incremental to rollback possible bottlenecks
Is this a synthetic full with transform or just a normal synthetic full? Is the the first run after the V6 upgrade? How big were your incremental backups prior to the run starting?
-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
Hi,
It's a synthetic full with transform. Yes, it's the first run. The initial run was full and only one incremental. I configured the Job to run Sun-Thursday Incremental. Friday, full Synthetic. The V6 is a new installation not an upgrade, I preferred to run a new setup rather than upgrading the existing V5.
Full Backup Size: 354+GB
Incremental Size: 26+GB
Synthetic .vrb Size: 40+GB
Thanks,
It's a synthetic full with transform. Yes, it's the first run. The initial run was full and only one incremental. I configured the Job to run Sun-Thursday Incremental. Friday, full Synthetic. The V6 is a new installation not an upgrade, I preferred to run a new setup rather than upgrading the existing V5.
Full Backup Size: 354+GB
Incremental Size: 26+GB
Synthetic .vrb Size: 40+GB
Thanks,
-
- VP, Product Management
- Posts: 6024
- Liked: 2853 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: transform incremental to rollback possible bottlenecks
That does sound like a pretty long time for the amount of data. Is there any other activity on the target storage? Are there multiple synthetic jobs running?
-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
hi,
when the job was running there was one job running on at job nother vm 800 GB synthetic as well. that job finished but on this vm still running 99% Transform 50%.
when the job was running there was one job running on at job nother vm 800 GB synthetic as well. that job finished but on this vm still running 99% Transform 50%.
-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
26hrs and 13minuts and the processed rate, read and transferred are same rates. Transform 50% Done, No progress at all.
-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
I'm afraid backup will not be running on it while the RollBack Transform still active
-
- VP, Product Management
- Posts: 6024
- Liked: 2853 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: transform incremental to rollback possible bottlenecks
I would suggest opening a support case and providing the logs.
-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
Hi,
Logged a support case [5176761].
I can't see in the Event Viewer that the VSS has release the Exchange Database as successful backed up. I can see only when the VSS started.
Thanks,
Logged a support case [5176761].
I can't see in the Event Viewer that the VSS has release the Exchange Database as successful backed up. I can see only when the VSS started.
Thanks,
-
- Novice
- Posts: 7
- Liked: never
- Joined: Dec 21, 2009 2:45 am
- Full Name: Tim Lane
- Contact:
Re: transform incremental to rollback possible bottlenecks
Same issue here with v6 on physical win2003 r2 server after upgrade from v5. Vmware vSphere 5, ESXi 5.
-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
hi timl,
I ended up doing the following to kill the job.
1. Exit out of Backup and Replication Console
2. Stop all Veeam Services
3. Open the Task Manager and click on Process Tab
4. Kill all Veeam Services and Veeam Agents
5. Open a VSphere Client
6. Check the Snapshot Manager and the Datastore of the VM in question for any Open SnapShots
7. If there any snapshots, manually create a snap and then hit the "Delete All" button inside the Snapshot Manager
8. Restart all Veeam Services
9. Open Backup and Replication Console.
10. If job is still stuck, then reboot Veeam box
Hope it helps.
Thanks,
I ended up doing the following to kill the job.
1. Exit out of Backup and Replication Console
2. Stop all Veeam Services
3. Open the Task Manager and click on Process Tab
4. Kill all Veeam Services and Veeam Agents
5. Open a VSphere Client
6. Check the Snapshot Manager and the Datastore of the VM in question for any Open SnapShots
7. If there any snapshots, manually create a snap and then hit the "Delete All" button inside the Snapshot Manager
8. Restart all Veeam Services
9. Open Backup and Replication Console.
10. If job is still stuck, then reboot Veeam box
Hope it helps.
Thanks,
-
- Novice
- Posts: 7
- Liked: never
- Joined: Dec 21, 2009 2:45 am
- Full Name: Tim Lane
- Contact:
Re: transform incremental to rollback possible bottlenecks
It is still running, the file is still being wrote to. I am going to wait it out and see where it is in the morning. It did the same with several VMs and they eventually finished.
-
- Veteran
- Posts: 391
- Liked: 32 times
- Joined: Jul 18, 2011 9:30 am
- Full Name: Hussain Al Sayed
- Location: Bahrain
- Contact:
Re: transform incremental to rollback possible bottlenecks
I waited 37hurs and 50 minutes, nothing change then finally I followed the above steps to kill the job and start full backup all over again.
-
- Novice
- Posts: 7
- Liked: never
- Joined: Dec 21, 2009 2:45 am
- Full Name: Tim Lane
- Contact:
Re: transform incremental to rollback possible bottlenecks
Well, it finally finished. It sat on "Transform 85%" around 12 hours. Maybe it is the server, but I don't remember having these issues with version 5.
VMware Backup job: server
Created by at 3/1/2012 4:43:59 PM. Success
1 of 1 VMs processed
Saturday, March 03, 2012 7:15:02 PM
Success 1 Start time 7:15:02 PM Total size 925.0 GB Backup size 61.2 GB
Warning 0 End time 7:12:19 AM + 2 Data read 84.3 GB Dedupe 1.0x
Error 0 Duration 35:57:16 Transferred 61.1 GB Compression 1.3x
Details
Name Status Start time End time Size Read Transferred Duration Details
storage Success 7:16:31 PM 10:11:04 PM 925.0 GB 84.3 GB 61.1 GB 2:54:32
VMware Backup job: server
Created by at 3/1/2012 4:43:59 PM. Success
1 of 1 VMs processed
Saturday, March 03, 2012 7:15:02 PM
Success 1 Start time 7:15:02 PM Total size 925.0 GB Backup size 61.2 GB
Warning 0 End time 7:12:19 AM + 2 Data read 84.3 GB Dedupe 1.0x
Error 0 Duration 35:57:16 Transferred 61.1 GB Compression 1.3x
Details
Name Status Start time End time Size Read Transferred Duration Details
storage Success 7:16:31 PM 10:11:04 PM 925.0 GB 84.3 GB 61.1 GB 2:54:32
-
- Novice
- Posts: 7
- Liked: never
- Joined: Dec 21, 2009 2:45 am
- Full Name: Tim Lane
- Contact:
Re: transform incremental to rollback possible bottlenecks
Okay, I am going to try the patch. I am running v6.0.0.153 and the patch is 6.0.0.181.
-
- Novice
- Posts: 7
- Liked: never
- Joined: Dec 21, 2009 2:45 am
- Full Name: Tim Lane
- Contact:
Re: transform incremental to rollback possible bottlenecks
Okay, I defragged the vm and ran backups, the defrag caused the incremental to be rather large. Now though everything seems to be running great. The last incremental only took 10 minutes.
-
- Novice
- Posts: 7
- Liked: never
- Joined: Dec 21, 2009 2:45 am
- Full Name: Tim Lane
- Contact:
Re: transform incremental to rollback possible bottlenecks
Okay, it is still slow. 43 hours and counting. I am going to try setting up a veeam server at our DR site and use it to backup that server.
-
- Novice
- Posts: 7
- Liked: never
- Joined: Dec 21, 2009 2:45 am
- Full Name: Tim Lane
- Contact:
Re: transform incremental to rollback possible bottlenecks
It finished successfully, but I have 3 huge files in the backup folder now. Two 750GB and one 800GB for that same VM.
-
- Enthusiast
- Posts: 32
- Liked: never
- Joined: Jan 20, 2010 6:59 pm
- Full Name: Ronny
- Contact:
Re: transform incremental to rollback possible bottlenecks
The problem is that this did not happen in v5, so something has changed since v6. I have a support case open for the same issue.Gostev wrote:Transform is a very I/O intensive operation, so it is normal for it to take long time if backup target cannot dish out good IOPS.
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: transform incremental to rollback possible bottlenecks
Any update on this? I'm down to 12 days of a 30 day trial and had my first Saturday Transform this past weekend where the backup job tried to "Transform previous full backup chains into rollbacks" and it took 25 hours and 25 minutes to get that one step done! Normal backup times are 1.5-2hrs for this job (incrementals) and I think about 2.5-3hrs full (first time).
Hmm, I hope I can get my trial extended, 30 days isn't enough so far to test everything under Hyper-V (current environment) and VMWare (what we are hoping to switch to in the next 3-6 months). Too many weird and wonky bugs to work through
P.S. Backup is "Processing" about 800GB of data (Hyper-V VMs). VBK file is around 400GB in size. VRBs are around 35-40GB each.
Hmm, I hope I can get my trial extended, 30 days isn't enough so far to test everything under Hyper-V (current environment) and VMWare (what we are hoping to switch to in the next 3-6 months). Too many weird and wonky bugs to work through
P.S. Backup is "Processing" about 800GB of data (Hyper-V VMs). VBK file is around 400GB in size. VRBs are around 35-40GB each.
-
- Chief Product Officer
- Posts: 31606
- Liked: 7097 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: transform incremental to rollback possible bottlenecks
What is your backup storage? Might be too slow to handle transform...
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: transform incremental to rollback possible bottlenecks
512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). Veeam is on this same server.
I still have Backup Exec on there and its constantly doing junk at about 2MB/sec for the dedupe folders but I'm in the process (as I write this) of removing this from the equation...
It just really seems like a small file (400GB isn't that big...) for it to be taking 25hrs even on this storage.
Bottleneck is always around 1-3% target, 99% source for the backup of the actual VMs.
I still have Backup Exec on there and its constantly doing junk at about 2MB/sec for the dedupe folders but I'm in the process (as I write this) of removing this from the equation...
It just really seems like a small file (400GB isn't that big...) for it to be taking 25hrs even on this storage.
Bottleneck is always around 1-3% target, 99% source for the backup of the actual VMs.
-
- Chief Product Officer
- Posts: 31606
- Liked: 7097 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: transform incremental to rollback possible bottlenecks
That would be perfect setup to investigate the issue deeply. What sort of I/O are you seeing during transform, is it very low? How much RAM does the system has, and all the available RAM occupied by the system cache? Are you using Windows 2008 or later for that server? Because I have just posted an update to old topic about performance degradation issue in the VMware subforum, which might be related.
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: transform incremental to rollback possible bottlenecks
Unfortunately I was not monitoring it during the transform this Saturday (I assumed it would magically work nice and faster without any issues, well, that and I was too tired/lazy to log in from home over the weekend to check things out lol).
Server has 24GB RAM, usually around 10% used. Windows 2008 R2 latest SP1/patches/fixes.
If that other degradation issue is related let me know
Server has 24GB RAM, usually around 10% used. Windows 2008 R2 latest SP1/patches/fixes.
If that other degradation issue is related let me know
-
- Influencer
- Posts: 12
- Liked: never
- Joined: Apr 28, 2011 3:04 pm
- Full Name: Kevin Beaumont
- Contact:
Re: transform incremental to rollback possible bottlenecks
Folks
We have a Veeam job which is 3x VMs, totaling 3tb in size. In VBK land, 1.8tb. Each daily VIB is around 70gb.
All runs fine, except the transforms take a loooong time. 72 hours or so from dailys into a full.
The Veeam backup repo has 16gb of RAM, CPU usage barely gets above 3% during the transforms. Storage is a shared HP MSA. Fast disk, really fast Veeam backups to this disk, slow transforms. Can't work out if it's the sheer size of the data, or if this is unusually slow.
We have a Veeam job which is 3x VMs, totaling 3tb in size. In VBK land, 1.8tb. Each daily VIB is around 70gb.
All runs fine, except the transforms take a loooong time. 72 hours or so from dailys into a full.
The Veeam backup repo has 16gb of RAM, CPU usage barely gets above 3% during the transforms. Storage is a shared HP MSA. Fast disk, really fast Veeam backups to this disk, slow transforms. Can't work out if it's the sheer size of the data, or if this is unusually slow.
Who is online
Users browsing this forum: Google [Bot] and 37 guests