Comprehensive data protection for all workloads
habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 03, 2012 5:44 am

Hi, will it reach 12 hours for just 1.+ TB of data?

Look at the screen shot. Job started last night at 8:00pm still didn't finish :)

http://www.freeimagehosting.net/jrivf

Any help or work around?

tsightler
VP, Product Management
Posts: 5383
Liked: 2217 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by tsightler » Mar 03, 2012 5:49 am

It might, what is the random R/W I/O performance of your target storage? Synthetic fulls will take longer that actual fulls in most environments. How long does a real full backup of that data take?

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 03, 2012 8:09 am

Hi, R/W around 100mb/ W - 75mb R. The full initial back it took around 5 hours + of 1.4 TB data of of it approximately 600GB used.

Thanks,
Hussain

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 03, 2012 11:15 am

Completed 17hrs and 35 minutes still Transform 50% Done :( Anybody experience this issue before? I'm afraid that it will corrupt the Mail Server.

Thanks,

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by Gostev » Mar 03, 2012 11:16 am

Transform is a post-processing operation, there is no way how it can affect your production mail server, so don't worry.

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 03, 2012 3:29 pm

22hrs and 20minuts with same status :(

tsightler
VP, Product Management
Posts: 5383
Liked: 2217 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by tsightler » Mar 03, 2012 4:30 pm

Is this a synthetic full with transform or just a normal synthetic full? Is the the first run after the V6 upgrade? How big were your incremental backups prior to the run starting?

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 03, 2012 5:29 pm

Hi,
It's a synthetic full with transform. Yes, it's the first run. The initial run was full and only one incremental. I configured the Job to run Sun-Thursday Incremental. Friday, full Synthetic. The V6 is a new installation not an upgrade, I preferred to run a new setup rather than upgrading the existing V5.

Full Backup Size: 354+GB
Incremental Size: 26+GB
Synthetic .vrb Size: 40+GB

Thanks,

tsightler
VP, Product Management
Posts: 5383
Liked: 2217 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by tsightler » Mar 03, 2012 5:56 pm

That does sound like a pretty long time for the amount of data. Is there any other activity on the target storage? Are there multiple synthetic jobs running?

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 03, 2012 6:36 pm

hi,
when the job was running there was one job running on at job nother vm 800 GB synthetic as well. that job finished but on this vm still running 99% Transform 50%.

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 03, 2012 7:18 pm

26hrs and 13minuts and the processed rate, read and transferred are same rates. Transform 50% Done, No progress at all.

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 03, 2012 8:26 pm

I'm afraid backup will not be running on it while the RollBack Transform still active :(

tsightler
VP, Product Management
Posts: 5383
Liked: 2217 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by tsightler » Mar 03, 2012 8:28 pm

I would suggest opening a support case and providing the logs.

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 04, 2012 5:27 am

Hi,
Logged a support case [5176761].

I can't see in the Event Viewer that the VSS has release the Exchange Database as successful backed up. I can see only when the VSS started.

Thanks,

timl
Novice
Posts: 7
Liked: never
Joined: Dec 21, 2009 2:45 am
Full Name: Tim Lane
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by timl » Mar 05, 2012 3:42 am

Same issue here with v6 on physical win2003 r2 server after upgrade from v5. Vmware vSphere 5, ESXi 5.

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 05, 2012 5:03 am

hi timl,

I ended up doing the following to kill the job.

1. Exit out of Backup and Replication Console
2. Stop all Veeam Services
3. Open the Task Manager and click on Process Tab
4. Kill all Veeam Services and Veeam Agents
5. Open a VSphere Client
6. Check the Snapshot Manager and the Datastore of the VM in question for any Open SnapShots
7. If there any snapshots, manually create a snap and then hit the "Delete All" button inside the Snapshot Manager
8. Restart all Veeam Services
9. Open Backup and Replication Console.
10. If job is still stuck, then reboot Veeam box

Hope it helps.

Thanks,

timl
Novice
Posts: 7
Liked: never
Joined: Dec 21, 2009 2:45 am
Full Name: Tim Lane
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by timl » Mar 05, 2012 6:37 am

It is still running, the file is still being wrote to. I am going to wait it out and see where it is in the morning. It did the same with several VMs and they eventually finished.

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by habibalby » Mar 05, 2012 6:48 am

I waited 37hurs and 50 minutes, nothing change then finally I followed the above steps to kill the job and start full backup all over again.

timl
Novice
Posts: 7
Liked: never
Joined: Dec 21, 2009 2:45 am
Full Name: Tim Lane
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by timl » Mar 05, 2012 2:16 pm

Well, it finally finished. It sat on "Transform 85%" around 12 hours. Maybe it is the server, but I don't remember having these issues with version 5.

VMware Backup job: server
Created by at 3/1/2012 4:43:59 PM. Success
1 of 1 VMs processed
Saturday, March 03, 2012 7:15:02 PM
Success 1 Start time 7:15:02 PM Total size 925.0 GB Backup size 61.2 GB
Warning 0 End time 7:12:19 AM + 2 Data read 84.3 GB Dedupe 1.0x
Error 0 Duration 35:57:16 Transferred 61.1 GB Compression 1.3x
Details
Name Status Start time End time Size Read Transferred Duration Details
storage Success 7:16:31 PM 10:11:04 PM 925.0 GB 84.3 GB 61.1 GB 2:54:32

timl
Novice
Posts: 7
Liked: never
Joined: Dec 21, 2009 2:45 am
Full Name: Tim Lane
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by timl » Mar 05, 2012 2:27 pm

Okay, I am going to try the patch. I am running v6.0.0.153 and the patch is 6.0.0.181.

timl
Novice
Posts: 7
Liked: never
Joined: Dec 21, 2009 2:45 am
Full Name: Tim Lane
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by timl » Mar 09, 2012 12:51 am

Okay, I defragged the vm and ran backups, the defrag caused the incremental to be rather large. Now though everything seems to be running great. The last incremental only took 10 minutes.

timl
Novice
Posts: 7
Liked: never
Joined: Dec 21, 2009 2:45 am
Full Name: Tim Lane
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by timl » Mar 12, 2012 8:45 pm

Okay, it is still slow. 43 hours and counting. I am going to try setting up a veeam server at our DR site and use it to backup that server.

timl
Novice
Posts: 7
Liked: never
Joined: Dec 21, 2009 2:45 am
Full Name: Tim Lane
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by timl » Mar 13, 2012 1:05 pm

It finished successfully, but I have 3 huge files in the backup folder now. Two 750GB and one 800GB for that same VM.

htwnrva
Enthusiast
Posts: 32
Liked: never
Joined: Jan 20, 2010 6:59 pm
Full Name: Ronny
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by htwnrva » Mar 13, 2012 1:16 pm

Gostev wrote:Transform is a very I/O intensive operation, so it is normal for it to take long time if backup target cannot dish out good IOPS.
The problem is that this did not happen in v5, so something has changed since v6. I have a support case open for the same issue.

trafsta
Enthusiast
Posts: 43
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by trafsta » Apr 30, 2012 5:55 pm

Any update on this? I'm down to 12 days of a 30 day trial and had my first Saturday Transform this past weekend where the backup job tried to "Transform previous full backup chains into rollbacks" and it took 25 hours and 25 minutes to get that one step done! Normal backup times are 1.5-2hrs for this job (incrementals) and I think about 2.5-3hrs full (first time).

Hmm, I hope I can get my trial extended, 30 days isn't enough so far to test everything under Hyper-V (current environment) and VMWare (what we are hoping to switch to in the next 3-6 months). Too many weird and wonky bugs to work through :(

P.S. Backup is "Processing" about 800GB of data (Hyper-V VMs). VBK file is around 400GB in size. VRBs are around 35-40GB each.

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by Gostev » Apr 30, 2012 6:16 pm

What is your backup storage? Might be too slow to handle transform...

trafsta
Enthusiast
Posts: 43
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by trafsta » Apr 30, 2012 6:38 pm

512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). Veeam is on this same server.

I still have Backup Exec on there and its constantly doing junk at about 2MB/sec for the dedupe folders but I'm in the process (as I write this) of removing this from the equation...

It just really seems like a small file (400GB isn't that big...) for it to be taking 25hrs even on this storage.

Bottleneck is always around 1-3% target, 99% source for the backup of the actual VMs.

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by Gostev » Apr 30, 2012 6:45 pm

That would be perfect setup to investigate the issue deeply. What sort of I/O are you seeing during transform, is it very low? How much RAM does the system has, and all the available RAM occupied by the system cache? Are you using Windows 2008 or later for that server? Because I have just posted an update to old topic about performance degradation issue in the VMware subforum, which might be related.

trafsta
Enthusiast
Posts: 43
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by trafsta » Apr 30, 2012 6:52 pm

Unfortunately I was not monitoring it during the transform this Saturday (I assumed it would magically work nice and faster without any issues, well, that and I was too tired/lazy to log in from home over the weekend to check things out lol).

Server has 24GB RAM, usually around 10% used. Windows 2008 R2 latest SP1/patches/fixes.

If that other degradation issue is related let me know :)

KevinBeaumont
Influencer
Posts: 12
Liked: never
Joined: Apr 28, 2011 3:04 pm
Full Name: Kevin Beaumont
Contact:

Re: transform incremental to rollback possible bottlenecks

Post by KevinBeaumont » May 01, 2012 3:55 pm

Folks

We have a Veeam job which is 3x VMs, totaling 3tb in size. In VBK land, 1.8tb. Each daily VIB is around 70gb.

All runs fine, except the transforms take a loooong time. 72 hours or so from dailys into a full.

The Veeam backup repo has 16gb of RAM, CPU usage barely gets above 3% during the transforms. Storage is a shared HP MSA. Fast disk, really fast Veeam backups to this disk, slow transforms. Can't work out if it's the sheer size of the data, or if this is unusually slow.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 9 guests