Incremental to rollback transform: possible bottlenecks?

[habibalby]

Hi, will it reach 12 hours for just 1.+ TB of data?

Look at the screen shot. Job started last night at 8:00pm still didn't finish

http://www.freeimagehosting.net/jrivf

Any help or work around?

[tsightler]

It might, what is the random R/W I/O performance of your target storage? Synthetic fulls will take longer that actual fulls in most environments. How long does a real full backup of that data take?

[habibalby]

Hi, R/W around 100mb/ W - 75mb R. The full initial back it took around 5 hours + of 1.4 TB data of of it approximately 600GB used.

Thanks,
Hussain

[habibalby]

Completed 17hrs and 35 minutes still Transform 50% Done Anybody experience this issue before? I'm afraid that it will corrupt the Mail Server.

Thanks,

[Gostev]

Transform is a post-processing operation, there is no way how it can affect your production mail server, so don't worry.

[habibalby]

22hrs and 20minuts with same status

[tsightler]

Is this a synthetic full with transform or just a normal synthetic full? Is the the first run after the V6 upgrade? How big were your incremental backups prior to the run starting?

[habibalby]

Hi,
It's a synthetic full with transform. Yes, it's the first run. The initial run was full and only one incremental. I configured the Job to run Sun-Thursday Incremental. Friday, full Synthetic. The V6 is a new installation not an upgrade, I preferred to run a new setup rather than upgrading the existing V5.

Full Backup Size: 354+GB
Incremental Size: 26+GB
Synthetic .vrb Size: 40+GB

Thanks,

[tsightler]

That does sound like a pretty long time for the amount of data. Is there any other activity on the target storage? Are there multiple synthetic jobs running?

[habibalby]

hi,
when the job was running there was one job running on at job nother vm 800 GB synthetic as well. that job finished but on this vm still running 99% Transform 50%.

[habibalby]

26hrs and 13minuts and the processed rate, read and transferred are same rates. Transform 50% Done, No progress at all.

[habibalby]

I'm afraid backup will not be running on it while the RollBack Transform still active

[tsightler]

I would suggest opening a support case and providing the logs.

[habibalby]

Hi,
Logged a support case [5176761].

I can't see in the Event Viewer that the VSS has release the Exchange Database as successful backed up. I can see only when the VSS started.

Thanks,

[timl]

Same issue here with v6 on physical win2003 r2 server after upgrade from v5. Vmware vSphere 5, ESXi 5.

[habibalby]

hi timl,

I ended up doing the following to kill the job.

1. Exit out of Backup and Replication Console
2. Stop all Veeam Services
3. Open the Task Manager and click on Process Tab
4. Kill all Veeam Services and Veeam Agents
5. Open a VSphere Client
6. Check the Snapshot Manager and the Datastore of the VM in question for any Open SnapShots
7. If there any snapshots, manually create a snap and then hit the "Delete All" button inside the Snapshot Manager
8. Restart all Veeam Services
9. Open Backup and Replication Console.
10. If job is still stuck, then reboot Veeam box

Hope it helps.

Thanks,

[timl]

It is still running, the file is still being wrote to. I am going to wait it out and see where it is in the morning. It did the same with several VMs and they eventually finished.

[habibalby]

I waited 37hurs and 50 minutes, nothing change then finally I followed the above steps to kill the job and start full backup all over again.

[timl]

Well, it finally finished. It sat on "Transform 85%" around 12 hours. Maybe it is the server, but I don't remember having these issues with version 5.

VMware Backup job: server
Created by at 3/1/2012 4:43:59 PM. Success
1 of 1 VMs processed
Saturday, March 03, 2012 7:15:02 PM
Success 1 Start time 7:15:02 PM Total size 925.0 GB Backup size 61.2 GB
Warning 0 End time 7:12:19 AM + 2 Data read 84.3 GB Dedupe 1.0x
Error 0 Duration 35:57:16 Transferred 61.1 GB Compression 1.3x
Details
Name Status Start time End time Size Read Transferred Duration Details
storage Success 7:16:31 PM 10:11:04 PM 925.0 GB 84.3 GB 61.1 GB 2:54:32

[timl]

Okay, I am going to try the patch. I am running v6.0.0.153 and the patch is 6.0.0.181.

[timl]

Okay, I defragged the vm and ran backups, the defrag caused the incremental to be rather large. Now though everything seems to be running great. The last incremental only took 10 minutes.

[timl]

Okay, it is still slow. 43 hours and counting. I am going to try setting up a veeam server at our DR site and use it to backup that server.

[timl]

It finished successfully, but I have 3 huge files in the backup folder now. Two 750GB and one 800GB for that same VM.

[htwnrva]

Transform is a very I/O intensive operation, so it is normal for it to take long time if backup target cannot dish out good IOPS.

The problem is that this did not happen in v5, so something has changed since v6. I have a support case open for the same issue.

[trafsta]

Any update on this? I'm down to 12 days of a 30 day trial and had my first Saturday Transform this past weekend where the backup job tried to "Transform previous full backup chains into rollbacks" and it took 25 hours and 25 minutes to get that one step done! Normal backup times are 1.5-2hrs for this job (incrementals) and I think about 2.5-3hrs full (first time).

Hmm, I hope I can get my trial extended, 30 days isn't enough so far to test everything under Hyper-V (current environment) and VMWare (what we are hoping to switch to in the next 3-6 months). Too many weird and wonky bugs to work through

P.S. Backup is "Processing" about 800GB of data (Hyper-V VMs). VBK file is around 400GB in size. VRBs are around 35-40GB each.

[Gostev]

What is your backup storage? Might be too slow to handle transform...

[trafsta]

512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). Veeam is on this same server.

I still have Backup Exec on there and its constantly doing junk at about 2MB/sec for the dedupe folders but I'm in the process (as I write this) of removing this from the equation...

It just really seems like a small file (400GB isn't that big...) for it to be taking 25hrs even on this storage.

Bottleneck is always around 1-3% target, 99% source for the backup of the actual VMs.

[Gostev]

That would be perfect setup to investigate the issue deeply. What sort of I/O are you seeing during transform, is it very low? How much RAM does the system has, and all the available RAM occupied by the system cache? Are you using Windows 2008 or later for that server? Because I have just posted an update to old topic about performance degradation issue in the VMware subforum, which might be related.

[trafsta]

Unfortunately I was not monitoring it during the transform this Saturday (I assumed it would magically work nice and faster without any issues, well, that and I was too tired/lazy to log in from home over the weekend to check things out lol).

Server has 24GB RAM, usually around 10% used. Windows 2008 R2 latest SP1/patches/fixes.

If that other degradation issue is related let me know

[KevinBeaumont]

Folks

We have a Veeam job which is 3x VMs, totaling 3tb in size. In VBK land, 1.8tb. Each daily VIB is around 70gb.

All runs fine, except the transforms take a loooong time. 72 hours or so from dailys into a full.

The Veeam backup repo has 16gb of RAM, CPU usage barely gets above 3% during the transforms. Storage is a shared HP MSA. Fast disk, really fast Veeam backups to this disk, slow transforms. Can't work out if it's the sheer size of the data, or if this is unusually slow.