Veeam B&R - Replication jobs sit idle at 99% for 20+ minute

[itdirector]

- Veeam B&R 6.0.0.181 (64-bit) on Dell R710 server - Win2k8 R2 64-bit - 32GB - Dual X5660 - Backup Repository on DAS MD1000 - Transfer rate to/from DAS is 500MB/s+

- Source - vSphere 5 server on Dell R710 hardware - 32GB - Dual X5675 - Win2k8 Veeam Proxy VM (4 vCPU - 2GB) running Virtual Appliance transfer mode - 8 Gigabit nics - VM's are on DAS - Transfer rate to/from DAS is 700MB/s+
- Target - vSphere 5 server on Dell R710 hardware - 32GB - Dual X5675 - Win2k8 Veeam Proxy VM (6 vCPU - 4GB) running Virtual Appliance transfer mode - 8 Gigabit nics - Replicated VM's are on DAS - Transfer rate to/from DAS is 600MB/s+


No matter if I am running one replication job at time (1 job = 1 VM replication) or 4 jobs (maximum set), most jobs finish transferring data between 3 minutes & 30 minutes, however, they just sit at 99% for 20 to 50 minutes (with no network traffic nor CPU usage nor disk usage on the source, target, nor Veeam B&R server.

I would like to know if this is normal? What is it doing & why is it stuck at 99% for that long of a period. Why does a job that finish transferring data in 3 minutes sit at 99% for 20+ minutes?

Thank you.

Example below: This job finished transferring data after 3 minutes, yet it just sat at 99% for 18 minutes:

Wednesday, February 22, 2012 9:33:03 AM
Success 1 Start time 9:33:03 AM Total size 100.0 GB Backup size 211.8 MB
Warning 0 End time 9:55:01 AM Data read 3.5 GB Dedupe 1.0x
Error 0 Duration 0:21:57 Transferred 1.8 GB Compression 1.0x
Details
Name Status Start time End time Size Read Transferred Duration Details
FDC2NYC Success 9:33:37 AM 9:54:54 AM 100.0 GB 3.5 GB 1.8 GB 0:21:17

[foggy]

What kind of VMs are you replicating? I'd assume that these are highly transactional ones and depending on how much time the VM runs off snapshot during replication, the snapshot commit time could become considerable.

[Gostev]

Could it be processing retention policy, doing replica VM snapshot commit?

[itdirector]

Foggy,

It doesn't matter which VM's: all the same issue with sitting at 99% for 20+ minutes. I've tried Exchange 2010 servers , & plain vanilla 10GB data server that is not being used & has no transactions.

Gostev,
This is a replica, so not sure where the retension policy would be set. We have all replica jobs set to 1 restore point. I don't think it is doing any snapshot commits, because again, there is no disk/nic/cpu activity during those 20+ minutes on the source/target/Veeam Win2k8 server & we only have 1 restore point.

Thx.

[itdirector]

& I forgot to mention that when running multiple jobs, at times, when one job is sitting at 99%, the next job won't start hot add until that first job has gone from 99% to 100%. The 2nd job just sits there on "Processing VM". This is obviously a huge issue for us, because we need to backup dozens of VM's.

[foggy]

Please open a support case, logs should be reviewed to define the reason of such behavior.

[Gostev]

I don't think it is doing any snapshot commits, because again, there is no disk/nic/cpu activity during those 20+ minutes on the source/target/Veeam Win2k8 server & we only have 1 restore point.

Well, of course there is no disk/nic/cpu activity during those 20+ minutes on the source/target/Veeam Win2k8 server, because VMware snapshot commits are handled by the actual ESX(i) hosts. Look for vCenter events, and see if there is a corresponding event.

And it does not matter how many restore points you have, you retention policy will be processed after each incremental run to keep the amount of restore points as required.

[itdirector]

Gostev,

The Veeam & vCenter + both of their databases are installed on the same MS SQL Express 2008. Could there be a freezing issue that is locking up the database, not allowing the next job to run? Has this issue been seen before?

[Gostev]

No. SQL unavailability would cause the job to timeout much, much sooner than 20 minutes.

[itdirector]

I think I found the issue : in the logs. Please let me know if there is anything I can try

[Gostev]

Please do not post the log snippets. All logs should be submitted to and investigated by our technical support, as per forum rules provided when you click New Topic. This is not a support forum, and the team behind this forum is unable to analyze debug logs for you.

[itdirector]

Gostev. I apologize; however, I have opened up two tickets with support (unrelated issues). The first ticket, it took 4 days for a generic response.

This 2nd ticket 5174859, which I opened up yesterday (I included all of the details of the issue, including specs, logs & my Veeam B&R version number 6.0.0.181), support, instead of reading through the notes, writes a generic email back saying maybe you should try to upgrade to version 6.0.0.181 & gives me a link.

I, along with the majority of people who use Veeam are in the business of providing IT support. These generic responses from Veeam support & lack of effort to read through notes & logs that we upload are disconcerting. I have found that Veeam's forums are the only venue to get real answers to these issues.

[foggy]

If you don't feel your support cases are handled properly, you should submit your feedback to the support management using their portal at http://vee.am/support (click Talk to a Manager button) and help us improve support service this way. We will not know there is a problem until you tell us.

[chris2005]

SO where is the log? What is the solution?