Host-based backup of VMware vSphere VMs.
Post Reply
itdirector
Enthusiast
Posts: 59
Liked: 3 times
Joined: Jan 19, 2012 8:53 pm
Full Name: friedman

Veeam B&R - Replication jobs sit idle at 99% for 20+ minute

Post by itdirector »

- Veeam B&R 6.0.0.181 (64-bit) on Dell R710 server - Win2k8 R2 64-bit - 32GB - Dual X5660 - Backup Repository on DAS MD1000 - Transfer rate to/from DAS is 500MB/s+

- Source - vSphere 5 server on Dell R710 hardware - 32GB - Dual X5675 - Win2k8 Veeam Proxy VM (4 vCPU - 2GB) running Virtual Appliance transfer mode - 8 Gigabit nics - VM's are on DAS - Transfer rate to/from DAS is 700MB/s+
- Target - vSphere 5 server on Dell R710 hardware - 32GB - Dual X5675 - Win2k8 Veeam Proxy VM (6 vCPU - 4GB) running Virtual Appliance transfer mode - 8 Gigabit nics - Replicated VM's are on DAS - Transfer rate to/from DAS is 600MB/s+


No matter if I am running one replication job at time (1 job = 1 VM replication) or 4 jobs (maximum set), most jobs finish transferring data between 3 minutes & 30 minutes, however, they just sit at 99% for 20 to 50 minutes (with no network traffic nor CPU usage nor disk usage on the source, target, nor Veeam B&R server.

I would like to know if this is normal? What is it doing & why is it stuck at 99% for that long of a period. Why does a job that finish transferring data in 3 minutes sit at 99% for 20+ minutes?

Thank you.

Example below: This job finished transferring data after 3 minutes, yet it just sat at 99% for 18 minutes:

Wednesday, February 22, 2012 9:33:03 AM
Success 1 Start time 9:33:03 AM Total size 100.0 GB Backup size 211.8 MB
Warning 0 End time 9:55:01 AM Data read 3.5 GB Dedupe 1.0x
Error 0 Duration 0:21:57 Transferred 1.8 GB Compression 1.0x
Details
Name Status Start time End time Size Read Transferred Duration Details
FDC2NYC Success 9:33:37 AM 9:54:54 AM 100.0 GB 3.5 GB 1.8 GB 0:21:17
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by foggy »

What kind of VMs are you replicating? I'd assume that these are highly transactional ones and depending on how much time the VM runs off snapshot during replication, the snapshot commit time could become considerable.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by Gostev »

Could it be processing retention policy, doing replica VM snapshot commit?
itdirector
Enthusiast
Posts: 59
Liked: 3 times
Joined: Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by itdirector »

Foggy,

It doesn't matter which VM's: all the same issue with sitting at 99% for 20+ minutes. I've tried Exchange 2010 servers , & plain vanilla 10GB data server that is not being used & has no transactions.

Gostev,
This is a replica, so not sure where the retension policy would be set. We have all replica jobs set to 1 restore point. I don't think it is doing any snapshot commits, because again, there is no disk/nic/cpu activity during those 20+ minutes on the source/target/Veeam Win2k8 server & we only have 1 restore point.

Thx.
itdirector
Enthusiast
Posts: 59
Liked: 3 times
Joined: Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by itdirector »

& I forgot to mention that when running multiple jobs, at times, when one job is sitting at 99%, the next job won't start hot add until that first job has gone from 99% to 100%. The 2nd job just sits there on "Processing VM". This is obviously a huge issue for us, because we need to backup dozens of VM's.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by foggy »

Please open a support case, logs should be reviewed to define the reason of such behavior.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by Gostev »

itdirector wrote:I don't think it is doing any snapshot commits, because again, there is no disk/nic/cpu activity during those 20+ minutes on the source/target/Veeam Win2k8 server & we only have 1 restore point.
Well, of course there is no disk/nic/cpu activity during those 20+ minutes on the source/target/Veeam Win2k8 server, because VMware snapshot commits are handled by the actual ESX(i) hosts. Look for vCenter events, and see if there is a corresponding event.

And it does not matter how many restore points you have, you retention policy will be processed after each incremental run to keep the amount of restore points as required.
itdirector
Enthusiast
Posts: 59
Liked: 3 times
Joined: Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by itdirector »

Gostev,

The Veeam & vCenter + both of their databases are installed on the same MS SQL Express 2008. Could there be a freezing issue that is locking up the database, not allowing the next job to run? Has this issue been seen before?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by Gostev »

No. SQL unavailability would cause the job to timeout much, much sooner than 20 minutes.
itdirector
Enthusiast
Posts: 59
Liked: 3 times
Joined: Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by itdirector »

I think I found the issue : in the logs. Please let me know if there is anything I can try
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by Gostev »

Please do not post the log snippets. All logs should be submitted to and investigated by our technical support, as per forum rules provided when you click New Topic. This is not a support forum, and the team behind this forum is unable to analyze debug logs for you.
itdirector
Enthusiast
Posts: 59
Liked: 3 times
Joined: Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by itdirector »

Gostev. I apologize; however, I have opened up two tickets with support (unrelated issues). The first ticket, it took 4 days for a generic response.

This 2nd ticket 5174859, which I opened up yesterday (I included all of the details of the issue, including specs, logs & my Veeam B&R version number 6.0.0.181), support, instead of reading through the notes, writes a generic email back saying maybe you should try to upgrade to version 6.0.0.181 & gives me a link.

I, along with the majority of people who use Veeam are in the business of providing IT support. These generic responses from Veeam support & lack of effort to read through notes & logs that we upload are disconcerting. I have found that Veeam's forums are the only venue to get real answers to these issues.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by foggy »

If you don't feel your support cases are handled properly, you should submit your feedback to the support management using their portal at http://vee.am/support (click Talk to a Manager button) and help us improve support service this way. We will not know there is a problem until you tell us.
chris2005
Lurker
Posts: 1
Liked: never
Joined: Jun 03, 2015 7:23 pm
Full Name: Chris B
Contact:

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Post by chris2005 »

SO where is the log? What is the solution?
Post Reply

Who is online

Users browsing this forum: Google [Bot], Ivan239 and 98 guests