Veeam B&R - Replication jobs sit idle at 99% for 20+ minute

VMware specific discussions

Veeam B&R - Replication jobs sit idle at 99% for 20+ minute

Veeam Logoby itdirector » Wed Feb 22, 2012 3:02 pm

- Veeam B&R 6.0.0.181 (64-bit) on Dell R710 server - Win2k8 R2 64-bit - 32GB - Dual X5660 - Backup Repository on DAS MD1000 - Transfer rate to/from DAS is 500MB/s+

- Source - vSphere 5 server on Dell R710 hardware - 32GB - Dual X5675 - Win2k8 Veeam Proxy VM (4 vCPU - 2GB) running Virtual Appliance transfer mode - 8 Gigabit nics - VM's are on DAS - Transfer rate to/from DAS is 700MB/s+
- Target - vSphere 5 server on Dell R710 hardware - 32GB - Dual X5675 - Win2k8 Veeam Proxy VM (6 vCPU - 4GB) running Virtual Appliance transfer mode - 8 Gigabit nics - Replicated VM's are on DAS - Transfer rate to/from DAS is 600MB/s+


No matter if I am running one replication job at time (1 job = 1 VM replication) or 4 jobs (maximum set), most jobs finish transferring data between 3 minutes & 30 minutes, however, they just sit at 99% for 20 to 50 minutes (with no network traffic nor CPU usage nor disk usage on the source, target, nor Veeam B&R server.

I would like to know if this is normal? What is it doing & why is it stuck at 99% for that long of a period. Why does a job that finish transferring data in 3 minutes sit at 99% for 20+ minutes?

Thank you.

Example below: This job finished transferring data after 3 minutes, yet it just sat at 99% for 18 minutes:

Wednesday, February 22, 2012 9:33:03 AM
Success1Start time9:33:03 AMTotal size100.0 GBBackup size211.8 MB
Warning0End time9:55:01 AMData read3.5 GBDedupe1.0x
Error0Duration0:21:57 Transferred1.8 GBCompression1.0x
Details
NameStatusStart timeEnd timeSizeReadTransferredDurationDetails
FDC2NYCSuccess9:33:37 AM9:54:54 AM100.0 GB3.5 GB1.8 GB0:21:17
itdirector
Enthusiast
 
Posts: 59
Liked: 3 times
Joined: Thu Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby foggy » Wed Feb 22, 2012 3:44 pm

What kind of VMs are you replicating? I'd assume that these are highly transactional ones and depending on how much time the VM runs off snapshot during replication, the snapshot commit time could become considerable.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby Gostev » Wed Feb 22, 2012 4:18 pm

Could it be processing retention policy, doing replica VM snapshot commit?
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby itdirector » Wed Feb 22, 2012 5:57 pm

Foggy,

It doesn't matter which VM's: all the same issue with sitting at 99% for 20+ minutes. I've tried Exchange 2010 servers , & plain vanilla 10GB data server that is not being used & has no transactions.

Gostev,
This is a replica, so not sure where the retension policy would be set. We have all replica jobs set to 1 restore point. I don't think it is doing any snapshot commits, because again, there is no disk/nic/cpu activity during those 20+ minutes on the source/target/Veeam Win2k8 server & we only have 1 restore point.

Thx.
itdirector
Enthusiast
 
Posts: 59
Liked: 3 times
Joined: Thu Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby itdirector » Wed Feb 22, 2012 6:01 pm

& I forgot to mention that when running multiple jobs, at times, when one job is sitting at 99%, the next job won't start hot add until that first job has gone from 99% to 100%. The 2nd job just sits there on "Processing VM". This is obviously a huge issue for us, because we need to backup dozens of VM's.
itdirector
Enthusiast
 
Posts: 59
Liked: 3 times
Joined: Thu Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby foggy » Wed Feb 22, 2012 9:01 pm

Please open a support case, logs should be reviewed to define the reason of such behavior.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby Gostev » Thu Feb 23, 2012 12:18 am

itdirector wrote:I don't think it is doing any snapshot commits, because again, there is no disk/nic/cpu activity during those 20+ minutes on the source/target/Veeam Win2k8 server & we only have 1 restore point.

Well, of course there is no disk/nic/cpu activity during those 20+ minutes on the source/target/Veeam Win2k8 server, because VMware snapshot commits are handled by the actual ESX(i) hosts. Look for vCenter events, and see if there is a corresponding event.

And it does not matter how many restore points you have, you retention policy will be processed after each incremental run to keep the amount of restore points as required.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby itdirector » Thu Feb 23, 2012 2:39 pm

Gostev,

The Veeam & vCenter + both of their databases are installed on the same MS SQL Express 2008. Could there be a freezing issue that is locking up the database, not allowing the next job to run? Has this issue been seen before?
itdirector
Enthusiast
 
Posts: 59
Liked: 3 times
Joined: Thu Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby Gostev » Thu Feb 23, 2012 4:23 pm

No. SQL unavailability would cause the job to timeout much, much sooner than 20 minutes.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby itdirector » Thu Feb 23, 2012 4:27 pm

I think I found the issue : in the logs. Please let me know if there is anything I can try
itdirector
Enthusiast
 
Posts: 59
Liked: 3 times
Joined: Thu Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby Gostev » Thu Feb 23, 2012 4:36 pm

Please do not post the log snippets. All logs should be submitted to and investigated by our technical support, as per forum rules provided when you click New Topic. This is not a support forum, and the team behind this forum is unable to analyze debug logs for you.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby itdirector » Fri Feb 24, 2012 4:28 pm

Gostev. I apologize; however, I have opened up two tickets with support (unrelated issues). The first ticket, it took 4 days for a generic response.

This 2nd ticket 5174859, which I opened up yesterday (I included all of the details of the issue, including specs, logs & my Veeam B&R version number 6.0.0.181), support, instead of reading through the notes, writes a generic email back saying maybe you should try to upgrade to version 6.0.0.181 & gives me a link.

I, along with the majority of people who use Veeam are in the business of providing IT support. These generic responses from Veeam support & lack of effort to read through notes & logs that we upload are disconcerting. I have found that Veeam's forums are the only venue to get real answers to these issues.
itdirector
Enthusiast
 
Posts: 59
Liked: 3 times
Joined: Thu Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby foggy » Fri Feb 24, 2012 4:34 pm

If you don't feel your support cases are handled properly, you should submit your feedback to the support management using their portal at http://vee.am/support (click Talk to a Manager button) and help us improve support service this way. We will not know there is a problem until you tell us.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam B&R - Replication jobs sit idle at 99% for 20+ mi

Veeam Logoby chris2005 » Wed Jun 03, 2015 7:24 pm

SO where is the log? What is the solution?
chris2005
Lurker
 
Posts: 1
Liked: never
Joined: Wed Jun 03, 2015 7:23 pm
Full Name: Chris B


Return to VMware vSphere



Who is online

Users browsing this forum: Google [Bot] and 11 guests