Comprehensive data protection for all workloads
Post Reply
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org » Dec 20, 2012 12:42 am

Support ID: 00168202

I am testing VEEAM Backup and Replication 6.5. I have installed it in our test lab. The lab has two ESXi hosts and 4 VMs. One of the VMs is the VEEAM server. There is NFS storage and only two VMs have files in it. Its flat network environment, only one phyiscal switch, no subnets, so its very simple.

I have created a backup job with one VM but the job status is at 0% for over 2hrs. I look at the job progress detail and everything has a green check mark. The last item on the list is the Hard Disk 1. The process rate 72kb/s. I have deleted this job and recreated another one, with another VM, same results.

Though, I was able to backup the VM which is running the VEEAM without any problem. It finised in less than an hour.

Am I supposed to install VEEAM on each VM? My understanding VEEAM is agent-less.

I am stuck with this... any help?

Vitaliy S.
Product Manager
Posts: 22999
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. » Dec 20, 2012 9:05 am

Hi Chris,

Yes, you're correct in your understanding that Veeam is using agent-less approach to backup virtual machines and there is no need to install Veeam on every machine you're going to back up. The debug logs that you have sent to our support team should clearly indicate the reason for such slow processing rate. Btw, what backup mode are you using? HotAdd? Network?

Thanks!

christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org » Dec 20, 2012 4:52 pm

Hi Vitaliy,

I left the backup running, I came in this morning it is at 20% and it has been running for 17hrs. This isn't right.... I am using a network. The NSF is plugged to the same switch with the two hosts. There isn't much traffic, its a small test lab. What is HotAdd? How come the VEEAM server backup in less than an hour?

Thanks,

Vitaliy S.
Product Manager
Posts: 22999
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. » Dec 20, 2012 5:01 pm

christiank@pcs.org wrote:it is at 20% and it has been running for 17hrs. This isn't right....
I agree that this is an awful job performance. Have you heard back from you support engineer? What does the bottleneck statistics window say about this job?
christiank@pcs.org wrote: What is HotAdd? How come the VEEAM server backup in less than an hour?
That's a good question! Can you please ask your engineer to compare job jog sessions from both runs?

Please see our sticky F.A.Q. for further details on the available processing modes: VMware : [FAQ] FREQUENTLY ASKED QUESTIONS

christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org » Dec 20, 2012 5:21 pm

No, I haven't heard from support engineer. The bottleneck status is source. I killed the job and configure proxy with virtual appliance then changed the job settings. I restarted the job its been running for over 10min now still at 0%.

Is there something wrong with my configuration?

Vitaliy S.
Product Manager
Posts: 22999
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. » Dec 20, 2012 6:09 pm

Your Veeam backup configuration looks fine to me, I guess there might be something wrong with the connection to vSphere environment.

Are you using vCenter Server connection to backup your VMs? Can you please try to add standalone hosts via IP address and configure a test backup job to see what the job performance will be?

christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org » Dec 20, 2012 7:48 pm

Yes, I am vCenter. I deleted the vCenter from the backup infrustucture, added one host (it has 2 VMs) I am using iSCSi target to the NFS storage. I added a proxy server (one of the VMs which has a connection to the NFS) I created a new backup job for one VM.

The job is running now at 0% bottleneck status is target.

I am skeptical about this product. Will it be able to run in our production environment of over 50 VMs? I am beating my head just to get it to backup 4 VMs.

Unless my configuration is missing something.

Vitaliy S.
Product Manager
Posts: 22999
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. » Dec 20, 2012 7:55 pm

Ok, let's wait what our support engineer figures out after reviewing your job debug log files.

christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org » Dec 20, 2012 8:21 pm

OK thanks, what is the expected time for a VM with 40GB to be backed up? I got a good feeling this can be a good solution my environment because I backed up one VM (the VEEAM server) for close to an hour, then Instant recover was very quick.

I just hope the engineers will shine some light on this....

tsightler
VP, Product Management
Posts: 5421
Liked: 2243 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by tsightler » Dec 20, 2012 8:32 pm

I rarely see speeds less than about 40-50MB/sec, even for small environments, so I'd say typical would be about 12-15 minutes for a 40GB VM. Perhaps much faster if your hardware can support it. What you're seeing is way, way below anything that would be "normal" so there must be something wrong.

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Yuki » Dec 20, 2012 11:24 pm

Hi Chris,
I've replied to your question in my thread http://forums.veeam.com/viewtopic.php?f ... 135#p69135

The issue we are seeing is that there is a corrupt OIB file from a job that was stuck and then canceled. You will need to do some cleanup to fix the problem (remove old backups, run Active full backup, import backups).

christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org » Dec 20, 2012 11:34 pm

Hi Yuki,

Please read my postings. I have deleted the jobs several times, tried to back up different VMs with different jobs, same results. I ran one job at a time. All VMs are connected to the NFS via Windows iSCSi.

What else should I do?

tsightler
VP, Product Management
Posts: 5421
Liked: 2243 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by tsightler » Dec 21, 2012 1:52 am

christiank@pcs.org wrote:All VMs are connected to the NFS via Windows iSCSi.
You've made this statement twice, but it confuses me as NFS and iSCSI are two different protocols. You can't connect to NFS via iSCSI as NFS is a file sharing protocol, and iSCSI is a block based protocol. Are you perhaps confusing some terms? Do you mean that you are connecting to the NAS via iSCSI? Also, which VMs are connected to it?

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Yuki » Dec 21, 2012 9:08 am

tsightler wrote: You've made this statement twice, but it confuses me as NFS and iSCSI are two different protocols. You can't connect to NFS via iSCSI as NFS is a file sharing protocol, and iSCSI is a block based protocol. Are you perhaps confusing some terms? Do you mean that you are connecting to the NAS via iSCSI? Also, which VMs are connected to it?
1) What tsightler says is correct - either one or the other. In my personal experience SMB/CIFS is faster than iSCSI on inexpensive setups. If you have iSCSI SAN - than it is a different story.

2) As i said - for us the fix was to 1- kill the job, 2- remove backups from the backup job in Veeam console, 3 - run FULL Active backup. If you attempt to rerun your backups without clearing old backup information from veeam and detaching those backup files - you will have the same problem of backups getting stuck.

christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org » Dec 24, 2012 4:29 am

Hi tsightler,

Sorry for the confusion. I meant to say NAS via iSCSi. All VMs are connected to it via iSCSi connector. I can backup the VEEAM server (one of the VMs that has VEEAM installed) without a problem. It takes less than and hour to complete.

I have deleted the backup jobs many times during my test. When the job runs for about 3hrs and the process status is 0% I delete it and then navigated in the NAS storaged to delete the files already created then start all over again.

What could cause one VM, with VEEAM install, to backup and others not? I have done lots of research on this I don't see many blogs about this version of VEEAM. Couldn't be a bug or because I am using the trial version?

Vitaliy S.
Product Manager
Posts: 22999
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. » Dec 24, 2012 7:34 am

Hi Chris, this behavior has nothing to do with the license version you're using. As far as I can see from your ticket status our engineer is still awaiting the debug log files from your backup jobs. Can you please upload them as it is hard to troubleshoot technical issues over the forums?

christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org » Dec 25, 2012 3:28 pm

Vitaliy,

I did upload the log files before starting this thread. There has been many test jobs and they all have the same problem. There was only one job passed the 0% and went up to 20% after 17hrs. I will created another job, ran it, then grub the log. How long should this job ran?

foggy
Veeam Software
Posts: 18278
Liked: 1564 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by foggy » Dec 25, 2012 3:37 pm

I think 5-10 minutes is enough to grab the logs with information required.

christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org » Dec 25, 2012 4:46 pm

OK. So upload the files to support engineers?

Vitaliy S.
Product Manager
Posts: 22999
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. » Dec 25, 2012 5:28 pm

Please do.

Aremac
Enthusiast
Posts: 58
Liked: 3 times
Joined: Sep 30, 2011 11:59 pm
Full Name: Sam Pierce
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Aremac » Jan 02, 2013 8:24 pm

Question about your storage. You said "I meant to say NAS via iSCSi. All VMs are connected to it via iSCSi connector" Do you mean that all your VM's use iSCSI based datastores (VMFS or RDM) for their disks and they are shared by the guest OS to provide NAS features (NFS / CIFS). Or, do you mean that your VM's use the Microsoft iSCSI initiator within the OS to connect to iSCSI LUNs. The latter would be a problem as you cannot backup a VM that has an iSCSI LUN attached via the Microsoft iSCSI initiator.

christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org » Jan 02, 2013 11:12 pm

I blow off the entire lab, started from scratch. Now my trial license has expired. I went to manage licenses and don't have the option for another trial.

Any ideas?

Gostev
SVP, Product Management
Posts: 24803
Liked: 3563 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Gostev » Jan 02, 2013 11:22 pm

Click the "Trial Key Expired" link in the License key section of the product download area, and it will open the form to contact sales for an extended trial license key.

Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Bing [Bot] and 19 guests