Comprehensive data protection for all workloads
Post Reply
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org »

Support ID: 00168202

I am testing VEEAM Backup and Replication 6.5. I have installed it in our test lab. The lab has two ESXi hosts and 4 VMs. One of the VMs is the VEEAM server. There is NFS storage and only two VMs have files in it. Its flat network environment, only one phyiscal switch, no subnets, so its very simple.

I have created a backup job with one VM but the job status is at 0% for over 2hrs. I look at the job progress detail and everything has a green check mark. The last item on the list is the Hard Disk 1. The process rate 72kb/s. I have deleted this job and recreated another one, with another VM, same results.

Though, I was able to backup the VM which is running the VEEAM without any problem. It finised in less than an hour.

Am I supposed to install VEEAM on each VM? My understanding VEEAM is agent-less.

I am stuck with this... any help?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. »

Hi Chris,

Yes, you're correct in your understanding that Veeam is using agent-less approach to backup virtual machines and there is no need to install Veeam on every machine you're going to back up. The debug logs that you have sent to our support team should clearly indicate the reason for such slow processing rate. Btw, what backup mode are you using? HotAdd? Network?

Thanks!
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org »

Hi Vitaliy,

I left the backup running, I came in this morning it is at 20% and it has been running for 17hrs. This isn't right.... I am using a network. The NSF is plugged to the same switch with the two hosts. There isn't much traffic, its a small test lab. What is HotAdd? How come the VEEAM server backup in less than an hour?

Thanks,
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. »

christiank@pcs.org wrote:it is at 20% and it has been running for 17hrs. This isn't right....
I agree that this is an awful job performance. Have you heard back from you support engineer? What does the bottleneck statistics window say about this job?
christiank@pcs.org wrote: What is HotAdd? How come the VEEAM server backup in less than an hour?
That's a good question! Can you please ask your engineer to compare job jog sessions from both runs?

Please see our sticky F.A.Q. for further details on the available processing modes: VMware : [FAQ] FREQUENTLY ASKED QUESTIONS
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org »

No, I haven't heard from support engineer. The bottleneck status is source. I killed the job and configure proxy with virtual appliance then changed the job settings. I restarted the job its been running for over 10min now still at 0%.

Is there something wrong with my configuration?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. »

Your Veeam backup configuration looks fine to me, I guess there might be something wrong with the connection to vSphere environment.

Are you using vCenter Server connection to backup your VMs? Can you please try to add standalone hosts via IP address and configure a test backup job to see what the job performance will be?
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org »

Yes, I am vCenter. I deleted the vCenter from the backup infrustucture, added one host (it has 2 VMs) I am using iSCSi target to the NFS storage. I added a proxy server (one of the VMs which has a connection to the NFS) I created a new backup job for one VM.

The job is running now at 0% bottleneck status is target.

I am skeptical about this product. Will it be able to run in our production environment of over 50 VMs? I am beating my head just to get it to backup 4 VMs.

Unless my configuration is missing something.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. »

Ok, let's wait what our support engineer figures out after reviewing your job debug log files.
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org »

OK thanks, what is the expected time for a VM with 40GB to be backed up? I got a good feeling this can be a good solution my environment because I backed up one VM (the VEEAM server) for close to an hour, then Instant recover was very quick.

I just hope the engineers will shine some light on this....
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by tsightler »

I rarely see speeds less than about 40-50MB/sec, even for small environments, so I'd say typical would be about 12-15 minutes for a 40GB VM. Perhaps much faster if your hardware can support it. What you're seeing is way, way below anything that would be "normal" so there must be something wrong.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Yuki »

Hi Chris,
I've replied to your question in my thread http://forums.veeam.com/viewtopic.php?f ... 135#p69135

The issue we are seeing is that there is a corrupt OIB file from a job that was stuck and then canceled. You will need to do some cleanup to fix the problem (remove old backups, run Active full backup, import backups).
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org »

Hi Yuki,

Please read my postings. I have deleted the jobs several times, tried to back up different VMs with different jobs, same results. I ran one job at a time. All VMs are connected to the NFS via Windows iSCSi.

What else should I do?
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by tsightler »

christiank@pcs.org wrote:All VMs are connected to the NFS via Windows iSCSi.
You've made this statement twice, but it confuses me as NFS and iSCSI are two different protocols. You can't connect to NFS via iSCSI as NFS is a file sharing protocol, and iSCSI is a block based protocol. Are you perhaps confusing some terms? Do you mean that you are connecting to the NAS via iSCSI? Also, which VMs are connected to it?
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Yuki »

tsightler wrote: You've made this statement twice, but it confuses me as NFS and iSCSI are two different protocols. You can't connect to NFS via iSCSI as NFS is a file sharing protocol, and iSCSI is a block based protocol. Are you perhaps confusing some terms? Do you mean that you are connecting to the NAS via iSCSI? Also, which VMs are connected to it?
1) What tsightler says is correct - either one or the other. In my personal experience SMB/CIFS is faster than iSCSI on inexpensive setups. If you have iSCSI SAN - than it is a different story.

2) As i said - for us the fix was to 1- kill the job, 2- remove backups from the backup job in Veeam console, 3 - run FULL Active backup. If you attempt to rerun your backups without clearing old backup information from veeam and detaching those backup files - you will have the same problem of backups getting stuck.
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org »

Hi tsightler,

Sorry for the confusion. I meant to say NAS via iSCSi. All VMs are connected to it via iSCSi connector. I can backup the VEEAM server (one of the VMs that has VEEAM installed) without a problem. It takes less than and hour to complete.

I have deleted the backup jobs many times during my test. When the job runs for about 3hrs and the process status is 0% I delete it and then navigated in the NAS storaged to delete the files already created then start all over again.

What could cause one VM, with VEEAM install, to backup and others not? I have done lots of research on this I don't see many blogs about this version of VEEAM. Couldn't be a bug or because I am using the trial version?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. »

Hi Chris, this behavior has nothing to do with the license version you're using. As far as I can see from your ticket status our engineer is still awaiting the debug log files from your backup jobs. Can you please upload them as it is hard to troubleshoot technical issues over the forums?
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org »

Vitaliy,

I did upload the log files before starting this thread. There has been many test jobs and they all have the same problem. There was only one job passed the 0% and went up to 20% after 17hrs. I will created another job, ran it, then grub the log. How long should this job ran?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by foggy »

I think 5-10 minutes is enough to grab the logs with information required.
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org »

OK. So upload the files to support engineers?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Vitaliy S. »

Please do.
Aremac
Enthusiast
Posts: 59
Liked: 3 times
Joined: Sep 30, 2011 11:59 pm
Full Name: Sam Pierce
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Aremac »

Question about your storage. You said "I meant to say NAS via iSCSi. All VMs are connected to it via iSCSi connector" Do you mean that all your VM's use iSCSI based datastores (VMFS or RDM) for their disks and they are shared by the guest OS to provide NAS features (NFS / CIFS). Or, do you mean that your VM's use the Microsoft iSCSI initiator within the OS to connect to iSCSI LUNs. The latter would be a problem as you cannot backup a VM that has an iSCSI LUN attached via the Microsoft iSCSI initiator.
christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by christiank@pcs.org »

I blow off the entire lab, started from scratch. Now my trial license has expired. I went to manage licenses and don't have the option for another trial.

Any ideas?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup job stuck at 0% for over 3hrs.

Post by Gostev »

Click the "Trial Key Expired" link in the License key section of the product download area, and it will open the form to contact sales for an extended trial license key.
Post Reply

Who is online

Users browsing this forum: Amazon [Bot] and 212 guests