Backup job stuck at 0% for over 3hrs.

[christiank@pcs.org]

Support ID: 00168202

I am testing VEEAM Backup and Replication 6.5. I have installed it in our test lab. The lab has two ESXi hosts and 4 VMs. One of the VMs is the VEEAM server. There is NFS storage and only two VMs have files in it. Its flat network environment, only one phyiscal switch, no subnets, so its very simple.

I have created a backup job with one VM but the job status is at 0% for over 2hrs. I look at the job progress detail and everything has a green check mark. The last item on the list is the Hard Disk 1. The process rate 72kb/s. I have deleted this job and recreated another one, with another VM, same results.

Though, I was able to backup the VM which is running the VEEAM without any problem. It finised in less than an hour.

Am I supposed to install VEEAM on each VM? My understanding VEEAM is agent-less.

I am stuck with this... any help?

[Vitaliy S.]

Hi Chris,

Yes, you're correct in your understanding that Veeam is using agent-less approach to backup virtual machines and there is no need to install Veeam on every machine you're going to back up. The debug logs that you have sent to our support team should clearly indicate the reason for such slow processing rate. Btw, what backup mode are you using? HotAdd? Network?

Thanks!

[christiank@pcs.org]

Hi Vitaliy,

I left the backup running, I came in this morning it is at 20% and it has been running for 17hrs. This isn't right.... I am using a network. The NSF is plugged to the same switch with the two hosts. There isn't much traffic, its a small test lab. What is HotAdd? How come the VEEAM server backup in less than an hour?

Thanks,

[Vitaliy S.]

it is at 20% and it has been running for 17hrs. This isn't right....

I agree that this is an awful job performance. Have you heard back from you support engineer? What does the bottleneck statistics window say about this job?

What is HotAdd? How come the VEEAM server backup in less than an hour?

That's a good question! Can you please ask your engineer to compare job jog sessions from both runs?

Please see our sticky F.A.Q. for further details on the available processing modes: VMware : [FAQ] FREQUENTLY ASKED QUESTIONS

[christiank@pcs.org]

No, I haven't heard from support engineer. The bottleneck status is source. I killed the job and configure proxy with virtual appliance then changed the job settings. I restarted the job its been running for over 10min now still at 0%.

Is there something wrong with my configuration?

[Vitaliy S.]

Your Veeam backup configuration looks fine to me, I guess there might be something wrong with the connection to vSphere environment.

Are you using vCenter Server connection to backup your VMs? Can you please try to add standalone hosts via IP address and configure a test backup job to see what the job performance will be?

[christiank@pcs.org]

Yes, I am vCenter. I deleted the vCenter from the backup infrustucture, added one host (it has 2 VMs) I am using iSCSi target to the NFS storage. I added a proxy server (one of the VMs which has a connection to the NFS) I created a new backup job for one VM.

The job is running now at 0% bottleneck status is target.

I am skeptical about this product. Will it be able to run in our production environment of over 50 VMs? I am beating my head just to get it to backup 4 VMs.

Unless my configuration is missing something.

[Vitaliy S.]

Ok, let's wait what our support engineer figures out after reviewing your job debug log files.

[christiank@pcs.org]

OK thanks, what is the expected time for a VM with 40GB to be backed up? I got a good feeling this can be a good solution my environment because I backed up one VM (the VEEAM server) for close to an hour, then Instant recover was very quick.

I just hope the engineers will shine some light on this....

[tsightler]

I rarely see speeds less than about 40-50MB/sec, even for small environments, so I'd say typical would be about 12-15 minutes for a 40GB VM. Perhaps much faster if your hardware can support it. What you're seeing is way, way below anything that would be "normal" so there must be something wrong.

[Yuki]

Hi Chris,
I've replied to your question in my thread http://forums.veeam.com/viewtopic.php?f ... 135#p69135

The issue we are seeing is that there is a corrupt OIB file from a job that was stuck and then canceled. You will need to do some cleanup to fix the problem (remove old backups, run Active full backup, import backups).

[christiank@pcs.org]

Hi Yuki,

Please read my postings. I have deleted the jobs several times, tried to back up different VMs with different jobs, same results. I ran one job at a time. All VMs are connected to the NFS via Windows iSCSi.

What else should I do?

[tsightler]

All VMs are connected to the NFS via Windows iSCSi.

You've made this statement twice, but it confuses me as NFS and iSCSI are two different protocols. You can't connect to NFS via iSCSI as NFS is a file sharing protocol, and iSCSI is a block based protocol. Are you perhaps confusing some terms? Do you mean that you are connecting to the NAS via iSCSI? Also, which VMs are connected to it?

[Yuki]

You've made this statement twice, but it confuses me as NFS and iSCSI are two different protocols. You can't connect to NFS via iSCSI as NFS is a file sharing protocol, and iSCSI is a block based protocol. Are you perhaps confusing some terms? Do you mean that you are connecting to the NAS via iSCSI? Also, which VMs are connected to it?

1) What tsightler says is correct - either one or the other. In my personal experience SMB/CIFS is faster than iSCSI on inexpensive setups. If you have iSCSI SAN - than it is a different story.

2) As i said - for us the fix was to 1- kill the job, 2- remove backups from the backup job in Veeam console, 3 - run FULL Active backup. If you attempt to rerun your backups without clearing old backup information from veeam and detaching those backup files - you will have the same problem of backups getting stuck.

[christiank@pcs.org]

Hi tsightler,

Sorry for the confusion. I meant to say NAS via iSCSi. All VMs are connected to it via iSCSi connector. I can backup the VEEAM server (one of the VMs that has VEEAM installed) without a problem. It takes less than and hour to complete.

I have deleted the backup jobs many times during my test. When the job runs for about 3hrs and the process status is 0% I delete it and then navigated in the NAS storaged to delete the files already created then start all over again.

What could cause one VM, with VEEAM install, to backup and others not? I have done lots of research on this I don't see many blogs about this version of VEEAM. Couldn't be a bug or because I am using the trial version?

[Vitaliy S.]

Hi Chris, this behavior has nothing to do with the license version you're using. As far as I can see from your ticket status our engineer is still awaiting the debug log files from your backup jobs. Can you please upload them as it is hard to troubleshoot technical issues over the forums?

[christiank@pcs.org]

Vitaliy,

I did upload the log files before starting this thread. There has been many test jobs and they all have the same problem. There was only one job passed the 0% and went up to 20% after 17hrs. I will created another job, ran it, then grub the log. How long should this job ran?

[foggy]

I think 5-10 minutes is enough to grab the logs with information required.

[christiank@pcs.org]

OK. So upload the files to support engineers?

[Vitaliy S.]

Please do.

[Aremac]

Question about your storage. You said "I meant to say NAS via iSCSi. All VMs are connected to it via iSCSi connector" Do you mean that all your VM's use iSCSI based datastores (VMFS or RDM) for their disks and they are shared by the guest OS to provide NAS features (NFS / CIFS). Or, do you mean that your VM's use the Microsoft iSCSI initiator within the OS to connect to iSCSI LUNs. The latter would be a problem as you cannot backup a VM that has an iSCSI LUN attached via the Microsoft iSCSI initiator.

[christiank@pcs.org]

I blow off the entire lab, started from scratch. Now my trial license has expired. I went to manage licenses and don't have the option for another trial.

Any ideas?

[Gostev]

Click the "Trial Key Expired" link in the License key section of the product download area, and it will open the form to contact sales for an extended trial license key.