Host-based backup of VMware vSphere VMs.
Post Reply
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Nine jobs hung

Post by ashman70 »

I have nine jobs that are presently hung, four back up and five replication jobs. I have opened a case with support case #00705356.

It appears that three jobs ran at the same time at 8pm last night, two replication and one backup, the first job, a replication job is stuck on 'processing (vm name) the rest of the jobs behind it are waiting for backup infrastructure resources availability. This is all in one environment on one host. All the VM's are production vm's and seem fine and unaffected. Not sure what I should do, but I'm hoping I don't have to wait 8 hours to hear from support. This is a VMware environment and I'm running the latest version 8 of Veeam Backup.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Nine jobs hung

Post by foggy »

If you click that VM in the list to the left in the job session log, what particular task the job is stuck on?
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 »

Not sure what you mean 'the list to the left'?

If I click on 'Running' on the left it shows all nine jobs a 0% completed.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Nine jobs hung

Post by foggy »

I mean the list of VMs in the job session log (available if you right-click the job and select Statistics).
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 »

What do you suggest I do? Can I high light all the jobs and select 'stop' or is that a bad idea? Do you think the snapshot is stuck on the first job that is stuck on 'processing' its a replication job. All the other jobs are waiting for resources to free up.
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 »

The first job that is an incremental replication job has been stuck on 'processing (vm name) for almost 12 hours. Its at 0%. I uploaded logs for the last 48hours when I opened the case.
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 »

There is a snapshot for this VM. I can see it in vsphere manager for this particular VM.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Nine jobs hung

Post by foggy »

ashman70 wrote:Do you think the snapshot is stuck on the first job that is stuck on 'processing' its a replication job.
I've just asked about the particular operation it is stuck on to understand that.
ashman70 wrote:There is a snapshot for this VM. I can see it in vsphere manager for this particular VM.
What kind of VM it is? You can stop all other jobs, but should wait for this one to finish (you cannot affect the process of snapshot commit).
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 »

Ok but if I stop all the other jobs they will just start again as I have other jobs that run very hour etc.

Its a Windows server vm, running Windows Server 2008.

Are you saying we just have to wait indefinitely for this stuck job to finish?
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 »

Am I going to have to open a support case with VMware?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Nine jobs hung

Post by foggy »

I recommend to contact our technical team first. Veeam B&R debug logs will tell what's the job is doing at that very moment and whether it is Veeam-related issue or not.
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 »

Ok but the SLA for standard support is 8 hours and I just opened the case 90 minutes ago. Should I call in?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Nine jobs hung

Post by foggy »

If this is a critical issue (severity 1), I believe you will be contacted within a shorter SLA.
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 »

I got through to support, they need some time to process the logs I uploaded and will get back to me.


Thanks.
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 »

Just an update, support found that the number of concurrent jobs in my setup was 1 or 2, that was changed to 6 since the server running Veeam has a single six core Xeon processor it should be able to handle that. We had to stop all Veeam services to get the hung job to stop, the snapshot seemed to take care of itself and all was well. This morning however I find the same replication job for the same VM is hung again. Called support, they had to kill processes for the job to fail (I had clicked stop on the job as I had yesterday but it didn't do anything) I have uploaded logs to support, they are recommending a reboot of the Veeam server which is difficult in my environment as its performing many roles or introducing a second proxy which is more viable, although it will have to be a windows 7 based workstation with 4GB of RAM for now. It is only one VM I am having this issue with and it for a replication job to a local host.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Nine jobs hung

Post by foggy »

Thanks for providing an update, I believe you will be able to track the issue with that VM down with the help of our engineer. What is your support case ID for reference, btw?
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 » 1 person likes this post

Thanks Foggy, it support case #00705356. It concerns me that this same vm replication job keeps hanging and has done so twice in the past 24 hours, I hope we can get it resolved soon. In the meantime I am going to go onsite today and deploy a workstation to act as a second proxy, hopefully that will help.
ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Nine jobs hung

Post by ashman70 »

Just another followup, had another job hang late this morning and got it resolved by support. They suggested deploying a second proxy to use just for this problematic replication job so I was onsite today and deployed a windows 7 64bit machine to act as a proxy just for this job. So far so good, we'll know in the next 12 hours if this has resolved the problem for now.
Post Reply

Who is online

Users browsing this forum: No registered users and 93 guests