Nine jobs hung

[ashman70]

I have nine jobs that are presently hung, four back up and five replication jobs. I have opened a case with support case #00705356.

It appears that three jobs ran at the same time at 8pm last night, two replication and one backup, the first job, a replication job is stuck on 'processing (vm name) the rest of the jobs behind it are waiting for backup infrastructure resources availability. This is all in one environment on one host. All the VM's are production vm's and seem fine and unaffected. Not sure what I should do, but I'm hoping I don't have to wait 8 hours to hear from support. This is a VMware environment and I'm running the latest version 8 of Veeam Backup.

[foggy]

If you click that VM in the list to the left in the job session log, what particular task the job is stuck on?

[ashman70]

Not sure what you mean 'the list to the left'?

If I click on 'Running' on the left it shows all nine jobs a 0% completed.

[foggy]

I mean the list of VMs in the job session log (available if you right-click the job and select Statistics).

[ashman70]

What do you suggest I do? Can I high light all the jobs and select 'stop' or is that a bad idea? Do you think the snapshot is stuck on the first job that is stuck on 'processing' its a replication job. All the other jobs are waiting for resources to free up.

[ashman70]

The first job that is an incremental replication job has been stuck on 'processing (vm name) for almost 12 hours. Its at 0%. I uploaded logs for the last 48hours when I opened the case.

[ashman70]

There is a snapshot for this VM. I can see it in vsphere manager for this particular VM.

[foggy]

Do you think the snapshot is stuck on the first job that is stuck on 'processing' its a replication job.

I've just asked about the particular operation it is stuck on to understand that.

There is a snapshot for this VM. I can see it in vsphere manager for this particular VM.

What kind of VM it is? You can stop all other jobs, but should wait for this one to finish (you cannot affect the process of snapshot commit).

[ashman70]

Ok but if I stop all the other jobs they will just start again as I have other jobs that run very hour etc.

Its a Windows server vm, running Windows Server 2008.

Are you saying we just have to wait indefinitely for this stuck job to finish?

[ashman70]

Am I going to have to open a support case with VMware?

[foggy]

I recommend to contact our technical team first. Veeam B&R debug logs will tell what's the job is doing at that very moment and whether it is Veeam-related issue or not.

[ashman70]

Ok but the SLA for standard support is 8 hours and I just opened the case 90 minutes ago. Should I call in?

[foggy]

If this is a critical issue (severity 1), I believe you will be contacted within a shorter SLA.

[ashman70]

I got through to support, they need some time to process the logs I uploaded and will get back to me.


Thanks.

[ashman70]

Just an update, support found that the number of concurrent jobs in my setup was 1 or 2, that was changed to 6 since the server running Veeam has a single six core Xeon processor it should be able to handle that. We had to stop all Veeam services to get the hung job to stop, the snapshot seemed to take care of itself and all was well. This morning however I find the same replication job for the same VM is hung again. Called support, they had to kill processes for the job to fail (I had clicked stop on the job as I had yesterday but it didn't do anything) I have uploaded logs to support, they are recommending a reboot of the Veeam server which is difficult in my environment as its performing many roles or introducing a second proxy which is more viable, although it will have to be a windows 7 based workstation with 4GB of RAM for now. It is only one VM I am having this issue with and it for a replication job to a local host.

[foggy]

Thanks for providing an update, I believe you will be able to track the issue with that VM down with the help of our engineer. What is your support case ID for reference, btw?

[ashman70]

Thanks Foggy, it support case #00705356. It concerns me that this same vm replication job keeps hanging and has done so twice in the past 24 hours, I hope we can get it resolved soon. In the meantime I am going to go onsite today and deploy a workstation to act as a second proxy, hopefully that will help.

[ashman70]

Just another followup, had another job hang late this morning and got it resolved by support. They suggested deploying a second proxy to use just for this problematic replication job so I was onsite today and deployed a windows 7 64bit machine to act as a proxy just for this job. So far so good, we'll know in the next 12 hours if this has resolved the problem for now.