Jobs freezing

[ec-inflxx]

So ever since I upgraded to v6 I have been having intermittent problems with jobs just freezing. I have about 4 jobs setup that run nightly. Sometimes I wont get any email status and I will look at the jobs and they are frozen at a certain percentage. No errors just frozen. Only way I have found to stop them was a reboot.

Has anyone experience anything like this and what could be a good fix? At this point I am thinking of uninstalling and reinstalling but dont feel like recreating all the jobs.

Also what is the best practice for having multiple jobs? Can they run together? I have the time spaced out so I am hoping that the job is finished before the next but that doesn't always happen especially when it does its full.

Any help would be kindly appreciated.

[Jfmoots]

So ever since I upgraded to v6 I have been having intermittent problems with jobs just freezing. I have about 4 jobs setup that run nightly. Sometimes I wont get any email status and I will look at the jobs and they are frozen at a certain percentage. No errors just frozen. Only way I have found to stop them was a reboot.

Has anyone experience anything like this and what could be a good fix? At this point I am thinking of uninstalling and reinstalling but dont feel like recreating all the jobs.

Also what is the best practice for having multiple jobs? Can they run together? I have the time spaced out so I am hoping that the job is finished before the next but that doesn't always happen especially when it does its full.

Any help would be kindly appreciated.

I've seen the console go "catatonic" when the machine was overloaded. Too many jobs running concurrently. I've even seen problems with the proxy had too much ram. (strange as that may sound). Tell us about your Veeam B&R server/proxy. Is it physical or virtual? How many CPUs? How much RAM?

Be sure you're running the latest patch. (go to cp.veeam.com and type "patch 3" in the search box)

You can run jobs together. Rule of thumb is one job per 2 CPUs. For instance, if you have a 4 CPU machine, you're good to run 2 jobs concurrently. Of course, if your proxy is running on a physical machine you can get away with pushing that limit.

One thing to double check. If your proxy is a VM and it's using hot-add, you're likely to have stuck hot-adds from your reboots while jobs were running. You'll need to go in to the settings on the proxy VM(s) and remove any hard disks that belong to other VMs. This could/will cause snapshot troubles on future jobs.

[Gostev]

The job process should keep logging its activity into the debug log (unless the job process literally halts for whatever reason), so this would be the first place to check to see what the job is really doing when it appears "frozen".

[ec-inflxx]

This is a physical server 32gb RAM, 2 Quad-Core processors. I have the latest patch installed.

Weird that all of a sudden the job did start and they all ran fine last night. But it seems to be sporatic. Maybe I will space the jobs out a little more so that they don't overlap as much?

[Vitaliy S.]

I don't believe that using different schedule will help here, so if you still want to nail down the issue you need to review the job logs as Anton suggested.

[Gostev]

It does not matter if the jobs overlap, the v6 scheduling engine will never run more jobs concurrently than your backup infrastructure allows. Other jobs will be simply waiting for proxy and repository resources to free up. Meaning, you can set all your jobs to start at the same time (up to 64 of them, actually), and this will not cause any issues.

[gcballard]

I have been having this issue as well. I had opened a ticket (5173983) and submitted logs. The tech had to have me kill the veeamagent.exe in order for the jobs to stop. Looking at the logs it appeared that the last thing was a call to delete the snapshot. But vcenter doesn't register this command as ever being sent and Veeam never aborts the job, it just sits there forever. Deleting the snapshot manually doesn't help either. This is frustrating because if it aborted, it could then process the other guests and then retry.

After I killed VeeamAgent.exe, I reran the job and it worked fine until last night (5 days or so no issues). I had the tech close the ticket.

Before I open (or reopen) the ticket, I just tried (today) recreating a job from scratch. I had two jobs fail (or should I say, "fail to fail") last night so I left the other alone as a "control" job. I also told Veeam to not run past the "window" in hopes that it will abort and fail and I will be notified at least.



A note about the environment:
My systems were upgraded from v5 (latest patch) and no real change to repositories. Just backing up locally. Also, brand new Cisco C-210 dual 6 core x5670 w/ not a lot running on it. System utilization is well below 50% at peak times.

[gcballard]

No joy on re-creating the jobs. My SAP production job hung at the end of the very first VM. I woke up @ 3 and checked it and then tried to re-run it but it ran out of time for it's job window and so it terminated it. I would post log snippets but that's a no-no. I will reopen the ticket and upload new logs to veeam support.

[fgw]

have you enabled indexing on any one of your jobs?

try to disable indexing and see if this helps!

had a similar problem and disabling indexing solved this.

pretty sure there is something wrong with indexing, as i had a lot of issues while indexing was enabled ...

case: ID#5170228

[htwnrva]

Yes! I have been having this problem constantly since v6. On a physical server with 24 cores and 32GB of ram. I'm interested in the "server has too much ram" theory, can you elaborate why that would be an issue.

[gcballard]

I don't have indexing enabled. I did make some changes such as move the virtual appliance to another host. I also hard coded the vcenter IP in the hosts file and tweaked the time the backup ran. So far, it's been running ok. But I'm not holding my breath. I still think a minor code change that would abort the job after a certain timeout would be greatly advisable.At least then the job would re-run

[gcballard]

Well, I had this happen again on a different backup server and different VM. the job hung and the snapshot grew for 14 hours. Because I was only backing up the system disk (0:0) I didn't realize what could happen. Turns out that the datastore ran out of space because the snapshot delta change was huge (data warehouse system).

There is one clue (maybe). The very next day I noticed issues w/ vcenter. We're running v5 which is upgraded from v4.1u1. One host was disonnected. I tried to reconnect it, but it would do anything. So, I restarted the vcenter service. It appeared to reconnect. But later when I went to vmotion a machine (for other reasons) it didn't do anything. It was like I hadn't told it to do anything at all. Long story short, I finally had to uninstall vcenter and reboot (tried disconnects/removals/reboots/service restarts) and reinstall to get the hosts to come back online and work properly.

So, if it's vcenter going wonky (new favorite word) I can't be mad at veeam. BUT (there's always a but), Veeam should time out and send an e-mail. I have now created a couple different snapshot alerts in vmware. One with a low threshold to trigger once a day to let me know that a system has a snapshot (people create them and forget) and one that alerts me more frequently (30 minutes) when a snapshot gets too big (75GB).

In the meantime, I think I'm going to work on creating a new non-upgraded vcenter DB.

[Gostev]

There are definitely timeouts in the code for everything, and they are quite short. I would say, we better investigate the logs for your hung job to understand what really happened when why those timeouts did not work. May be they do not cover all places (those where freeze issues were completely unexpected by devs)? I had a similar complaint once from our own backup admin recently.

[Gostev]

So, it looks like this freeze happens in the VDDK libraries of VMware vStorage API. We are researching this now - if we are able to find and hook the faulting function, then this will fix the issue.

[gcballard]

Thanks Anton. At the behest of support, I made a VDDK timeout change on my systems. We'll see if this helps or not. It usually happens once a week or so.