Jobs freezing

[ec-inflxx]

So ever since I upgraded to v6 I have been having intermittent problems with jobs just freezing. I have about 4 jobs setup that run nightly. Sometimes I wont get any email status and I will look at the jobs and they are frozen at a certain percentage. No errors just frozen. Only way I have found to stop them was a reboot.

Has anyone experience anything like this and what could be a good fix? At this point I am thinking of uninstalling and reinstalling but dont feel like recreating all the jobs.

Also what is the best practice for having multiple jobs? Can they run together? I have the time spaced out so I am hoping that the job is finished before the next but that doesn't always happen especially when it does its full.

Any help would be kindly appreciated.

[Jfmoots]

So ever since I upgraded to v6 I have been having intermittent problems with jobs just freezing. I have about 4 jobs setup that run nightly. Sometimes I wont get any email status and I will look at the jobs and they are frozen at a certain percentage. No errors just frozen. Only way I have found to stop them was a reboot.

Has anyone experience anything like this and what could be a good fix? At this point I am thinking of uninstalling and reinstalling but dont feel like recreating all the jobs.

Also what is the best practice for having multiple jobs? Can they run together? I have the time spaced out so I am hoping that the job is finished before the next but that doesn't always happen especially when it does its full.

Any help would be kindly appreciated.

I've seen the console go "catatonic" when the machine was overloaded. Too many jobs running concurrently. I've even seen problems with the proxy had too much ram. (strange as that may sound). Tell us about your Veeam B&R server/proxy. Is it physical or virtual? How many CPUs? How much RAM?

Be sure you're running the latest patch. (go to cp.veeam.com and type "patch 3" in the search box)

You can run jobs together. Rule of thumb is one job per 2 CPUs. For instance, if you have a 4 CPU machine, you're good to run 2 jobs concurrently. Of course, if your proxy is running on a physical machine you can get away with pushing that limit.

One thing to double check. If your proxy is a VM and it's using hot-add, you're likely to have stuck hot-adds from your reboots while jobs were running. You'll need to go in to the settings on the proxy VM(s) and remove any hard disks that belong to other VMs. This could/will cause snapshot troubles on future jobs.

[Gostev]

The job process should keep logging its activity into the debug log (unless the job process literally halts for whatever reason), so this would be the first place to check to see what the job is really doing when it appears "frozen".

[ec-inflxx]

This is a physical server 32gb RAM, 2 Quad-Core processors. I have the latest patch installed.

Weird that all of a sudden the job did start and they all ran fine last night. But it seems to be sporatic. Maybe I will space the jobs out a little more so that they don't overlap as much?

[Vitaliy S.]

I don't believe that using different schedule will help here, so if you still want to nail down the issue you need to review the job logs as Anton suggested.

[Gostev]

It does not matter if the jobs overlap, the v6 scheduling engine will never run more jobs concurrently than your backup infrastructure allows. Other jobs will be simply waiting for proxy and repository resources to free up. Meaning, you can set all your jobs to start at the same time (up to 64 of them, actually), and this will not cause any issues.

[gcballard]

I have been having this issue as well. I had opened a ticket (5173983) and submitted logs. The tech had to have me kill the veeamagent.exe in order for the jobs to stop. Looking at the logs it appeared that the last thing was a call to delete the snapshot. But vcenter doesn't register this command as ever being sent and Veeam never aborts the job, it just sits there forever. Deleting the snapshot manually doesn't help either. This is frustrating because if it aborted, it could then process the other guests and then retry.

After I killed VeeamAgent.exe, I reran the job and it worked fine until last night (5 days or so no issues). I had the tech close the ticket.

Before I open (or reopen) the ticket, I just tried (today) recreating a job from scratch. I had two jobs fail (or should I say, "fail to fail") last night so I left the other alone as a "control" job. I also told Veeam to not run past the "window" in hopes that it will abort and fail and I will be notified at least.



A note about the environment:
My systems were upgraded from v5 (latest patch) and no real change to repositories. Just backing up locally. Also, brand new Cisco C-210 dual 6 core x5670 w/ not a lot running on it. System utilization is well below 50% at peak times.

[gcballard]

No joy on re-creating the jobs. My SAP production job hung at the end of the very first VM. I woke up @ 3 and checked it and then tried to re-run it but it ran out of time for it's job window and so it terminated it. I would post log snippets but that's a no-no. I will reopen the ticket and upload new logs to veeam support.

[fgw]

have you enabled indexing on any one of your jobs?

try to disable indexing and see if this helps!

had a similar problem and disabling indexing solved this.

pretty sure there is something wrong with indexing, as i had a lot of issues while indexing was enabled ...

case: ID#5170228

[htwnrva]

Yes! I have been having this problem constantly since v6. On a physical server with 24 cores and 32GB of ram. I'm interested in the "server has too much ram" theory, can you elaborate why that would be an issue.

[gcballard]

I don't have indexing enabled. I did make some changes such as move the virtual appliance to another host. I also hard coded the vcenter IP in the hosts file and tweaked the time the backup ran. So far, it's been running ok. But I'm not holding my breath. I still think a minor code change that would abort the job after a certain timeout would be greatly advisable.At least then the job would re-run

[gcballard]

Well, I had this happen again on a different backup server and different VM. the job hung and the snapshot grew for 14 hours. Because I was only backing up the system disk (0:0) I didn't realize what could happen. Turns out that the datastore ran out of space because the snapshot delta change was huge (data warehouse system).

There is one clue (maybe). The very next day I noticed issues w/ vcenter. We're running v5 which is upgraded from v4.1u1. One host was disonnected. I tried to reconnect it, but it would do anything. So, I restarted the vcenter service. It appeared to reconnect. But later when I went to vmotion a machine (for other reasons) it didn't do anything. It was like I hadn't told it to do anything at all. Long story short, I finally had to uninstall vcenter and reboot (tried disconnects/removals/reboots/service restarts) and reinstall to get the hosts to come back online and work properly.

So, if it's vcenter going wonky (new favorite word) I can't be mad at veeam. BUT (there's always a but), Veeam should time out and send an e-mail. I have now created a couple different snapshot alerts in vmware. One with a low threshold to trigger once a day to let me know that a system has a snapshot (people create them and forget) and one that alerts me more frequently (30 minutes) when a snapshot gets too big (75GB).

In the meantime, I think I'm going to work on creating a new non-upgraded vcenter DB.

[Gostev]

There are definitely timeouts in the code for everything, and they are quite short. I would say, we better investigate the logs for your hung job to understand what really happened when why those timeouts did not work. May be they do not cover all places (those where freeze issues were completely unexpected by devs)? I had a similar complaint once from our own backup admin recently.

[Gostev]

So, it looks like this freeze happens in the VDDK libraries of VMware vStorage API. We are researching this now - if we are able to find and hook the faulting function, then this will fix the issue.

[gcballard]

Thanks Anton. At the behest of support, I made a VDDK timeout change on my systems. We'll see if this helps or not. It usually happens once a week or so.

[htwnrva]

So did the VDDK timeout fix your issue? I am still having the same problem.

[Davd]

I have the same problem here with backup jobs freezing at a certain percentage during backup... VM's beeing on snapshot.... I have a ticket open with support ID#5190966

I also have one job that was freezed on "Transforming previous full backup chain into rollbacks" for 31 hours before I noticed it... Don't know if its the same issue causing it as for freezes during backup..but it was frozen.. VIB was "only" 11 GB and normaly transformation for that job completes within 20-30 minutes.

Since this thread is from February... its kind of worrying for me that there might not be an available solution for this problem yet... cause we just started to get this problems last week and were more or less forced to disable all backup jobs since I cannot "babysit" all backups during the night...

Our Veeam is latest version/fully patched.

[paulglavin]

I'm also having a similar issue with a replica job.
One of our servers will randomly time out and not delete the snapshot. If I manually delete the snapshot then the job will normally complete on the next run.
Ticket logged with support #5192069

[rconway]

Same problem here, ever since upgrading to Veeam 6. Jobs "freeze" intermittently, and cannot be stopped short of rebooting or killing the Veeam processes. This happens to upgraded jobs and newly created jobs, to backup jobs and replica jobs. Any fix in sight?

[SimonCurtiss]

I also have this issue on Veeam 5 - with latest patch - on ESXi 5 - with latest patches. Server is Windows 2003 SP2

99% of the time it is stuck at the end of a servers backup, so removing the snapshot seems to be the problem.

Killing the Veeam agent/services on both server and VM do not affect it and I have to reboot the Backup Server. A retry will always work

There is no pattern to when it will happen (sometimes twice a week, sometimes not for a fortnight) or with which VM it happens.

Cheers

Simon

[cteps]

Similar issue here and only when we upgraded to V6. Have installed the latest veeam patch and its on a physical server (12GB, 2xQuad core). Case logged ID# 5189402

I reboot to kill the Veeam jobs as of this morning I had 3 that had frozen at 99%

[bhwong]

If there is a freeze issue on version 6 not found on version 5, should I deploy version 5 instead?

[KiwiJJ]

Hi,
I had the same issue last week with freezing jobs, just after I upgraded vCenter to version 5. I am still running Veeam version 5. Deleted all of the jobs and re-added them. Will see how it goes this week.

John

[Gostev]

The issue sits in the VMware vStorage API components (VDDK), so Veeam version does not matter.

[Eamonn Deering]

I’m having the same issue. Everything was running ok on V6 with just the odd failure. I then updated to patch 3 some weeks back. Not sure why but the last week my backups are a bit of a mess. Freezing on almose every backup now (3 out of 4 jobs). I'm on vmware ESXi 4.1.0.502767
Should we be looking to vmware for a patch (VDDK)?

[Gostev]

You should open a support case with Veeam, once the patch is available from VMware or we find a way to work around this, you will be notified.

The problem I am talking about is actually not wide spread, and affects only a handful of customers, and so may have to deal with some environment specifics which cause VDDK to misbehave. For example, we never ever faced this issue in our own production deployment at Veeam. Also, the support cases of many earlier posters in this topic are already closed, because in those cases, the freeze was caused by something else (such as indexing), and not this rare VDDK issue.