Job hangs - 0%

VMware specific discussions

Job hangs - 0%

Veeam Logoby cvacanti » Mon May 18, 2015 7:31 pm

Support case 00887944 - Opened April 23

Hoping someone can assist - I've had this case open with support for almost a month. I've uploaded multiple log bundles, we've done webex sessions, and nothing yet is a smoking gun why these jobs fail. It should be known that other jobs on the Veeam server are running successfully. DR jobs are working too. I've left Backup Job1 running and it is sitting at 161 hours run time currently with 0% processed.

Backup Job1 - 3VMs - yes use storage integration w netapp, no do not fail over to network backups, yes use application aware processing, no do not index guest file system
VM1 - MS SQL server
VM2 - MS App server
VM3 - MS App server

If I create three new backup jobs they work because I did not choose to use application aware processing and yes use storage snapshots.
VM1-Backup - ran this job 100% successful with no checks for application aware processing or indexing.
VM2-Backup - ran this job 100% successful with no checks for application aware processing or indexing.
VM3-Backup - ran this job 100% successful with no checks for application aware processing or indexing.

If I change the individual backup jobs to use application aware processing things fail:
VM1-Backup - job hangs at 0% with this entry: 5/1/2015 2:59:31 PM :: Inventorying guest system. In order to stop this job I have to kill the Veeam processes in Task Manager.
VM2-Backup - Job fails with "Failed to prepare guest for hot backup. VSSControl: Freeze state is none."
VM3-Backup - Job fails with "Failed to prepare guest for hot backup. VSSControl: Freeze state is none."

Here is the output of statistics for VM2-Backup
Code: Select all
5/1/2015 3:21:50 PM :: Job started at 5/1/2015 3:21:48 PM
5/1/2015 3:21:50 PM :: Building VM list
5/1/2015 3:22:11 PM :: VM size: 460.0 GB (64.8 GB used)
5/1/2015 3:22:11 PM :: Changed block tracking is enabled
5/1/2015 3:22:15 PM :: Queued for processing at 5/1/2015 3:22:15 PM
5/1/2015 3:22:15 PM :: Required backup infrastructure resources have been assigned
5/1/2015 3:22:15 PM :: Preparing next VM for processing
5/1/2015 3:22:38 PM :: Creating storage snapshot
5/1/2015 3:22:48 PM :: Preparing next VM for processing
5/1/2015 3:22:53 PM :: All VMs have been queued for processing
5/1/2015 3:22:53 PM :: Processing VM2 Error: Failed to prepare VM for processing.
5/1/2015 3:23:01 PM :: Deleting storage snapshot
5/1/2015 3:24:02 PM :: Job finished with error at 5/1/2015 3:24:02 PM

Statistics for VM3-Backup
5/1/2015 3:22:18 PM :: VMware Tools are not up to date, some functionality
may be affected (this only shows on one of the jobs)
5/1/2015 3:22:23 PM :: Inventorying guest system
5/1/2015 3:22:30 PM :: Failed to prepare guest for hot backup. Error: VSSControl: Freeze state is none
5/1/2015 3:22:48 PM :: Queued for processing at 5/1/2015 3:22:48 PM
5/1/2015 3:22:48 PM :: Required backup infrastructure resources have been assigned
5/1/2015 3:22:53 PM :: VM processing started at 5/1/2015 3:22:53 PM
5/1/2015 3:22:53 PM :: VM size: 460.0 GB (64.8 GB used)
5/1/2015 3:22:55 PM :: Error: Failed to prepare VM for processing.
5/1/2015 3:22:55 PM :: Network traffic verification detected no corrupted blocks
5/1/2015 3:22:55 PM :: Processing finished with errors at 5/1/2015 3:22:55PM
cvacanti
Enthusiast
 
Posts: 35
Liked: 4 times
Joined: Fri Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti

Re: Job hangs - 0%

Veeam Logoby Vitaliy S. » Mon May 18, 2015 9:53 pm

Do you have any error messages in Windows Event log at the time when freeze operation fails?
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Job hangs - 0%

Veeam Logoby cvacanti » Tue May 19, 2015 7:30 pm

No event log errors. I've worked with support as well to remove the veeam vss support folders on each VM and then reboot and try the jobs again. No change.
cvacanti
Enthusiast
 
Posts: 35
Liked: 4 times
Joined: Fri Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti

Re: Job hangs - 0%

Veeam Logoby Vitaliy S. » Tue May 19, 2015 9:18 pm

Need to review VSS logs for understanding why guest OS cannot freeze itself for backup. Please keep working with our support team, as there is little we can say without seeing full debug logs from the job.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Job hangs - 0%

Veeam Logoby cvacanti » Wed May 20, 2015 1:15 am 1 person likes this post

We are setting up another webex with support for Thursday to review where things are at.
cvacanti
Enthusiast
 
Posts: 35
Liked: 4 times
Joined: Fri Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti

Re: Job hangs - 0%

Veeam Logoby rzuber » Thu May 21, 2015 9:07 am

I have the same issue.
Exchange VM on Hyper-V
If i enable application-aware processing the job sits there at 0% forever blocking all the other jobs.
I have to kill the Veem.Backup.Manager process to unfreeze the situation.
Had 2 webex sessions with Veeam Support and uploaded logs but there is no indication of what the problem is.
We need the application-aware processing to process the Exchange Log.
As a workaround we do local backup of the Exchange and Veeam backup without application-aware processing
rzuber
Novice
 
Posts: 3
Liked: 1 time
Joined: Thu May 21, 2015 9:02 am

Re: Job hangs - 0%

Veeam Logoby cvacanti » Thu May 21, 2015 1:38 pm

Thanks rzuber - we've had this case open for a month with no success. I asked for an escalation and was told the tech I was working with was the escalation. Now the case is over to another escalations tech with another webex scheduled for today.

I think it's important to note - this was working for months. And then just stopped...no knows why yet. I'll update after my webex this afternoon.
cvacanti
Enthusiast
 
Posts: 35
Liked: 4 times
Joined: Fri Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti

Re: Job hangs - 0%

Veeam Logoby Vitaliy S. » Thu May 21, 2015 5:01 pm

@rzuber, can you post your case ID, so that we could use it during internal discussions with QA team (if needed)?
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Job hangs - 0%

Veeam Logoby cvacanti » Wed May 27, 2015 7:29 pm 1 person likes this post

Another long call with support today. Turning the issue over to development. Not sure how this was working, then broke, now we need a hot fix coded.
cvacanti
Enthusiast
 
Posts: 35
Liked: 4 times
Joined: Fri Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti

Re: Job hangs - 0%

Veeam Logoby rzuber » Mon Jun 01, 2015 1:14 pm 1 person likes this post

Veeam Support - Case # 00891649
we just had webex and figured out that the dynamic port range on the Windows 2012 is from 6005 till 59530 and was not fully open from the backup server to the backed up Hyper-V VM.
But the same problem persists after opening the port range from 6005 till 65535.

The secondary issue is that while this jobs is hanging, no other jobs can run with a message "Resource not ready: Hyper-V backup proxy" but we have it configured for 6 parallel jobs on Host and on Datastore.
rzuber
Novice
 
Posts: 3
Liked: 1 time
Joined: Thu May 21, 2015 9:02 am

Re: Job hangs - 0%

Veeam Logoby foggy » Mon Jun 01, 2015 4:39 pm

rzuber wrote:The secondary issue is that while this jobs is hanging, no other jobs can run with a message "Resource not ready: Hyper-V backup proxy" but we have it configured for 6 parallel jobs on Host and on Datastore.

How many VMs/disks does this job have in it?
foggy
Veeam Software
 
Posts: 14743
Liked: 1081 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Job hangs - 0%

Veeam Logoby cvacanti » Thu Jun 04, 2015 5:43 pm

So my case was escalated to development. What I find agonizingly frustrating is that this was working and then stopped. They did provide a patch up to v 2029 - and that didn't fix it.
cvacanti
Enthusiast
 
Posts: 35
Liked: 4 times
Joined: Fri Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti

Re: Job hangs - 0%

Veeam Logoby rzuber » Tue Jun 16, 2015 11:28 am

We have stopped trying to enable the Application Aware Processing.
Instead we have enabled Hyper-V guest quiescing and that looks like it triggers the purge of the exchange logs.
I have closed the case with that.
rzuber
Novice
 
Posts: 3
Liked: 1 time
Joined: Thu May 21, 2015 9:02 am

Re: Job hangs - 0%

Veeam Logoby cvacanti » Wed Jun 24, 2015 6:21 pm 1 person likes this post

Ok - so after multiple support techs we are able to process a sql server with multiple instances and IPs. There was a reg edit that support had us do to solve that. Now the job stalls at truncating sql transaction logs. I've sent that to support too for help.
cvacanti
Enthusiast
 
Posts: 35
Liked: 4 times
Joined: Fri Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti

Re: Job hangs - 0%

Veeam Logoby MrBero » Mon Aug 17, 2015 2:23 pm

My problem is very similar
I have migrated my 2 VM from one host to another with "quick migration" option.
After this migration my VM's didn't backup, I am using "enable application-aware processing" option, after unchecking this option both machines backing well.. but I need this option...
I don't know how to resolve this problem please help me :(

OS: windows 2012 R2
Veeam version: Veeam Backup & Replication 8.0.0.2030

my VM:
1. windows 2008 r2 SP1
2. windows 7 Sp1

Image
MrBero
Novice
 
Posts: 4
Liked: 1 time
Joined: Wed Nov 26, 2014 4:57 pm
Full Name: Mr Bero

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 12 guests