Discussions specific to the VMware vSphere hypervisor
Post Reply
cvacanti
Enthusiast
Posts: 35
Liked: 4 times
Joined: Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti
Contact:

Job hangs - 0%

Post by cvacanti » May 18, 2015 7:31 pm

Support case 00887944 - Opened April 23

Hoping someone can assist - I've had this case open with support for almost a month. I've uploaded multiple log bundles, we've done webex sessions, and nothing yet is a smoking gun why these jobs fail. It should be known that other jobs on the Veeam server are running successfully. DR jobs are working too. I've left Backup Job1 running and it is sitting at 161 hours run time currently with 0% processed.

Backup Job1 - 3VMs - yes use storage integration w netapp, no do not fail over to network backups, yes use application aware processing, no do not index guest file system
VM1 - MS SQL server
VM2 - MS App server
VM3 - MS App server

If I create three new backup jobs they work because I did not choose to use application aware processing and yes use storage snapshots.
VM1-Backup - ran this job 100% successful with no checks for application aware processing or indexing.
VM2-Backup - ran this job 100% successful with no checks for application aware processing or indexing.
VM3-Backup - ran this job 100% successful with no checks for application aware processing or indexing.

If I change the individual backup jobs to use application aware processing things fail:
VM1-Backup - job hangs at 0% with this entry: 5/1/2015 2:59:31 PM :: Inventorying guest system. In order to stop this job I have to kill the Veeam processes in Task Manager.
VM2-Backup - Job fails with "Failed to prepare guest for hot backup. VSSControl: Freeze state is none."
VM3-Backup - Job fails with "Failed to prepare guest for hot backup. VSSControl: Freeze state is none."

Here is the output of statistics for VM2-Backup

Code: Select all

5/1/2015 3:21:50 PM :: Job started at 5/1/2015 3:21:48 PM
5/1/2015 3:21:50 PM :: Building VM list
5/1/2015 3:22:11 PM :: VM size: 460.0 GB (64.8 GB used)
5/1/2015 3:22:11 PM :: Changed block tracking is enabled
5/1/2015 3:22:15 PM :: Queued for processing at 5/1/2015 3:22:15 PM
5/1/2015 3:22:15 PM :: Required backup infrastructure resources have been assigned
5/1/2015 3:22:15 PM :: Preparing next VM for processing
5/1/2015 3:22:38 PM :: Creating storage snapshot
5/1/2015 3:22:48 PM :: Preparing next VM for processing
5/1/2015 3:22:53 PM :: All VMs have been queued for processing
5/1/2015 3:22:53 PM :: Processing VM2 Error: Failed to prepare VM for processing.
5/1/2015 3:23:01 PM :: Deleting storage snapshot
5/1/2015 3:24:02 PM :: Job finished with error at 5/1/2015 3:24:02 PM

Statistics for VM3-Backup
5/1/2015 3:22:18 PM :: VMware Tools are not up to date, some functionality
may be affected (this only shows on one of the jobs)
5/1/2015 3:22:23 PM :: Inventorying guest system
5/1/2015 3:22:30 PM :: Failed to prepare guest for hot backup. Error: VSSControl: Freeze state is none
5/1/2015 3:22:48 PM :: Queued for processing at 5/1/2015 3:22:48 PM
5/1/2015 3:22:48 PM :: Required backup infrastructure resources have been assigned
5/1/2015 3:22:53 PM :: VM processing started at 5/1/2015 3:22:53 PM
5/1/2015 3:22:53 PM :: VM size: 460.0 GB (64.8 GB used)
5/1/2015 3:22:55 PM :: Error: Failed to prepare VM for processing.
5/1/2015 3:22:55 PM :: Network traffic verification detected no corrupted blocks
5/1/2015 3:22:55 PM :: Processing finished with errors at 5/1/2015 3:22:55PM

Vitaliy S.
Product Manager
Posts: 23006
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job hangs - 0%

Post by Vitaliy S. » May 18, 2015 9:53 pm

Do you have any error messages in Windows Event log at the time when freeze operation fails?

cvacanti
Enthusiast
Posts: 35
Liked: 4 times
Joined: Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti
Contact:

Re: Job hangs - 0%

Post by cvacanti » May 19, 2015 7:30 pm

No event log errors. I've worked with support as well to remove the veeam vss support folders on each VM and then reboot and try the jobs again. No change.

Vitaliy S.
Product Manager
Posts: 23006
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job hangs - 0%

Post by Vitaliy S. » May 19, 2015 9:18 pm

Need to review VSS logs for understanding why guest OS cannot freeze itself for backup. Please keep working with our support team, as there is little we can say without seeing full debug logs from the job.

cvacanti
Enthusiast
Posts: 35
Liked: 4 times
Joined: Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti
Contact:

Re: Job hangs - 0%

Post by cvacanti » May 20, 2015 1:15 am 1 person likes this post

We are setting up another webex with support for Thursday to review where things are at.

rzuber
Novice
Posts: 3
Liked: 1 time
Joined: May 21, 2015 9:02 am
Contact:

Re: Job hangs - 0%

Post by rzuber » May 21, 2015 9:07 am

I have the same issue.
Exchange VM on Hyper-V
If i enable application-aware processing the job sits there at 0% forever blocking all the other jobs.
I have to kill the Veem.Backup.Manager process to unfreeze the situation.
Had 2 webex sessions with Veeam Support and uploaded logs but there is no indication of what the problem is.
We need the application-aware processing to process the Exchange Log.
As a workaround we do local backup of the Exchange and Veeam backup without application-aware processing

cvacanti
Enthusiast
Posts: 35
Liked: 4 times
Joined: Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti
Contact:

Re: Job hangs - 0%

Post by cvacanti » May 21, 2015 1:38 pm

Thanks rzuber - we've had this case open for a month with no success. I asked for an escalation and was told the tech I was working with was the escalation. Now the case is over to another escalations tech with another webex scheduled for today.

I think it's important to note - this was working for months. And then just stopped...no knows why yet. I'll update after my webex this afternoon.

Vitaliy S.
Product Manager
Posts: 23006
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job hangs - 0%

Post by Vitaliy S. » May 21, 2015 5:01 pm

@rzuber, can you post your case ID, so that we could use it during internal discussions with QA team (if needed)?

cvacanti
Enthusiast
Posts: 35
Liked: 4 times
Joined: Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti
Contact:

Re: Job hangs - 0%

Post by cvacanti » May 27, 2015 7:29 pm 1 person likes this post

Another long call with support today. Turning the issue over to development. Not sure how this was working, then broke, now we need a hot fix coded.

rzuber
Novice
Posts: 3
Liked: 1 time
Joined: May 21, 2015 9:02 am
Contact:

Re: Job hangs - 0%

Post by rzuber » Jun 01, 2015 1:14 pm 1 person likes this post

Veeam Support - Case # 00891649
we just had webex and figured out that the dynamic port range on the Windows 2012 is from 6005 till 59530 and was not fully open from the backup server to the backed up Hyper-V VM.
But the same problem persists after opening the port range from 6005 till 65535.

The secondary issue is that while this jobs is hanging, no other jobs can run with a message "Resource not ready: Hyper-V backup proxy" but we have it configured for 6 parallel jobs on Host and on Datastore.

foggy
Veeam Software
Posts: 18287
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Job hangs - 0%

Post by foggy » Jun 01, 2015 4:39 pm

rzuber wrote:The secondary issue is that while this jobs is hanging, no other jobs can run with a message "Resource not ready: Hyper-V backup proxy" but we have it configured for 6 parallel jobs on Host and on Datastore.
How many VMs/disks does this job have in it?

cvacanti
Enthusiast
Posts: 35
Liked: 4 times
Joined: Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti
Contact:

Re: Job hangs - 0%

Post by cvacanti » Jun 04, 2015 5:43 pm

So my case was escalated to development. What I find agonizingly frustrating is that this was working and then stopped. They did provide a patch up to v 2029 - and that didn't fix it.

rzuber
Novice
Posts: 3
Liked: 1 time
Joined: May 21, 2015 9:02 am
Contact:

Re: Job hangs - 0%

Post by rzuber » Jun 16, 2015 11:28 am

We have stopped trying to enable the Application Aware Processing.
Instead we have enabled Hyper-V guest quiescing and that looks like it triggers the purge of the exchange logs.
I have closed the case with that.

cvacanti
Enthusiast
Posts: 35
Liked: 4 times
Joined: Feb 28, 2014 7:36 pm
Full Name: Chris Vacanti
Contact:

Re: Job hangs - 0%

Post by cvacanti » Jun 24, 2015 6:21 pm 1 person likes this post

Ok - so after multiple support techs we are able to process a sql server with multiple instances and IPs. There was a reg edit that support had us do to solve that. Now the job stalls at truncating sql transaction logs. I've sent that to support too for help.

MrBero
Novice
Posts: 4
Liked: 1 time
Joined: Nov 26, 2014 4:57 pm
Full Name: Mr Bero
Contact:

Re: Job hangs - 0%

Post by MrBero » Aug 17, 2015 2:23 pm

My problem is very similar
I have migrated my 2 VM from one host to another with "quick migration" option.
After this migration my VM's didn't backup, I am using "enable application-aware processing" option, after unchecking this option both machines backing well.. but I need this option...
I don't know how to resolve this problem please help me :(

OS: windows 2012 R2
Veeam version: Veeam Backup & Replication 8.0.0.2030

my VM:
1. windows 2008 r2 SP1
2. windows 7 Sp1

Image

foggy
Veeam Software
Posts: 18287
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Job hangs - 0%

Post by foggy » Aug 17, 2015 4:37 pm

Have you contacted support already? Logs investigation is required here.

caitlin.butler
Novice
Posts: 3
Liked: never
Joined: Dec 23, 2014 8:13 pm
Full Name: Caitlin Butler
Contact:

Re: Job hangs - 0%

Post by caitlin.butler » Jun 08, 2016 3:04 pm

I have an update regarding this post -- The original ticket Support case 00887944 - Opened April 23, ended without resolution as the Customer that opened the ticket stopped responding to find root cause. They ended up applying a few registry values that allowed Application Aware processing to work again, but they couldn't get SQL Log truncation to work.

I have worked with a Customer who opened a ticket recently for a similar issue who referenced back to this ticket, and they are on Version 9 (OC was V8 when issue was worked on) and the suggested registry key fix resolved Application Aware Processing as well as SQL Log Truncation it seems so far. We are monitoring right now, but the registry key suggested seems to have worked well.

The newer ticket with the noted fix is Case 0181928. If you feel you are experiencing a similar issue, please feel free to open a Support Ticket and reference Case 01819282 or this forum post and our Support Team can provide the suggested fix.

PKaufmann
Enthusiast
Posts: 36
Liked: 5 times
Joined: Oct 05, 2016 8:00 am
Contact:

Re: Job hangs - 0%

Post by PKaufmann » Mar 20, 2018 2:15 pm

what Reg key fix did you use ?

Vitaliy S.
Product Manager
Posts: 23006
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Job hangs - 0%

Post by Vitaliy S. » Mar 21, 2018 4:56 pm

Hello PKaufmann. To make sure that you're experiencing exactly the same issue (should be clear after reviewing job logs), please contact our technical team directly and reference the support case ID mentioned above.

PKaufmann
Enthusiast
Posts: 36
Liked: 5 times
Joined: Oct 05, 2016 8:00 am
Contact:

Re: Job hangs - 0%

Post by PKaufmann » Mar 21, 2018 8:40 pm 1 person likes this post

done, case 02689863

Post Reply

Who is online

Users browsing this forum: No registered users and 29 guests