Job hangs - 0%

[cvacanti]

Support case 00887944 - Opened April 23

Hoping someone can assist - I've had this case open with support for almost a month. I've uploaded multiple log bundles, we've done webex sessions, and nothing yet is a smoking gun why these jobs fail. It should be known that other jobs on the Veeam server are running successfully. DR jobs are working too. I've left Backup Job1 running and it is sitting at 161 hours run time currently with 0% processed.

Backup Job1 - 3VMs - yes use storage integration w netapp, no do not fail over to network backups, yes use application aware processing, no do not index guest file system
VM1 - MS SQL server
VM2 - MS App server
VM3 - MS App server

If I create three new backup jobs they work because I did not choose to use application aware processing and yes use storage snapshots.
VM1-Backup - ran this job 100% successful with no checks for application aware processing or indexing.
VM2-Backup - ran this job 100% successful with no checks for application aware processing or indexing.
VM3-Backup - ran this job 100% successful with no checks for application aware processing or indexing.

If I change the individual backup jobs to use application aware processing things fail:
VM1-Backup - job hangs at 0% with this entry: 5/1/2015 2:59:31 PM :: Inventorying guest system. In order to stop this job I have to kill the Veeam processes in Task Manager.
VM2-Backup - Job fails with "Failed to prepare guest for hot backup. VSSControl: Freeze state is none."
VM3-Backup - Job fails with "Failed to prepare guest for hot backup. VSSControl: Freeze state is none."

Here is the output of statistics for VM2-Backup

Code: Select all

5/1/2015 3:21:50 PM :: Job started at 5/1/2015 3:21:48 PM
5/1/2015 3:21:50 PM :: Building VM list
5/1/2015 3:22:11 PM :: VM size: 460.0 GB (64.8 GB used)
5/1/2015 3:22:11 PM :: Changed block tracking is enabled
5/1/2015 3:22:15 PM :: Queued for processing at 5/1/2015 3:22:15 PM
5/1/2015 3:22:15 PM :: Required backup infrastructure resources have been assigned
5/1/2015 3:22:15 PM :: Preparing next VM for processing
5/1/2015 3:22:38 PM :: Creating storage snapshot
5/1/2015 3:22:48 PM :: Preparing next VM for processing
5/1/2015 3:22:53 PM :: All VMs have been queued for processing
5/1/2015 3:22:53 PM :: Processing VM2 Error: Failed to prepare VM for processing.
5/1/2015 3:23:01 PM :: Deleting storage snapshot
5/1/2015 3:24:02 PM :: Job finished with error at 5/1/2015 3:24:02 PM

Statistics for VM3-Backup
5/1/2015 3:22:18 PM :: VMware Tools are not up to date, some functionality
may be affected (this only shows on one of the jobs)
5/1/2015 3:22:23 PM :: Inventorying guest system
5/1/2015 3:22:30 PM :: Failed to prepare guest for hot backup. Error: VSSControl: Freeze state is none
5/1/2015 3:22:48 PM :: Queued for processing at 5/1/2015 3:22:48 PM
5/1/2015 3:22:48 PM :: Required backup infrastructure resources have been assigned
5/1/2015 3:22:53 PM :: VM processing started at 5/1/2015 3:22:53 PM
5/1/2015 3:22:53 PM :: VM size: 460.0 GB (64.8 GB used)
5/1/2015 3:22:55 PM :: Error: Failed to prepare VM for processing.
5/1/2015 3:22:55 PM :: Network traffic verification detected no corrupted blocks
5/1/2015 3:22:55 PM :: Processing finished with errors at 5/1/2015 3:22:55PM

[Vitaliy S.]

Do you have any error messages in Windows Event log at the time when freeze operation fails?

[cvacanti]

No event log errors. I've worked with support as well to remove the veeam vss support folders on each VM and then reboot and try the jobs again. No change.

[Vitaliy S.]

Need to review VSS logs for understanding why guest OS cannot freeze itself for backup. Please keep working with our support team, as there is little we can say without seeing full debug logs from the job.

[cvacanti]

We are setting up another webex with support for Thursday to review where things are at.

[rzuber]

I have the same issue.
Exchange VM on Hyper-V
If i enable application-aware processing the job sits there at 0% forever blocking all the other jobs.
I have to kill the Veem.Backup.Manager process to unfreeze the situation.
Had 2 webex sessions with Veeam Support and uploaded logs but there is no indication of what the problem is.
We need the application-aware processing to process the Exchange Log.
As a workaround we do local backup of the Exchange and Veeam backup without application-aware processing

[cvacanti]

Thanks rzuber - we've had this case open for a month with no success. I asked for an escalation and was told the tech I was working with was the escalation. Now the case is over to another escalations tech with another webex scheduled for today.

I think it's important to note - this was working for months. And then just stopped...no knows why yet. I'll update after my webex this afternoon.

[Vitaliy S.]

@rzuber, can you post your case ID, so that we could use it during internal discussions with QA team (if needed)?

[cvacanti]

Another long call with support today. Turning the issue over to development. Not sure how this was working, then broke, now we need a hot fix coded.

[rzuber]

Veeam Support - Case # 00891649
we just had webex and figured out that the dynamic port range on the Windows 2012 is from 6005 till 59530 and was not fully open from the backup server to the backed up Hyper-V VM.
But the same problem persists after opening the port range from 6005 till 65535.

The secondary issue is that while this jobs is hanging, no other jobs can run with a message "Resource not ready: Hyper-V backup proxy" but we have it configured for 6 parallel jobs on Host and on Datastore.

[foggy]

The secondary issue is that while this jobs is hanging, no other jobs can run with a message "Resource not ready: Hyper-V backup proxy" but we have it configured for 6 parallel jobs on Host and on Datastore.

How many VMs/disks does this job have in it?

[cvacanti]

So my case was escalated to development. What I find agonizingly frustrating is that this was working and then stopped. They did provide a patch up to v 2029 - and that didn't fix it.

[rzuber]

We have stopped trying to enable the Application Aware Processing.
Instead we have enabled Hyper-V guest quiescing and that looks like it triggers the purge of the exchange logs.
I have closed the case with that.

[cvacanti]

Ok - so after multiple support techs we are able to process a sql server with multiple instances and IPs. There was a reg edit that support had us do to solve that. Now the job stalls at truncating sql transaction logs. I've sent that to support too for help.

[MrBero]

My problem is very similar
I have migrated my 2 VM from one host to another with "quick migration" option.
After this migration my VM's didn't backup, I am using "enable application-aware processing" option, after unchecking this option both machines backing well.. but I need this option...
I don't know how to resolve this problem please help me

OS: windows 2012 R2
Veeam version: Veeam Backup & Replication 8.0.0.2030

my VM:
1. windows 2008 r2 SP1
2. windows 7 Sp1

[foggy]

Have you contacted support already? Logs investigation is required here.

[caitlin.butler]

I have an update regarding this post -- The original ticket Support case 00887944 - Opened April 23, ended without resolution as the Customer that opened the ticket stopped responding to find root cause. They ended up applying a few registry values that allowed Application Aware processing to work again, but they couldn't get SQL Log truncation to work.

I have worked with a Customer who opened a ticket recently for a similar issue who referenced back to this ticket, and they are on Version 9 (OC was V8 when issue was worked on) and the suggested registry key fix resolved Application Aware Processing as well as SQL Log Truncation it seems so far. We are monitoring right now, but the registry key suggested seems to have worked well.

The newer ticket with the noted fix is Case 0181928. If you feel you are experiencing a similar issue, please feel free to open a Support Ticket and reference Case 01819282 or this forum post and our Support Team can provide the suggested fix.

[PKaufmann]

what Reg key fix did you use ?

[Vitaliy S.]

Hello PKaufmann. To make sure that you're experiencing exactly the same issue (should be clear after reviewing job logs), please contact our technical team directly and reference the support case ID mentioned above.

[PKaufmann]

done, case 02689863