Application aware processing causes job to stall at the star

[pidthepiper]

So I have a support ticket raised with Veeam (Case # 01237802), as a job just starts and then freezes. Only recently has it done this. The job never completes and just stays at 0%

I have tried cloning the job, creating a new job with the same vms or a variation of vms and they all do the same. all my other jobs are fine.

As soon as I disable application aware processing the job kicks off fine and runs.

The database is running on SQL 2008 R2 on an external DB VM.

Support are keen for me to upgrade to sql 2012 which I am sure has plenty of new features, but I dont want to overhaul a major db server over this issue. Its in the pipe line in the future but I dont believe its needed yet.

http://www.veeam.com/kb1995 I have done all this.

I have been told application aware processing puts extra load on the database etc, but I can see the current loads and there is nothing making me go oh nooooooooo!

So has anyone any ideas, I dont want to run the job without application aware processing (DC,Exchange) but also its a feature we pay for and should use, and is being used currently and was used by this job for a while before.

I am open to any ideas?

[pidthepiper]

I have created a new database restored a configuration backup, re-scanned by repositories and still the job fails to run. If I disable application aware and use vss and disable cbt it runs.

The fact that I cant run it with application aware is puzzling and support have gone pretty quite

[foggy]

Bilal, does it get stuck on the same stage in the job session log each time (and what stage it is, if yes)? Is it always the same VM? What if you remove this VM from the job, if yes?

[pidthepiper]

It doesnt matter what Vm it is in that job, I have tried creating a new job with a different VM fromt he same bunch and the issue still happens.

I ran the job last night using just normal VSS and no application aware and it ran, this morning I have tried to enable application aware and it freezes at the same point again

I have then re run the job using vmtools vss only and enabled cbt tracking and the job runs, creates it snap shot.

I have tested the application aware credentials and they are fine.

I would like this support case escalating now

[foggy]

At what point does it freeze (what do you see in the job session log last)? What kind of VMs are in the job?

[Vitaliy S.]

SQL Server hosting Veeam B&R database has nothing to do with this issue. If I understood your post correctly, your backup jobs freeze as soon as you enable application-aware image processing for the VMs in the job. What applications do you have installed in these VMs? Is there any VM without SQL server/Exchange, so you could try the AAIP option against it?

[pidthepiper]

Hi there, I agree it doesn't look sql related but the support engineer was very much trying to make me install a whole new sql 2012 build?!

it is an AD controllers and an exchange server, and a couple of non descriptive vms, no matter how I mix them the job stalls with application aware processing enabled. From what I can tell it doesn't even log into the VM?

It stalls at the getting VM info from vSphere. I have supplied endless logs to support and I do feel like I am hitting a brick wall. They did say it could be windows update related. But group policy has it disabled I have even stopped the update service and we use WSUS to download but not install any updates.

I am trying to stress to support that the job can run without application aware processing but as far as I am concerned this isn't a valid backup and it is unacceptable.

What is the AAIP option? I am open to try anything

I am asking for this to be escalated a second time with support now,as I am pretty disappointed with their response so far

[foggy]

Escalation is the right step in this case.

You can read about AAIP here, it is indeed recommended to have it enabled for all the VMs running VSS-aware applications. However running the job without this option enabled doesn't mean the backup is not valid, it is a question of VM state consistency and when different levels of it are acceptable.

[pidthepiper]

Oh AAIp is short hand for application aware processing. it was running perfectly using application aware till last week, when I raised my initial call. When I say the backup isn't valid, I mean I am going from fully application aware to not being application aware, which to me is a step in the wrong direction, and I want to get to the bottom of why this has happened

[foggy]

You definitely need to get to the bottom of this issue and I believe our support engineers will be able to help you with this, once you escalate the case.

[pidthepiper]

I have emailed the current support engineer/support email address stating my original issue is still unresolved and that I would like it escalating a second time and I have provided more logs to their ftp.

[pidthepiper]

Update:

Supported asked me to do:
InverseVssProtocolOrder
Type: REG_DWORD
Value = 1

To the HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ on a Veeam server.

this appears to have worked. Is there a detailed Kb about what it does, from initial look it appears it reverse the order in which application aware backups run?

[foggy]

Yes, what it does is inverse the order of protocols used to connect to VM for VSS processing (by default, Veeam B&R attempts to use network and then VIX, if network connection is unavailable). With that registry value set to 1, it attempts VIX first. So probably something has changed in your environment recently that prevents network connection to VM and/or proper failover to VIX.

[pidthepiper]

Well its solved that issue, and the backups are running as they should!

Horrayyyyyyyyyyyyyyyyyyyyyyyyyyyy