Issues backing up SQL 2012 SP3 with Veeam.. SQLVDI errors

[cerberus]

We have been using Veeam for about 2 months and our backups have been successful, including backing up our SQL Server 2012 SP3 VM.

Over the weekend I upgraded vSphere from 5.1 to 6.0, including our ESXi hypervisors and VMware tools and Hardware Version of each VM to be the latest... Everything seems to work OK, backups are working EXCEPT for our SQL VM... I am getting some very strange errors.. we have other SQL VMs that are backing up fine, just this 1 out of 3 SQL VMs is not backing up and it happens to be the only one on SQL Server 2012 SP3.. the others are SQL 2008.

On Veeam I get..

Code: Select all

12/15/2015 2:53:34 PM :: Unable to release guest. Error: VSSControl: Failed to freeze guest, wait timeout 
12/15/2015 2:54:13 PM :: Error: VSSControl: Failed to freeze guest, wait timeout 

On the SQL VM I get these events..

Code: Select all

SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=2716. Thread=10364. Client. Instance=. VD=Global\{9D5E6728-DF63-4D2D-9456-0DED3B6C95FE}1_SQLVDIMemoryName_0.
BackupVirtualDeviceFile::SendFileInfoBegin:  failure on backup device '{9EA00DDC-5D4F-4853-895B-38E6457B99D8}22'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).
BACKUP failed to complete the command BACKUP DATABASE NPS_Accounting. Check the backup application log for detailed messages.

If I try to backup the databases through the SQL Management Studio or TSQL it works fine.. If I run "vsswriter list writers" right after a failed Veeam backup I see..

Code: Select all

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {99145e76-f525-4d46-ad18-1fb816a23aeb}
   State: [9] Failed
   Last error: Timed out

Any ideas? It is a relatively small SQL VM (32GB/4vCPU) with about 15 databases that are less than 250GB in total on disk.

We tried to increase the max_thread_count to 1024 as suggested by Veeam Support but no luck.. The only thing we did is upgrade our virtual infrastructure stack as I described above and I deployed a new vCenter Server and updated my jobs to use the new vCenter Server.. for my testing I created a dummy backup job that would backup the affected SQL VM so I can reproduce this error quickly.

I am using SAN based backup over FC and leveraging change block tracking. Our Veeam s/w is up to the latest available update for v8.

Any ideas to try? In the mean time I'll run some Windows OS updates and try to reboot the VM after hours to see if that helps..

Support case #01235500

[cerberus]

More digging around and it looks like https://www.veeam.com/kb1051 fixes my issue.

This SQL instance hosts our vCenter Server database, I think because it was also being frozen during VSS backup it caused these types of errors. I included it as a db to be excluded and no errors.

Wohoo