job error SQL Server is not available after upgrade Veeam9

[svp007]

Hi

Since 2 weeks we upgraded from Veeam 8 to Veeam 9.
Since the upgrade we had 1 backup-job and 4 backup-jobs failing with the error "SQL Server is not available" (local Veeam configuration database).
So it is not Always the case.
Extract from the detailed Veeam logs :

Code: Select all

[13.03.2016 16:12:07] <01> Error    Microsoft SQL server hosting the configuration database is currently unavailable. Possible reasons are heavy load, networking issue, server reboot, or hot backup.
[13.03.2016 16:12:07] <01> Error    Please wait, and try again later.
[13.03.2016 16:12:07] <01> Error    Error information:
[13.03.2016 16:12:07] <01> Error    Microsoft SQL server hosting the configuration database is currently unavailable. Possible reasons are heavy load, networking issue, server reboot, or hot backup.
[13.03.2016 16:12:07] <01> Error    Please wait, and try again later.
[13.03.2016 16:12:07] <01> Error    Error information:
[13.03.2016 16:12:07] <01> Error    SQL server is not available
[13.03.2016 16:12:07] <01> Error    SQL server is not available (Veeam.Backup.Common.CSqlException)
[13.03.2016 16:12:07] <01> Error       at Veeam.Backup.Common.CExceptionUtil.ThrowSqlException(Exception innerException)

Our Veeam backupserver use the default local SQL Server Express database (no separate SQL Server).
For some reason sometimes the Veeam backup service is unable to connect to its own local SQL Server database.
We have never had this kind of error when using Veeam 8 (on the same backup server; we did an in-place upgrade).

I wondered if any of you also had this problem after upgrading to Veeam 9.
Has perhaps some timeout-setting been made more strict in Veeam 9 ? Why is not retrying the connection ?
Know issue ? sny solutions to avoid this ?

Best regards

Stefaan

[foggy]

Stefaan, please review this thread for some hints and contact support for a closer investigation (support case ID is always required when posting about technical issues).

[svp007]

I do not quite see how referencing a Veeam7 (!) post would help an error occuring after Veeam9 upgrade when we did not have the issues in Veeam8.
I can assure you the Veeam-backup server has 8 GB of RAM (almost 4 GB free) and is a VM with 4 vCPU; it is not underdimensioned.

The problem occured AFTER upgrading to Veeam 9 so please focus on that aspect.
You also do not react at all to the specific points I mentionned :

- Has perhaps some timeout-setting been made more strict in Veeam 9 ?
- Why is not retrying the connection ?
- Know issue ? any solutions to avoid this ?

I will open a support case and hopefully be taken more seriously.
Case ID : # 01730258

Stefaan

[foggy]

My intention was to show one of the possible reasons for SQL Server unavailability, possible memory issues are actually not Veeam B&R version-related. In other cases using SQL authentication solved similar issues. The actual reason can be defined only after reviewing the log files, that's why I directed you to support (note that this is not a support forum at all).

Btw, do you see any SQL Server related messages in Event Viewer?

[svp007]

No SQL Server related messages in Event Viewer.
No specific errors in the SQL Server logs (C:\Program Files\Microsoft SQL Server\MSSQL11.VEEAMSQL2012\MSSQL\Log\ERRORLOG.*)

[briguyiu]

Same thing is happening to me as well....Already switched to SQL Authentication...

Error: SQL server is not available

It's also local to the server, and doesn't happen every night, and rarely on an entire job. Most of the time it's just one VM in a job, and upon retry it works.

I have 8 GB of RAM and 12 CPUs in this thing, so I hardly think resources are to blame.

What's even odder is that in the location for SQL logs, there is NOTHING for today when it JUST happened...but there is stuff from a few days ago...

Code: Select all

2016-03-19 21:24:57.10 Logon       Error: 17189, Severity: 16, State: 1.
2016-03-19 21:24:57.10 Logon       SQL Server failed with error code 0xc0000000 to spawn a thread to process a new login or connection. Check the SQL Server error log and the Windows event logs for information about possible related problems. [CLIENT: <local machine>]
2016-03-19 21:24:57.10 Logon       Error: 17189, Severity: 16, State: 1.
2016-03-19 21:24:57.10 Logon       SQL Server failed with error code 0xc0000000 to spawn a thread to process a new login or connection. Check the SQL Server error log and the Windows event logs for information about possible related problems. [CLIENT: <local machine>]
2016-03-19 21:24:57.10 Logon       Error: 17189, Severity: 16, State: 1.
2016-03-19 21:24:57.10 Logon       SQL Server failed with error code 0xc0000000 to spawn a thread to process a new login or connection. Check the SQL Server error log and the Windows event logs for information about possible related problems. [CLIENT: <local machine>]
2016-03-19 21:24:57.12 Logon       Error: 17189, Severity: 16, State: 1.
2016-03-19 21:24:57.12 Logon       SQL Server failed with error code 0xc0000000 to spawn a thread to process a new login or connection. Check the SQL Server error log and the Windows event logs for information about possible related problems. [CLIENT: <local machine>]
2016-03-21 21:20:41.27 spid71      Error: 7884, Severity: 20, State: 1. (Params:). The error is printed in terse mode because there was error during formatting. Tracing, ETW, notifications etc are skipped.
2016-03-21 21:40:50.23 spid76      Error: 701, Severity: 17, State: 123.
2016-03-21 21:40:50.23 spid76      There is insufficient system memory in resource pool 'internal' to run this query.
2016-03-21 21:40:50.23 spid58      Error: 701, Severity: 17, State: 219.
2016-03-21 21:40:50.23 spid58      There is insufficient system memory in resource pool 'internal' to run this query.
2016-03-21 21:40:50.23 spid64      Error: 701, Severity: 17, State: 123.
2016-03-21 21:40:50.23 spid64      There is insufficient system memory in resource pool 'internal' to run this query.
2016-03-21 21:40:50.26 Logon       Error: 17189, Severity: 16, State: 1.
2016-03-21 21:40:50.26 Logon       SQL Server failed with error code 0xc0000000 to spawn a thread to process a new login or connection. Check the SQL Server error log and the Windows event logs for information about possible related problems. [CLIENT: 172.31.16.74]
2016-03-21 21:40:50.26 Logon       Error: 17189, Severity: 16, State: 1.
2016-03-21 21:40:50.26 Logon       SQL Server failed with error code 0xc0000000 to spawn a thread to process a new login or connection. Check the SQL Server error log and the Windows event logs for information about possible related problems. [CLIENT: 172.31.16.74]
2016-03-21 21:40:50.26 Logon       Error: 17189, Severity: 16, State: 1.
2016-03-21 21:40:50.26 Logon       SQL Server failed with error code 0xc0000000 to spawn a thread to process a new login or connection. Check the SQL Server error log and the Windows event logs for information about possible related problems. [CLIENT: 172.31.16.74]
2016-03-21 21:40:50.26 Logon       Error: 17189, Severity: 16, State: 1.

Also, see this is the event viewer for the 21st, but not today....even though the same error occured???

SQL Server failed with error code 0xc0000000 to spawn a thread to process a new login or connection. Check the SQL Server error log and the Windows event logs for information about possible related problems. [CLIENT: 172.31.16.74]

[foggy]

Brian, do you have a case open for this issue as well? Stefaan's case is still under investigation.

[briguyiu]

Not yet....planning on getting one in there today hopefully.

[vBalmer]

I had this issue as well. Two things fixed it for me:

- Changing the 4 vCPUs on the Veeam VM from 4 sockets to 1 socket with 4 cores (SQL Express only uses 1 socket). This fixed the majority of the time it occurred.
- Splitting out Enterprise Manager into its own VM. During backups the CPU and RAM max out (4 vCPU, 12GB RAM) and it was choking. This seemed to give the backup the extra room it needed. This fixed the few times it would still happen after fixing the vCPU.

[briguyiu]

I had no idea about SQL express and only seeing 1 socket. Just changed it to 2 sockets 4 cores each vs. 8 sockets. We'll see if that fixes the issue!

[svp007]

After spending much time debugging the problem (support case # 01730258) a solution has been to avoid the problem.

The Logon Account for "Veeam backup Service" = LocalService (by default). This was also the case in Veeam8.
Solution proposed by Veeam support: modify Logon Account for that particular service to a DomainAdmin-account and restart the service.
We modified this on our 4 VeeamBackup-servers and the problem did not re-occur during last 5 days (we keep our fingers crossed).

I must admit I do not really understand why this solves the problem since the "Veeam backup Service" is communication with al local SQL*Server Express.
In Veeam8 the situation was exactly the same but in Veeam9 it poses a problem.

[BGA-Robert]

We had a similar issue (case 01713366), but had seen SQL Server not available on v8 too.
Solution was to increase memory and to throttle Veeam proxy
I was told that we need to provision 2 GB RAM for each proc
My Feeling is that I also needed to leave a little headroom for the SQL server
Currently, our backup server is running on VMware, 16 procs, 32GB RAM
We were seeing consistent SQL errors at 16GB RAM (and 16 procs)
Veeam proxy is set to 12 tasks. I'm leaving a couple procs for other tasks, like the SQL server

We push our backup server - lots of VM's, lots of data. Lots of concurrent usage

Lesson learned: Veeam needs more resources