SQL Server network issues during Veeam backup

[shlomia]

Hi,

I have SQL server that is have on going connection and transaction during all day.
I backup him in different times every 1-2 days.
Every time when I start the backup, the SQL server seems to have network issues, I receive this error:

Code: Select all

 ex.TargetSite:System.Data.DataTable ExecuteDataTable(System.String, System.String, System.Data.SqlClient.SqlParameter[])/n ex.InnerException:System.ComponentModel.Win32Exception (0x80004005): [b]The specified network name is no longer available/n ex.Message:A transport-level error has occurred when receiving results from the server[/b]. (provider: [b]TCP Provider, error: 0 - The specified network name is no longer available[/b].)/n StackTrace:   at DAL.InternalClasses.ManageSql.ExecuteDataTable(String connectionString, String procName, SqlParameter[] SqlParams) in

I was wondering, is that normal with Veeam backup? or can I somehow solve this issue?
The SQL server is VM on ESXi 5.5.
Running Windows server 2012 R2 and SQL Server 2014.

Thank you.

[cby]

I've seen this problem at snapshot merge time after the backup has completed but not when the backup starts. It could be the VM stun as the snapshot is merged/consolidated and SQL loses the n/work connection. N/work loss can take seconds during the consolidation with large snapshots (though I've seen minutes in some cases). Nothing to do with Veeam.

I understand that vSphere 6 has reduced snapshot consolidation times dramatically so that may be an option.

[foggy]

Shlomi, do you see similar behavior when taking this VM snapshot manually?

[shlomia]

Hi, I'm sorry for the very late reply.
I just tried to take manual Quiesce snapshot, and seems like the machine is having network issues for the first minutes, or until the snapshot is taken.
After, the machine is working properly while on snapshot.

What does it mean? is it a vmware problem i suppose?
It's a big problem for me, because I have transactions and connections to this SQL Server all the time during the day.

[foggy]

Yes, it's a typical behavior on VMware snapshot creation/deletion.

[skrause]

You are going to have stuns at VMware snapshot removal on SQL, especially with highly transactional database servers as there are a lot of changes to consolidate in the removal process.

We got around this by having Always On synchronous replication between two servers with active-failover enabled. We then chain the jobs so both SQL servers are not running on snapshots (or removing them) at the same time, when the cluster loses contact with the "active" server it fails connections over to the other server and then fails back once the snapshot stun is over.

But, you need to have Enterprise SQL licensing for that which is not an inexpensive proposition.

[zuldan]

We have a 2Tb DB in a SQL 2016 AOAG that VMware snapshots (ESXi v6.0 U3) were causing havoc with. We resolved the issue by increasing the SameSubnetDelay setting from 1000 to 2000 using Powershell. This won't help the OP but hopefully will with people running Availability Groups.

(get-cluster).SameSubnetDelay = 2000

See for more info http://dbinsight.com.au/dbinsight/vmwar ... up-cluster

[DaveWatkins]

VMWare 6 also made this stun much better. That's probably the easiest thing to do, if it's still an issue then the fixes get more difficult. Putting the SQL server on SSD's will help immensely, using SAN snapshot integration with Veeam is the last remaining option if the previous ones don't help.

We've got some decent sized SQL servers on VMWare 6 on an all flash 3PAR without using the integration and snapshot creation and merge don't drop connections at all

[skrause]

I was actually talking to my DBA about this yesterday afternoon and he said that there haven't been any actual cluster failover actions that he can recall at all (he gets alerted when they happen through his monitoring system) due to backup actions, just reboots for patching once a month. At the very least, none of the snapshot removals have caused us an issue since our main database on this cluster grew from 150GB to over 500GB. He does see that there are brief network events logged by the cluster during the removal every night but that it is not enough to cause the AOAG to fail over.

We did tweak the cluster sensitivity as zuldan mentions above when we first set things up and were having issues so maybe that was all that was needed? /shrug

We are still on Vmware 5.5.

We also do Active Fulls every week for compliance with the organization policy.