SQL Server network issues during Veeam backup

Availability for the Always-On Enterprise

SQL Server network issues during Veeam backup

Veeam Logoby shlomia » Wed May 17, 2017 6:09 am

Hi,

I have SQL server that is have on going connection and transaction during all day.
I backup him in different times every 1-2 days.
Every time when I start the backup, the SQL server seems to have network issues, I receive this error:

Code: Select all
 ex.TargetSite:System.Data.DataTable ExecuteDataTable(System.String, System.String, System.Data.SqlClient.SqlParameter[])/n ex.InnerException:System.ComponentModel.Win32Exception (0x80004005): [b]The specified network name is no longer available/n ex.Message:A transport-level error has occurred when receiving results from the server[/b]. (provider: [b]TCP Provider, error: 0 - The specified network name is no longer available[/b].)/n StackTrace:   at DAL.InternalClasses.ManageSql.ExecuteDataTable(String connectionString, String procName, SqlParameter[] SqlParams) in


I was wondering, is that normal with Veeam backup? or can I somehow solve this issue?
The SQL server is VM on ESXi 5.5.
Running Windows server 2012 R2 and SQL Server 2014.

Thank you.
shlomia
Influencer
 
Posts: 10
Liked: never
Joined: Mon Mar 20, 2017 3:40 pm
Full Name: Shlomi

Re: SQL Server network issues during Veeam backup

Veeam Logoby cby » Wed May 17, 2017 11:05 am

I've seen this problem at snapshot merge time after the backup has completed but not when the backup starts. It could be the VM stun as the snapshot is merged/consolidated and SQL loses the n/work connection. N/work loss can take seconds during the consolidation with large snapshots (though I've seen minutes in some cases). Nothing to do with Veeam.

I understand that vSphere 6 has reduced snapshot consolidation times dramatically so that may be an option.
cby
Expert
 
Posts: 106
Liked: 6 times
Joined: Tue Feb 24, 2009 5:02 pm

Re: SQL Server network issues during Veeam backup

Veeam Logoby foggy » Wed May 17, 2017 4:57 pm

Shlomi, do you see similar behavior when taking this VM snapshot manually?
foggy
Veeam Software
 
Posts: 14904
Liked: 1096 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: SQL Server network issues during Veeam backup

Veeam Logoby shlomia » Tue Jun 06, 2017 5:50 pm

Hi, I'm sorry for the very late reply.
I just tried to take manual Quiesce snapshot, and seems like the machine is having network issues for the first minutes, or until the snapshot is taken.
After, the machine is working properly while on snapshot.

What does it mean? is it a vmware problem i suppose?
It's a big problem for me, because I have transactions and connections to this SQL Server all the time during the day.
shlomia
Influencer
 
Posts: 10
Liked: never
Joined: Mon Mar 20, 2017 3:40 pm
Full Name: Shlomi

Re: SQL Server network issues during Veeam backup

Veeam Logoby foggy » Tue Jun 06, 2017 5:57 pm

Yes, it's a typical behavior on VMware snapshot creation/deletion.
foggy
Veeam Software
 
Posts: 14904
Liked: 1096 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: SQL Server network issues during Veeam backup

Veeam Logoby skrause » Tue Jun 06, 2017 6:11 pm

You are going to have stuns at VMware snapshot removal on SQL, especially with highly transactional database servers as there are a lot of changes to consolidate in the removal process.

We got around this by having Always On synchronous replication between two servers with active-failover enabled. We then chain the jobs so both SQL servers are not running on snapshots (or removing them) at the same time, when the cluster loses contact with the "active" server it fails connections over to the other server and then fails back once the snapshot stun is over.

But, you need to have Enterprise SQL licensing for that which is not an inexpensive proposition.
Steve Krause
Veeam Certified Architect
skrause
Expert
 
Posts: 300
Liked: 45 times
Joined: Mon Dec 08, 2014 2:58 pm
Full Name: Steve Krause

Re: SQL Server network issues during Veeam backup

Veeam Logoby zuldan » Tue Jun 06, 2017 8:25 pm

We have a 2Tb DB in a SQL 2016 AOAG that VMware snapshots (ESXi v6.0 U3) were causing havoc with. We resolved the issue by increasing the SameSubnetDelay setting from 1000 to 2000 using Powershell. This won't help the OP but hopefully will with people running Availability Groups.

(get-cluster).SameSubnetDelay = 2000

See for more info http://dbinsight.com.au/dbinsight/vmwar ... up-cluster
zuldan
Enthusiast
 
Posts: 45
Liked: 5 times
Joined: Wed Feb 15, 2017 9:51 am

Re: SQL Server network issues during Veeam backup

Veeam Logoby DaveWatkins » Tue Jun 06, 2017 9:12 pm

VMWare 6 also made this stun much better. That's probably the easiest thing to do, if it's still an issue then the fixes get more difficult. Putting the SQL server on SSD's will help immensely, using SAN snapshot integration with Veeam is the last remaining option if the previous ones don't help.

We've got some decent sized SQL servers on VMWare 6 on an all flash 3PAR without using the integration and snapshot creation and merge don't drop connections at all
DaveWatkins
Expert
 
Posts: 263
Liked: 61 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: SQL Server network issues during Veeam backup

Veeam Logoby skrause » Wed Jun 07, 2017 12:33 pm

I was actually talking to my DBA about this yesterday afternoon and he said that there haven't been any actual cluster failover actions that he can recall at all (he gets alerted when they happen through his monitoring system) due to backup actions, just reboots for patching once a month. At the very least, none of the snapshot removals have caused us an issue since our main database on this cluster grew from 150GB to over 500GB. He does see that there are brief network events logged by the cluster during the removal every night but that it is not enough to cause the AOAG to fail over.

We did tweak the cluster sensitivity as zuldan mentions above when we first set things up and were having issues so maybe that was all that was needed? /shrug

We are still on Vmware 5.5.

We also do Active Fulls every week for compliance with the organization policy.
Steve Krause
Veeam Certified Architect
skrause
Expert
 
Posts: 300
Liked: 45 times
Joined: Mon Dec 08, 2014 2:58 pm
Full Name: Steve Krause


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot] and 25 guests