Sage Line 500 Backups & Replication

[MadAsToast]

Morning All,

We are experiencing issues with backups and replication on Sage line 500 - we have tried both application aware and non application aware (block level).

Sage Line 500 is hosted on one virtual machine, MS SQL database on a second virtual machine.

The problem we are experiencing is identical on both application aware and not, and whether we do backups/replication on either VM - The end user loses connection to the database from the client, we get the following error in the log file:

Code: Select all

[08S01]:[[Microsoft][SQL Native Client]TCP Provider: An existing connection was forcibly closed by the remote host.]:iebuildquery:iestmt_opencurs:table scheme.stockm

We have spoken to our Sage support guys, however it seems their stock reply to anything in depth is "network issues" which as we run a full reporting Cisco network we know the network is running at about 7% utilisation without any issues.

If anyone has any insight into backups/replication on a sage system I would appreciate any advice, I can rely on out of hours backups and replications but I'm fairly sure this shouldn't be happening especially on block level.

TIA

MadAsToast.

[Gostev]

Hi Chris, the only suspect here would be long VMware snapshot operation, but those problems would result connection to timeout instead, and here the connection is forcibly (actively) closed by SQL Server. Something is actually actively closing the connection, and you say this even happens if you do crashconsistent backups (no app-aware processing or VSS freeze). I have never seen something like this, and too suspect some network problems actually...

[MadAsToast]

Not sure I agree with the network problems. Perhaps I need to supply a little more information.

The Veeam server is connected to the storage through two fibre channel controllers connected to fabric switches, the SAN holds all the data again connected to the fabric switches.

The SQL database server AND the Sage Line 500 servers are both VM machines and tied to sit on the same physical host, therefore the communication between these servers NEVER hits the physical network, only ever connect through a vSwitch.

The disconnection of client only ever happens when the backup is running and that as stated before is both Application aware AND block level. I cant see how this will be network related as the only network being used is a virtual network to communicate between the two virtual servers!

I agree the long snapshot could have something to do with it, I'm more inclined to agree that this might be connected to the issue, HOWEVER the disconnects happen almost IMMEDIATELY after the backup/replication has been started, so I'm not convinced that the length of time on the snapshot is causing the issue. Perhaps there is a issue on the actual disk utilisation on the snapshot being taken and when the snapshot is "in use" but I'm slightly sceptic.

I'm open to any and all suggestions on this one as currently I'm having to schedule outside office hours and outside backup time replications, once I have the "window" to do the initial replication.

TIA

Chris

[ThomasMc]

We run Sage 200 with a similar setup and although I can snapshot it fine(Backup/Replicate) its one of the only things I can't vMotion during business hours due to a similar issue we get when we do it, and when we spoke to Sage Support funny enough they said it was a Network Issue lol(Cisco kit throughout and Solarwinds monitoring it all), this info probably won't really help your issue though

[Vitaliy S.]

I'm more inclined to agree that this might be connected to the issue, HOWEVER the disconnects happen almost IMMEDIATELY after the backup/replication has been started

Chris, have you tried creating snapshot manually through vSphere Client to check if it has a similar impact on your VMs?

[tsightler]

Is the problem occurring during the snapshot of the Sage server, or the SQL server? I've seen this issue on Dell server with certain Broadcom NIC's whenever there is high load (a snapshot might be created similar behavior to high load). I think it was caused by TCP Offload.

Ah..here the Microsoft KB article:

http://support.microsoft.com/kb/942861

The note even mentions that the error may occur when you "replicate SQL databases". Obviously they're not referring to Veeam replication, probably just due to the load involved. Might not be the same issue, but it's an easy solution to try.

[tsightler]

BTW, the article at http://blogs.msdn.com/b/mssqlisv/archiv ... issue.aspx has some great resources for troubleshooting intermittent connectivity issues. It specifically mentions the TCP Offload (TCP Chimney Offload) issue I mentioned, as well as a couple of other known causes of intermittent connectivity causing timeouts that really should just be retries.