SQL restore over WAN/VPN fails (iSCSI issue?)

[Regnor]

Hi,

Our customer does have problems restoring SQL databases over a WAN/VPN connection to different SQL servers in branch offices; Case ID #05279119.

The restore process starts, takes somewhere between 10-45 minutes and then aborts with an IO or "cannot find the path specified" error. In fact the restore never gets to restore any data and fails during the mount process.

Backups are stored centrally at the main location, the other locations only have local Veeam proxies but no repositories.
During a restore the backup gets mounted via iSCSI on the target server, but this takes quite some time (10m+) and the mounted folders can only be access with very high latency. Through the whole process there are disk, iSCSiPrt and NTFS warnings/errors in the event log.
One brach office for example has a 20/20 connection with two digit latency.
This made me wondering if there are any connection requirements or limitations in order to successfully use iSCSI over WAN connections?
Does anyone have some real world numbers/examples?

[PetrM]

Hi Max,

I'd suggest checking the list of required ports on this page of our help center. If everything is configured according to the requirements, our support engineers will need to research the issue. Also, why do you think that the problem is related to a failed connection?

Thanks!

[Regnor]

Hi Petr,

all required ports are open and nothing is blocked between the locations.
In my opinion it's the unstable iSCSI connection because of all the events logged in parallel; timeouts, IO errors, non-retryable errors and at some points iSCSI disconnects.
If I manually browse the FLR folder on the target server it takes forever to open a folder, sometimes also resulting in Windows explorer freezes.

We've checked network, firewall, etc and everything's fine. It's just the iSCSI connection which is having problems.

[PetrM]

In fact, there are no strictly defined requirements for connection performance or reliability. I saw a lot of different cases related to the unstable iSCSI link and it would be challenging to make explorer work unless all connectivity issues are resolved. Let's see what our support team can find out.

Thanks!

[Regnor]

Were there any solutions or workarounds to get the iSCSI connection stable in the other cases?

[PetrM]

It's hard to recall, it always depends on a specific infrastructure, network equipment etc. Anyway, "iSCSI over WAN" is not something that makes me think about reliability but I admit it can work in theory. That's why I think it would be best to continue working with our support team.

Thanks!

[micoolpaul]

Hi Regnor!

What speed are the links between these sites?

If you performed a packet capture at the iSCSI initiator endpoint, do you see traffic reaching it from the iSCSI target?

[Regnor]

@Petr: Exactly; iSCSI isn't meant for WAN connections. But for SQL direct restores there's no way around it. Let's hope there's something to tweak or at least make it reliable enough to complete the restore.

@Michael: It seems the connection is much slower than I thought; it's a 4 Mbit line and itself probably not stable as we see retransmissions.
Actually the iSCSI target gets successfully connected and the restores starts; the mount points can be accessed. But after some minutes it either times out or causes IO errors.

[micoolpaul]

That 4Mbit line is the culprit unfortunately. That bandwidth is too thin for QoS to fix, but could be a start

[Regnor]

I'll look at QoS and how much bandwidth we actually get during the restore next week.

It's just unfortunate that we require the use of iSCSI for SQL restores. Doing an Export, manually copying it to the target server and restoring/importing it there does work, but makes the whole process more difficult. The MS SQL plugin in v12 might change something there but won't work for this case because of the Veeam licenses.

[PetrM]

Hi Max,

Just FYI: you will need Enterprise Plus to use SQL plug-in.

Thanks!

[Regnor]

Hi Petr, that's the reason why I wrote that we can't use it in this case