How does Veeam B&R handle network disconnections

[Shogan]

Hi guys,

A question about Veeam Backup - how does it handle network disconnections? i.e. between the source and target when doing a backup? My testing thus far is that even if a short disconnection occurs on the network, the backup fails. Is there any built in protection against this, or a certain number of retries that occur before the job fails?

Thanks!

[foggy]

Minor packet loss is not a problem, backup will still be able to proceed correctly, but in case of a tangible disconnection the job fails.

[Shogan]

Thanks Foggy, so for example, a disconnection of 5 seconds would probably result in failure then? Most definitely if there was a disconnection of 30 seconds?

Reason I ask is we have a fibre connection that sometimes gets disconnections and needs to be failed over via an automated script - this causes a disconnection of around 30 seconds, meaning veeam jobs fail.

I (and I am sure others) would really like to see a feature in Veeam Backup that can handle disconnections like this.

[Gostev]

Anything that causes TCP/IP connection to drop will result in job failure.

[Shogan]

Hey Gostev,

Any plans for a new feature in the next release / update release of v6 to include an automatic dropped connection retry feature for jobs?

I would like to formerly request this feature!

[Gostev]

Actually, we have job retry feature since v3. Job will automatically retry, and process only VMs which failed during main job run. This behavior is controlled on the same page where you set the job schedule. Thanks!

[Shogan]

Thanks Anton,

Maybe I should have been more specific - I would like to request a feature where the current VM being backed up in a job will resume backing up if there is a drop in network connection, without the need for the VM to be retried again from the start. So if 20GB out of 50GB have been processed so far, and there is a drop, then the job tries to resume from that 20GB point and continue on with things.

Our issue is we back up between Data Centres using a fibre connection across the city, and sometimes there can be short periods of maintenance / times when the supplier re-route the connection and there is a 5 - 30 second drop in network connectivity. It would be a bit of a pain for large (500GB) VM backups to need to retry completely, rather than just resume.

[Gostev]

Resume, may be... don't recall many requests for this feature though. Possibly because nowadays, most connections have a very high uptime, especially those enterprise class ones which cost a lot.

[arsprod]

True that most enterprise class san's have high uptime, but I (and I would guess other folks) are using other devices (like simple nas or software iscsi like starwind) that don't necessarily have high uptime. I'm having a heck of a time with Veeam jobs becoming corrupted because of disconnects. Re-running the job doesn't help because the interruption corrupts the whole job.

[Vitaliy S.]

Hello Aaron,

Out of curiosity how often do you have disconnects? Have you been able to track down the initial reason for that? Thanks

[arsprod]

well, it's a long story! At the moment at least a couple times a week. I've been working with support and through that process (over a couple months) we've determined it's the iscsi datastores disconnecting, which in turn corrupts the replica job and I have to delete all the history and start over. I was using a Windows 2008r2 server and MS iscsi. I recently removed all the MS stuff and installed Starwind which seems much more stable, but still getting disconnected. Just today I moved that backup SAN to the dedicated iscsi switch - we'll see if that helps. Sorry you asked?

[Vitaliy S.]

Ouch...that sounds painful. I'm not a storage guru, but trying a dedicated switch might help here.

[st0623]

A couple of HyperIP customers have asked us to tune our timeouts for network disconnects in the WAN. To date, if Veeam B&R is running over HyperIP, we can keep the session/job up for up to 3 seconds, without causing the backup job to fail. We haven't tried a longer disconnect, but could test it with you. 3 seconds is forever in the network world, but we are aware of some links disconnecting for 5 seconds or longer. Sometimes the nature of MPLS or cheaper public internet connections. I'd be happy to talk with you, Shogan, if you want to reach out via my contact info below.

[Shogan]

A couple of HyperIP customers have asked us to tune our timeouts for network disconnects in the WAN. To date, if Veeam B&R is running over HyperIP, we can keep the session/job up for up to 3 seconds, without causing the backup job to fail. We haven't tried a longer disconnect, but could test it with you. 3 seconds is forever in the network world, but we are aware of some links disconnecting for 5 seconds or longer. Sometimes the nature of MPLS or cheaper public internet connections. I'd be happy to talk with you, Shogan, if you want to reach out via my contact info below.

Hi Steve,

Thanks for your reply. At the moment we are running a script which monitors an end point on the other side - when this fails to ping a certain amount of times, it runs another script which fails over to a completely different supplier connection. This works out so that the disconnection is actually around 1 minute in some cases depending on the issue as the script needs time to ensure there is more than just a little bit of packet loss causing dropped pings before it switches over to the other connection.

I've noted down your details though in case things change here - I don't think much could be done about a long disconnection like the above In terms of Veeam though, we could really do with a feature that allows the job to resume / retry at times like these.